Author: Irwin Collier

Edward Chamberlin was a member of the graduate examination committee of the Harvard economics department in the early 1960s and in his files I have found copies of the theory exams from 1961, 1962, and 1963 along with a few memos that  circulated among members of the committee that together provide a description of the procedures used for grading.

Of related interest is the following report that was transcribed and posted earlier:

Report on the General Examination for an Economics PhD, 1970

_________________________________

Other Written Exams
in Economic Theory

April 11, 1961
November 13, 1962

_________________________________

HARVARD UNIVERSITY
Department of Economics

Written Economic Theory Examination
April 8, 1963

You are to answer a total of 6 questions.

All three questions in Part A.
One question each in Parts B, C, and D.

Use a separate book for each question.

PART A: Answer all THREE

1. Explain the phenomena of “external economies” and “external diseconomies.” Describe how they affect the efficiency of the competitive pricing mechanism, and discuss measures which have been proposed to improve welfare when external economies or diseconomies are present.
2. State and explain several leading principles from the field of “non-price competition.” Comment on the problems that arise in combining this type of theory with the more orthodox “price competition.”
3. Interpret the Marshallian concept of Consumers’ Surplus in terms of a theory of utility based solely on Indifference Lines.

PART B: Answer ONE of the two.

4. Contrast the “liquidity preference” and the “loanable-funds” theories of interest. Discuss the implications of these two theories for monetary policies intended to maintain full employment.
5. Discuss the purely theoretical proposition that if all prices everywhere were sufficiently responsive in both directions to supply and demand there would, in a free market economy, be no persistent unemployment. Be equally interested in pointing out what may be right and what may be wrong about the statement. State what assumptions you would want explicitly stated if you had to support the proposition.

PART C: Answer ONE of the two.

6. Compare the main ideas of Adam Smith and David Ricardo about economic growth — its mechanism and its consequences.
7. Formulate a simple, highly aggregated model of economic growth. Incorporate technological change in it by including an industry called Research with a production function of a specified shape. Its inputs are capital (stock) and labor (flow). It is up to you to give a definition of its output that is appropriate to your model.

PART D: Answer ONE of the two.

8. Explain and compare some of the conclusions that economists have reached about the interest rate in a static or stationary state.
9. Discuss the similarities and differences among the principles of economic choice that are applicable to the three following:
   1. an individual consumer;
   2. a trade union, producers’ cartel, or other interest group;
   3. society as a whole.

You may keep this question sheet when you hand in your exam books.

_________________________________

HARVARD UNIVERSITY
CENTER FOR INTERNATIONAL AFFAIRS

Robert R. Bowie, Director
Alex Inkeles
Henry A. Kissinger
Edward S. Mason
Thomas C. Schelling
Raymond Vernon

6 Divinity Avenue
Cambridge 38
Massachusetts

April 23, 1963

From: T. C. Schelling
To: Messrs. Chamberlin, Leontief, Vanek

I enclose a sheet with the names and grades for your information.

The outcome of our regrading was as follows. You will recall that there were five students for whom we were rereading one or more books. You will also recall that we were to count the third reading as equal in weight to the other two. The results were:

Book 3, down from 1.4 to 1.2, Fail
Book 5, down from 1.4 to 1.1, Fail
Book 6, up from 1.5 to 1.6, Fair
Book 7, up from 1.25 to 1.4, Fair –
Book 10, down from 1.5 to 1.0, Fail

To recapitulate, three of these failed, and we had five clean failures, making a total of eight failures. On the rereading, three Fair minuses went down to Fail, one Fail went up to Fair -, one Fair – went up to Fair. I think this is about what we could have expected, and I am glad we did the rereading. Incidentally, two of the three who failed after the rereading had three books reread with two different readers involved, so I think we can feel they got fair treatment.

Next week I shall circulate to you my thoughts about a report to the Department and, if you wish, we can get together or alternatively you can add your comments. If it is convenient I should prefer to get together, but not until I have given you at least my thoughts on what we should report.

TCS: ac

_________________________________

HARVARD UNIVERSITY
CENTER FOR INTERNATIONAL AFFAIRS

Robert R. Bowie, Director
Alex Inkeles
Henry A. Kissinger
Edward S. Mason
Thomas C. Schelling
Raymond Vernon

6 Divinity Avenue
Cambridge 38
Massachusetts

June 4, 1963

From: T. C. Schelling
To: Messrs. Chamberlin, Leontief, Vanek
Subject:  Report to the Department on the Graduate Theory Examination

We promised the Department a report. And we had some things we wanted to report.

Some of our experiences we can communicate to next year’s committee over the lunch  table. Some really require the Department’s cognizance. I am listing below some of the points I think we should like to report. This is not a draft, but just a chance to check with you. If you agree, disagree, or want to add anything, I suggest you do so in writing with copies to each other. There is no great hurry, but next year’s committee will want some Departmental instruction by the time of the second Department meeting next fall. I would like to get this done before my memory fades, and submit it if possible to the Department as soon as everybody is back from the summer.

1. I would propose that individual questions be graded not Excellent, Good, Fair, and Fail, but either numerically on the base one-hundred or with letter grades A, B, C, with the committee to decide — subject to any advice the Department wishes to make explicit — what kind of average or combination of grades should qualify a person as a “pass.” The Department should either make clear that the committee may do as it pleases or express itself on such things as how many failures on individual questions make a failing exam in spite of the average. The Department might also express itself on how large or how small the failing fraction might be without being considered “abnormal.” Just to get a proposal in the works, I would propose that questions be graded A, B, C, and Fail, with a B- required for passing, but with the committee empowered to make individual exceptions in either direction on the basis of the whole exam, and that the committee expect to fail somewhere from one-tenth to one-fourth without considering a “policy issue” being involved.
2. I would strongly recommend that we experiment next fall with typewritten examinations. This raises a number of technical questions, ranging from who provides the typewriter to how noisy the room is, and it surely discriminates somewhat according to typing skill. The present scheme also discriminates according to longhand skill. Students who cannot type, or choose not to type, should have their examinations transcribed, either at their own expense or at the Department’s expense. This seems to me the one exam that, because it is for graduates and because it interferes with no individual’s course, lends itself to the experiment. I feel quite sure that the reading of examinations will be much more reliable if the material is typed, and that disputed grades could be discussed more readily if the exams can be easily and quickly read. The number of students taking the exam in the fall is usually small, and that is therefore a good time to try it out.
3. I am surely persuaded that anonymous examination books make a real difference and the difference is a good one. I think the committee should avoid as far as possible putting students in special categories like the few who this year were offered the option of presenting to the oral exam with a re-examination in theory. At the same time, the committee cannot avoid having an opinion (or opinions) about the success of its own examinations; and the committee may, as I think we did, have some doubts after the examination about its reliability. If these are strong doubts, they should, as we did, consider special treatment of a few individual cases.
4. Especially if we go in for typed exams, the Department should consider making this a six-hour exam rather than a three-hour exam, just to increase its reliability. Reading time, I believe, would be sufficiently cut by having a typed examination to make the six-hour exam feasible for the committee.
5. The questions are also up to the committee but I would pass along the advice that the questions be as concrete and as problem-oriented as possible in contrast to general essay or discussions of what economists have said, proposed, etc. I think I say this not out of a priori prejudice but because I have felt more confident of the grade I gave when the student was responding to a very direct question or problem with little scope for inadvertent or deliberate evasion and with the obligation to give his own answer and not to repeat [what]others have [said]. This kind of advice surely is not suitable for Departmental action, but, if we share some experience we might try to articulate it for the next committee.

TCS: ac

_________________________________

Schelling’s Memo to Dunlop
and the Exam Committee

TO: CHAIRMAN [John Dunlop], DEPARTMENT OF ECONOMICS

FROM: THOMAS C. SCHELLING, CHAIRMAN

DEPARTMENTAL COMMITTEE ON THE GRADUATE THEORY EXAMINATION

RE: WRITTEN THEORY EXAMINATION

DATE: SEPTEMBER 18, 1963

Last year’s committee, consisting of Chamberlin, Leontief, Vanek, and me, reached several conclusions we would like to report for the benefit of the new committee that follows us. Most of the observations we want to pass along arose out of our dissatisfaction, not our contentment, with the examination process. We would like this report to go to the whole Department, or the Executive Committee, or whatever part of the Department is appropriate; if you want to postpone this a month while your new committee decides what it would like to do, if anything, about these recommendations, that is agreeable to us.

1. Our grading scheme, which we accepted without much thought, was to grade each question excellent, good, fair, or fail, with plus and minus, then to convert these to a numerical scale to facilitate averaging, and then to grade the whole examination. This procedure led to an anomaly that in turn produced some real misunderstanding among the graders. The anomaly was that, after collecting all of the books together and looking at the distribution of grades, the committee might wish to fail people whose average grade was “fair” or to give an “excellent” to a man whose average grade was “good.” This can lead to disputed interpretations of what the grades meant as well as what the grading standards should be. I doubt whether any committee would want, either in principle or in practice, to rely on a straight forward averaging to determine good, fair, fail, etc. I strongly recommend — and this may sound trivial but it is not — that the initial grading be on some arbitrary numerical scale with the final determination of over-all grades from fair to excellent being determined afterward. My committee agrees with this. I personally do not see that a matter of principle is involved here that ought to go to the Department, but I foresee that some eventual controversy may be forestalled if the Department is apprised of this problem and of the new committee’s intentions.
2. The committee is bound to have some notion of what proportion of those taking the examination might normally be expected to fail it. Different members of the committee may have very different notions. I believe this is meant to be a hard examination, and that the fraction failing it might be comparable to Written Theory Examination the fraction of students who fail their Generals. It is, I believe, also meant to weed out students who would likely fail their Generals. And it is an examination in which the committee ought to feel that anywhere from one-tenth to one-quarter of the candidates might be failed without the result seeming to be abnormal. It might be helpful if the Department would at least discuss the matter briefly so the committee would have a pretty good idea how much leeway it had in grading. It is not quite enough to say that this is completely within the committee’s competence; the philosophy of the examination derives somewhat from the Department’s notion of how strict this examination ought to be and how great a variation in outcomes needs to be expected.
3. We recommend that the committee experiment in the fall term with typewritten examinations. There are some practical questions here, such as who provides the typewriter, how noisy the room will be, and so forth. Typewritten exams will discriminate according to typing skill; but the present exam discriminates according to long-hand skill. Students who cannot type, or who choose not to type, should have their examinations transcribed, either at their expense or at the Department’s expense. Because this exam is for graduates, and because it interferes with no individual course, it lends itself to experiment; in particular, the small group in the fall term presents an opportunity on a small scale. All of us on the committee believe that the grading will be more reliable if the material is typed, and that disputed grades can be discussed better, and reread more easily, if they are typed. We, therefore, strongly urge that the experiment be made this year.
4. We are quite persuaded that anonymous examination books (books from which student names have been removed) make a real difference and that the difference is a good one. In case of borderline grades, it is hard to resist the temptation, after the exam has been graded, to get out the student’s record and see whether or not he deserves the benefit of the doubt. We did this, and we believed it was right to do so, but maybe as a matter of principle it should not be done. Let me point out that an awful lot hinges on a single examination if one does not fall back on the student’s theory record in borderline cases. In the oral examination I think it is fair to say that the student’s course background does count in the examiner’s evaluation of him. If the Department really does not want the written theory exam to be anything but an anonymous exam graded solely on its merits, a flat rule would relieve the committee of a philosophical problem that can be quite a nuisance. If the Department wishes the committee to use its own judgment, it will probably help the committee to have it understood in advance that the committee may decide this one. We recommend that the committee be free to use the additional information after the books have once been graded but that the committee avoid this expedient if possible.
5. If the typed examination is adopted, there is much to be said for making this a six-hour examination to increase its reliability. Reading time will be cut by the typing enough to compensate the greater number of books read.
6. Our final recommendation involves something that cannot be legislated. It is that the questions be as concrete and as problem-oriented as possible, in contrast to general essays or discussions of what economists have written about a subject. Our impression was that grading was much more reliable on the more direct questions and problems. There was both deliberate and inadvertent evasion on the more general questions, as well as more ambiguity on the committee itself as to what the question called for. The common occurrence of a bluebook that was a decent essay on a question that wasn’t asked might be averted by using questions that are fairly direct and unambiguous. Another common occurrence was the bluebook that indiscriminately gave the positions of various writers without the student’s accepting responsibility for his own analysis or evaluation. Our feeling was that these rather indirect questions provided quite unreliable evidence on which to grade students.

_________________________________

Chamberlin’s Memo
to the Exam Committee

This letter was “in the works” when Tom’s report to John Dunlop of September 18, 1963, came in the mail. It is now sent as a supplement to Tom’s report.

FROM: E. H. CHAMBERLIN
TO: MESSERS. SCHELLING, LEONTIEF, VANEK

SUBJECT: REPORT TO THE DEPARTMENT ON THE GRADUATE THEORY EXAMINATION (letter from Tom Schelling, June 4)

                  First I should admit that I was against the written theory examination when it was first proposed, but without any question the experience this year has made me more opposed than ever. In my letter to the other members of this committee on April twenty-fourth (?), I urged that the attempt to shake some more failures out of the group of eight between the figures of 1.2 and 1.5 had been a “fiasco” and that we should simply allow all of them (i.e., everyone excepting [name deleted], [name deleted], and [name deleted]) to take the orals, with the decision whether to pass or fail to be made at that time. As a compromise, we finally settled on three students: [name deleted], [name deleted], and [name deleted], and offered them the opportunity of taking the oral examination, in which  they might do well enough to pass, even though they had “failed” theory. (The fact that no one of them accepted was certainly not surprising, since they had all been told already that they had failed theory and therefore had two strikes against them if they risked the orals.)

Although we all concurred in the decisions, I feel that I was mainly responsible for the matter coming up at all. As certainly evident in my letter of April twenty-fourth, I was extremely critical of the attempt to fail people who had already been graded at or near the good- to fair+ line merely because we “needed” more failures. In this respect especially I think the policy worked badly this year. The whole matter is probably one for the Executive Committee, rather than for this one. I hope it is understood that from many years of examining in economic theory I have the matter very much at heart. Certainly the treatment of our graduate students at the end of the second year is of the first importance, and I think the Executive Committee should devote some time to reconsidering the whole problem.

I still hope that this group may issue a unanimous report to the Department although as will be seen from the following comments there are son important differences between Tom and me. Perhaps we ought to have a meeting. Comments:

1. Questions of grading.
   1. The approach of failing a certain percentage of those who take the exam (“one-tenth” to “one fourth”) must absolutely be dropped. It is contrary to the practice , both of this Department and of Harvard University, for as far back as I can remember. One only needs to recall the recent principle that “all (undergraduate) students are potential honors candidates” and to consider the grading processes with respect to these latter, to realise how far astray the concept of “failing or passing a certain percentage” is from the general practice at Harvard, and, in the past, in this Department. In any event, it is clearly unjustifiable with a group as small as we normally expect in the written theory examination — this year 37, of whom we failed, by a great effort, 8, or more than 20 per cent. I think it was the obsession that 3 was not enough failures and that we ought to increase the number, that led to a compounding of arbitrary decisions at the “margin”, and to results which, as I think I demonstrated in my earlier letter, made passing or failure for the group of 8 to which it was applied, almost a matter of pure chance. It was a witch’s brew if there ever was one. However, it was described in some detail in my earlier letter, and I refrain from another lengthy demonstration here.

Only one example from later developments: there were three books at the same grade of 1.25, (a Fair+ by the first reading). Two of them, having no questions eligible for re-reading by our rules, were below the new line of 1.4, and left as “failures”; a third, however, qualified for having two questions re-read (the intervals of discrepancy being 4 and 5 in the two cases), was converted into a pass and finished with a “Good” in the Generals. Why should he have had the opportunity to take Generals while two at the same grade had to wait six months? The conclusions: 1. I think we should admit that the methods we used to make distinctions within this “marginal” group were at fault (to put it mildly) and were future committees against them. 2. We should revert to an “absolute”, not a “relative”” or percentage, standard of quality for passing, and for the several grades of Excellent, Good and Fair, rather than trying to fail a particular number or percentage of people. 3. We should recommend to the Executive Committee that they reconsider whether we really want to “raise standards” in the Economic Theory part of the General examination as much as we appear to have done.

As for the second point, my own conviction is that only those conspicuously deficient in Theory should be failed. It should be not only possible, but a goal of the Department that all who take the examination should be well enough prepared to pass. After all, this only means that the Admissions Committee has done its work well, that the student has been well-advised as to courses, and that he has not outrageously neglected his work. Realistically, of course, there will usually be a few failures, either in the Theory exam or in the Generals. But in my opinion, failures should be voted by the Executive Committee upon recommendation of the Committee on the examination. In all cases of recommended failures, the members of this latter committee should each read the entire book with full knowledge of the identity of the persons involved and decide upon the fate of the student only in consultation, (as at present after the general oral examination).

1. 2. As for grading terminology, this year it was Excellent, Good, Fair, and Fail, as we all know, and when numerical values were given to these categories later, the space between then was assumed to be equal: 6, 3, 0 (=Fair!) and -3. Tom has made another proposal in his letter (of June fourth). The important thing, it seems to me, is to put more space between Fair, which has always been a passing grade, and Failure. Indeed, one could easily explain the fact that 23 out of 37 books, approximately two-thirds, received a good- (2) in the first reading by the fact that 2 is mid-point between 6 and -3! Although these (good-) books were later broken down and distributed between the levels of “good” and “fair-”, this merely disguises the fact that the grades given actually had very little difference between them. In fact, with the exception of only four books, 33 of the 37 lay between the limits of 2.8 and 1.2, the former .2 below the good average, and the latter .2 above the fair+ average. Clearly the method of grading used this year, in spite of later adjustments, did a poor job of revealing the differences which must exist among the candidates who took the examination.

However, using the same figures, I experimented with breaking up the concentration at good- by introducing mechanically several considerations which ought to enter in anyway. Since this was actually done (out of the sheer fascination of the problem) I attach copies of the result for what they may be worth, perhaps only in suggesting the other ways in which the objective might be achieved. The 6, 3, 0, were kept for Excellent, Good, and Fair, but Fail became -9 (i.e., -6 more in every case of a -3). A more normal scale would evidently be: six questions with a value of 15 each; highest possible total grade a 90. Each question graded 15 = Excellent, 12 = Good, 9 = Fair, 0 a total failure. Also +3 whenever two different readers agreedthat an answer was a Good or better, and -3 whenever two readers both gave 0 (=Fair) or less. These several devices spread out the grades, [name deleted] actually got his Excellent, [name deleted] and [name deleted] showed up as clear failures instead of getting fairs, with [name deleted] such a low Fair that he might easily be added in, the number of Good’s was reduced to 15, etc., etc. (The applause is accepted). No re-readings, either.

To return to Tom’s letter:

2. I do not think it is fair to require students to type-write their examinations or pay to have it done (I think it is optional now). But they should be warned to write legibly and told that if they do not, they will have to pay to have their written examinations transcribed.
3. I agree that anonymous examination books are desirable up to a point. But no one should ever be failed without knowing the candidate’s identity and all we can about him.
4. A three hour examination seems to me long enough, or four at the very most. We should not forget that each student has already been examined for three hours per semester in his courses.
5. I think the questions should be of all kinds. Just as I refuse assent to the proposition that the scope of Economic theory should be limited to what can be treated in mathematical symbols, as I should not want an examination in theory to be cast in one particular mold.

_________________________________

Leontief Letter to Schelling

September 23, 1963

TO: T. C. Schelling

FROM: W. Leontief.

cc: B. H. Chamberlin, J. Vanek

I heartily approve of all recommendations contained in your memorandum on Written Theory Examinations dated September 18th.

The typing of all examinations — which, incidentally, I proposed at the very beginning before we started them — might not be easy to arrange since the secretaries in the Department offices have no less difficulty in reading the handwritten bluebooks than we do.

A six-hour examination might be rather hard on the students unless it is made quite clear that the additional two hours are allotted for preparing a clear typescript or readable long-hand.

WL: kd

Source: Duke University. David M. Rubenstein Rare Book & Manuscript Library. Economists’ Papers Archive. Edward H. Chamberlin Papers, Box 18, Folder “Written Theory Committee, 1963-64”.