Herbert Scarf - Career at Rand, Stanford, and Yale

Career At Rand, Stanford, and Yale

Scarf worked at Bell Labs in the summer of 1953 and travelled every day between Princeton and the laboratory with John Tukey, an eminent statistician. At Bell Labs Scarf encountered Claude Shannon, the inventor of information theory. In June 1954, Scarf left Princeton to join the Rand Corporation. He chose Rand instead a more conventional academic job, because he desired to be involved in applied rather than abstract mathematics. The Rand Corporation was founded by the US Defense Department in 1948 in order to apply a variety of analytical tools to the economic, political and strategic problems of the Cold War and provided an ideal environment for researchers with applied interests.

Among his colleagues at Rand were Lloyd Shapley, George Dantzig, Richard Bellman, Ray Fulkerson, and Lester Ford. Dantzig, the inventor of the simplex method, had arrived a bit earlier and was applying his methods to a large variety of basic problems. Bellman was trying to formulate and solve all possible optimization problems with a dynamic structure as dynamic programming problems. Fulkerson and Ford were working together on network flow problems which became the springboard for the flourishing field of combinatorial optimization. At Rand, Scarf worked with Shapley on games with partial information and differential games with survival payoffs and was occasionally joined by John Nash when he visited as a consultant. This activity resulted in two early papers of Scarf and Shapley on game theory.

At Rand, Scarf was first assigned to the Mathematics Department but after a year the organization was visited by a budgetary crisis and Scarf was transferred to the Department of Logistics - a junior subset of the Department of Economics. His colleagues in the logistics group were mainly concerned with maintenance, repair, scheduling and inventory management which had little to do with the economic and strategic questions of the Cold War. Scarf was not assigned to any specific research topic. He learned about inventory problems by himself and wrote his first paper in this field. He met Samuel Karlin and Kenneth Arrow at Rand. They were both interested in inventory problems (Arrow had already written a remarkable paper on inventory theory with Harris and Marschak) and they invited Scarf to spend the academic year of 1956-1957 at the Department of Statistics, Stanford University.

At Stanford, Scarf worked intensively on inventory problems and demonstrated his extraordinary analytical skill and penetrating discernment on the nature of fundamental problems, when he published his two epoch-making papers on dynamic inventory problems: the first (1959) is on the optimality of policies and the second paper (1960), with Andrew Clark, on optimal policies for a multi-echelon inventory problem. Scarf also collaborated intensively with Arrow and Karlin on inventory problems. This collaboration resulted in three landmark volumes: Studies in Mathematical Theory of Inventory and Production, 1958, Contributions to the Theory of Inventory and Replacement, 1961, and Multistage Inventory Models and Techniques, 1963. Arrow and Karlin also became Scarf’s good friends and mentors.

Scarf’s visit was originally for a single year but the invitation was extended and in the fall of 1957 he was appointed as assistant professor in the Department of Statistics and subsequently an associate professor until he left Stanford in 1963. While working on inventory problems, Scarf became very interested in economics from discussions with Arrow and Hirofumi Uzawa and by attending the seminars on Mathematics in the Social Sciences organized by Arrow, Karlin and Patrick Suppes. He was particularly fascinated by general equilibrium models which he considered to be the central paradigm of economic theory.

In 1958 and 1959, Arrow and Leonid Hurwicz published two basic papers (the latter one with Robert Block) in Econometrica. They proved that the Walrasian price adjustment process formalized by Paul Samuelson (1941) converges globally to an equilibrium for exchange economies with divisible goods when all goods are gross substitutes. It was much speculated that such processes would converge in any reasonable economy with divisible goods. But Scarf (1960) soon dashed such hopes by producing a simple example with three consumers and three commodities that was globally unstable. This was Scarf’s first classic article in economic theory and was the very beginning of his remarkable career in the economics profession.

On Tjalling Charles Koopmans’ invitation, Scarf spent the academic year of 1959-1960 at the Cowles Foundation at Yale University. Koopmans, whom Scarf had met earlier at Rand, became a very close friend and mentor of Scarf. During his visit Scarf gave a seminar talk on his counter-examples. The seminar was chaired by James Tobin who was then the director. Among his audience were Gerard Debreu, Donald Hester, Alan Manne, Art Okun, Edmund Phelps, Bob Summers, and Jacob Marschak. During the same academic year, Scarf was invited to give a talk at Columbia University on his counter-examples. His old colleague Martin Shubik was in the audience. After the talk Scarf and Shubik took a long walk from 125th street to Shubik’s apartment in Sutton Place, New York. During the walk, Shubik passionately talked about and tried to persuade Scarf to solve the so-called Edgeworth conjecture that the core of an exchange economy would converge to its set of competitive equilibria if the number of traders in the economy tends to infinity.

Shubik’s enthusiasm sparked Scarf’s interest in this question and he started thinking seriously about the topic. He read von Neumann and Morgenstern’s book: The Theory of Games and Economic Behavior, Edgeworth’s analysis of the contract curve with two goods and two types of traders in his book: Mathematical Psychics, and Shubik’s 1959 paper on this subject. Several months later a decisive moment came when Scarf found a way, albeit extremely complicated, of proving the Edgeworth conjecture; see his 1961 paper: ``An analysis of markets with a large number of participants”. Debreu subsequently improved Scarf’s argument and published it in his 1963 paper: ``On a theorem of Scarf”. But a significant simplification of Scarf’s argument came when Scarf met Debreu on one occasion in December 1961, as Debreu eloquently described it in his 1983 Nobel Prize lecture: ``Associated with our joint paper is one of my vivid memories of the instant when a problem is solved. Scarf, then at Stanford, had met me at the San Francisco Airport in December 1961, and as he was driving to Palo Alto on the freeway, one of us, in one sentence, provided a key to the solution; the other, also in one sentence, immediately provided the other key; and the lock clicked open.” This collaboration yielded their 1963 paper: ``A limit theorem on the core of an economy,” which is one of the most fundamental results in general equilibrium theory. It is an important milestone for at least three reasons: First, it provides an important justification for the assumption of perfect competition that is fundamental in the treatment of neoclassical economic equilibrium models; second, it shows that competition and cooperation are just two sides of a coin for economic activities under the right circumstances; third, it became the starting point for a large literature on the core equivalence.

In 1963, Scarf moved to the Cowles Foundation and the Department of Economics at Yale University and was appointed as a full professor. In 1979 he became a Sterling Professor—the highest recognition for academic staff at Yale. He was the Director of the Cowles Foundation for the periods of 1967-71 and 1981-84. Since 1963 Scarf has remained at Cowles except for visiting appointments at Cambridge, Stanford and other institutes. He found the environment at Cowles extremely suited to him, as he describes it in the preface of his 1973 book: ``The standard of mathematical rigor and clarity of thought which prevail at Cowles are well known to the economics profession. But perhaps more important is the persistent though subtle suggestion that the highest aim of even the most theoretical work in economics is an ultimate practical applicability.”

During his first few years at Cowles Scarf concentrated on the problem of finding a method for computing economic equilibria. His work on the core equivalence result had suggested a roadmap. If he could find a way to calculate a point in the core of a game based on a general equilibrium model, then this method would serve to find an approximate equilibrium allocation, at least in an economy with a large number of traders. This activity resulted in the first major core existence theorem for a large class of cooperative games without side payments. He proved that an N-person game has a nonempty core if the game is balanced. Scarf’s first proof of this theorem relied on Brouwer’s fixed point theorem, but his hope was to provide a numerical method for computing a point in the core, making no use of fixed point theorems. Good fortune loves those who are well-prepared. Robert Aumann was visiting the Cowles Foundation during the academic year 1964-65. Scarf described his problem to Aumann, who suggested that he take a look at a recent paper by Lemke and Howson (1964). In this article, they proposed an algorithm for computing a Nash equilibrium in a finite two person non zero-sum game. In a single evening, Scarf realized that he could directly translate the Lemke-Howson’s algorithm through a limiting process into an elementary and constructive proof of his core existence theorem. This result was reported in his 1967 classic article: ``The core of an N-person game,” and became one of the most important theorems in cooperative game theory.

Having found an algorithm for the core, in November 1965, Scarf finally realized that he could explore this technique to design a novel algorithm for approximating equilibrium prices directly, without relying on the relation between the core and the competitive equilibrium. This path-breaking work marked the successful culmination of his long battle for transforming abstract general equilibrium analysis into a practical tool for the evaluation of economic policy. The result is published in his 1967 article: ``The approximation of fixed points of a continuous mapping.”

Since the early 1970s, Scarf launched his longest, hardest and most ambitious struggle: to tackle economies with indivisibilities, increasing returns and nonconvexity. In fact in 1963 he already wrote: ``Notes on the core of production economy,” which was widely circulated but was not published until 1986. In this note, he studied economies where the production set exhibits increasing returns. He showed that if the production possibility set satisfies customary properties, but is not a cone, then there is a collection of consumers with conventional preferences and specific initial endowments for which the core is empty. His seminal article with Shapley in 1974: ``On cores and indivisibilities,” marked the first victory in his battle tackling indivisibilities and has become a most-cited classic article in the field.

In the 1940s and 1950s, Dantzig and Koopmans had developed the activity analysis model of a production possibility set with constant returns to scale. When factor endowments are specified, the model leads directly to a linear program which can be solved by Dantzig’s simplex method. The method makes use of competitive prices to test for the optimality of a proposed feasible solution.

However, neither decreasing returns nor constant returns reflect economic reality. Since the beginning of the Industrial Revolution in the 1760s, economies of scale and increasing returns based on large indivisible pieces of machinery or forms of productive organization such as the assembly line are prominent features of every industrialized nation. Unfortunately, economic theory based on the assumption of convexity and perfect divisibility does not offer any clue to this challenging economic problem. The difficulty of dealing with indivisibilities has long been recognized by many leading economists including Lerner (1944), Koopmans and Beckmann (1957), and Debreu (1959), as Lerner (1944) points out: ``We see then that indivisibility leads to an expansion in the output of the firm, and this either makes the output big enough to render the indivisibility insignificant, or it destroys the perfection of competition. Significant indivisibility destroys perfect competition.”

Scarf was interested in economies with indivisibilities in production, i.e., where activity levels are constrained to be integers, an extreme form of non-convexity. When factor endowments are specified we are led to the general integer program for which there is no pricing test to detect whether a feasible production plan is indeed optimal. His major goals have been (1) to replace the pricing test by a local neighbourhood search and (2) to develop a mechanism for efficiently finding this test set. In the early 1980s, he made a decisive victory in achieving his first goal. Using his early concept of primitive sets arising in his research on the core and the computation of equilibria, Scarf succeeded in developing a quantity test set. He proved that this test set is unique and minimal, depending on the technology matrix alone and not on the specification of the particular factor endowment. It consists of a finite number of integral production plans. When this test set is available, one can easily use it to verify if a production plan is optimal or not, and if it is not optimal, one can use the test set to obtain a better production plan.

Scarf has worked with a group of mathematicians on this subject for many years. He has found several important special classes of technology matrices for which the test set can be easily identified. However, important questions remain open and the battle is not yet over, as he states in his 1983 Presidential Address of the Econometric Society (1986, Econometrica):``At the present time, I am far from being able to present a convincing argument which relates the structure of neighbourhood systems (i.e., test sets) to the administrative arrangements that might be taken by a large industrial enterprise.” Up to this very moment, his struggle goes on. Indeed, as a Chinese poem says: ``An old war-horse may be stabled, yet still it longs to gallop a thousand miles; and a noble-hearted man though advanced in years never abandons his proud aspirations.”