International crude oil markets don’t function like textbook markets and never have, and they don’t work like they did 15, 25, or 35 years ago either.

With high and increasing crude oil prices this year, everybody who comments on economics is weighing in on what he/she thinks are the causes and what should be done—or more often, what should not be done—about the situation. A large majority of commentators make the threshold error of assuming real-world crude oil markets work like the idealized models in their economics textbooks. They don't. Here is an excellent summary of how the methods of pricing OPEC crude oils at the producer level have changed several times over 40 years and how it really works today. Here's my summary of the summary.

In the mid-20th Century, international oil ownership and production was very much in the control of the Seven Sisters and a few smaller international oil companies like Occidental. In a foreign province, they obtained the concessions from the government, made the investments, owned the oil, determined how fast to pump it, set the prices, and simply paid a royalty to the host governments. Prices were administered entirely by the oil companies; they "posted" the selling prices and, very occasionally, changed them in their own discretion.

Then in the 1970s the big oil producing nations in rapid succession partially or totally nationalized their oil concessions and took over price administration. During this period, which lasted until the mid-1980s, the host governments set the prices and constrained production as necessary to enforce price discipline within the producers' cartel. But then demand plummeted, non-OPEC producers became significant, and there was so much excess capacity that discipline among producing nations could not be maintained. Companies that bought directly from the producing States refused to pay the posted prices, and started buying in the spot market or from non-OPEC members. OPEC's administered pricing system collapsed.

Saudi Arabia fought back by adopting a netback pricing system, where the price of the crude oil would be calculated from the selling prices of refined products by deducting a negotiated refining margin and transportation costs. Prices crashed from $26 to $10, and that ended netback pricing.

This was followed by a system of long-term contracts between OPEC members and their first purchasers in which the price of each shipment was set by reference to published reports of spot prices for marker crudes such as West Texas Intermediate and/or dated Brent (North Sea) crude oil. These reports were recognized as problematic for several reasons. Because actual sales of wet barrels in the reference markets were fairly infrequent, it was possible and useful for players to manipulate prices on critical dates. The reported prices were third-party estimates of prices based on surveys that were vulnerable to misrepresentations by the persons interviewed. Marker crudes were quite different from typical Middle East crudes in both quality and location. Marker crude production rates declined as Middle East crude production increased, making the tail smaller and the dog bigger.

To replace the unsatisfactory system based on reported spot prices, producing nations adopted the current system, which is to sell on long-term contracts with terms that price each shipment by reference to the crude oil futures market(s). The change was made because the futures markets seemed more transparent, larger and more liquid, and less susceptible to manipulation or misrepresentation than reported spot pricing. Long term contracts with this type of pricing provision are now the norm.

Today there are three distinct—but interrelated—crude oil markets: the primary long-term, reference-price contract market between OPEC members and first purchasers, spot markets for actual delivery of wet barrels, and the commodities futures markets for paper barrels. Probably a substantial number of transactions and prices in all three markets are never publicly disclosed. Thus, actual crude oil markets are not at all like the ideal markets assumed in textbooks and by most commentators on economics. The critical question, to which I have seen no convincing answer is how, to what extent, and under which circumstances each of these three markets—and the players in them—affect the other two markets.

Coda on blogging

The foregoing summary and most of my understanding about what's going on came from the blogosphere. Mainstream media reports were limited by space, reportorial understanding, and a bias for reporting controversies instead of information. In wonderful contrast, relevant parts of the blogosphere have experts arguing with one another in depth. Sure, the blogosphere contains vast quantities of wasted and erroneous words, but I have found several sites run by real and careful experts who elicit some very informed and insightful comments. Now I'd give up my subscription to the Los Angeles Times before I would stop following my favorite blogs.

Most of my improved understanding of oil prices came directly or indirectly from Paul Krugman's blog, where he started posting on oil prices in April 2008 and followed up with at least 6 more. At first, PK thought he would just explain to everybody that there is no speculative bubble because there is no physical hoarding. Full stop. He illustrated this with diagrams from a textbook. He got a storm of criticism, at least 10-20% of which was obviously well-informed, about practical realities of crude oil and/or futures markets. Some of the critics and supporters were obviously professional economists and some were recognized industry experts like Phil Verleger. Apparently, PK was also communicating with his peers off-line, and he linked to other blogs even when he didn't agree with them. I believe PK's own understanding changed considerably during these 10 weeks of dialog. Mine certainly did. It was a fascinating, collaborative, civil, intellectual process in which anybody could join.

PK's blog enriched me in another unexpected way. The citation to the report I summarized above was provided by JD, comment #35 to this Krugman post. [UPDATE 2/8/2011: The JD comment is now #16; apparently some comments were purged.] Before that, I had made a targeted online search for an explanation like this but could not find one and had given up. (JD provided other useful links too, as did other commenters.) Before I started regularly visiting selected blogs, I had assumed they would be useless as a source of basic facts and resources. I have found that the right blog can be a good tool for finding relevant and reliable information.

Update on Wednesday, July 23, 2008 at 09:16AM by Skeptic

Here's a news report about current futures market conditions that provides some insight into how actions of some market players affect, or might affect, other players, how the number of open postitions now is smaller and held in a more "normal" pattern than two weeks ago when crude oil prices were $20 higher, and the potential impact of some players entering into long term contracts. 

Update on Thursday, July 24, 2008 at 02:39PM by Skeptic

For specific examples of how it used to be and perspective from a former oil trader go here.  And for the same commentator's explanation about how current markets work and whether financial players are contributing to recent price increases go here.

Update on Tuesday, February 8, 2011 at 07:42AM by Skeptic

Among those engaging Krugman in the spring of 2008 over the question whether there are alternatives to the textbook position that commodities speculators can only affect prices by hoarding the commodity was Yves Smith. (I had not yet discovered her excellent blog.)  She engages Krugman again in her current blog post, Krugman, Commodity Prices, and Speculators. She links there to her July 3, 2008 post "Futures Prices Determine Physical Oil Prices" where she quotes from JD (presumably the same JD whose comment on PK's blog led me to the Oxford Energy paper I summarized in my post above) who has a really good, well-documented post on the subject here at Peak Oil Debunked. Yves' post today says she summarized the 2008 exchange in her book ECONNED.

I think Krugman lost the argument about crude oil price speculation because his explanation did not take into account the fact that spot prices for the largest volumes of transactions are contractually set by a basket of futures prices. On the other hand, his classical explanation that physical hoarding is necessary for speculation in food and other commodities could be correct--or not.  For me, it's an open question. 

Update on Monday, April 16, 2012 at 02:45PM by Skeptic

The Oxford Institute for Energy Studies paper I linked in the first paragraph of the original post has been taken down and superseded by An Anatomy of the Crude Oil Pricing System (January 2011). The author, Bassam Fattouh, greatly expands the earlier analysis and gives particular attention to the influence of futures markets on spot prices: 

The report also calls for broadening the empirical research to include the trading strategies of physical players. In recent years, the futures markets have attracted a wide range of financial players including swap dealers, pension funds, hedge funds, index investors, technical traders, and high net worth individuals. There are concerns that these financial players and their trading strategies could move the oil price away from the "true" underlying fundamentals. The fact remains however that the participants in many of the OTC markets such as forward markets and CFDs which are central to the price discovery process are mainly "physical" and include entities such as refineries, oil companies, downstream consumers, physical traders, and market makers. Financial players such as pension funds and index investors have limited presence in many of these markets. Thus, any analysis limited to non-commercial participants in the futures market and their role in the oil price formation process is incomplete and also potentially misleading. 

At 9. Among his conclusions (at 78):  

The assumption that the process of identifying the price of benchmarks in the current oil pricing system can be isolated from financial layers is rather simplistic. The analysis in this report shows that the different layers of the oil market are highly interconnected  and form a complex web of links, all of which play a role in the price discovery process. The information derived from financial layers is essential for identifying the price level of the benchmark. One could argue that without these financial layers it would not be possible to "discover" or "identify" oil prices in the current oil pricing system. In effect, crude oil prices are jointly co-determined and identified in both layers, depending on differences in timing, location and quality.

. . . .  The issue of whether the paper market drives the physical or the other way around is difficult to construct theoretically and test empirically in the context of the oil market.