People respond to incentives. When a product is profitable, that gives people an incentive to develop substitutes. These developments do not magically come about. They are the result of hard work, luck, and opportunity, and sometimes it takes awhile. From the Wall Street Journal:
From the time of the California energy crisis at the beginning of this decade, it appeared that the U.S. was headed for an extended period of tight supplies, even shortages, of natural gas.
While gas has many favorable attributes—as a clean, relatively low-carbon fuel—abundance did not appear to be one of them. Prices had gone up, but increased drilling failed to bring forth additional supplies. The U.S., it seemed, was destined to become much more integrated into the global gas market, with increasing imports of liquefied natural gas (LNG).
But a few companies were trying to solve a perennial problem: how to liberate shale gas—the plentiful natural gas supplies locked away in the impermeable shale. The experimental lab was a sprawling area called the Barnett Shale in the environs of Fort Worth, Texas.
The companies were experimenting with two technologies. One was horizontal drilling. Instead of merely drilling straight down into the resource, horizontal wells go sideways after a certain depth, opening up a much larger area of the resource-bearing formation.
The other technology is known as hydraulic fracturing, or "fraccing." Here, the producer injects a mixture of water and sand at high pressure to create multiple fractures throughout the rock, liberating the trapped gas to flow into the well.
The critical but little-recognized breakthrough was early in this decade—finding a way to meld together these two increasingly complex technologies to finally crack the shale rock, and thus crack the code for a major new resource. It was not a single eureka moment, but rather the result of incremental experimentation and technical skill. The success freed the gas to flow in greater volumes and at a much lower unit cost than previously thought possible.
In the last few years, the revolution has spread into other shale plays, from Louisiana and Arkansas to Pennsylvania and New York State, and British Columbia as well.
The supply impact has been dramatic. In the lower 48, states thought to be in decline as a natural gas source, production surged an astonishing 15% from the beginning of 2007 to mid-2008. This increase is more than most other countries produce in total.
The low-hanging fruit has already been picked. But the more profitable it becomes to pick ever-higher fruit, the more likely it will be to be picked. That's the profit motive at work.