The puzzle of the vacuum energy, and Einstein's mistake about Friedmann

In quantum field theory (string theory is another story, which Guth mostly sets aside) there are, he says, three contributions.

1. Zero-point fluctuations of boson fields

The simplest example is the electromagnetic field. In a classical vacuum the electric and magnetic fields would be zero, the lowest-energy state. But just as a particle cannot have a well-defined position and momentum at the same time, $\vec E$ and $\vec B$ cannot both be well-defined at once. So the fields cannot be zero and stay zero: they constantly fluctuate, and the fluctuations carry energy.

The mathematics is simple. Put the field in a box and describe it by standing waves, each fitting a whole number of half-wavelengths across the box. Each standing wave behaves — in quantum mechanics too — exactly like a harmonic oscillator, so each has a zero-point energy:

(Guth assigned this as a homework problem.) But there is no limit to how short a wavelength can be, so there are infinitely many standing waves, and the total diverges.

The divergence comes from short distances, which gives us an excuse: we only know the physics down to the wavelengths we can test in the laboratory. Beyond some length the reasoning that gives $\frac{1}{2}\hbar\omega$ may simply fail, so we stop adding there. A cutoff often invoked is the Planck length, where quantum gravity should become important — and our field theory leaves gravity out, and nobody yet knows how to put it in:

Two student questions concern the box. Is it legitimate to treat infinite space as the limit of a big box? Yes, Guth says: as the box grows, the energy grows in proportion to its volume, so the energy density settles to a fixed value. If it didn't — if the answer depended on how space is infinite — a laboratory experiment would tell you whether the universe is closed or infinite, which is far-fetched; and calculations in a large closed universe, with no box, give the same density. So is the box the observable universe? No: the energy density doesn't depend on the box at all once it is large, and only the energy density matters.

Now the catch. With the Planck-length cutoff, does the result come anywhere near the vacuum energy the astronomers measure? Emphatically not: it is too large by about 120 orders of magnitude — "even in cosmology", Guth says, a significant embarrassment. Nobody knows how to get a number as small as the one observed.

The energy-density formula and the numbers are this page's own estimate; Guth quotes only the result, about 120 orders of magnitude.

2. Fluctuations of fermion fields — with a minus sign

In quantum field theory every particle is described by a field, including the electron. Because electrons are fermions and obey the exclusion principle, their field behaves differently. Its fluctuations are of the same order, $\hbar\omega$ per mode, but — strangely — the contribution is negative.

Textbooks explain the sign by pointing to an anticommutator in some equation. Guth finds that a statement of where the minus sign appears rather than what it means, and offers Dirac's old picture instead. The energy levels of the Dirac equation are symmetric about zero: for every positive-energy state there is a negative-energy one. Since the vacuum is by definition the lowest-energy state, Dirac proposed that in the vacuum all the negative-energy states are filled — the Dirac sea. An electron is an extra particle in a positive-energy state; a positron is a hole in the sea.

The picture is little used today because it makes electrons and positrons look different, though Dirac knew the mathematics treats them symmetrically. But it shows where the negative energy comes from: start from zero with every level empty, and each negative-energy level you fill lowers the energy, without limit. The algebra is exactly right — the textbook anticommutator expression is precisely the energy of filling the sea.

Can the positive boson energy and the negative fermion energy cancel? asks a student. Very likely they do, Guth says — but we have no idea why. Cut off both, each enormous, and add them: our ignorance allows any answer. That the sum comes out incredibly close to zero, far smaller than either piece, means a cancellation is happening that we cannot explain.

There is one setting where cancellation is expected: supersymmetry, a perfect symmetry between bosons and fermions. Each particle would have a partner — the photon a spin-½ photino, the electron a spin-0 selectron — and in an exactly supersymmetric theory the vacuum energy is exactly zero. But no partner has ever been seen, so the world is not exactly supersymmetric. Many physicists think, for good reasons, that supersymmetry exists as a broken symmetry. Then the cancellation is no longer guaranteed; estimating the mismatch from the supersymmetry-breaking scale reduces the discrepancy from about 120 to about 50 orders of magnitude — much better, but not good enough.

3. Fields with non-zero values in the vacuum

Finally, some fields are believed to be non-zero even in the vacuum. The famous example is the Higgs field, whose particle was discovered at CERN in 2012, a year before this lecture, after a search lasting decades. [Guth says over 50 years; the field was proposed in 1964, so the search had lasted nearly 50.] In the Standard Model it may be the only such field; grand unified theories have many more. This contribution is probably finite, so it is less troublesome than the first two.

The bottom line: it is easy for particle physicists to understand why the vacuum energy should be non-zero, and very hard to see why it has the value it has.

Guth mentions, to return to at the end of the course, an explanation that shows how desperate physicists are. String theory suggests not one vacuum but an enormous number of them — perhaps $10^{500}$. A typical one would have a vacuum energy of order the Planck scale, but the values would spread from plus to minus the Planck scale, so a tiny fraction would have a very small vacuum energy like ours.

Why would we live in such an unusual vacuum? Perhaps because life can only form there. This is not built from whole cloth: vacuum energy enters the Friedmann equation. A Planck-scale positive vacuum energy would blow the universe apart within roughly a Planck time ($10^{-40}$-something seconds, as Guth puts it), and a Planck-scale negative one would make it collapse just as fast. If life needs billions of years to evolve, and assuming nothing else about it, life can only exist where the vacuum energy is incredibly small. Every observer in such a multiverse would see a tiny vacuum energy, even though a random place would most likely have one near the Planck scale.

Before moving on, Guth tells a story about the Friedmann equations and Einstein. He likes it for two reasons: it is comforting that perhaps the greatest physicist of all time could make simple mistakes like everyone else, and it shows how important it is to keep an open mind. Einstein was so convinced the universe was static that he made real mistakes defending it.

Alexander Friedmann was a Russian meteorologist — an expert in partial differential equations — who became interested in the new theory of general relativity. He wrote two papers on what we now call the Friedmann equations, one for closed universes and one for open ones. The first appeared in the leading physics journal of the day, the Zeitschrift für Physik, received on 29 June 1922. [Guth first calls this the publication date, then correctly the date the paper was received.]

Within a few months Einstein sent in a comment, received 18 September 1922, claiming the paper was mathematically wrong. He argued that the field equations, which require energy and momentum to be conserved, together with Friedmann's assumptions, force the density to be constant in time, $\partial\rho/\partial x_4 = 0$ — and hence the radius of the universe, our $a(t)$, too. On that reading, all Friedmann's paper showed was that Einstein's static solution is the only one. This had no basis in mathematics. Guth quotes Einstein's notes in the English translation collected by Jeremy Bernstein and Gary Feinberg in Cosmological Constants.

The rest of the story follows Friedmann's friend Yuri Krutkov, who was visiting Berlin:

• Friedmann learned of the rebuttal from Krutkov in December 1922, tracked it down, and wrote Einstein a detailed letter explaining his error — a gutsy thing to do. Einstein was travelling and did not read it until much later.
• In May 1923 Krutkov and Einstein met at Leiden, at Lorentz's farewell lecture. Krutkov's letters to his sister in Petrograd (now St Petersburg), quoted in the biography Alexander A. Friedmann: The Man Who Made the Universe Expand by Tropp, Frenkel and Chernin, record that on 7 May he read Friedmann's paper together with Einstein, and on 18 May that he had won the argument — "Petrograd's honor is saved!"
• On 31 May 1923 Einstein submitted a retraction: his objection had rested on a calculational error, as Krutkov in person and Friedmann's letter had convinced him; Friedmann's results were correct and clarifying, and the field equations have time-varying solutions as well as the static one.

A final twist, from Pierre Kerszberg's The Invented Universe: Einstein's draft of the retraction survives, and in it he had written, and then crossed out, a remark that the new solutions could hardly have any physical significance. Even convinced that they solved the equations, his instinct was that they couldn't be physical, because the universe is static. Logic prevailed before he submitted it — fortunately, since Friedmann's expanding solution is exactly the one we now consider physically significant.

In the last minutes Guth introduces two problems of the conventional Big Bang — the theory developed so far, without inflation. Both concern the initial conditions needed to make it work: the horizon (or horizon–homogeneity) problem and the flatness problem.

The CMB has the same temperature in all directions to about one part in 100,000. Systems do reach a uniform temperature — a room heated in one corner evens out, a hot slice of pizza cools — so one might hope the universe simply had time to equalise (the zeroth law of thermodynamics). It did not. Tracing back CMB photons arriving from opposite directions, their emission points turn out to be about 50 horizon distances apart: no physical influence, limited by the speed of light, could have connected them. Asked whether the uniformity could have been set up when the universe was smaller, Guth answers that in the conventional model the separation is about 50 horizon distances no matter how early you look. One can simply assume the singularity started everything uniform — that contradicts nothing — but it is speculation, not a theory. Inflation, he promises, gives a dynamical explanation.

• Quantum field theory gives the vacuum energy from (1) boson zero-point energies, $\frac{1}{2}\hbar\omega$ per mode, positive and divergent; (2) fermion fluctuations, negative (the filled Dirac sea); (3) fields like the Higgs field that are non-zero in the vacuum.
• Cutting off at the Planck length $\sqrt{\hbar G/c^3} \approx 1.6 \times 10^{-33}$ cm overshoots the observed density by about 120 orders of magnitude; broken supersymmetry reduces this to about 50. The near-cancellation is unexplained.
• One speculative answer: among perhaps $10^{500}$ string-theory vacua, life can only arise in the rare ones with tiny vacuum energy.
• In 1922 Einstein wrongly "refuted" Friedmann's expanding solutions; in May 1923, persuaded by Krutkov and Friedmann's letter, he retracted — after crossing out a remark that they had no physical significance.
• Next: the horizon and flatness problems of the conventional Big Bang.