Casimir Puzzle and Casimir Conundrum: Discovery and Search for Resolution

Abstract

This paper provides a review of the complicated problems in Lifshitz theory describing the Casimir force between real material plates composed of metals and dielectrics, including different approaches to their resolution. For both metallic plates with perfect crystal lattices and any dielectric plates, we show that the Casimir entropy calculated in the framework of Lifshitz theory violates the Nernst heat theorem when the well-approved dielectric functions are used in computations. The respective theoretical Casimir forces are excluded by the measurement data of numerous precision experiments. In the literature, this situation has been called the Casimir puzzle and the Casimir conundrum for the cases of metallic and dielectric plates, respectively. This review presents a summary of both the main theoretical and experimental findings on this subject. Next, a discussion is provided of the main approaches proposed in the literature to bring the Lifshitz theory into agreement with the measurement data and with the laws of thermodynamics. Special attention is paid to the recently suggested spatially nonlocal Drude-like response functions, which consider the relaxation properties of conduction electrons, as does the standard Drude model, but lead to the theoretical results being in agreement with both thermodynamics and the measurement data through the alternative response to quantum fluctuations of the mass shell. Further advances and trends in this field of research are discussed.

1. Introduction

The Casimir effect is one of the counterintuitive physical phenomena. It is somewhat against expectations that two parallel, electrically neutral ideal metal plates at zero temperature spaced in vacuum at a distance a attract each other. However, Casimir [1] proved that there is an attractive force per unit area of the plates in this situation, which depends on a, the Planck constant ℏ, and the speed of light c

$$F_{IM}\left(a\right)=-\frac{{\pi }^2}{240}\frac{\hslash c}{a^4}.$$

(1)

The Casimir force arises due to the zero-point oscillations of a quantized electromagnetic field. In free space, the energy density of these oscillations is infinitely large and unobservable. All the physical energies are measured from this infinity. In the presence of parallel plates, however, the tangential component of the electric field and the normal component of the magnetic induction vanish on their surfaces, and the third component of the wave vector perpendicular to the plates becomes discrete. In so doing, the vacuum energy density in the presence of plates remains infinitely large but if we, following the common procedure, subtract from it the vacuum energy density in free space, the result is finite

$$E_{IM}\left(a\right)=-\frac{{\pi }^2}{720}\frac{\hslash c}{a^3}.$$

(2)

This is the Casimir energy per unit area of the plates. In this case, (1) follows from (2) by the negative differentiation with respect to a. Thus, the Casimir effect is directly connected with the concept of the quantum vacuum, which is the most fundamental and not clearly understood type of physical reality. Despite the renormalization and regularization procedures that have been elaborated in detail, which help to obtain physically meaningful calculation results, the enormously large vacuum energy obtained after the momentum cut-off is sometimes considered as catastrophic [2]. The energy of the quantum vacuum is directly connected with the cosmological constant [3], dark energy, and the acceleration of the Universe expansion [4]. This places the Casimir effect on the list of closely connected greatest problems of modern physics. It is not surprising then that experimental and theoretical investigations of the Casimir force have been met with outstanding problems, being named the Casimir puzzle and the Casimir conundrum.

At first, no fundamental problems in the theory of the Casimir effect were anticipated. Lifshitz [5,6] developed the general theory expressing the Casimir free energy and force between two thick plates (semispaces) kept at any temperature T and composed of real materials described by the frequency-dependent dielectric permittivities. This theory was generalized for plates possessing magnetic properties described by the frequency-dependent magnetic permeability [7]. It was shown that the familiar van der Waals force is nothing more than the Casimir force at separations between the plates of below a few nanometers. At so short separations the speed of light can be considered as infinitely large. The Casimir results (1) and (2) were obtained from Lifshitz theory at $T=0$ for the plates composed of a perfectly reflecting material. The Lifshitz results for two thick plates were generalized for the case of planar systems containing any number of material layers with different dielectric permittivities [8].

The first experiments measuring the Casimir force were not sufficiently precise and have only been found to be in qualitative agreement with theoretical predictions of Lifshitz theory (see [9] for a review). It was shown [10], however, that at large separations, the thermal Casimir force between metallic plates calculated using Lifshitz theory and the dielectric permittivity of the dissipative Drude model is equal to one-half of that found for ideal metal plates at nonzero temperature. This result generated serious incomprehension. At low frequencies, which determine the force behavior at large separations, the Drude model was well-tested in numerous experiments and in technical applications. However, it was expected that with increasing separation, the plate metal would behave increasingly closer to the ideal one. This is why the special prescription was proposed [10], which is supplementary to the formalism of Lifshitz theory (see below).

Attention to the internal problems of this theory was attracted in 2000 after a publication of the following result. It was found [11] that at short separations the thermal correction to the Casimir force computed using the Drude model is relatively large and decreases the force magnitude, i.e., corresponds to a repulsion, which is quite unexpected. By contrast, Lifshitz theory combined with the dielectric permittivity of the dissipationless plasma model (which should not be applicable at low frequencies) predicted a smooth approach of the Casimir force to the ideal metal value at large separations and a very small thermal correction at short separation. The latter is of the same sign as the zero-temperature force and only slightly increases its magnitude [12].

These problems were highlighted by an inconsistency of the Lifshitz theory using the Drude model with the laws of thermodynamics. It was proven [13,14,15] that in the configuration of two metallic plates with perfect crystal lattices described by the Drude model the Casimir entropy goes with vanishing temperature to a nonzero negative constant depending on the parameters of a system in violation of the third law of thermodynamics (the Nernst heat theorem). It was also shown [13,14,15] that Lifshitz theory using the plasma model leads to positive Casimir entropy, which goes to zero when the temperature vanishes in agreement with the Nernst heat theorem. More recently, the same results were obtained for several other Casimir configurations [16,17,18,19].

Precise measurements of the Casimir interaction between metallic test bodies added to the complexity of this situation. In a series of experiments performed using a micromechanical torsional oscillator and an atomic force microscope, Lifshitz theory using the Drude model at low frequencies was unambiguously excluded by the measurement data, whereas the same theory using the plasma model was found to be in good agreement with the measurement results [20,21,22,23,24,25,26,27]. The dielectric permittivity of a metal in these experiments was found from the measured optical data for its complex index of refraction extrapolated using either the Drude or the plasma model in the region of low frequencies where optical data are not available. Note that in the single experiment, which was interpreted in favor of the Drude model [28], the Casimir force was not directly measured but extracted using the fitting procedure from up to an order of magnitude larger force presumably determined by the surface patches. It was shown [29,30] that the interpretation of this experiment suffers from serious uncertainties.

The situation with dielectric test bodies is closely parallel to that with metallic ones. It was found that the Casimir entropy calculated using Lifshitz theory for ideal dielectrics (insulators possessing zero electrical conductivity) satisfies the Nernst heat theorem [31,32,33,34,35,36]. However, at any nonzero temperature, the dielectric permittivity of real dielectric bodies includes a term describing small but nonzero electric conductivity at a constant current (the DC conductivity). If this term is taken into account, as it should be, the Casimir entropy calculated using Lifshitz theory goes to a nonzero positive constant depending on the parameters of a system with vanishing temperature, i.e., violates the Nernst heat theorem [31,32,33,34,35,36].

The experimental results obtained for dielectric test bodies also resemble the case of metallic plates. If the DC conductivity of the plate material is taken into account, the predictions of Lifshitz theory are excluded by the measurement data [37,38,39,40,41]. If, however, the DC conductivity is simply omitted in calculations, the theoretical results are found to be in agreement with the data [37,38,39,40,41,42]. Once again, as in the case of metals, the thermodynamically consistent theory agrees with the experimental results. This is reached, however, by disregarding the real physical phenomena: the dissipation of conduction electrons for metals and the DC conductivity for dielectrics. This is why the above problems have been called in the literature the Casimir puzzle and the Casimir conundrum [43,44,45,46,47]. For metals and dielectrics, the above inconsistencies originate from different sources (see below), so it was suggested [43,47] that they be called a puzzle and a conundrum, respectively.

In this article, we review different efforts to understand both the puzzle and the conundrum in Casimir physics undertaken in the literature during the last 20 years. We start with the more rigorous mathematical formulation of the essence of these problems and elucidate in more detail the theoretical and experimental parts of the Casimir puzzle for metallic and the Casimir conundrum for dielectric test bodies. The basically viable approaches to the resolution of these problems can be of two types. The approaches of the first type search for some physical effects that could impact the measurement results but were not taken into account (or were accounted for improperly) in the comparison between experiment and theory. Among them is the role of impurities of a crystal lattice and the respective residual relaxation, a possibility of the alternative sets of optical data for the complex index of refraction, an impact of the surface roughness and patch potentials, etc. (see below for the references to each of these approaches).

The approaches of the second type admit that some serious modifications of Lifshitz theory might be necessary for a resolution of the above problems. These approaches deal with generalizations of Lifshitz theory for the case of configurations with nonplanar boundaries, considering spatial dispersion, screening effects, etc. (the related references are provided below). In this review, we consider both types of attempts to find a resolution to the Casimir puzzle and the Casimir conundrum, including a very recent one that shows considerable promise.

The review is organized as follows: In Section 2, we briefly present the formulations of Lifshitz theory in terms of real and imaginary frequencies. In Section 3, a correlation is described between the thermal Casimir forces for ideal and real metals. The thermodynamic and experimental parts of the Casimir puzzle for real metals are considered in Section 4. Section 5 is devoted to the thermal Casimir force between ideal and real dielectrics and Section 6 to the thermodynamic and experimental parts of the Casimir conundrum. In Section 7, we summarize the main approaches to a resolution to the Casimir puzzle and the Casimir conundrum. In Section 8, the new way toward resolving the Casimir puzzle is considered. Section 9 is devoted to the discussion, and Section 10 provides our conclusions.

2. Lifshitz Theory of the Casimir Force

There are different approaches to derivation of the Lifshitz theory starting from the fluctuation-dissipation theorem of statistical physics [5,6,48] and quantum electrodynamics with continuity boundary conditions on the boundary surfaces [9,49,50]. For a discussion of the Casimir puzzle and conundrum, we only need the expression for the Casimir free energy and force in the configuration of two thick plates (semispaces) separated by a gap of width a and kept at temperature T in thermal equilibrium with the environment. In the framework of Lifshitz theory, it is assumed that the material of the plates is described by the frequency-dependent dielectric permittivity $\epsilon \left(\omega \right)$ and magnetic permeability $\mu \left(\omega \right)$. This means that the original formulation of the Lifshitz theory does not consider the spatial dispersion. If the plate material is nonmagnetic, $\mu$ should be put equal to unity.

2.1. Formulation in Terms of Real Frequencies

The immediate result of derivation using the fluctuation-dissipation theorem is the following Lifshitz formula for the Casimir free energy per unit area of the plates

$$\begin{array}{cc}\hfill \mathcal{F}(a,T)& =\frac{\hslash }{4{\pi }^2}{\int }_0^{\infty }{k}_{\perp }d{k}_{\perp }{\int }_0^{\infty }d\omega coth\frac{\hslash \omega }{2k_{B}T}\hfill \\ & \times \mathrm{Im}\left\{ln\left[1-r_{\mathrm{TM}}^2(\omega ,k_{\perp })e^{-2aq}\right]+ln\left[1-r_{\mathrm{TE}}^2(\omega ,k_{\perp })e^{-2aq}\right]\right\}.\hfill \end{array}$$

(3)

where $k_{\perp }=\sqrt{k_1^2+k_2^2}$ is the projection of the wave vector on a plane of the plates (it is perpendicular to the Casimir force), $\omega$ is the frequency, $k_B$ is the Boltzmann constant, and $q=q(\omega ,k_{\perp })={(k_{\perp }^2-{\omega }^2/c^2)}^{1/2}$. The reflection coefficients for the transverse magnetic (TM) and transverse electric (TE) polarizations of the electromagnetic field are given by

$$\begin{array}{ccc}& & r_{\mathrm{TM}}(\omega ,k_{\perp })=\frac{\epsilon \left(\omega \right)q(\omega ,k_{\perp })-k(\omega ,k_{\perp })}{\epsilon \left(\omega \right)q(\omega ,k_{\perp })+k(\omega ,k_{\perp })},\hfill \\ & & r_{\mathrm{TE}}(\omega ,k_{\perp })=\frac{\mu \left(\omega \right)q(\omega ,k_{\perp })-k(\omega ,k_{\perp })}{\mu \left(\omega \right)q(\omega ,k_{\perp })+k(\omega ,k_{\perp })},\hfill \end{array}$$

(4)

where

$$k(\omega ,k_{\perp })={\left[k_{\perp }^2-\epsilon \left(\omega \right)\mu \left(\omega \right)\frac{{\omega }^2}{c^2}\right]}^{1/2}.$$

(5)

Equation (4) provides the familiar Fresnel reflection coefficients of classical electrodynamics.

The Casimir force per unit area of the plates is obtained from (3) by the negative differentiation with respect to a

$$\begin{array}{cc}\hfill F(a,T)& =-\frac{\hslash }{2{\pi }^2}{\int }_0^{\infty }{k}_{\perp }d{k}_{\perp }{\int }_0^{\infty }d\omega coth\frac{\hslash \omega }{2k_{B}T}\hfill \\ & \times \mathrm{Im}\left\{q{\left[r_{\mathrm{TM}}^{-2}(\omega ,k_{\perp })e^{2aq}-1\right]}^{-1}+q{\left[r_{\mathrm{TE}}^{-2}(\omega ,k_{\perp })e^{2aq}-1\right]}^{-1}\right\}.\hfill \end{array}$$

(6)

The integration with respect to $k_{\perp }$ in (3) and (6) is performed from 0 to ∞ at any fixed $\omega$. In so doing, the integration over $k_{\perp }>\omega /c$ describes the contribution from the off-the-mass-shell electromagnetic fluctuations, which are also often called evanescent waves, as opposed to propagating waves, which are characterized by $k_{\perp }⩽\omega /c$ and correspond to the on-the-mass-shell fluctuations.

Note that the factor $coth[\hslash \omega /\left(2k_{B}T\right)]$ in (3) and (6) implies an origin of the Casimir force at $T\ne 0$ from both the zero point and thermal fluctuations of the electromagnetic field. This factor arises from the free energy of an oscillator

$$\frac{\hslash \omega }{2}+k_{B}Tln\left(1-e^{-\frac{\hslash \omega }{k_{B}T}}\right)=k_{B}Tln\left(2sinh\frac{\hslash \omega }{2k_{B}T}\right)$$

(7)

after an application of Cauchy theorem [9].

For the off-the-mass-shell fluctuations, the quantity q is real. Because of this, the contribution of these fluctuations to the Casimir free energy and force can be calculated using (3) and (6). However, for the on-the-mass-shell fluctuations, the quantity q becomes purely imaginary. As a result, both (3) and (6) become the integrals of rapidly oscillating functions, which plagues their calculation. Because of this, another mathematically equivalent formulation of the expressions (3) and (6) was suggested [5].

2.2. Formulation in Terms of Imaginary Matsubara Frequencies

This formulation is based on the fact that dielectric permittivity is a causal function which does not possess either poles or zeros in the upper half plane of complex frequency. As a consequence, the dielectric permittivity along the imaginary frequency axis takes the real values $\epsilon \left(\mathrm{i}\xi \right)$. Finally, the equivalent formulation of (3) is given by the Lifshitz formula

$$\begin{array}{ccc}\hfill \mathcal{F}(a,T)& =\frac{k_{B}T}{2\pi }\hfill & \sum _{l=0}^{\infty }{\phantom{sum}}^{\prime }{\int }_0^{\infty }{k}_{\perp }d{k}_{\perp }\left\{ln\left[1-r_{\mathrm{TM}}^2({\mathrm{i}\xi }_l,k_{\perp })e^{-2a{q}_l}\right]\right.\hfill \\ & & +\left.ln\left[1-r_{\mathrm{TE}}^2({\mathrm{i}\xi }_l,k_{\perp })e^{-2a{q}_l}\right]\right\},\hfill \end{array}$$

(8)

where the prime on the summation sign divides the term with $l=0$ by 2 and

$${\xi }_l=\frac{2\pi k_{B}Tl}{\hslash },l=0,1,2,\dots$$

(9)

are the Matsubara frequencies. The quantities $q_l$, $r_{\mathrm{TM}}({\mathrm{i}\xi }_l,k_{\perp })$, and $r_{\mathrm{TE}}({\mathrm{i}\xi }_l,k_{\perp })$ are obtained from $q(\omega ,k_{\perp })$, $r_{\mathrm{TM}}(\omega ,k_{\perp })$, and $r_{\mathrm{TE}}(\omega ,k_{\perp })$ defined above by a replacement of $\omega$ with ${\mathrm{i}\xi }_l$.

Similarly, the expression (6) for the Casimir force takes the form

$$\begin{array}{c}\hfill F(a,T)=-\frac{k_{B}T}{\pi }\sum _{l=0}^{\infty }{\phantom{sum}}^{\prime }{\int }_0^{\infty }{q}_l{k}_{\perp }d{k}_{\perp }\left\{{\left[r_{\mathrm{TM}}^{-2}({\mathrm{i}\xi }_l,k_{\perp })e^{2a{q}_l}-1\right]}^{-1}\right.\hfill \\ \hfill +\left.{\left[r_{\mathrm{TE}}^{-2}({\mathrm{i}\xi }_l,k_{\perp })e^{2a{q}_l}-1\right]}^{-1}\right\}.\hfill \end{array}$$

(10)

The expressions (8) and (10) are convenient for both analytical and numerical calculations because $q_l$ is a real positive number at any l.

3. Thermal Casimir Force between Ideal and Real Metals: First Surprise

The Casimir free energy (8) and force (10) are defined at nonzero temperature T and, therefore, are called thermal. In the limiting case $T\to 0$, we have

$$k_{B}T\sum _{l=0}^{\infty }{\phantom{sum}}^{\prime }\to \frac{\hslash }{2\pi }{\int }_0^{\infty }d\xi ,$$

(11)

the discrete Matsubara frequencies ${\xi }_l$ are replaced with the continuous frequency $\xi$, $q_l$ is replaced with $q={(k_{\perp }^2+{\xi }^2/c^2)}^{1/2}$, and we obtain from (8) and (10) the expressions for the Casimir energy $E\left(a\right)$ and force $F\left(a\right)$ between real metal plates at zero temperature.

The ideal metal is a perfect conductor at all frequencies including the zero frequency, so that

$$r_{\mathrm{TM}}(\mathrm{i}\xi ,k_{\perp })=-r_{\mathrm{TE}}(\mathrm{i}\xi ,k_{\perp })=1.$$

(12)

This is in line with the prescription [10], which demands that we consider the limit of $\epsilon$ to infinity in (4) first and the limit of frequency to zero second.

Introducing into the Lifshitz formula (8) written at $T=0$ with account of (11) the dimensionless variables $y=2aq$ and $\zeta =2a\xi /c$, and using (12), we find the Casimir energy between ideal metal plates

$$E_{\mathrm{IM}}(a,0)=\frac{\hslash c}{16{\pi }^2{a}^3}{\int }_0^{\infty }d\zeta {\int }_{\zeta }^{\infty }ydyln(1-e^{-y})=-\frac{{\pi }^2}{720}\frac{\hslash c}{a^3},$$

(13)

which agrees with the Casimir result (2).

Similarly, the Lifshitz formula (10) written at $T=0$ for ideal metal plates results in

$$F_{\mathrm{IM}}(a,0)=-\frac{\hslash c}{16{\pi }^2{a}^4}{\int }_0^{\infty }d\zeta {\int }_{\zeta }^{\infty }\frac{y^{2}dy}{e^y-1}=-\frac{{\pi }^2}{240}\frac{\hslash c}{a^4},$$

(14)

as was originally obtained by Casimir [1] considering the zero-point oscillations of a quantized electromagnetic field.

An application of the Lifshitz theory at nonzero temperature is more involved. We illustrate it first in the case of large separation distances between the plates. In this case, all terms of (8) and (10) with $l⩾1$ are exponentially small, and only the term with $l=0$ determines the total result. The formal condition for the application of large separation limit is [9]

$$a\gg \frac{\hslash c}{4\pi k_{B}T}.$$

(15)

At room temperature $T=300$ K, this gives $a\gg 0.64$ μm; so that at $a⩾6$ μm, we can safely use only the terms of (8) and (10) with $l=0$.

Then, for ideal metal plates, i.e., under condition (12), the Casimir free energy and force at large separations are given by

$$\begin{array}{ccc}& & {\mathcal{F}}_{\mathrm{IM}}(a,T)=\frac{k_{B}T}{2\pi }{\int }_0^{\infty }{k}_{\perp }d{k}_{\perp }ln(1-e^{-2a{k}_{\perp }})=-\frac{k_{B}T}{8\pi a^2}\zeta \left(3\right),\hfill \\ & & F_{\mathrm{IM}}(a,T)=-\frac{k_{B}T}{\pi }{\int }_0^{\infty }\frac{k_{\perp }^{2}d{k}_{\perp }}{e^{2a{k}_{\perp }}-1}=-\frac{k_{B}T}{4\pi a^3}\zeta \left(3\right),\hfill \end{array}$$

(16)

where $\zeta \left(z\right)$ is the Riemann zeta function.

For real metals at large separations between the plates, we might expect a similar result. This expectation, however, lacks of support from calculations. The point is that the low-frequency response of metals to the electromagnetic field is described by the dissipative Drude model

$${\epsilon }_D\left(\omega \right)=1-\frac{{\omega }_p^2}{\omega [\omega +\mathrm{i}\gamma (T\left)\right]},{\epsilon }_D\left(\mathrm{i}\xi \right)=1+\frac{{\omega }_p^2}{\xi [\xi +\gamma (T\left)\right]},$$

(17)

where ${\omega }_p$ is the plasma frequency and $\gamma \left(T\right)\ll {\omega }_p$ is the relaxation parameter.

For the nonmagnetic metals, we have from (5)

$$\underset{\xi \to 0}{lim}{\xi }^2{\epsilon }_D\left(\mathrm{i}\xi \right)=0,\underset{\xi \to 0}{lim}k(\mathrm{i}\xi ,k_{\perp })=k_{\perp }.$$

(18)

Then, from (4), we obtain

$$r_{\mathrm{TM},D}(0,k_{\perp })=1,r_{\mathrm{TE},D}(0,k_{\perp })=0.$$

(19)

This is in contradiction with (12) which is valid for an ideal metal. As a result, in the limiting case of large separations, the terms of (8) and (10) with $l=0$ are equal to

$$\begin{array}{ccc}& & {\mathcal{F}}_D(a,T)=\frac{k_{B}T}{4\pi }{\int }_0^{\infty }{k}_{\perp }d{k}_{\perp }ln(1-e^{-2a{k}_{\perp }})=-\frac{k_{B}T}{16\pi a^2}\zeta \left(3\right),\hfill \\ & & F_D(a,T)=-\frac{k_{B}T}{2\pi }{\int }_0^{\infty }\frac{k_{\perp }^{2}d{k}_{\perp }}{e^{2a{k}_{\perp }}-1}=-\frac{k_{B}T}{8\pi a^3}\zeta \left(3\right),\hfill \end{array}$$

(20)

i.e., by a factor of 2 smaller in magnitude than for an ideal metal in (16).

If, instead of (17), the dissipationless plasma model

$${\epsilon }_p\left(\omega \right)=1-\frac{{\omega }_p^2}{{\omega }^2},{\epsilon }_D\left(\mathrm{i}\xi \right)=1+\frac{{\omega }_p^2}{{\xi }^2},$$

(21)

is used, we find

$$\underset{\xi \to 0}{lim}{\xi }^2{\epsilon }_p\left(\mathrm{i}\xi \right)={\omega }_p^2,\underset{\xi \to 0}{lim}k(\mathrm{i}\xi ,k_{\perp })={\left(k_{\perp }^2+\frac{{\omega }_p^2}{c^2}\right)}^{1/2}.$$

(22)

This leads to the following values of the reflection coefficients at zero frequency:

$$r_{\mathrm{TM},p}(0,k_{\perp })=1,r_{\mathrm{TE},p}(0,k_{\perp })=\frac{c{k}_{\perp }-\sqrt{c^2{k}_{\perp }^2+{\omega }_p^2}}{c{k}_{\perp }+\sqrt{c^2{k}_{\perp }^2+{\omega }_p^2}}.$$

(23)

For the Casimir free energy and force defined using the plasma model at large separations, Lifshitz theory leads to

$$\begin{array}{ccc}& & {\mathcal{F}}_D(a,T)=-\frac{k_{B}T}{16\pi a^2}\zeta \left(3\right)+\frac{k_{B}T}{4\pi }{\int }_0^{\infty }{k}_{\perp }d{k}_{\perp }ln[1-r_{\mathrm{TE},p}^2(0,k_{\perp })e^{-2a{k}_{\perp }}],\hfill \\ & & F_D(a,T)=-\frac{k_{B}T}{8\pi a^3}\zeta \left(3\right)-\frac{k_{B}T}{2\pi }{\int }_0^{\infty }\frac{k_{\perp }^{2}d{k}_{\perp }}{r_{\mathrm{TE},p}^{-2}(0,k_{\perp })e^{2a{k}_{\perp }}-1},\hfill \end{array}$$

(24)

where $r_{\mathrm{TE},p}(0,k_{\perp })$ is defined in (23). Taking into account that in the limiting case of ideal metal (${\omega }_p\to \infty$) it holds

$$\underset{{\omega }_p\to \infty }{lim}{r}_{\mathrm{TE},p}(0,k_{\perp })=-1$$

(25)

in accordance with (23), we conclude that the results (24) obtained for real metal described by the plasma model join smoothly with the respective results (16) obtained for ideal metal plates. This could be considered an advantage of the plasma model as compared with the Drude model.

It should be remembered, however, that in the region of quasistatic frequencies the relaxation properties of conduction electrons in metals, disregarded by the plasma model, play an important role and should be taken into account. Moreover, according to the Bohr-van Leeuwen theorem of classical statistical physics, the TE electromagnetic fields should decouple from matter in the classical limit. According to the results of [51], this leads to the zero value of $r_{\mathrm{TE}}$ in this limit (see, however, Section 7 concerning the role of quantum fluctuations off the mass shell at all separations).

Complexity was added to this problem by a calculation of the thermal correction to the Casimir force between metallic plates described by the Drude model [11]. It turned out that this correction takes relatively large in magnitude negative values and, thus, decreases the magnitude of the Casimir force, which is a counterintuitive result. We recall that for the ideal metal plates at room temperature the thermal correction to the zero-temperature force at separations below $1$ μm is negligibly small. This result is justified by the smallness of parameter $T/T_{\mathrm{eff}}$, where $k_B{T}_{\mathrm{eff}}=\hslash c/\left(2a\right)$.

To illustrate a fundamental difference between the thermal effects in the Casimir force predicted by the Drude and plasma models, we consider the quantity

$${\delta }_T{F}_{D\left(p\right)}(a,T)=\frac{F_{D\left(p\right)}(a,T)-F_{D\left(p\right)}(a,0)}{F_{D\left(p\right)}(a,T)}$$

(26)

computed by (10) using the Drude model (17) and the plasma (21) dielectric permittivities of gold with ${\omega }_p=9.0$ eV and $\gamma \left(T\right)=0.035$ eV at $T=300$ K. The computational results in percent are shown in Figure 1 as the functions of separation between the plates by the solid and dashed lines for the Drude and plasma models, respectively.

As is seen in Figure 1, Lifshitz theory using the Drude model predicts rather large in magnitude thermal correction to the Casimir force of the opposite sign to the zero-temperature contribution up to the separation of $6.3$ μm, where it vanishes [11]. For example, at $a=500$ nm, 700 nm, and $1$ μm this correction is equal to −6.4%, −9.4%, and −13.8%, respectively (see the solid line). In contrast to this, Lifshitz theory using the plasma model predicts the thermal correction of the same sign as the zero-temperature Casimir force, the magnitude of which is in qualitative agreement with that for ideal metal plates, i.e., very small at separations below $1$ μm [12]. Thus, at separations $a=500$ nm and $1$ μm, we have ${\Delta }_T{F}_p=0.058$% and 0.29%, respectively. These results remain almost unchanged when we do not use the simple Drude and plasma models (17) and (21), but the dielectric permittivities obtained from the measured optical data for the complex index of refraction of gold [52], extrapolated by the Drude and plasma models down to zero frequency, where the optical data are unavailable.

Starting from 2000, the problems in Lifshitz theory have attracted much attention. The surprising thing is that the use of the well-tested Drude model leads to somewhat strange and unexpected results, whereas the results obtained using the seemingly inapplicable plasma model are quite reasonable from the theoretical point of view. As described in the next section, at a later time, this problem was further aggravated both theoretically and experimentally.

4. Thermodynamic and Experimental Parts of the Casimir Puzzle for Real Metals

As mentioned in Section 1, first measurements of the Casimir force were not precise enough and demonstrated only a qualitative agreement with theoretical predictions of Lifshitz theory. With the advent of micromechanical devices, it became possible to perform precise and reproducible measurements of small forces at separations below a micrometer. At the same time, quick progress in analytical calculations enabled by powerful computers made investigations of the asymptotic behaviors of complicated mathematical expressions more accessible. Taken together, these achievements provided the possibility to compare in more detail the predictions of Lifshitz theory with the measurement data and to correlate its formalism with the fundamental laws of thermodynamics. Below, we consider both the experimental and thermodynamic aspects of Lifshitz theory. We show that they constitute two parts of the serious problem that received the name of the Casimir puzzle. We begin with the thermodynamic part.

4.1. The Casimir Entropy for Metallic Plates and the Nernst Heat Theorem

The Casimir free energy per unit area of two parallel metallic plates kept at temperature T in thermal equilibrium with the environment is expressed by the Lifshitz formula (8). As there are problems with the Lifshitz theory considered in Section 3, it would be interesting to perform some kind of thermodynamic checking for the behavior of the Casimir free energy and respective entropy at vanishing temperature when the Drude and the plasma models of the dielectric response are used. It is well known that in accordance with the third law of thermodynamics (the Nernst heat theorem), with vanishing temperature, the entropy of a closed system in the state of thermal equilibrium must go to zero or to some universal constant that does not depend on the parameters of this system [53,54].

To check whether the Casimir entropy satisfies the Nernst heat theorem, the metal of the plates with perfect crystal lattice was considered [13,14,15]. Perfect crystal lattice is a basic theoretical model of condensed matter physics possessing the nondegenerate ground state. For perfect crystal lattices, the relaxation parameter of the Drude model (17) goes to zero with vanishing temperature as $\gamma \left(T\right)\sim T^2$. This dependence is caused by electron–electron scattering and is followed at all T below the temperature of liquid helium [55]. At higher T, the relaxation parameter decreases following the power law with different powers, but the condition $\gamma \left(T\right)\ll {\xi }_1\left(T\right)$ is satisfied in all cases. This is shown in [15].

For obtaining the asymptotic behavior of the Casimir free energy ${\mathcal{F}}_D$ when the Drude model is used, we should substitute $\epsilon \left(\mathrm{i}\xi \right)={\epsilon }_D\left(\mathrm{i}\xi \right)$ in (8). It is convenient to separate the term with $l=0$ and expand the sum of all other terms in powers of the small parameter $\gamma \left(T\right)/{\xi }_1\left(T\right)\ll 1$. Then, calculating the Casimir entropy as

$$S_{D\left(p\right)}(a,T)=-\frac{\partial {\mathcal{F}}_{D\left(p\right)}(a,T)}{\partial T},$$

(27)

we can present $S_D$ in the form [9,15]

$$S_D(a,T)=S_p(a,T)+\frac{k_B}{4\pi }{\int }_0^{\infty }{k}_{\perp }d{k}_{\perp }ln\left[1-r_{\mathrm{TE},p}^2(0,k_{\perp })e^{-2a{k}_{\perp }}\right]+O\left(Tln\frac{T}{T_{\mathrm{eff}}}\right).$$

(28)

The Casimir entropy $S_p$ calculated using the plasma model was found [14,15] as a perturbation expansion in the powers of the small parameter ${\delta }_0/a\ll 1$, where ${\delta }_0=c/{\omega }_p$ is the skin depth of a metal

$$\begin{array}{c}\hfill S_p(a,T)=\frac{k_B^3{T}^2}{\pi {\hslash }^2{c}^2}\left\{\phantom{{\left(\frac{{\delta }_0}{a}\right)}^2\frac{20\zeta \left(5\right)a^2{k}_B^2{T}^2}{{\hslash }^2{c}^2}}\frac{3}{2}\zeta \left(3\right)-\frac{4{\pi }^{3}a{k}_{B}T}{45\hslash c}+\frac{{\delta }_0}{a}\left[3\zeta \left(3\right)-\frac{16{\pi }^{3}a{k}_{B}T}{45\hslash c}\right]\right.\hfill \\ \hfill \left.-{\left(\frac{{\delta }_0}{a}\right)}^2\frac{20\zeta \left(5\right)a^2{k}_B^2{T}^2}{{\hslash }^2{c}^2}\right\}.\hfill \end{array}$$

(29)

Note that the contribution of the zeroth order in ${\delta }_0/a$ in (29) coincides with the Casimir entropy for the ideal metal plates at low temperature [56]. We have

$$\underset{T\to 0}{lim}{S}_p(a,T)=S_p(a,0)=0,$$

(30)

i.e., Lifshitz theory using the plasma model satisfies the Nernst heat theorem.

This is not the case, however, when the Drude model is used. From (28), with the help of (30), we obtain

$$\underset{T\to 0}{lim}{S}_D(a,T)=S_D(a,0)=\frac{k_B}{4\pi }{\int }_0^{\infty }{k}_{\perp }d{k}_{\perp }ln\left[1-r_{\mathrm{TE},p}^2(0,k_{\perp })e^{-2a{k}_{\perp }}\right].$$

(31)

Expanding this expression in the powers of the small parameter ${\delta }_0/a$, we obtain

$$S_D(a,0)=-\frac{k_B\zeta \left(3\right)}{16\pi a^2}\left[1-4\frac{{\delta }_0}{a}+12{\left(\frac{{\delta }_0}{a}\right)}^2-\dots \right]<0.$$

(32)

The quantities (31) and (32) depend on the parameters of a system such as the separation between the plates and the plasma frequency, which means that the Nernst heat theorem is violated. Note that in [57,58], this violation was attributed to the role of eddy currents, which explain the big difference between theoretical predictions for the Casimir force when the Drude and the plasma models are used. According to [57,58], the nonzero entropy (32) at $T=0$ is explained by an initiation of the correlated glassy (i.e., nonequilibrium) state.

Until now, we considered parallel plates composed of a nonmagnetic metal. The same results are, however, valid for other geometric configurations and for magnetic metals [16,17,18,19]. These results are puzzling because Lifshitz theory conflicts the fundamental law of thermodynamics when using the Drude response function in the frequency region where it is well-applicable. It is no less surprising that the agreement of Lifshitz theory with thermodynamics is restored when the plasma response function is used in the region of quasistatic frequencies, i.e., outside of its application region.

4.2. Lifshitz Theory in Experiments with Metallic Test Bodies

First precise measurements of the Casimir force allowing a discrimination between theoretical predictions using the plasma and Drude response functions have been performed using a micromechanical torsional oscillator in the configuration of an Au-coated sapphire sphere above an Au-coated polysilicon plate [20,21,22,23]. The thicknesses of Au coatings were sufficiently large so that the test bodies could be considered all gold. The dielectric permittivity of Au along the imaginary frequency axis was found by means of the Kramers–Kronig relations using the available tabulated optical data of Au [52] extrapolated down to zero frequency either by the Drude or the plasma model [59].

As discussed in Section 2, Lifshitz theory was formulated for a configuration of two parallel plates. However, the sphere-plate configuration is much more convenient for precise measurements of the Casimir interaction. In the early 21st century, the exact expression for the Casimir force between a sphere and a plate was not available, so experiment and theory were compared using the proximity force approximation (PFA) [9,59,60].

According to the PFA, the Casimir force between a sphere and a plate $F_{sp}$ is approximately equal to

$$F_{sp}(a,T)=2\pi R\mathcal{F}(a,T),$$

(33)

where $\mathcal{F}(a,T)$ is the Casimir free energy per unit area of two parallel plates defined in (8), a is the minimum sphere–plate separation, and R in the sphere radius. The error of this approximation was not known precisely at that time but it was expected to be of the order of $a/R$. As the measurements [20,21,22,23] were performed at $a<1$ μm with the sphere radius $R\approx 150$ μm, this error does not exceed 0.7%. The most precise experiments measuring the Casimir interaction were performed in dynamic mode, where not the Casimir force but its gradient $F_{sp}^{\prime }$ was measured. By differentiating (33) with respect to a, the force gradient is expressed by the Lifshitz formula (10) for the force per unit area of two parallel plates

$$F_{sp}^{\prime }(a,T)\equiv \frac{\partial F_{sp}(a,T)}{\partial a}=-2\pi RF(a,T).$$

(34)

Another point that was considered when comparing experiments with theory was the impact of surface roughness. The roughness profiles on both interacting surfaces were examined using an atomic force microscope. The heights of the maximum roughness peaks were found to be much less than the minimum separation between a sphere and a plate. In this case, the Casimir force $F(a,T)$ between two parallel plates with account of surface roughness was found using an additive method of geometrical averaging of the forces (10) calculated at the local separations $a_i$ over the profiles of both surfaces [9,59,61]. The gradient of the Casimir force with account of surface roughness was then calculated using (34).

In Figure 2, we present a typical comparison between the measurement data obtained using a micromechanical torsional oscillator in a high vacuum [22] and the theoretical predictions of Lifshitz theory. The predictions found with the Drude and plasma extrapolations of the optical data are shown by the blue and red bands, respectively, as functions of separation. The widths of the bands reflect the total theoretical error, which includes inaccuracies of the PFA, and errors in the optical data and in the sphere radius. The mean experimental data are shown as crosses. The arms of the crosses indicate the total experimental errors. All errors were found at the 95% confidence level. Note that in [22] experiment and theory were compared in terms of the effective Casimir pressure between two parallel plates, which is equivalently presented here in terms of the originally measured gradients of the Casimir force between a sphere and a plate.

As shown in Figure 2, Lifshitz theory using the Drude model for extrapolation of the optical data is experimentally excluded. The same result was obtained over the separation region from 160 to 750 nm. From Figure 2, it is apparent that Lifshitz theory, using the plasma model for extrapolation, is in very good agreement with the measurement data. This is also the case over the entire measurement range [22].

As the second example of comparison between experiment and theory, we consider an experiment measuring the gradient of the Casimir force between a sphere of $R\approx 62$ μm radius and a plate with coated by magnetic metal Ni surfaces [25,26]. These experiments were performed in a high vacuum using an atomic force microscope operated in dynamic mode. The metallic coatings of the test bodies were not magnetized, so that the magnetic force did not contribute to the measurement results. It should be recalled that at room temperature the magnetic permeability $\mu \left(\mathrm{i}\xi \right)$ falls to unity at frequencies much below the first Matsubara frequency [62]. Because of this, the magnetic properties of Ni impact the Casimir force only through the term of (10) with $l=0$, where $\mu \left(0\right)=110$, whereas in all terms with $l⩾1$ we should use $\mu \left({\mathrm{i}\xi }_l\right)=1$.

Experiment and theory were compared similar to the above using the optical data for Ni [52] extrapolated to zero frequency either by the Drude or by the plasma model, the proximity force approximation, and the additive approach to account for the surface roughness. In Figure 3, the predictions of Lifshitz theory using the Drude and plasma models are shown as the blue and red bands, respectively, whereas the measurement data are shown as crosses with all errors determined at the 67% confidence level [25]. Lifshitz theory using the Drude model is excluded by the measurement data, which are in good agreement with the same theory using the plasma model. Altogether, Lifshitz theory using the Drude model was excluded over the separation region from 223 nm (the minimum separation of this experiment) to 420 nm.

Notably, measurements of the Casimir interaction between magnetic metals have an important difference compared with experiments using the gold test bodies. As shown in Figure 2, the predictions of Lifshitz theory using the Drude model (the blue band) are situated below the experimental crosses. This means that for these predictions to agree with the data, some additional attractive force would be needed. By contrast, for the magnetic test bodies, the predictions of Lifshitz theory using the Drude model lie above the measurement data. An agreement between them could only be reached at the expense of some extra repulsive force. Thus, both sets of experiments make any attempt to bring the Lifshitz theory combined with the Drude model in agreement with measurements a challenging task (Section 7).

From Figure 2 and Figure 3 it is seen that, although Lifshitz theory using the Drude model is clearly excluded by the measurement data, its predictions differ from these data (and from the predictions of the same theory using the plasma model) by only a few percent of the force gradient. However, it was shown that by using the differential force measurements, originally suggested for searching the Yukawa-type corrections to Newton’s gravitational law [63,64], it is possible to increase the difference between the predictions of Lifshitz theory using the Drude and plasma response functions by several orders of magnitude [65,66]. Using this idea, the differential Casimir force was measured between an Ni-coated sphere and Au and Ni sectors covered with an Au overlayer [27]. A presence of the overlayer provides almost equal contributions of all nonzero Matsubara frequencies in the positions of a sphere above the Au and Ni sectors. As a result, the differences in the Casimir forces acting on a sphere in these positions, calculated by Lifshitz theory using the Drude and the plasma model extrapolations of the optical data, diverge by up to a factor of 1000. The measurement results for the differential Casimir force were found to be in good agreement with the predictions of Lifshitz theory using the plasma model extrapolations of the optical data. The alternative predictions using the Drude model, which differ from the confirmed values by up to a factor of 1000, were found to be out of all proportion to the actual size of the measured differential force.

The comparison between the predictions of Lifshitz theory and the measurement data of precise experiments presented above is puzzling because the commonly used and considered as correct Drude response function failed, whereas the evidently inapplicable plasma response function demonstrated that it can be used successfully.

5. Casimir Force between Ideal and Real Dielectrics

The ideal dielectric or insulator is a material that does not conduct an electric current at any temperature. The dielectric permittivity of an ideal dielectric at the pure imaginary Matsubara frequencies can be presented in the oscillator form [67]

$$\epsilon \left({\mathrm{i}\xi }_l\right)=1+\sum _{j=1}^K\frac{g_j}{{\omega }_j^2+{\xi }_l^2+{\gamma }_j{\xi }_l},$$

(35)

where ${\omega }_j$ are the oscillator frequencies (all of which are nonzero), ${\gamma }_j$ are the relaxation parameters, $g_j$ are oscillators strengths, and K is the number of oscillators.

At zero Matsubara frequency, ${\xi }_0=0$, the permittivity of an ideal dielectric takes the finite value

$$\epsilon \left(0\right)=1+\sum _{j=1}^K\frac{g_j}{{\omega }_j^2}.$$

(36)

By putting $\omega =\mathrm{i}\xi =0$ in (4) for an ideal dielectric with no magnetic properties ($\mu =1$), we obtain

$$r_{\mathrm{TM}}(0,k_{\perp })=\frac{\epsilon \left(0\right)-1}{\epsilon \left(0\right)+1},r_{\mathrm{TE}}(0,k_{\perp })=0.$$

(37)

As is seen in (37), the value of the TE reflection coefficient is the same as for the Drude model in (19), whereas the value of the TM reflection coefficient is different.

According to Section 3, the zero-frequency terms of the Lifshitz formulas determine the behaviors of the Casimir free energy and force at large separations. Substituting (37) in (8) and integrating, we find the Casimir free energy per unit area of the plates composed of an ideal dielectric

$${\mathcal{F}}_{\mathrm{ID}}(a,T)=\frac{k_{B}T}{4\pi }{\int }_0^{\infty }{k}_{\perp }d{k}_{\perp }ln\left[1-r_{\mathrm{TM}}^2(0,k_{\perp })e^{-2a{k}_{\perp }}\right]=-\frac{k_{B}T}{16\pi a^2}L{i}_3\left[r_{\mathrm{TM}}^2\left(0\right)\right],$$

(38)

where $r_{\mathrm{TM}}(0,k_{\perp })\equiv r_{\mathrm{TM}}\left(0\right)$ is defined in (37) and $L{i}_3\left(z\right)$ is the polylogarithm function.

Similarly, substituting (37) in (10), we find the asymptotic behavior of the Casimir force per unit area between two ideal dielectric plates at large separations

$$F_{\mathrm{ID}}(a,T)=-\frac{k_{B}T}{2\pi }{\int }_0^{\infty }\frac{k_{\perp }^{2}d{k}_{\perp }}{r_{\mathrm{TM}}^{-2}(0,k_{\perp })e^{2a{k}_{\perp }}-1}=-\frac{k_{B}T}{8\pi a^3}L{i}_3\left[r_{\mathrm{TM}}^2\left(0\right)\right].$$

(39)

Real dielectrics are also characterized by the zero electric conductivity at zero temperature. This property differentiates them from metals that have a nonzero conductivity at any temperature. In this regard, all materials are either dielectrics or metals [68]. There are several types of real dielectrics, but all of them have some nonzero conductivity at $T>0$.

The dielectric permittivity of a real dielectric at temperature T considered at the pure imaginary Matsubara frequencies can be presented in the form [52]

$${\epsilon }_{\mathrm{RD}}\left({\mathrm{i}\xi }_l\right)=\epsilon \left({\mathrm{i}\xi }_l\right)+\frac{4\pi {\sigma }_0\left(T\right)}{{\xi }_l},$$

(40)

where $\epsilon \left({\mathrm{i}\xi }_l\right)$ is determined in (35) and ${\sigma }_0$ is the temperature-dependent static conductivity. This static conductivity of dielectric material vanishes with temperature exponentially fast

$${\sigma }_0\left(T\right)\sim exp\left(-\frac{\Delta }{2k_{B}T}\right),$$

(41)

where $\Delta$ is the band gap. It is convenient to call the dielectric material an insulator if $\Delta >2-3$ eV, and an intrinsic semiconductor if $\Delta <2-3$ eV [68].

Substituting (40) in (4), we obtain the following expressions for the reflection coefficients of real dielectrics at zero frequency:

$$r_{\mathrm{TM}}(0,k_{\perp })=1,r_{\mathrm{TM}}(0,k_{\perp })=0,$$

(42)

which coincide with (19) obtained for metals described by the Drude model, but are different from (37) obtained for ideal dielectrics. Thus, the Casimir free energy and force for dielectric plates at sufficiently large separations are given by expression (20) obtained in Section 3 for metallic plates described by the Drude model.

Given this result, we suggest that an inclusion of the DC conductivity in calculation of the Casimir free energy and force between dielectric plates plays the same role as an inclusion of the relaxation properties of conduction electrons for the metallic ones. This hypothesis is confirmed by calculations of the relative thermal correction to the Casimir force between dielectric plates defined as

$${\delta }_T{F}_{\mathrm{RD}\left(\mathrm{ID}\right)}(a,T)=\frac{F_{\mathrm{RD}\left(\mathrm{ID}\right)}(a,T)-F_{\mathrm{RD}\left(\mathrm{ID}\right)}(a,0)}{F_{\mathrm{RD}\left(\mathrm{ID}\right)}(a,0)}.$$

(43)

Computations were performed for fused silica using the optical data for its complex index of refraction [52] with $\epsilon \left(0\right)=3.81$ (ideal dielectric) and considering the typical value of the DC conductivity of fused silica at room temperature ${\sigma }_0=29.7 {\mathrm{s}}^{-1}$ (real dielectric). Note that the computational results for real fused silica do not depend on the value of ${\sigma }_0$ but only on it being not equal to zero.

Figure 4 shows the predictions of Lifshitz theory for the thermal correction (43) in percent for the real and ideal fused silica (i.e., with included and omitted DC conductivity) at $T=300$ K as functions of the separation between the plates by the dashed and solid lines, respectively. As shown in this figure, at short separations the thermal correction is reasonably small if the DC conductivity is omitted in computations (see the solid line). Thus, at $a=1$ and $2$ μm it is equal to only 3.9% and 15.4%, respectively. The situation reverses if the DC conductivity is taken into account in computations (see the dashed line). In this case, the relative thermal correction (43) is equal to 182% and 314% at the same separations, respectively.

The very large difference between the predictions of Lifshitz theory with omitted and included DC conductivity for dielectrics arises from the difference between the TM reflection coefficients in (37) and (42). We recall that a similar difference between predictions of the same theory for metallic plates described by the Drude and plasma models is due to a difference between the TE reflection coefficients in (19) and (23). In any case, the above results suggest that the serious problems met by Lifshitz theory in the case of metals (Section 4) may appear for dielectrics as well. This conclusion is confirmed in the next section.

6. Thermodynamic and Experimental Parts of the Casimir Conundrum for Dielectrics

Consideration of the thermal Casimir force between dielectric plates demonstrates that there is ample evidence of problems between Lifshitz theory using the dielectric response function for real dielectrics and thermodynamics, on the one hand, and measurement data, on the other hand. Experiments measuring the Casimir interaction with dielectric test bodies are more complicated in comparison with metallic ones due to the problem of localized electric charges, which are present on dielectric surfaces. Until now, only three relatively precise measurements using the dielectric test bodies have been performed. However, the thermodynamic test similar to that elucidated in Section 4.1 for metals is even more conclusive because it does not require an assumption of a perfect crystal lattice. We begin with the thermodynamic argument and continue with a comparison between experiment and theory.

6.1. Casimir Entropy for Dielectric Plates and Nernst Heat Theorem

The thermodynamic test for two dielectric plates with either included or omitted DC conductivity can be performed in close analogy to that presented in Section 4.1 for metallic plates described either by the Drude or the plasma model. We should substitute the dielectric permittivity of a real dielectric (40) with $\epsilon \left({\mathrm{i}\xi }_l\right)$ given by (35) in (8) to obtain the Casimir free energy ${\mathcal{F}}_{\mathrm{RD}}$ for real dielectric plates, and then find the low-temperature behavior of the Casimir entropy

$$S_{\mathrm{RD}}(a,T)=-\frac{\partial {\mathcal{F}}_{\mathrm{RD}}(a,T)}{\partial T}.$$

(44)

Similarly substituting the dielectric permittivity (35) in (8), we obtain the Casimir free energy ${\mathcal{F}}_{\mathrm{ID}}$ for ideal dielectric plates with omitted DC conductivity. Then, as in (44), we find their Casimir entropy $S_{\mathrm{ID}}$.

The Casimir entropy $S_{\mathrm{RD}}$ can be presented in the form [9,31,32]

$$\begin{array}{c}\hfill S_{\mathrm{RD}}(a,T)=S_{\mathrm{ID}}(a,T)+\frac{k_B}{4\pi }{\int }_0^{\infty }{k}_{\perp }d{k}_{\perp }\left\{ln\left[1-r_{\mathrm{TM}}^2(0,k_{\perp })e^{-2a{k}_{\perp }}\right]\right.\hfill \\ \hfill \left.-ln\left(1-e^{-2a{k}_{\perp }}\right)\right\}+O\left(\frac{1}{T^3}{e}^{-\frac{\Delta }{k_{B}T}}\right),\hfill \end{array}$$

(45)

where the reflection coefficient $r_{\mathrm{TM}}(0,k_{\perp })$ defined in (37), does not depend on $k_{\perp }$: $r_{\mathrm{TM}}(0,k_{\perp })=r_{\mathrm{TM}}\left(0\right)$. Because of this, the integral in (45) can be calculated with the result

$$\begin{array}{c}\hfill S_{\mathrm{RD}}(a,T)=S_{\mathrm{ID}}(a,T)+\frac{k_B}{16\pi a^2}\left\{\zeta \left(3\right)-L{i}_3\left[r_{\mathrm{TM}}^2\left(0\right)\right]\right\}\hfill \\ \hfill +O\left(\frac{1}{T^3}{e}^{-\frac{\Delta }{k_{B}T}}\right).\hfill \end{array}$$

(46)

The asymptotic behavior of the Casimir entropy for an ideal dielectric at arbitrary low temperature is given by [9,31,32]

$$S_{\mathrm{ID}}(a,T)=\frac{k_B^{2}T}{2\hslash c{a}^2}\left\{\frac{GL{i}_3\left[r_{\mathrm{TM}}^2\left(0\right)\right]}{3[{\epsilon }^2\left(0\right)-1]}+\frac{3\zeta \left(3\right)r_{\mathrm{TM}}^2\left(0\right)[{\epsilon }^2\left(0\right)+1]}{2\pi }\frac{a^2{k}_{B}T}{\hslash c}+O{\left(\frac{T}{T_{\mathrm{eff}}}\right)}^2\right\},$$

(47)

where the quantity G is expressed via the oscillator parameters introduced in (35)

$$G=c\sum _{j=1}^K\frac{g_j{\gamma }_j}{{\omega }_j^4}.$$

(48)

As seen from (47),

$$\underset{T\to 0}{lim}{S}_{\mathrm{ID}}(a,T)=S_{\mathrm{ID}}(a,0)=0,$$

(49)

i.e., Lifshitz theory satisfies Nernst heat theorem in the case of Casimir plates composed of an ideal dielectric. In this sense, the ideal dielectric plates are similar to the ideal metal plates and to the plates composed of a metal described by the plasma model.

The situation reverses when we consider the Casimir entropy of real dielectric plates composed of real dielectric possessing some DC conductivity at any nonzero temperature. In the limiting case of zero temperature, (47) results in

$$\underset{T\to 0}{lim}{S}_{\mathrm{RD}}(a,T)=S_{\mathrm{RD}}(a,0)=\frac{k_B}{16\pi a^2}\left\{\zeta \left(3\right)-L{i}_3\left[r_{\mathrm{TM}}^2\left(0\right)\right]\right\}>0,$$

(50)

where $r_{\mathrm{TM}}\left(0\right)$ is given in (37). Thus, for the plates composed of a real dielectric possessing some DC conductivity at any nonzero temperature, Lifshitz theory leads to a positive Casimir entropy depending on the separation between the plates and the static dielectric permittivity, i.e., violates the Nernst heat theorem. Note that for both metallic and dielectric plates the Casimir entropy is not the entropy of a closed system, which should also include, for instance, the entropy of plate materials. Notably, the Casimir entropy alone depends on the separation distance between the plates. Because of this, it cannot be compensated by some other contribution in the total entropy of a closed system, making the violation of the Nernst heat theorem unavoidable.

The presented results demonstrate that when taking into account the DC conductivity of the plate material, Lifshitz theory violates the Nernst heat theorem and satisfies it when the DC conductivity is disregarded, which is called the Casimir conundrum. These results are strange as DC conductivity does exist and has been measured in numerous experiments.

6.2. Lifshitz Theory in Experiments with Dielectric Test Bodies

As mentioned in the beginning of Section 6, measurements of the Casimir force between dielectric test bodies are more complicated than between metallic ones due to the localized electric charges that are present on dielectric surfaces. Because of this, it is preferable to use dielectrics with sufficiently high concentration of charge carries (for instance, doped semiconductors with doping concentration below the critical value at which the dielectric-to-metal phase transition occurs).

We start with an experiment on the optically modulated Casimir force where the force difference between an Au-coated sphere of $R=98.9$ μm radius and a silicon membrane illuminated with laser pulses has been measured in a high vacuum in the presence and absence of light [37,38]. Measurements of the force difference,

$$F_{\mathrm{diff}}(a,T)=F_{sp}^l(a,T)-F_{sp}(a,T),$$

(51)

were performed using an atomic force microscope.

In the absence of a laser pulse on the membrane composed of a p-type silicon, the concentration of charge carriers in the membrane was equal to approximately $5\times {10}^{14}{\mathrm{cm}}^{-3}$ [52]. Thus, the membrane was in a dielectric state. In the presence of a laser pulse, the charge carrier concentration increased by five orders of magnitude, up to (1–2)$\times {10}^{19}{\mathrm{cm}}^{-3}$, depending on the absorbed power. This means that in the bright phase silicon was in a metallic state.

The difference of the Casimir forces between an Au-coated sphere and a silicon membrane in the bright and dark phases was calculated using Lifshitz theory and the proximity force approximation (33) with account of surface roughness by means of geometrical averaging at the laboratory temperature $T=300$ K. Figure 5 shows the computational results for $F_{\mathrm{diff}}$ by solid and dashed lines as functions of sphere–plate separation (the absorbed power was 8.5 mW). The solid and dashed lines were computed with omitted and taken into account DC conductivity of silicon, respectively, in the absence of a laser pulse, i.e., in the dark phase. In the presence of a laser pulse (the bright phase), the charge carriers were taken into account using the Drude model (the use of the plasma model in the bright phase for silicon and Au leads to only minor differences in this case, which cannot be distinguished experimentally). The experimental data are shown as crosses plotted at the 95% confidence level.

As shown in Figure 5, the predictions of Lifshitz theory taking into account the DC conductivity of silicon in the dielectric state (the dashed line) are experimentally excluded, whereas the predictions with omitted DC conductivity of dielectric silicon are in very good agreement with the measurement data. Thus, Lifshitz theory is experimentally consistent only if the real physical phenomenon—small but quite measurable electric conductivity—is ignored. It is highly meaningful that only in this case the theory is in agreement with the requirements of thermodynamics and is otherwise in contradiction with them.

One more experiment performed using an atomic force microscope was devoted to measurements of the Casimir force between an Au-coated sphere of radius $R=101.2$ μm and an indium tin oxide (ITO) film deposited on a quartz substrate. These measurements were performed twice: before and after UV irradiation of the ITO film [40,41]. At room temperature, ITO is a transparent conductor. Based on this, this material was suggested for use in measurements of Casimir force [14]. The first experiments of this kind were performed earlier [69,70]. The main novelty of the experiments [40,41] is the UV irradiation of the ITO film with subsequent measurement of the Casimir force. The UV irradiation of ITO led to a lower mobility of charge carriers [71] with no detectable changes in the optical data and, as was hypothesized [40,41], to a phase transition from a metallic to a dielectric state.

The experimental results and their comparison with theory confirmed this hypothesis. Figure 6a shows pairs of blue and red lines indicating the boundaries of theoretical bands for the Casimir force between an Au sphere and an ITO film computed before and after UV irradiation of this film with taken into account and omitted contribution of free charge carriers. The experimental data in both cases are shown as crosses plotted at the 95% confidence level. The theoretical predictions are in very good agreement with the measurement data both before and after irradiation.

In Figure 6b, we again plot the upper set of crosses obtained after UV irradiation of ITO film in comparison with the theoretical band computed with taken into account contribution of free charge carriers in the UV irradiated sample. The predictions of the Lifshitz theory are excluded by the measurement data. This result is in line with the results found from measuring the optically modulated Casimir force, which showed that in the dielectric state experiment and theory are in agreement with disregarded contribution of free charge carriers to the dielectric response of a material.

The last experiment using the dielectric test body involved measuring the thermal Casimir–Polder force between ${}^{87}$ Rb atoms belonging to the Bose-Einstein condensate and a dielectric fused silica plate at separations of a few micrometers [42]. In the first series of measurements, the plate was kept at the same temperature as the environment; in the second and third, it was kept at higher temperatures. This means that in the last two cases the Casimir–Polder force had two contributions: the equilibrium given by Lifshitz theory and from its generalization for a nonequilibrium case [72,73] (see also the case of phase-change wall material at nonequilibrium conditions [74]). It was shown [42] that the measurement data were in good agreement with theory when the DC conductivity of fused silica was omitted from the computations. However, by repeating the same calculations including the DC conductivity of the plate material, the theoretical results are excluded by the measurement data [39]. This happens exclusively due to the equilibrium contribution to the Casimir–Polder force given by Lifshitz theory (the nonequilibrium one is almost independent of whether or not the DC conductivity is included in computations).

Thus, experiments measuring the Casimir interaction with dielectric test bodies are consistent with theoretical predictions of the Lifshitz theory only under the condition that the DC conductivity of he dielectric material is omitted in computations. In so doing, as shown in Section 6.1, the theory is consistent with the laws of thermodynamics, but at the sacrifice of the phenomenon of DC conductivity. The approaches to the resolution of the Casimir conundrum originating from this situation are discussed below.

7. Different Approaches to the Resolution of the Casimir Puzzle and the Casimir Conundrum

The above problems attracted much attention of experts in the field during the last 20 years. Many different explanations and suggestions were introduced as to why the predictions of Lifshitz theory appear to disagree with the laws of thermodynamics and the measurement data and how the agreement could be restored. Certain of the proposed approaches are seeking for some unaccounted systematic effects in the performed experiments, such as an impact of an additional electric force due to electrostatic patches on metallic surfaces or the nonadditive effects in the surface roughness. According to other approaches, the roots of the problems could lay in the simplifications used in Lifshitz theory, which does not account for spatial dispersion. There was a suggestion [75] to modify the Planck distribution by including in it the special damping parameter. This parameter, however, should take different values for bringing theoretical predictions into agreement with the measurement data of different experiments. A separate line of investigation was the generalization of Lifshitz theory for more complicated geometries used in experiments, including the configuration of a sphere above a plate, based on the first principles of quantum field theory with no use of uncontrolled additive methods such as PFA. Below, we consider several suggested approaches directed to the elimination of contradictions between the Lifshitz theory and thermodynamics and/or experiments.

7.1. Variations in the Optical Data

Many precise experiments measuring the Casimir force between metallic test bodies used the Au-coated surfaces of a sphere and a plate. The frequency-dependent dielectric permittivity of Au was found using the tabulated optical data for the complex index of refraction [52] extrapolated down to zero frequency using either the Drude or plasma models as discussed above. However, the optical properties of Au films depend on the sample and the used method of deposition. Most importantly, different sets of optical data lead to different values of the Drude parameters ${\omega }_p$ and $\gamma$ used in the extrapolation to zero frequency. This may influence the values of the Casimir force calculated using Lifshitz theory and the comparison between experiments and theory. The question arises whether it is possible to bring the theoretical predictions in agreement with the experimental results if the optical data are not taken from tables but measured for the laboratory bodies used in this specific experiment [76,77]. To answer this question, the Casimir force was computed using different sets of the optical data for Au available in the literature. It was concluded [76,77] that the variations in the optical data of Au with respective changes in the plasma frequency used in extrapolations to zero frequency may result in differences up to 5–10% of the Casimir force. These differences are of the same size as the discrepancies between experiment and theory found in [20,21,22,23,24,25,26].

To determine the actual role of the optical data variations in the resolution of the Casimir puzzle in an experiment [23], the plasma frequency and the relaxation parameter were found for the specific Au coatings used in measurements of the Casimir force. The obtained values were very close to the tabulated ones [52]. In addition, a complete set of the optical data for experimental Au coatings was measured using ellipsometry and was found [78] to be in agreement with the tabulated one. These results were confirmed using the weighted Kramers–Kronig relations [79]. All sets of the optical data of Au available in the literature with the respective plasma frequencies varying from 6.85 to 9.0 eV were used in calculations of the Casimir interaction for subsequent comparison with the measured values of [23]. The results of this comparison are shown in Figure 7, where the measured gradients of the Casimir force are shown as crosses (the same as in Figure 2) and the blue theoretical band was obtained using all sets of the optical data of Au extrapolated to zero frequency using the Drude model. Figure 7 shows that the use of alternative sets of the optical data only increases the disagreement between experiment and theory. Thus, it was concluded [59] that it is impossible to bring the predictions of Lifshitz theory using the Drude model in agreement with the measurement data at the cost of variation of the optical data.

7.2. Impact of Surface Patches

The Au coatings on the test bodies in experiments measuring the Casimir force have a polycrystal structure. In addition, even in a high vacuum, some contaminants and dust exist on the surfaces. As a result, some spatial distribution of electrostatic potential appears even on the grounded surfaces of a sphere and a plate, which are called the patch potentials [80]. The patch potentials lead to some additional attractive force acting between a sphere and a plate and to a dependence on the separation of the residual potential difference between them. The magnitude of this force depends on the size of the surface patches or contaminants.

It was hypothesized [81] that an additional force gradient due to surface patches could compensate the difference between the blue and red bands in Figure 2 and thus bring Lifshitz theory using the Drude model into agreement with the measurement data of [22]. Sufficiently large patches and contaminants up to $2$ μm can lead to the force gradient of the required magnitude but also result in a strong dependence of the residual potential difference on the sphere–plate separation, which was not present in the calibration measurements of [20,21,22,23,24,25,26] due to a special selection of samples. An investigation [82] of the patch potentials on Au surfaces of the samples similar to those used in the performed experiments [22,23] with the help of Kelvin probe force microscopy demonstrated that the force of the patch origin cannot explain the difference between the measured force gradients and the theoretical predictions of Lifshitz theory using the Drude model.

Further clarification of the role of patches in the measurements of Casimir force was provided by experiments with magnetic test bodies [25,26]. As noted in Section 4.2, to bring the theoretical predictions obtained using the Drude model into agreement with the measurement data in this case, some additional repulsive force is needed, which does not occur due to surface patches. Recently, several experiments measuring the Casimir interaction have been performed where the role of surface patches and contaminants was strongly suppressed by Ar-ion and UV-cleaning of the interacting surfaces [83,84,85]. The results of these experiments are consistent with the theoretical predictions of Lifshitz theory using the plasma model and exclude the predictions of the same theory using the Drude model up to the separation distance of $1.1$ μm. Thus, by considering surface patches, it is impossible to reconcile Lifshitz theory using the Drude model with the measurement data.

7.3. The Role of Surface Roughness

The interacting surfaces in the measurements of Casimir force are not perfectly plane or spherical, but covered with some roughness. If there are large-scale deviations from perfect geometry, the roughness can be described by a regular function. The short-scale roughness can be considered as stochastic and described by random functions. When comparing the precise experiments measuring the Casimir force mentioned above and theory, the roughness was considered in an additive way [9,59,61]. So, in the case of regular roughness, the Casimir force was computed using Lifshitz theory at all local separations and was then averaged over the actual geometrical profiles of the interacting surfaces determined using an atomic force microscope. In the case of stochastic roughness with small dispersions ${\delta }_{1,2}\ll a$, the roughness correction can be found perturbatively in powers of ${\delta }_{1,2}/a$ and considered in a multiplicative way. For example, corrected for the presence of a stochastic roughness gradient of the Casimir force between a sphere and a plate takes the form [9,59]

$$F_{sp,R}^{\prime }(a,T)=F_{sp}^{\prime }(a,T)\left[1+10\frac{{\delta }_1^2+{\delta }_2^2}{a^2}+105\frac{{({\delta }_1^2+{\delta }_2^2)}^2}{a^4}\right].$$

(52)

Different approaches to the account of surface roughness in experiments measuring Casimir force were the subject of many studies [86,87,88,89,90]. One may doubt whether the additive approach to the comparison between experiment and theory is exact enough. Strictly speaking, the larger and smaller roughness corrections to the Casimir force could decrease the difference between the predictions of Lifshitz theory using the Drude model and the measurement data in experiments with the nonmagnetic and magnetic surfaces, respectively.

A more fundamental scattering approach to the Casimir force between rough surfaces was developed [91,92]. It was shown that the additive approach leads to sufficiently precise results at much smaller separations between the interacting bodies compared with the roughness correlation length. For a typical correlation length of ${\Lambda }_c\approx 200$ nm, the additive approach is already applicable at $a<2{\Lambda }_c/3\approx 130$ nm. This is the most important region where the roughness correction contributes up to a few percent of the Casimir force. At larger separations, the additive approach underestimates the role of surface roughness, so that more accurate results are provided by the scattering approach. For instance, at $a=2{\Lambda }_c=400$ nm, the scattering approach leads to by a factor of 1.5 larger roughness correction than the additive approach. However, this does not damage the comparison between experiment and theory. The point is that with increasing separation the roughness correction decreases much faster than the Casimir force between perfectly shaped surfaces. Because of this, for surfaces of sufficiently good quality at separations of a few hundred nanometers it does not play any role in the comparison between experiment and theory and can be neglected. This means that the surface roughness is not helpful for resolution of the Casimir puzzle.

7.4. Deviations from the Proximity Force Approximation

As mentioned in Section 4.2, calculations of the Casimir force and its gradient in the sphere–plate geometry used in the comparison between experiment and theory were performed applying Lifshitz theory and the proximity force approximation (33) and (34). Although it was believed that for small $a/R$ the error introduced using the PFA should be of the order of $a/R$, the exact information concerning the size of this error was missing. Because of this, one could hope that a disagreement between the theoretical predictions of Lifshitz theory using the Drude model and the measurement data may be caused by some deviations from the PFA in the region of experimental separations and sphere radii.

A major step forward was made in 2006 when it was shown that the Casimir energy between nonplanar surfaces can be written in terms of functional determinants [93,94]. The obtained results were applied to calculate the Casimir energy of an ideal metal cylinder and a sphere in front of an ideal metal plane [93,94,95]. Later, Lifshitz theory was generalized for real material bodies of arbitrary shape kept at any temperature in thermal equilibrium with the environment [96,97,98,99]. Application of the developed theory to a sphere and a plate composed of real metals spaced at separations below a micrometer turned out to be a complicated problem. The first computations [100] produced an impression that in the limiting case of perfect reflectors the exact results rapidly depart from the PFA expectations and this should be taken into account in theory–experiment comparisons. For real metals, the computations were performed in [101] but for not too small values of $a/R>0.2$ compared to the experimental values of $a/R<0.01$.

Calculations of the Casimir force in a sphere–plate geometry for the experimental values of $a/R$ have been performed both analytically using the gradient expansion method and numerically [102,103,104,105,106,107,108,109]. As only one example of the obtained results, we present an expansion of the exact gradient of the Casimir force between a sphere and a plate, which includes the first-order correction to the PFA [107]

$$F_{sp,\mathrm{ex}}^{\prime }(a,T)=F_{sp}^{\prime }(a,T)\left[1+{\beta }_{D,p}(a,R)\frac{a}{R}\right],$$

(53)

where $F_{sp}^{\prime }$ is the PFA result presented in (34). The values of the expansion coefficient ${\beta }_{D,p}$ were computed at different separations for the sphere radii varying from 10 to $100$ μm using the tabulated optical data for the complex index of refraction of Au extrapolated to zero frequency by the Drude (${\beta }_D$) and plasma (${\beta }_p$) models. According to the obtained results [107], within the region of separations from 200 to 600 nm, the coefficient ${\beta }_D$ varies from −0.35 to −0.45, whereas ${\beta }_p$ varies from −0.5 to −0.6. Note that in the interpretation of experiments measuring Casimir force [20,21,22,23,24,25,26], the conservative estimation $|{\beta }_{D,p}|⩽1$ has been used. Thus, inaccuracies in the PFA are irrelevant to the Casimir puzzle and an exact computation of the Casimir force in the sphere–plate geometry does not help to resolve it.

7.5. Impurities in a Crystal Lattice and the Nernst Heat Theorem

The above consideration of the variations of the optical data, impact of surface patches, nonadditive effects in the surface roughness, and deviations of the Casimir force from PFA is aimed at resolving the second, experimental, part of the Casimir puzzle. All these research directions are unrelated to the inconsistency of Lifshitz theory with thermodynamics discussed in Section 4.1 and Section 6.1 (note that in the sphere–plate geometry Lifshitz theory faces the same problems with violation of the Nernst heat theorem as for two parallel plates [16]). Notably, after the crucial experiment [27] where the theoretical predictions using the plasma and Drude models differed by up to a factor of 1000, an exclusion of the latter by the measurement data was conclusively established. This attaches particular significance to the thermodynamic parts of the Casimir puzzle and Casimir conundrum.

The historical first approach to the problem of the violation of the Nernst heat theorem in the configuration of two parallel metallic plates was made with no modifications in Lifshitz theory. It was suggested [110] to take into account that the crystal lattice of any real metal has some small fraction of impurities. As a result, the relaxation parameter $\gamma$ of the Drude model (17) does not go to zero with vanishing temperature, but to some residual value ${\gamma }_0$, which depends on the impurity concentration [55]. Based on this, it was numerically shown [110] that at sufficiently low temperature the Casimir entropy abruptly jumps to zero starting from the negative value (32), i.e., the Nernst heat theorem is formally satisfied.

However, the numerical proof of vanishing cannot be considered as completely satisfactory because it is burdened with some computational error. Because of this, the case of metallic plates with impurities was also considered analytically [111,112]. As a result, it was shown that in the asymptotic limit of low T, the Casimir entropy calculated using the Drude model with the residual relaxation ${\gamma }_0$ vanishes as [112]

$$S_D(a,T)=-D_1\left(a\right)T+D_2\left(a\right)T^{3/2}-\dots .$$

(54)

For the typical value of ${\gamma }_0=5.32\times {10}^{10}$ rad/s, we find from (54) that the Casimir entropy jumps to zero at about 0.001 K starting from the negative value of $-2{\mathrm{M}\mathrm{e}\mathrm{V}\mathrm{m}}^{-2}{\mathrm{K}}^{-1}$ [9].

Although account of impurities provides an apparent resolution to the thermodynamic part of the Casimir puzzle, it cannot be considered as completely satisfactory. A perfect crystal lattice with no impurities serves as the basis for quantum condensed matter physics. It possesses a nondegenerate ground state and the Nernst heat theorem must be satisfied in this case as well [113,114]. In Section 8, we return to this point in connection with the recently proposed Drude-like nonlocal response functions.

As discussed in Section 6.1, the thermodynamic part of the Casimir conundrum for dielectrics is due to the exponentially fast vanishing of the conductivity (41) with decreasing temperature. This leads to a violation of Nernst heat theorem if the conductivity is taken into account in Lifshitz theory. To avoid this conclusion similar to metals with impurities, it was suggested [115] to consider some imaginary dielectric material possessing a constant conductivity at low temperature. Although the proposed model does not carry the thermodynamic anomaly, it is incapable of solving the Casimir conundrum for real dielectric materials because they are characterized by exponentially fast vanishing conductivity at low temperatures. Thus, neither the Casimir puzzle nor the Casimir conundrum can be solved without some radical changes in Lifshitz theory and/or in the used response functions.

7.6. The Anomalous Skin Effect and Spatial Nonlocality

The original formulation of Lifshitz theory assumes that the material of the plates is described by the dielectric permittivity depending on the frequency but not the wave vector. In the case of metallic plates, however, there is a frequency region where a connection between the electric field and the current becomes nonlocal and the concept of the frequency-dependent dielectric permittivity loses its meaning. This is the region of the anomalous skin effect where the spatial nonlocality plays an important role. At room temperature for Au the frequency region of the anomalous skin effect is from ${10}^{12}$ to ${10}^{13}$ rad/s, i.e., it is rather narrow. With decreasing temperature, however, this frequency region widens at the cost of a decrease in its left boundary. The questions arise as to the impact of the anomalous skin effect on the Casimir force and its possible role in the resolution of the Casimir puzzle.

In the presence of spatial nonlocality, the reflection coefficients in the Lifshitz formulas (8) and (10) are expressed via the surface impedances for the TM and TE polarizations [116,117,118]

$$r_{\mathrm{TM}}({\mathrm{i}\xi }_l,k_{\perp })=\frac{c{q}_l-{\xi }_l{Z}_{\mathrm{TM}}({\mathrm{i}\xi }_l,k_{\perp })}{c{q}_l+{\xi }_l{Z}_{\mathrm{TM}}({\mathrm{i}\xi }_l,k_{\perp })},r_{\mathrm{TE}}({\mathrm{i}\xi }_l,k_{\perp })=\frac{c{q}_l{Z}_{\mathrm{TE}}({\mathrm{i}\xi }_l,k_{\perp })-{\xi }_l}{c{q}_l{Z}_{\mathrm{TE}}({\mathrm{i}\xi }_l,k_{\perp })+{\xi }_l}.$$

(55)

The impedances in turn are expressed via the longitudinal ${\epsilon }^L({\mathrm{i}\xi }_l,k)$ and transverse ${\epsilon }^T({\mathrm{i}\xi }_l,k)$ dielectric permittivities, which describe the dielectric response of metal parallel and perpendicular to $k$ electric fields, respectively [117,118]

$$\begin{array}{ccc}& & Z_{\mathrm{TM}}({\mathrm{i}\xi }_l,k_{\perp })=\frac{{\xi }_l}{\pi c}{\int }_{-\infty }^{\infty }\frac{d{k}_z}{k^2}\left(\frac{c^2{k}_{\perp }^2}{{\xi }_l^2{\epsilon }^L({\mathrm{i}\xi }_l,k)}+\frac{k_z^2}{{k^T}^2({\mathrm{i}\xi }_l,k)+k_z^2}\right),\hfill \\ & & Z_{\mathrm{TE}}({\mathrm{i}\xi }_l,k_{\perp })=\frac{{\xi }_l}{\pi c}{\int }_{-\infty }^{\infty }\frac{d{k}_z}{{k^T}^2({\mathrm{i}\xi }_l,k)+k_z^2}.\hfill \end{array}$$

(56)

Here, $k_z$ is the third component of the wave vector, so that $k^2=k_{\perp }^2+k_z^2$ and

$${k^T}^2({\mathrm{i}\xi }_l,k)=k_{\perp }^2+{\epsilon }^T({\mathrm{i}\xi }_l,k)\frac{{\xi }_l^2}{c^2}.$$

(57)

The nonlocal generalizations ${\epsilon }^{L,T}$ of the Drude dielectric function ${\epsilon }_D$ for the description of the anomalous skin effect suggested in [117] were used [119] to calculate the correction to the Casimir force due to the effect of nonlocality. It was shown that with increasing separation from 100 to 300 nm the relative nonlocal correction is negative and its magnitude decreases from 0.3% to 0.1%. Thus, taking into account the effects of nonlocality arising in the dielectric response due to the anomalous skin effect cannot solve the experimental part of the Casimir puzzle.

Concerning the thermodynamic part of the Casimir puzzle, consideration of the anomalous skin effect leads to the same conclusions as discussed above for the simple Drude model. Nernst heat theorem is formally restored if there is some nonzero value of the effective relaxation parameter at zero temperature, and is violated otherwise [120]. In this sense, the spatial dispersion can play the same role as the relaxation of conduction electrons [121].

7.7. Inclusion of the Screening Effects

One more attempt to solve the Casimir puzzle and Casimir conundrum was made by considering the screening of the electric field created by an external source in a medium containing free charge carriers. This effect leads to the penetration of the static electric field into a conducting material to a depth of the screening length [122].

In Casimir physics, inclusion of the screening effects was first proposed in [123] in connection with an experiment measuring the Casimir–Polder force between a ${}^{87}$ Rb atom and a dielectric fused silica plate [42] (Section 6.2). If the DC conductivity of fused silica is taken into account in calculations of the Casimir–Polder force, the theoretical predictions of the standard Lifshitz theory are excluded by the measurement data. The Casimir–Polder entropy at zero temperature is equal to

$$S(a,0)=\frac{k_B\alpha \left(0\right)}{4a^3}[1-r_{\mathrm{TM}}\left(0\right)]>0,$$

(58)

where $r_{\mathrm{TM}}\left(0\right)$ is defined in (37) and $\alpha \left(0\right)$ is the static polarizability of the ${}^{87}$ Rb atom in violation of Nernst heat theorem [39].

With account of screening, the TM reflection coefficient at zero frequency (37) is replaced with [123]

$$r_{\mathrm{TM}}^{\mathrm{src}}(0,k_{\perp })=\frac{\epsilon \left(0\right)\sqrt{{\kappa }^2+k_{\perp }^2}-k_{\perp }}{\epsilon \left(0\right)\sqrt{{\kappa }^2+k_{\perp }^2}+k_{\perp }},$$

(59)

where $\kappa$ is the inverse quantity to the Debye-Hückel screening length

$$\frac{1}{\kappa }\equiv \frac{1}{{\kappa }_{DH}}=\sqrt{\frac{{\epsilon }_0{k}_{B}T}{4\pi e^{2}n}}$$

(60)

and $n=n\left(T\right)$ is the concentration of charge carriers in a dielectric material.

According to the results of [123], Lifshitz theory using the screened reflection coefficient (59) taking into account the presence of free charge carriers is consistent with the measurement data. This solves the experimental part of the Casimir conundrum in application to this specific experiment. However, it was shown [124] that the same theory still remains in disagreement with the measurement data of an experiment on the optically modulated Casimir force (Section 6.2).

The Lifshitz theory with the screened reflection coefficient (59) also provides a partial resolution of the thermodynamic part of the Casimir conundrum. For the insulators and intrinsic semiconductors, whose concentration of charge carriers $n\left(T\right)$ vanishes with temperature exponentially fast, Nernst heat theorem is satisfied. However, there are dielectric materials (dielectric-type semimetals, doped semiconductors with the dopant concentration below critical, dielectrics with ionic conductivity, etc.) whose static conductivity $\sigma \left(0\right)$ vanishes with temperature not due to the vanishing n but to the vanishing mobility of charge carriers. For these dielectric materials, Lifshitz theory using the screened reflection coefficient (59) still leads to a violation of Nernst heat theorem [124].

At the end of this section, we note that the screening effects were also considered in the reflection coefficients with any l. The developed formalism was presented in the form applicable to bodies with arbitrarily large charge carrier density including metals [125]. For this purpose, the Debye-Hückel screening length should be replaced with the Thomas–Fermi one [122]. It was shown [124] that this approach leads to approximately the same Casimir forces between metallic plates, as given by standard Lifshitz theory using the Drude model, and thus remains in contradiction with the measurement data. For metals with perfect crystal lattices the modified Lifshitz theory accounting for the screening effects violates Nernst heat theorem. Thus, despite some encouraging results, consideration of the screening effects did not resolve the Casimir puzzle and Casimir conundrum.

8. The Nonlocal Drude-Like Response to Quantum Fluctuations off the Mass Shell and the Casimir Puzzle

Many unsuccessful attempts to solve the Casimir puzzle and conundrum in the last 20 years suggest that there should be some alternative approach to their resolution. In this respect, our attention is attracted by graphene which is a 2D sheet of carbon atoms in the form of a hexagonal crystal lattice. The outstanding property of graphene is that at energies below a few eV it is well described by the Dirac model characterized by the linear dispersion relation, where the speed of light c is replaced by the Fermi velocity $v_F$ [126]. This allows to calculate many graphene’s properties, including its dielectric response to the electromagnetic field, on the basis of first principles of quantum electrodynamics at nonzero temperature. In so doing, the phenomenological response functions provided by the Drude or plasma models become unneeded.

Thus, it was found that graphene is described by the transverse and longitudinal dielectric permittivities depending on both the frequency and the wave vector, which are expressed via the polarization tensor [127,128,129,130]. In doing so, the predictions of Lifshitz theory using the exact graphene response functions were found to be in good agreement with an experiment on measuring the Casimir force between an Au-coated sphere and a graphene-coated plate [131,132]. The Casimir and Casimir–Polder entropies calculated for both the pristine graphene sheets and for graphene possessing a nonzero energy gap and chemical potential satisfy the Nernst heat theorem [133,134,135,136,137]. This means that the Casimir puzzle does not exist for graphene.

The case of graphene suggests that the problems discussed above might be rooted in the inadequate response functions of conventional 3D materials used in Lifshitz theory. The Casimir free energy (3) and force (6) are obtained by the integration over all $k_{\perp }$ from zero to infinity at each fixed $\omega$. This means that both the propagating waves, which are on the mass shell, and evanescent waves, which are off the mass shell, contribute to the final result. The Drude response function has been well checked experimentally and provides the correct response to real electromagnetic fields with a nonzero field strength. There is no direct experimental confirmation of this model, however, if we are seeking the response to quantum fluctuations that are off the mass shell. Additionally, the Casimir puzzle can be considered as an indirect indication that the Drude model incorrectly describes the response to fluctuations of this kind.

At this point, it is reasonable to search for nonlocal Drude-like response functions that provide approximately the same response as the standard Drude model to the electromagnetic fluctuations and fields on the mass shell but adequately describe the response to the off-shell fluctuations. The response functions of this kind were recently suggested [138]. They are given by

$$\begin{array}{ccc}& & {\tilde{\epsilon }}_D^T(\omega ,k_{\perp })=1-\frac{{\omega }_p^2}{\omega [\omega +\mathrm{i}\gamma (T\left)\right]}\left(1+\mathrm{i}\frac{v^T{k}_{\perp }}{\omega }\right),\hfill \\ & & {\tilde{\epsilon }}_D^L(\omega ,k_{\perp })=1-\frac{{\omega }_p^2}{\omega [\omega +\mathrm{i}\gamma (T\left)\right]}{\left(1+\mathrm{i}\frac{v^L{k}_{\perp }}{\omega }\right)}^{-1},\hfill \end{array}$$

(61)

where the quantities $v^{T,L}$ are the constants of the order of the Fermi velocity $v_F$. In the local limit, both ${\tilde{\epsilon }}_D^T$ and ${\tilde{\epsilon }}_D^L$ coincide with the standard Drude model

$${\tilde{\epsilon }}_D^T(\omega ,0)={\tilde{\epsilon }}_D^L(\omega ,0)={\epsilon }_D\left(\omega \right).$$

(62)

For the electromagnetic fields on the mass shell, we have $c{k}_{\perp }⩽\omega$ and

$$\frac{v^{T,L}{k}_{\perp }}{\omega }\sim \frac{v_F}{c}\frac{c{k}_{\perp }}{\omega }⩽\frac{v_F}{c}\ll 1.$$

(63)

Thus, in this case, the dielectric functions (61) lead to approximately the same results as the standard Drude model. This is different from the nonlocal response functions discussed in Section 7.6. For example, the nonlocal response describing the anomalous skin effect [117] used in the theory of the Casimir force [119,120] (see Section 7.6) was introduced to describe the physical phenomenon occurring in the fields on the mass shell. The nonlocal response functions describing electron gas without and with collisions were introduced [139,140]. In the static limit, they describe the screening effects considered in Section 7.7. These again occur in real fields on the mass shell.

By contrast, the suggested permittivities (61) are not intended for a description of small nonlocal effects in real fields and deviate from the standard Drude model only for the off-the-mass-shell fluctuations. Note also that the permittivities (61) depend only on $k_{\perp }$ and do not depend on $k_z$, as in the case of [117,139,140], because in the presence of Casimir plates the translational invariance along the z-axis is violated and it is, strictly speaking, impossible to define the response functions depending on the 3D vector $k$ [141,142].

For the permittivities depending only on $k_{\perp }$ (56) results in

$$\begin{array}{ccc}& & Z_{\mathrm{TM}}({\mathrm{i}\xi }_l,k_{\perp })=\frac{c}{{\xi }_l}\left[\frac{k_{\perp }}{{\epsilon }^L(\mathrm{i}{\xi }_l,k_{\perp })}+\frac{k^T({\mathrm{i}\xi }_l,k_{\perp })-k_{\perp }}{{\epsilon }^T(\mathrm{i}{\xi }_l,k_{\perp })}\right],\hfill \\ & & Z_{\mathrm{TE}}({\mathrm{i}\xi }_l,k_{\perp })=\frac{{\xi }_l}{c{k}^T({\mathrm{i}\xi }_l,k_{\perp })}.\hfill \end{array}$$

(64)

Then, the reflection coefficients (55) valid in the nonlocal case are given by

$$r_{\mathrm{TM}}({\mathrm{i}\xi }_l,k_{\perp })=\frac{{\epsilon }^T(\mathrm{i}{\xi }_l,k_{\perp })q_l-k^T({\mathrm{i}\xi }_l,k_{\perp })-k_{\perp }\left[{\epsilon }^T(\mathrm{i}{\xi }_l,k_{\perp })-{\epsilon }^L(\mathrm{i}{\xi }_l,k_{\perp })\right]{\left[{\epsilon }^L(\mathrm{i}{\xi }_l,k_{\perp })\right]}^{-1}}{{\epsilon }^T(\mathrm{i}{\xi }_l,k_{\perp })q_l+k^T({\mathrm{i}\xi }_l,k_{\perp })+k_{\perp }\left[{\epsilon }^T(\mathrm{i}{\xi }_l,k_{\perp })-{\epsilon }^L(\mathrm{i}{\xi }_l,k_{\perp })\right]{\left[{\epsilon }^L(\mathrm{i}{\xi }_l,k_{\perp })\right]}^{-1}},$$

$$r_{\mathrm{TE}}({\mathrm{i}\xi }_l,k_{\perp })=\frac{q_l-k^T({\mathrm{i}\xi }_l,k_{\perp })}{q_l+k^T({\mathrm{i}\xi }_l,k_{\perp })}.$$

(65)

Now it is possible to substitute the specific Drude-like permittivities (61) taken at $\omega ={\mathrm{i}\xi }_l$ in (65), to calculate the Casimir force between two Au plates (10) and, by using the PFA (34), find the gradient of the Casimir force between a sphere and a plate. After introducing small corrections due to surface roughness and inaccuracies of the PFA, considered in Section 7.3 and Section 7.4, we can compare the theoretical results obtained using Lifshitz theory and the nonlocal Drude-like response functions (61) with the measurement data.

Figure 8 shows the obtained theoretical predictions in the experimental configuration of the micromechanical torsional oscillator [22] as a function of separation by the black band. The measurement results are shown as crosses (as in Figure 2). We also reproduce from Figure 2 the red and blue theoretical bands found using the extrapolations of the optical data by means of the plasma and Drude models. As shown in Figure 8, the theoretical predictions using both the nonlocal Drude-like and plasma response functions agree with the measurement data. An important advantage of the Drude-like response is that it properly takes into account the relaxation properties of free charge carriers, which are disregarded by the plasma model.

As one more example, in Figure 9 we plot by the black band the predictions of Lifshitz theory using the optical data of Au extrapolated by the nonlocal Drude-like response functions (61) in the experiment measuring the gradient of the Casimir force between an Au-coated sphere and an Au-coated plate at larger separations using a dynamic atomic force microscope [85] (Section 7.2). The theoretical predictions using the Drude and plasma models are indicated by the blue and red bands, respectively, whereas the measured gradients of the Casimir force are shown as crosses. It is again seen that the theoretical approach using the nonlocal Drude-like response functions agrees with the measurement data along with the plasma model. The advantages of the Drude-like functions are that they properly account for the dissipation of conduction electrons and correctly describe the reflectances of the electromagnetic waves on the mass shell incident on an Au plate [138].

Based on the above, we can conclude that the nonlocal Drude-like response functions (61) provide a resolution to the experimental part of the Casimir puzzle for nonmagnetic metals. A detailed examination of the asymptotic behavior of the Casimir entropy calculated using the Drude-like functions (61) at low temperature demonstrates that these response functions also solve the thermodynamic part of the Casimir puzzle. Thus, for nonmagnetic metals with perfect crystal lattices the low-temperature behavior of the Casimir entropy is given by

$${\tilde{S}}_D(a,T)=C_1\left(a\right)\sqrt{T},$$

(66)

where the coefficient $C_1$, in addition to separation, depends on ${\omega }_p$, $v^T$, and the fundamental constants c, ℏ, and $k_B$.

For a metal with impurities possessing some residual relaxation ${\gamma }_0$, the low-temperature behavior of the Casimir entropy found using the nonlocal Drude-like response functions (61) takes the form

$${\tilde{S}}_D(a,T)=C_2\left(a\right)T,$$

(67)

where, in addition to the parameters indicated above, the coefficient $C_2$ also depends on ${\gamma }_0$.

From (66) and (67) it is seen that both for metals with perfect crystal lattices and for metals with impurities the Casimir entropy goes to zero with vanishing temperature. This means that the Lifshitz theory using the nonlocal Drude-like response functions is consistent with the laws of thermodynamics.

9. Discussion: The Present Status of the Casimir Puzzle and Casimir Conundrum

In the foregoing, we considered several theoretical and experimental results that are not yet fully understood and are called the Casimir puzzle and the Casimir conundrum in the literature. These results are related to the Casimir effect, which is the physical phenomenon determined by the properties of the quantum vacuum and its interaction with matter. As discussed in Section 1, several basic challenges of modern physics are directly connected with the concept of a vacuum. Because of this, it is probably not surprising that the Casimir puzzle and the Casimir conundrum do not yet have an ultimate resolution despite the work performed by many researchers.

Several precise experiments measuring the Casimir interaction between metallic and dielectric test bodies confirmed that the standard dielectric functions commonly used for a description of the response of matter to real electromagnetic fields with a nonzero field strength lead to incorrect predictions in the framework of Lifshitz theory. For a long time, serious doubts have been raised regarding the precision and interpretation of these measurements, but after the experiments with magnetic test bodies [25,26] and differential force measurement, where the measured signal differed from the theoretical prediction of Lifshitz theory by a factor of 1000 (Section 4.2), the facts have been established with certainty.

As the Drude response function also has an unambiguous experimental confirmation in the area of real electromagnetic fields with a nonzero field strength, it is reasonable to suggest that it may incorrectly describe the response of metals to quantum fluctuations off the mass shell. This suggestion finds support in the fact that some indirect experimental evidence can be obtained only about the form of the longitudinal permittivity ${\tilde{\epsilon }}^L$ in the off-the-mass-shell fields but not about ${\tilde{\epsilon }}^T$ [116], which should be derived theoretically as was performed for graphene using the polarization tensor.

No fewer difficulties are connected with the thermodynamic parts of the Casimir puzzle and the Casimir conundrum. Contradictions between Lifshitz theory and thermodynamics arise when we use the response functions that have been excluded by measurements of the Casimir force. Thus, it is reasonable to hope that agreement will be restored if the used response to the off-the-mass-shell quantum fluctuations, which is extrapolated from the area of real fields with a nonzero field strength and has no direct experimental confirmation, will be somehow corrected.

In the case of nonmagnetic metals, possible realization of this program has already been suggested (Section 8) in the form of nonlocal Drude-like response functions, which lead to approximately the same results, as the standard Drude model, in the on-the-mass-shell fields, but to significantly different responses to the off-the-mass-shell quantum fluctuations. It was shown that Lifshitz theory using the nonlocal Drude-like model agrees with the measurement data, as it does when using the plasma model, which simply disregards the relaxation properties of conduction electrons. Additionaly, the Casimir entropy calculated using the Drude-like response functions satisfies the Nernst heat theorem for metals with both perfect crystal lattices and lattices with impurities. Thus, the suggested Drude-like model points the way to a resolution to the Casimir puzzle.

However, we emphasize that unlike the case of graphene, this is a phenomenological solution. It demonstrates the possibility of solving the Casimir puzzle by modifying the dielectric response to quantum fluctuations off the mass shell, but this solution might not be unique. Furthermore, it is necessary to extend the proposed approach to the case of magnetic metals and to apply similar ideas to dielectric materials to find a resolution of the Casimir conundrum. Thus, further investigations of these challenging problems are necessary.

10. Conclusions

To conclude, we described the current situation regarding the complicated problems faced by Lifshitz theory during the last 20 years. The experimental findings in this field of research are largely settled. However, their theoretical understanding is far from being complete. Some ideas and even examples of considerable promise have been proposed on how the Casimir puzzle and the Casimir conundrum could be solved, but they need further theoretical justification.

The thermodynamic test of Lifshitz theory using one or other type of the dielectric response confirmed its usefulness. It is not by chance that Lifshitz theory using the Drude model for metals or the DC conductivity for dielectrics was found to violate the Nernst heat theorem. A development of Casimir physics confirmed that the thermodynamic and experimental problems occur concurrently. This means that if Lifshitz theory using some model of the dielectric response leads to a violation of Nernst heat theorem for the Casimir entropy, we should expect that theoretical predictions using this model will disagree with the measurement data.

The suggested nonlocal Drude-like response functions provide a supposed resolution to both the experimental and thermodynamic parts of the Casimir puzzle for nonmagnetic metals. Future trends should provide a final resolution of the complicated problems of Casimir physics for both metallic and dielectric materials and widen the scope of fluctuational electrodynamics using a more reliable description of the dielectric response of matter to quantum fluctuations off the mass shell.