There is a method which allows one to solve – at least in principle – the Schrödinger equation of an n-electron system. The recipe is to choose a basis set of m orbitals or 2m spin orbitals and to construct all possible n-electron determinants from this basis. Their number is $\left(\begin{array}{c}\hfill 2m\hfill \\ \hfill n\hfill \end{array}\right)$. One expands the Hamiltonian in this basis and diagonalizes it. This is called full CI (CI for configuration interaction). Next one increases the dimension of the orbital basis such that eventually it becomes complete (more precisely complete in the l^st Sobolev space [78]) and proceeds until convergence of the lowest eigenvalue(s). In the limit of a complete basis full CI becomes complete CI and virtually exact [79]. Unfortunately this method is completely inpracticable. For large m the number of Slater determinants increases as ~ mⁿ/n!, i.e. exponentially with the electron number n.

It is clear that full CI is only possible for relatively small basis sets. In view of this the progress in full CI calculations [83, 84, 85] is rather impressive. The main practical interest in full CI is to use it as benchmark to check the performance of approximations to full CI for the same basis, with the hope that good approximations to full CI can then be used successfully with larger basis sets.

The scaling of the computational effort with the number n of electrons (we shall henceforth simply say n-scaling) is particularly unfortunate for full CI, other current methods scale – as we shall see – as some power n^k of n. There is evidence that one does something basically wrong if one uses a method that scales unphysically with the particle number n, where it remains to be seen what is a physical n-scaling. If one compares calculations of related molecules with a different number N of similar atoms, the number n of electrons scales essentially with N such that n-scaling and N-scaling show the same pattern. One is also interested in the scaling with the number m of basis functions. This has a different meaning depending on whether one compares calculations on one system with constant n and varying m, or whether one consider different systems consistently with m essentially proportional to N.

The improvement of the n-scaling will definitely be one of the great challenges to quantum chemistry in the 21^st century.

The n-electron wave function is a rather complicated quantity and it contains much more information than is usually needed. In order to evaluate the energy one has – in a modern Fock space notation [86, 87] –

$H=h_q^p{a}_p^q+\frac{1}{2}{g_r^p}_s^q{a_p^r}_q^s$

$\begin{array}{c}E=<\mathrm{\Psi }\left|H\right|\mathrm{\Psi }>=h_q^p<\mathrm{\Psi }\left|a_p^q\right|\mathrm{\Psi }>+\frac{1}{2}{g_r^p}_s^q<\mathrm{\Psi }\left|{a_p^r}_q^s\right|\mathrm{\Psi }>\\ =h_q^p{\mathrm{\gamma }}_p^q+\frac{1}{2}{g_q^p}_s^q{{\mathrm{\gamma }}_p^r}_q^s\end{array}$

Here a_p^q etc. are excitation operators with respect to an orthonormal spinorbital basis {ψ_p}, while h_q^p and g_r^p_s^q are one- and two-particle matrix elements

$a_q^p=a^p{a}_q=a_p^{†}{a}_q;{a_r^p}_s^q=a^p{a}^q{a}_s{a}_r$

$h_q^p=<{\psi }_q\left|h\right|{\psi }_p>;{g_r^p}_s^p=<{\psi }_r\left(1\right){\psi }_s\left(2\right)\left|g\left(1,2\right)\right|{\psi }_p\left(1\right){\psi }_q\left(2\right)>$

with h and g the one- and two-particle parts of the Hamiltonian respectively, and γ_p^q, γ_p^r_q^s and the one- and two-particle density matrices respectively

${\gamma }_p^q=<\mathrm{\Psi }\left|a_p^q\right|\mathrm{\Psi }>;{{\gamma }_p^r}_q^s=<\mathrm{\Psi }\left|{a_p^r}_q^s\right|\mathrm{\Psi }>$

In the above expressions the Einstein summation convention is implied.

When the one electron basis {ψ_p} on which our Fock space is built, is finite (of dimension 2m), the Hamiltonian H given by (3) is exactly that of full CI [87]. The eigenfunctions and eigenvalues that one obtains in full CI, are the eigenfunctions and eigenvalues of this Hamiltonian.

3.2 The 2-particle density matrix and the n-representability problem

The number of density matrix elements γ_p^r_s^s scales with m⁴, i.e. it does not directly depend on n. However, since m should be choosen roughly proportional to n, there is a scaling ~ n⁴. If it were possible to take the γ_p^r_q^s rather than the CI coefficients as variational parameters, we would have got rid of the scaling problem of full CI. Unfortunately the γ_p^r_q^s cannot be regarded as variational parameters, unless one can impose conditions which guarantee that a γ-matrix is derivable from an n-particle wave function. This n-representability problem has played a big role in the late sixties and has been most thoroughly been formulated by Coleman [88, 89]. Unfortunately a simple solution of the n-representability problem which allows one to replace the n-electron wave function Ф by the two-particle density matrix γ_p^r_q^s, in a variational approach has not been found. Note that γ_p^q is not independent of γ_p^r_q^s, but can be derived from this by partial contraction

$\left(n-1\right){\mathrm{\gamma }}_p^q={\displaystyle \sum _r{{\mathrm{\gamma }}_p^q}_r^r}$

Although the idea to formulate the n-electron problem in terms of density matrices has been more or less abandoned, it is likely that it will be revived in the future, especially in view of the success and the limitations of the much more simple-minded density functional approaches (see sec. 6). For interpretational purposes the density matrices have always been rather central quantities [90, 91, 92, 93, 94, 95].

The eigenfunctions and eigenvalues of {γ_q^p} are known as natural spinorbitals and their occupation numbers [80], those of {γ_r^p_s^q} the natural geminals and their occupation numbers.

For typical closed shell atoms or molecules {γ_q^p} is close to idempotent, i.e. its eigenvalues are either close to 1 or to 0. In this case a single determinant Ф built up from the n first natural spin orbitals (with occupation numbers close to 1) is a good first order approximation to the exact Ψ. Then Hartree-Fock theory, in which < Ф|H|Ф > is minimized with respect to variation of the orbitals contained in Ф, is a good first order approximation. In fact, everything not contained in Ф means electron correlation.

For Ф a single Slater determinant all information is contained in its one particle density matrix [96]. For this special case we use instead of γ.

${\mathrm{\varrho }}_q^p=<\Phi \left|a_q^p\right|\Phi >$

${{\mathrm{\varrho }}_r^p}_s^p=<\Phi \left|{a_r^p}_s^q\right|\Phi >={\mathrm{\varrho }}_q^p{\mathrm{\varrho }}_s^q-{\mathrm{\varrho }}_s^p{\mathrm{\varrho }}_r^q\mathit{etc}\text{.}$

If one defines a two-particle cumulant matrix

${{\mathrm{\lambda }}_r^p}_s^q={{\mathrm{\gamma }}_r^p}_s^q-{\mathrm{\gamma }}_r^p{\mathrm{\gamma }}_s^q+{\mathrm{\gamma }}_s^p{\mathrm{\gamma }}_r^q$

and analogously higher order cumulants [15, 97, 98], a simple Slater determinant wave function is characterized by vanishing of all cumulants. Obviously the two-particle cumulant describes two-particle correlations etc.. On the basis of cumulants [15] which are (at variance with the k-particle density matrices) extensive (additively separable quantities) a definition of the correlation energy is possible that does not require a single Slater-determinant reference state and the corresponding Hartree-Fock energy. We define

$E_{\mathit{nc}}=h_q^p{\mathrm{\gamma }}_p^q+\frac{1}{2}{g_r^p}_s^q\left({\mathrm{\gamma }}_p^r{\mathrm{\gamma }}_q^s-{\mathrm{\gamma }}_q^r{\mathrm{\gamma }}_p^s\right)$

$E_{\mathit{corr}}=\frac{1}{2}{g_r^p}_s^q{{\lambda }_p^r}_q^s$

Where the non-correlated energy E_nc is of the same form as the Hartree-Fock energy, but with {γ_q^p} the exact one-particle density matrix, which is usually not idempotent. This concept has not yet been applied in practice.

3.3 Many-body perturbation theory

Noting that full CI is – in principle (when the basis limit is reached) – exact but unfeasible (except for very small basis set), and that Hartree-Fock is feasible, but a very crude approximation, some compromise between these two extrema has to be found.

One possibility is to use perturbation theory, truncated at some low order. There are various options for perturbation theory applied to an n-electron system. One of them consists in dividing the Hamiltonian H in the following way into an unperturbed Hamiltonian H₀ and a perturbation V

$H=H_0+\mathit{\lambda V};H_0=h_q^p{a}_p^q;V=\frac{1}{2}{g_r^p}_s^q{a_p^r}_q^s$

On this partition many-body perturbation theory (MBPT) as developed by Brueckner, Goldstone and others is based [99, 100, 101, 102, 103, 104]. This is also the basis of the Z-dependent perturbation theory of atoms of Layzer and others [105, 106, 107]. Strictly speaking one should distinguish between a perturbation theory in terms of the original Hamiltonian (not projected to a finite one-electron basis) and perturbation theory of the Fock space Hamiltonian in a finite basis, which is equivalent to perturbation theory of the matrix eigenvalue problem of full CI. When one refers to perturbation theory for electron correlation, almost exclusively the latter choice is implied, and we shall only consider this option.

The partition (14) corresponds to treating the bare nuclear Hamiltonian (BNH) as unperturbed and the entire electron interaction as perturbation. It is much better (i.e. leads to better results to low orders and gives a better overall convergence pattern), to include part of the electron interaction in H₀. Again there are various possibilities, but the Møller-Plesset partition [108, 109, 110] has turned out to be most convenient. Here

$H=E_0+H_0^{\prime }+V^{\prime }$

$H_0^{\prime }=f_q^p{\tilde{a}}_p^q;V^{\prime }=\frac{1}{2}{g_r^p}_s^q{{\tilde{a}}_p^r}_q^s$

$E_0=h_q^p{\mathrm{\varrho }}_p^q+\frac{1}{2}{g_r^p}_s^q{{\mathrm{\varrho }}_p^r}_q^s$

$f_q^p=h_q^p+\left({g_q^p}_s^r-{g_s^p}_q^r\right){\mathrm{\varrho }}_r^s$

where _q^p are density matrices corresponding to a Hartree-Fock reference function Ф, and where ${\tilde{a}}_q^p$ and ${{\tilde{a}}_p^r}_q^s$ are excitation operators with respect to Ф as so-called physical vacuum. These have the property

$<\Phi \left|{\tilde{a}}_p^q\right|\Phi >=0;<\Phi \left|{{\tilde{a}}_q^r}_p^s\right|\Phi >=0$

and are related to the a_p^q and a_q^r_p^s with respect to the genuine vacuum as [111, 97]

${\tilde{a}}_p^q=a_p^q-{\mathrm{\gamma }}_p^q$

${{\tilde{a}}_q^r}_p^s={a_q^r}_p^s-{\mathrm{\gamma }}_q^r{a}_p^s+\dots$

For the Møller-Plesset partition the first order perturbation vanishes

$E_1=<\Phi \left|V^{\prime }\right|\Phi >=0$

If one chooses canonical orbitals, for which f_q^p is diagonal

$H_0^{\prime }={\epsilon }_p^{\left(0\right)}{\tilde{a}}_p^p$

the second order energy is simply

$E_2={{\displaystyle \sum _{i,j,a,b}\left({\epsilon }_i^{\left(0\right)}+{\epsilon }_j^{\left(0\right)}-{\epsilon }_a^{\left(0\right)}-{\epsilon }_b^{\left(0\right)}\right)}}^{-1}{\left|<\Phi \left|V^{\prime }{a_i^a}_j^b\right|\Phi >\right|}^2$

with i,j counting spin-orbitals occupied in Ф, a, b virtual spin-orbitals.

Only doubly excited configurations

${{\Phi }_i^a}_j^b={a_i^a}_j^b\Phi$

contribute to E₂. Also E₃ only requires double excitations, while for the evaluation of E₄ triple excitations are needed and so forth.

MP-perturbation theory is formally simple to low orders, but becomes increasingly complicated to higher orders. The most serious drawback is that convergence of MP-PT is by no means guaranteed. Depending on the choice of the basis, divergence can be enforced or avoided [112, 113], which indicates that much that one finds may be an artifact of the matrix representation, that has other singularities – which determine the radius of convergence – than the original Hamiltonian. Anyway even if the MP-PT series converges, convergence may be very slow [114] and one cannot break up the series, unless sophisticated extrapolation methods are used.

Actually only MP2 has become a kind of standard, MP3 is regarded as hardly superior to MP2, while MP4 is occassionally used, but it requires about as much computational effort as the usually much better CCSD(T) method.

Let us also mention that, independent of the partition of H that one chooses, always the Rayleigh-Schrödinger form of PT is used, because – at variance with the Brillouin-Wigner PT – it is extensive (connected). See later.

3.4 Coupled-cluster and related methods

Related to perturbation theory are methods that have been inspired by perturbation theory, but which are essentially non-perturbative. Early attempts on these lines were partial summations of certain classes of diagrams [115] (in a diagrammatic formulation of MBPT), where it was not always clear whether these diagrams were summed because this was formally easy or because these diagrams were regarded as particularly important. More simple-minded but rather effective has been selective CI based on arguments from MP-PT [116].

Noting that to the leading orders of MP-PT only doubly excited configurations Φ_i^a_j^b contribute, one can decide to do better than MP2 or MP3 by including all Φ_i^a_j^b in a CI (i.e. variational) calculation. The number of unknown coefficients scales then as n²m², i. e. essentially as n⁴, while the overall computational effort goes as n⁴m² i.e. essentially as n⁶. The scaling is much better than for full CI, but the quality of the result is, of course, much poorer. One can so define a hierarchy of CI methods, called CI-SD, CI-SDT, CISDTQ etc. where S, D, T, Q stands for singly, doubly, triply, quadruply excited configurations. This hierarchy has the advantage over MP-PT that it does converge to full CI if one let the excitation rank go to the number of electrons. This is, however, just what one wants to avoid.

A serious disadvantage of these truncated CI schemes is that they are not extensive (size consistent). This means essentially that the energy of a supersystem consisting of two identical subsystems at large distances computed by CI is not twice the energy of the subsystem. For a system of N He atoms at mutually infinite distance the correlation energy goes for large N not as ~ N but as ~ $\sqrt{N}$ [12, 117].

The size-consistency problem has been removed by using the coupled cluster (CC) ansatz [12, 118, 119, 120, 121, 122, 123, 124, 125]. While in CI one writes the wave function as (in intermediate normalization)

$\mathrm{\Psi }=\left(1+X_a^i{a}_i^a+{X_a^i}_b^j{a_i^a}_j^b+X_a^i{{}_b^j}_c^k{a}_i^a{{}_j^b}_k^c+\dots \right)\Phi$

in CC theory the formulation is

$\mathrm{\Psi }=exp\left\{S_a^i{a}_i^a+{S_a^i}_b^j{a_i^a}_j^b+\dots \right\}\Phi$

If one truncates the CI expansion one gets a result which is variational, but not extensive, one can truncate the CC-expansion (in the exponential) without loosing extensivity. It is then, however, practically impossible to be variational as well [125]. Like for CI one can define CC-SD, CC-SDT, CC-SDTQ etc. The n-scaling of the computational demands is similar to that for the corresponding steps in the CI hierarchy. Like for the CI hierarchy the CC hierarchy also converges to full CI if the excitation rank goes to n, but the convergence is definitely faster.

While CCSD has become a kind of standard, full CCSDT calculations [126, 127, 128] are still too computer-resource demanding that their use is limited to rather small basis sets. Among the simplifications of CCSDT, the CCSD(T) method [129] has become very popular. There single and double substitutions are treated non-perturbativity, but triple substitutions by means of perturbation theory. An alternative to CCSD(T) is CCSD[T] – formerly called CCSD+T(CCSD) – that is probably better [131], although formal arguments are rather in favour of CCSD(T) [130]. While CCSDT scales with n⁸ (and CCSDTQ with n¹⁰), CCSD(T) and CCSD[T] scale only with n⁷, which matters a lot. CCSD(T) is these days regarded as the best that one can do for sufficiently small molecules. Nevertheless cases have been reported, where CCSD(T) is definitely not enough, e.g. if one wants to evaluate the harmonic vibration frequency of HF with an error of less than 1 cm^-1 [131].

Closely related to CC theory (and also to MP-theory) are the IEPA [3, 10, 12] and CEPA [11, 12, 13, 132, 133, 134] methods, that were used in molecular calculations prior even to the large scale studies in terms of MP2. IEPA (independent electron pair approximation) has in common with MP2 that the various electron pairs are decoupled, and MP2 has, in fact, been a first step on the way to IEPA [3, 10, 12]. Unlike in MP2 the various decoupled pairs are treated exactly (in the limitations due to the use of a finite basis) in IEPA. In CEPA (coupled electron pair approximation) the coupling of the pairs is taken into account, but unlike in CCSD, to which CEPA is closely related, some (generally small) indirect couplings are ignored. Although CEPA can be formulated as an approximation to CCSD [11, 12], in practical applications it has turned out that CEPA performs even better than CCSD (although it is cheaper), because apparently effects of triple substitutions are, to some extent simulated in CEPA [135, 136]. IEPA and CEPA share with MP and CC that they are extensive and not variational.

It is often forgotten that the probably first large-scale ab-initio implementation of CCSD by Taylor et al. [120] was based on the CEPA code of Ahlrichs et al. [13]. As to a comparison of CEPA and CC methods see [137]. We come back to an interesting aspect of IEPA and CEPA in sec. 5.

Closely related to CEPA are the method of self-consistency electron pairs (SCEP) [138] and also the coupled-pair functional approach (CPF) [139].

3.5 Multi-configuration-based methods

Most of the effort in coupled-cluster theory has concentrated on systems which are essentially of closed shell type, where the wave function is dominated by a single Slater determinant Ф, such that the wave function can be expanded around this Ф and good convergence is expected.

The situation is much more complicated if to 1^st order a single Slater determinant is not sufficient and rather a multiconfiguration reference Ф must be used. A completely satisfactory coupled-cluster theory for this case has not yet been formulated. This has both fundamental and practical reasons.

Among the fundamental reasons is the dilemma that the most straightforward formulation of an extensive theory leads inevitably to the appearance of the intruder problem and that it is hard to eliminate this problem without violating extensivity. In fact extensivity requires a Fock space formulation with a multiplicatively separable wave operator [12, 87]. This means that one formulates the wave operator and an effective Hamiltonian for the full valence space, for all possible particle numbers, i.e. that one uses a so-called valence universal theory. However then one can generally not avoid that external orbitals (i.e. which are not not in the valence space) get energies close to those of valence orbitals and dangerous energy denominators arise. These intruder problems are largely avoided in a state-specific theory limited to a single state, but this means to loose a formulation in Fock space and extensivity is no longer guaranteed. This dilemma has been analysed in detail by Mukherjee, [140, 141] who has also discussed possible solutions, which we can only mention without going into details. One possibility is the use of intermediate Hamiltonians, mainly proposed by Malrieu et al. [142], but see also refs. [143, 144]. Another possibility consists in the use of incomplete model spaces [145, 146, 148], a disadvantage of which is that they can usually not be applied to potential energy surfaces, since for different ranges of geometry different incomplete model spaces must be used. There are further methods in the framework of a state universal theory both for complete and incomplete model spaces [149, 150, 151], which are somewhat intermediate between valence universal and state specific, and which are to some extent based on the projection of a Fock space theory to a Hilbert space for finite particle number. Particularly promising appears the combination of extensivity and a state-specific theory using the recently developed concept of normal ordering (and the corresponding Wick theorem) with respect to a multiconfiguration wave function[15, 97, 98]. On this line there are certainly challenges for the next century.

From a practical point of view one has to decide whether one chooses a contracted [152, 153] or uncontracted excitation scheme. In the latter the number of excitations to include is tremendous, in the former one must find a way how to deal with the so-called excitations with spectator-lines (unless one neglects these, which appears to be justified to some extent). Contracted means that one considers formal single, double etc. excitations with respect to the full MC-SCF wave function Ψ, e.g. a_q^pΨ, a_r^p_s^qΨ, uncontracted means that one decomposes Ψ into a sum of Slater determinants and takes the excitations with respect to these.

As long as a satisfactory multireference coupled-cluster theory is missing, there are various options for states that need a zeroth-order multiconfiguration wave function. One possibility is to start from an MC-SCF calculation and to improve this by selected CI. Since the MC-SCF part is basically extensive, while the CI part is not, and since one can hardly go beyond external double excitations, one tends to include as many configurations in the MC-SCF part as possible. However, MC-SCF is usually of CAS (complete active space) [154] type, e.g. like full CI, which restricts the possible size of the active space. Such multireference CI scheme have been very popular for describing excited states, reaction barriers, dissociation processes etc.

Another possibility is to start from MC-SCF and to improve it by means of perturbation theory. The most popular approach on these lines is CASPT2 of Roos et al. [155]. This is almost extensive and one is not obliged to make the active space as large as possible.

One can further try to formulate a generalization of CEPA or rather CPF to a multiconfiguration reference function. On these lines the average coupled pair functional (ACPF) has been derived by Gdanitz and Ahlrichs [157]. A straightforward MC-CEPA method has been proposed by Fink and Staemmler [158]. This is extensive, while ACPF is only approximatively so.

One can finally try to apply single reference CC, but to go to very high excitations, at least selectively [159]. It is hard to avoid a lack of balance in such treatments.

In this context one should also mention the rather old MRD-CI-method of Buenker and Peyerimhoff [160]. There one starts from a multideterminant reference function and improves it by CI with single and double substitution. This method has not been designed for single states, but rather for the computation of various states simultaneously.

Density functional (DF) [201] methods (for reviews see [202, 203, 204, 205]) were very popular in solid state physics since about 1965, but were hardly applied in chemistry for quite a while (among the applications to structural chemistry see e.g. ref. [206]), before suddenly around 1988 they really conquered chemistry. Even before this the rather good performance in solid theory did not remain unobserved by quantum chemists, but the reluctance to consider density functional methods more seriously had various reasons.

(a) The justification of density functional methods in terms of the Hohenberg-Kohn theorem [207, 208, 209] appeared somewhat unorthodox and had a touch of mystery.

(b) DF methods performed surprisingly well at rather low cost. The results were usually superior to those of Hartree-Fock calculations, i. e. correlation effects were somehow taken care of, but not as accurately as in standard quantum chemical methods.

(c) There appeared to be no way to improve DF calculations systematically, when they were not good enough. The usual DF schemes did not appear to be steps in a hierarchy that eventually lead to an exact theory.

(d) A precursor of DF methods, the X α-scheme [210, 211] had been used in quantum chemistry before, but got a rather bad reputation, mainly due to overselling and the marriage with muffin-tin potentials [212].

(e) The existing DFT-codes of the first generation were numerically often rather inaccurate. Therefore calculations with one code could not be reproduced by another code, using, of course, the same functional. More serrious was that the limited numerical accuracy made the calculation of potential curves rather problematic.

Things changed drastically when independently Becke [213] and Perdew [214] invented new gradient-corrected functionals, which performed much better than the previously used local density functionals and which turned out to be competitive with MP2 at a cost somewhat lower than that of SCF calculations. Various groups implemented DF calculations taking advantage of all the knowhow of traditional quantum chemistry, in particular using expansions in a Gaussian basis, and shortly afterwards DF codes became available in commercial program packages. Also the problems of accurately (and reproduceably) evaluating the exchange correlation functionals were solved [215, 216, 217]. Further the evaluation of energy derivatives became routine [218, 219]. For a black-box user it is now as easy to perform an SCF as a B3LYP calculation. ’B3LYP’ stands for Becke-3-parameter for the exchange part and Lee-Yang-Parr for the corelation part. It is one of the most popular density functionals for use in chemistry and is implemented in the commerical GAUSSIAN program package.

The attitude of most users is pragmatic. One does not worry why DF methods work and uses them as one uses other methods implemented in the same black box. There are quite a few different functionals available and one can choose between them depending on the problem that one wants to solve, where the superiority of one functional over another is usually based on statistical comparison with experiment rather than on formal arguments. This pragmatic attitude is encouraged by the fact that some of the density functionals in current use, contain parameters that were adjusted to fit experimental data. This – in order not to say more – puts DF schemes in the neighborhood of semiempirical methods.

The density functionals which perform best at present, like B3LYP are mixtures of Kohn-Sham type functionals (see later) and Hartree-Fock functionals and exploit the fact that Kohn-Sham type functionals tend to overestimate binding, while Hartree-Fock underestimates it.

The acceptance of DFT by the ab-initio community necessarily implied some change of paradigm. In the ab-initio world it had always been regarded as problematic and even as a kind of cheating if one got good agreement with experiment due to a fortunate cancellation of errors (unless this cancellation was ’controlled’). In DFT one can judge the quality of a functional only from a comparison of the results wit experiment or with some benchmark calculation. There is hence no criterion for the extent of error compensation on which the performance of the functional is based. So the refutation of error cancellation can hardly be maintained. Nevertheless if one wants to get ’the right answer for the right reason’ one should not rely on DFT.

6.2 The Hohenberg-Kohn theorem as a Legendre transformation

The essential problem with DF methods is to understand why they work, or rather which of their features is responsible for the good agreement with experiment in many cases. A systematic improvement but also a guide to scope and limitations rests on the answer to this question. Although some current studies concentrate on this aspect [220, 221, 222, 223], for the majority of people grown up with DF approaches even to ask this question is a kind of sacrilege. The argument often put forward is that density functional theory is exact – in principle – and that deviations from exact results are simply due to the approximations inherent in the used functionals. This argument, usually given with reference to the so-called Hohenberg-Kohn theorem, is rather meaningless and reveals a basic misunderstanding of the situation.

To appreciate what the Hohenberg-Kohn theorem [207] really implies it is useful to formulate it in a language that it is slightly different from that given in most papers in this field. The key reference is that to a study of E. H. Lieb [224, 225], who gave a mathematically rigorous analysis of density functional theory on the basis of the modern theory of convex functionals, in which the concept of a Legendre transformation plays a central role. This concept was, in a related context even been alluded to in a paper [208] contemporary with that of Hohenberg and Kohn. The main concern of Lieb was related to the appropriate definition of the domains of the important functionals E(V) and F(). In fact Lieb criticized Hohenberg and Kohn for having been somewhat too careless with these domains. Nevertheless we can here only give a nonrigorous presentation of Lieb’s approach, not worrying about the domains of E(V) and F() and other mathematical subtleties. Readers interested in the mathematical details are referred to Lieb’s original paper [225] or to the excellent survey by Eschrig [205].

Let us consider a family of (exact) Hamiltonians for the same number n of electrons. These Hamiltonians can only differ in the external potential V, we hence have a V-dependent family of Hamiltonians. Let us further assume that V is a continuous set of local (multiplicative) potentials, and that for the whole family of potentials considered, the ground state is non-degenerate (the latter restriction is actually not necessary). The energy E of the ground state is then a functional of V, i.e. E = E(V). It is also clear how one has to proceed to evaluate this functional: one constructs the Hamiltonians H(V), and solves the Schrödinger equations

$H\left(V\right)\psi \left(V\right)=E\left(V\right)\psi \left(V\right)$

for the respective ground states. E(V) is even defined if there is no bound ground state, but this possibility causes some problems. For a potential $\tilde{V}=V+\mathrm{\Delta V}\text{,}$ that only differs infinitesimally from a given potential V the change in E is given by 1^st order perturbation theory as

$\mathrm{\Delta E}=E\left(V+\mathrm{\Delta V}\right)-E\left(V\right)=<\psi \left(V\right)\left|\mathrm{\Delta V}\right|\psi \left(V\right)>={\displaystyle \int \mathrm{\Delta V}\mathrm{\varrho }\mathit{d\tau }}$

where is the electron density corresponding to ψ(V).

It can easily be shown that (for fixed particle number) the functional E(V) is concave [205, 225], i. e. that

$E\left(\alpha V_1+\left[1-\alpha \right]V_2\right)\ge \mathit{\alpha E}\left(V_1\right)+\left(1-\alpha \right)E\left(V_2\right);0\le \alpha \le 1$

The functional derivative of the ground state energy with respect to V is the density

$\frac{\mathrm{\Delta E}}{\mathrm{\Delta V}}=\mathrm{\varrho }$

This relation between and V, together with (40) implies an invertibly unique mapping between V and ρ and suggests a change of variables (where the variables are functions) as is familiar from the Legendre transformation in thermodynamics (change from U(V, T) to H(p,T) with $p=-\frac{\partial U}{\partial V}$ or classical mechanics (change from $L\left(q,\dot{q}\right)$ to H(q,p) with $p=\frac{\partial L}{\partial \dot{q}}$). The corresponding change is now

$E\left(V\right)\to F\left(\mathrm{\varrho }\right)=E\left(V\right)-\mathit{\int }V\mathrm{\varrho }\mathit{d\tau }$

The functional F() is the internal energy, i. e. the total ground state energy minus the interaction of the ground state density with the external potential. It is equal to the sum of the kinetic energy and the electron interaction energy. The internal energy F() is independent of V and a functional of only. Hohenberg and Kohn have introduced the functional F() without seeing that (for appropriately chosen domains) it is just the Legendre transformation of E(V) and have given F() the somewhat unfortunate name ’universal functional of the density’. Of course F() is a functional of in the same sense as E(V) is a functional of V. There is no indication that there should be a simple and direct way to evaluate this functional.

In spite of the universal nature of F(), a change of can (we always have the exact expressions in mind) only be caused by a change of V. Only those are admissible as arguments of F, which correspond to the considered V. Although F() does not explicitly depend on V, it depends parametrically on V, and one finds that

$\frac{\mathrm{\Delta F}}{\delta \mathrm{\varrho }}=\frac{\mathrm{\Delta E}}{\delta \mathrm{\varrho }}-V=-V$

since the variation of E with respect to variations of vanishes, provided that is chosen such that it corresponds to the chosen V. Relation (43) actually allows one to invert the Legendre transformation and to arrive at E(V) starting from F(). This somewhat delicate nature of F() must be kept in mind if one really wants to appreciate density functional theory. The role of F() is somewhat obscured if one regards E as a functional of . Since E depends both on V and and since we are searching for the which corresponds t o V the double dependence of E on g and V is somewhat confusing. It can be shown that (for an appropriately chosen domain) F() + ∫ Vdτ is convex and hence attains its minimum for the exact , provided, of course, that F() is the exact (unknown) functional. This is known as the Hohenberg-Kohn variational principle. For a mathematically rigorous fomulation of the Hohenberg-Kohn theorem and especially of the Legendre transformation see [205, 225].

The main problem with F() is that it is entirely unknown, that it is at least as complicated as E(V) which is unknown as well, and that there is no way how one should construct F(). One can formally decompose F() into three parts

(a) the kinetic energy T

(b) the Coulomb contribution to the electron interaction energy

$E_{\mathit{Cout}}=\frac{1}{2}\mathit{\int }\mathrm{\varrho }\left(1\right)\frac{1}{r_{12}}\mathrm{\varrho }\left(2\right)\mathit{d\tau }$

(c) the remaining contributions to the electron interaction energy, i. e. the exchange and correlation energies.

Of these contributions only (b) happens to be known exactly, while for the other two one can at best best hope to approximate them somehow.

Density functional methods have been used even before the formulation of the Hohenberg-Kohn-theorem. The classical of these methods is the Thomas-Fermi model [226, 227, 228, 229]. It is based on an analysis of the homogeneous electron gas. This is characterized by a single parameter, namely the density (i. e. the number of electrons per unit volume), which is a constant, i.e. which is independent of the position $\overrightarrow{r}\text{.}$ Other properties, like the density of the kinetic energy, must be expressible in terms of , the kinetic energy density is e. g. proportional to ^2/3 and the exchange density to ^1/3. These relations were applied to atomic theory, ignoring that the electron density in an atom is very far from homogeneous. Although various kinds of corrections were added to the simple Thomas-Fermi-model [228, 229], its success was very limited, it could e.g. not account for the shell structure of atoms and not for chemical binding [230]. A careful analysis has shown that the most serious defect of the Thomas-Fermi model lies in the approximation of the kinetic energy as a functional of the density. Even till now no acceptable expression of the kinetic energy as functional of the density has been proposed.

6.3 Slater-Kohn-Sham type methods

Modern density functional methods, that can be traced back to a paper by Kohn and Sham [231], avoid the evaluation of the kinetic energy as a functional of the density. One rather introduces an artificial non-interacting system – in a modified external potential – with the same density as the considered system and one evaluates the kinetic energy of this system as the kinetic energy of a Slater determinant. So the density functional methods in current use, are strictly speaking not genuine density functional methods.

An ingredient of Kohn-Sham type DF methods is that the modified external potential for the artificial non-interacting system should be local (multiplicative).

Since in DF methods the unknown is the density, only basis sets to describe the density well are required. So much smaller basis sets are sufficient than for the genuine many-body methods described in sec. 6.

If one evaluates the kinetic energy (a) in the indicated way (plus an unkown correction, since the two systems – i. e. the real and the artificial one – only have approximatively the same kinetic energy), the two remaining parts (b) and (c) cause less problems. In fact (b) is a genuine and universal density functional, and (c) is regarded as relatively small, such that it does not matter too much how one approximates this term.

A curious aspect is that the dominant part of the exchange-correlation contribution is the removal of the unphysical self-interaction of the electrons contained in (b). Attempts to leave this out from the very beginning, were not too successful and have become popular only recently.

All essential ingredients of the Kohn-Sham approach were already contained in the much older Xα method of Slater [210]. The main difference is that the Xα method was designed as an approximation to Hartree Fock theory, replacing the (non-local) exact Hartree-Fock exchange by a local approximation, inspired by Thomas-Fermi theory. It was then somewhat surprising that the Xα method performed often much better than the Hartree-Fock method that it was designed to approximate. The origin of this apparent superiority remained rather obscure for a long time. It has to do with the fact that in replacing the exact Hartree-Fock exchange by a local exchange one simulates to some extent the ’left-right-correlation’, that is ignored in the Hartree-Fock approximation. Now one tends to say that Xα should rather be regarded as a density functional theory of Kohn-Sham type. Therefore it appears fair to refer to this class of functionals as of Slater-Kohn-Sham type, and to regard Xα so to say as their ’zeroth generation’. That Xα did not turn out too successful, had less to do with its shortcomings, e.g. that it did not treat exchange and correlation separately, but rather with its somewhat unlucky marriage with the ’muffin-tin approximation’.

Following the paper by Kohn and Sham various types of local exchange-correlation functionals were used successfully in solid state physics and with moderate success in chemistry. A big step in advance, especially as far as chemical applications were concerned, came with the generalized gradient-corrected functionals of Becke [213] and Perdew [214].

Meanwhile one counts three generations of density functional methods, (in addition to Xα as zeroth order) namely

1. Local density (LD) functionals or local spin density (LSD) functionals

2. Generalized gradient-corrected (GGA) functionals

3. Optimized potential method (OPM).

While methods of the first generation were only of limited use for chemical applications, those of the second generation had a large impact on applied quantum chemistry. They were especially successful for transition metal compounds for which traditional quantum chemical methods has serious problems, due to the strong dependence of the correlation energy on the occupancy of the d-shell. Even systems with partially filled f-shells could be treated successfully by DFT [232].

Methods of the third generation have so far mainly been applied to atoms with rather good success, while for molecules they have not yet made the methods of the second generation obsolete. These involve to some extent orbital functionals [233, 234] rather than density functionals, and they are much closer in spirit to quantum chemical ab-initio methods than were the older DF methods. Like in older DF schemes one still insists on a local effective potential, but self-interaction is usually – unlike in the older schemes – eliminated, which makes the remaining exchange and correlation effects much smaller. In fact the bulk of the exchange contribution consists in eliminating the unphysical self interaction, that is characteristic for the older DF approaches.

In the OPM schemes one starts from a Hartree-Fock like exchange energy, but the energy is optimized under the restriction that the effective potential is local. So ’exchange only’ OPM is as close to Hartree-Fock theory as a scheme with a local potential can be.

If one tries to forecast the future of density functional theory, the following statements can be safely made:

1. Density functional calculations with black-box programs will become very popular, and this even for relatively large molecules, since one is now very close to linear scaling [235, 236] of the computational effort with the size of the molecule.

2. On the methodologic side there will be an increased pragmatic attitude, and functionals with adjustable parameters will continue to be proposed, so making the theory to a large extent semiempirical.

3. Simultaneously studies will be continued to understand the physical basis of DF theory better, in particular to understand which are the essential ingredients that make DFT work and whether it is possible to arrive at any desired accuracy. If one is very optimistic one will hope that a hierarchy of density functionals will be found, such that it is also possible to go to the next higher level, if the lower one was not good enough. At present one is very far from this situation. The history of DFT is rich in examples of improvements based on formal arguments which rather deteriorated the agreement with experiment.

4. There are indications that there may be some convergence between traditional attempts to solve the Schrödinger equation by ab-initio methods and the so far very different philosophy of DFT. The two approaches to manyelectron systems have so far almost been orthogonal and there was hardly a communication between them. For DF methods of the third generation this has drastically changed.

If one compares the attempts reviewed in sec. 3.2 to base many-electron quantum mechanics on the two-particle density matrix, i.e. a 2-particle density matrix functional theory with the current density functional theory one realizes that for the former the functional is exactly known, while the full n-representability condition is unknown. For DFT on the other hand, the functional is unknown, but the n representability does not cause problems. Why should one take incomplete information on n-representability as more serious as lack of information on the exact functional? Possibly there was just more reluctance in the two-particle-density matrix functional community to be satisfied with approximate n-representability conditions than in the density functional community to accept approximate density functionals, and that this different attitude was decisive for the historical development.