2003 INTERTEK EUROPEAN STUDY

The 2003 Intertek European study deals with the use of financial modeling at European asset management firms.² It is based on studies conducted by The Intertek Group to evaluate model performance following the fall of the markets from their peak in March 2000, and explores changes that have occurred since then. In total, 61 managers at European asset management firms in the Benelux countries, France, Germany, Italy, Scandinavia, Switzerland, and the United Kingdom were interviewed. (The study does not cover alternative investment firms such as hedge funds.) At least half of the firms interviewed are among the major players in their respective markets, with assets under management ranging from €50 to €300 billion.

The major findings are summarized next.

Greater Role for Models

In the two years following the March 2000 market highs, quantitative methods in the investment decision-making process began to play a greater role. Almost 75% of the firms interviewed reported this to be the case, while roughly 15% reported that the role of models had remained stable. The remaining 10% noted that their processes were already essentially quantitative. The role of models had also grown in another sense; a higher percentage of assets were being managed by funds run quantitatively. One firm reported that over the past two years assets in funds managed quantitatively grew by 50%.

Large European firms had been steadily catching up with their U.S. counterparts in terms of the breadth and depth of use of models. As the price of computers and computer software dropped, even small firms reported that they were beginning to adopt quantitative models. There were still differences between American and European firms, though. American firms tended to use relatively simple technology but on a large scale; Europeans tended to adopt sophisticated statistical methods but on a smaller scale.

Demand pull and management push were among the reasons cited for the growing role of models. On the demand side, asset managers were under pressure to produce returns while controlling risk; they were beginning to explore the potential of quantitative methods. On the push side, several sources remarked that, after tracking performance for several years, their management has made a positive evaluation of a model-driven approach against a judgment-driven decision-making process. In some cases, this led to a corporate switch to a quantitative decision-making process; in other instances, it led to shifting more assets into quantitatively managed funds.

Modeling was reported to have been extended over an ever greater universe of assets under management. Besides bringing greater structure and discipline to the process, participants in the study remarked that models helped contain costs. Unable to increase revenues in the period immediately following the March 2000 market decline, many firms were cutting costs. Modeling budgets, however, were reported as being largely spared. About 68% of the participants said that their investment in modeling had grown over the prior two years, while 50% expected their investments in modeling to continue to grow over the next year.

Client demand for risk control was another factor that drove the increased use of modeling. Pressure from institutional investors and consultants in particular continued to work in favor of modeling.

More generally, risk management was widely believed to be the key driving force behind the use of models.

Some firms mentioned they had recast the role of models in portfolio management. Rather than using models to screen and rank assets—which has been a typical application in Europe—they applied them after the asset manager had acted in order to measure the pertinence of fundamental analysis, characterize the portfolio style, eventually transform products through derivatives, optimize the portfolio, and track risk and performance.

Performance of Models Improves

Over one-half of the study's participants responded that models performed better in 2002 than two years before. Some 20% evaluated 2002 model performance as stable with respect to two years ago, while another 20% considered that performance had worsened. Participants often noted that it was not models in general but specific models that had performed better or more poorly.

There are several explanations for the improved performance of models. Every model is, ultimately, a statistical device trained and estimated on past data. When markets began to fall from their peak in March 2000, models had not been trained on data that would have allowed them to capture the downturn—hence, the temporary poor performance of some models. Even risk estimates, more stable than expected return estimates, were problematic. In many cases, it was difficult to distinguish between volatility and model risk. Models have since been trained on new sets of data and are reportedly performing better.

From a strictly scientific and economic theory point of view, the question of model performance overall is not easy to address. The basic question is how well a theory describes reality, with the additional complication that in economics uncertainty is part of the theory. We cannot object to financial modeling but we cannot pretend a priori that model performance be good. Modeling should reflect the objective amount of uncertainty present in a financial process. The statement that “models perform better” implies that the level of uncertainty has changed. To make this discussion meaningful, clearly somehow we have to restrict the universe of models under consideration. In general, the uncertainty associated with forecasting within a given class of models is equated to market volatility. And as market volatility is not an observable quantity but a hidden one, it is model-dependent.³ In other words, the amount of uncertainty in financial markets depends on the accuracy of models. For instance, an ARCH-GARCH model will give an estimate of volatility different from that of a model based on constant volatility. On top of volatility, however, there is another source of uncertainty, which is the risk that the model is misspecified. The latter uncertainty is generally referred to as model risk.

The problem experienced when markets began to fall was that models could not forecast volatility simply because they were grossly misspecified. A common belief is that markets are now highly volatile, which is another way of saying that models do not do a good job of predicting returns. Yet models are now more coherent; fluctuations of returns are synchronized with expectations regarding volatility. Model risk has been reduced substantially.

Overall, the global perception of European market participants who participated in the study was that models are now more dependable. This meant that model risk had been reduced; although their ability to predict returns had not substantially improved, models were better at predicting risk. Practitioners' evaluation of model performance can be summarized as follows: (1) models will bring more and more insight to risk management; (2) in stock selection, we will see some improvement due essentially to better data, not better models; and (3) in asset allocation, the use of models will remain difficult as markets remain difficult to predict.

Despite the improved performance of models, the perception European market participants shared was one of uncertainty as regards the macroeconomic trends of the markets. Volatility, structural change, and unforecastable events continue to challenge models. In addition to facing uncertainty related to a stream of unpleasant surprises as regards corporate accounting at large public firms, participants voiced the concern that there is considerable fundamental uncertainty on the direction of financial flows.

A widely shared evaluation was that, independent of models themselves, the understanding of models and their limits had improved. Most traders and portfolio managers had at least some training in statistics and finance theory; computer literacy was greatly increased. As a consequence, the majority of market participants understand at least elementary statistical analyses of markets.

Use of Multiple Models on the Rise

According to the 2003 study's findings, three major trends had emerged in Europe over the prior few years: (1) a greater use of multiple models, (2) the modeling of additional new factors, and (3) an increased use of value-based models.

Let's first comment on the use of multiple models from the point of view of modern financial econometrics and, in particular, from the point of view of the mitigation of model risk. The present landscape of financial modeling applied to investment management is vast and well articulated.⁴ Financial models are typically econometric models, they do not follow laws of nature but are approximate models with limited validity. Every model has an associated model risk, which can be roughly defined as the probability that the model does not forecast correctly. Note that it does not make sense to consider model risk in abstract terms and against every possible assumption; model risk can be meaningfully defined only by restricting the set of alternative assumptions. For instance, we might compute measures of the errors made by an option pricing model if the underlying follows a distribution different from the one on which the model is based. Clearly, what must be specified are the families of alternative distributions.

Essentially every model is based on some assumption about the functional form of dependencies between variables and on the distribution of noise. Given the assumptions, models are estimated, and decisions made. The idea of estimating model risk is to estimate the distribution of errors that will be made if the model assumptions are violated. For instance, are there correlations or autocorrelations when it is assumed there are none? Are innovations fat-tailed when it is assumed that noise is white and normal? From an econometric point of view, combining different models in this way means constructing a mixture of distributions. The result of this process is one single model that weights the individual models.

Some managers interviewed for the 2003 study reported they were using judgment on top of statistical analysis. This entails that models be reviewed when they begin to produce results that are below expectations. In practice, quantitative teams constantly evaluate the performance of different families of models and adopt those that perform better. Criteria for switching from one family of models to another are called for, though. This, in turn, requires large data samples.

Despite these difficulties, application of multiple models has gained wide acceptance in finance. In asset management, the main driver is the uncertainty related to estimating returns.

Focus on Factors, Correlation, Sentiment, and Momentum

Participants in the 2003 study also reported efforts to determine new factors that might help predict expected returns. Momentum and sentiment were the two most cited phenomena modeled in equities. Market sentiment, in particular, was receiving more attention.

The use of factor models is in itself a well-established practice in financial modeling. Many different families of models are available, from the widely used classic static return factor analysis models to dynamic factor models. What remains a challenge is determination of the factors. Considerable resources have been devoted to studying market correlations. Advanced techniques for the robust estimation of correlations are being applied at large firms as well as at boutiques.

According to study respondents, over the three years prior to 2001, quantitative teams at many asset management firms were working on determining which factors are the best indicators of price movements. Sentiment was often cited as a major innovation in terms of modeling strategies. Asset management firms typically modeled stock-specific sentiment, while sentiment as measured by business or consumer confidence was often the responsibility of the macroeconomic teams at the mother bank, at least in continental Europe. Market sentiment is generally defined by the distribution of analyst revisions in earnings estimates. Other indicators of market confidence are flows, volume, turnover, and trading by corporate officers.

Factors that represent market momentum were also increasingly adopted according to the study. Momentum means that the entire market is moving in one direction with relatively little uncertainty. There are different ways to represent momentum phenomena. One might identify a specific factor that defines momentum, that is, a variable that gauges the state of the market in terms of momentum. This momentum variable then changes the form of models. There are models for trending markets and models for uncertain markets.

Momentum can also be represented as a specific feature of models. A random walk model does not have any momentum, but an autoregressive model might have an intrinsic momentum feature.

Some participants also reported using market-timing models and style rotation for the active management of funds. Producing accurate timing signals is complex, given that financial markets are difficult to predict. One source of predictability is the presence of mean reversion and cointegration phenomena.

Back to Value-Based Models

At the time of the 2003 study, there was a widespread perception that value-based models were performing better in post-2000 markets. It was believed that markets were doing a better job valuing companies as a function of the value of the firm rather than price trends, notwithstanding our remarks on the growing use of factors such as market sentiment. From a methodological point of view, methodologies based on cash analysis had increased in popularity in Europe. A robust positive operating cash flow is considered to be a better indication of the health of a firm than earnings estimates, which can be more easily massaged.

Fundamental analysis was becoming highly quantitative and automated. Several firms mentioned that they were developing proprietary methodologies for the automatic analysis of balance sheets. For these firms, with the information available on the Internet, fundamental analysis could be performed without actually going to visit firms. Some participants remarked that caution might be called for in attributing the good performance of value-tilted models to markets. One of the assumptions of value-based models is that there is no mechanism that conveys a large flow of funds through preferred channels, but this was the case in the telecommunications, media, and technology (TMT) bubble, when value-based models performed so poorly. In the last bull run prior to the study, the major preoccupation was to not miss out on rising markets; investors who continued to focus on value suffered poor performance. European market participants reported that they are now watching both trend and value.

Risk Management

Much of the attention paid to quantitative methods in asset management prior to the study had been focused on risk management. According to 83% of the participants, the role of risk management had evolved significantly over the prior two years to extend across portfolios and across processes.

One topic that has received a lot of attention, both in academia and at financial institutions, is the application of extreme value theory (EVT) to financial risk management. The RiskLab in Zurich, headed by Paul Embrechts, advanced the use of EVT and copula functions in risk management. At the corporate level, universal banks such as HSBC CCF have produced theoretical and empirical work on the applicability of EVT to risk management.⁵ European firms were also paying considerable attention to risk measures.

For participants in the Intertek study, risk management was the area where quantitative methods had made their biggest contribution. Since the pioneering work of Harry Markowitz in the 1950s, the objective of investment management has been defined as determining the optimal risk-return trade-off in an investor's profile. Prior to the diffusion of modeling techniques, though, evaluation of the risk-return trade-off was left to the judgment of individual asset managers. Modeling brought to the forefront the question of ex ante risk-return optimization. An asset management firm that uses quantitative methods and optimization techniques manages risk at the source. In this case, the only risk that needs to be monitored and managed is model risk.⁶

Purely quantitative managers with a fully automated management process were still rare according to the study. Most managers, although quantitatively oriented, used a hybrid approach calling for models to give evaluations that managers translate into decisions. In such situations, risk is not completely controlled at the origin.

Most firms interviewed for the study had created a separate risk management unit as a supervisory entity that controls the risk of different portfolios and eventually—although still only rarely—aggregated risk at the firm-wide level. In most cases, the tools of choice for controlling risk were multifactor models. Models of this type have become standard when it comes to making risk evaluations for institutional investors. For internal use, however, many firms reported that they made risk evaluations based on proprietary models, EVT, and scenario analysis.

Integrating Qualitative and Quantitative Information

More than 60% of the firms interviewed for the 2003 Intertek study reported they had formalized procedures for integrating quantitative and qualitative input, although half of these mentioned that the process had not gone very far; 30% of the participants reported no formalization at all. Some firms mentioned they had developed a theoretical framework to integrate results from quantitative models and fundamental views. Assigning weights to the various inputs was handled differently from firm to firm; some firms reported establishing a weight limit in the range of 50% to 80% for quantitative input.

A few quantitative-oriented firms reported that they completely formalized the integration of qualitative and quantitative information. In these cases, everything relevant was built into the system. Firms that both quantitatively managed and traditionally managed funds typically reported that formalization was implemented in the former but not in the latter.

Virtually all firms reported at least a partial automation in the handling of qualitative information. For the most part, a first level of automation—including automatic screening and delivery, classification, and search—is provided by suppliers of sell-side research, consensus data, and news. These suppliers are automating the delivery of news, research reports, and other information.

About 30% of the respondents note they have added functionality over and above that provided by third-party information suppliers, typically starting with areas easy to quantify such as earnings announcements or analysts' recommendations. Some have coupled this with quantitative signals that alert recipients to changes or programs that automatically perform an initial analysis.

Only the braver will be tackling difficult tasks such as automated news summary and analysis. For the most part, news analysis was still considered the domain of judgment. A few firms interviewed for this study reported that they attempted to tackle the problem of automatic news analysis, but abandoned their efforts. The difficulty of forecasting price movements related to new information was cited as a motivation.