Relative Importance for Linear Regression in R : The Package relaimpo

Relative importance is a topic that has seen a lot of interest in recent years, particularly in applied work. The R package relaimpo implements six diﬀerent metrics for assessing relative importance of regressors in the linear model, two of which are recommended - averaging over orderings of regressors and a newly proposed metric (Feldman 2005) called pmvd . Apart from delivering the metrics themselves, relaimpo also provides (exploratory) bootstrap conﬁdence intervals. This paper oﬀers a brief tutorial introduction to the package. The methods and relaimpo ’s functionality are illustrated using the data set swiss that is generally available in R . The paper targets readers who have a basic understanding of multiple linear regression. For the background of more advanced aspects, references are provided.


Introduction
"Relative importance" refers to the quantification of an individual regressor's contribution to a multiple regression model. Assessment of relative importance in linear models is simple, as long as all regressors are uncorrelated: Each regressor's contribution is just the R 2 from univariate regression, and all univariate R 2 -values add up to the full model R 2 . In sciences with predominance of observational data, regressors are typically correlated, so that it is no longer straightforward to break down model R 2 into shares from the individual regressors. Various methods have been proposed in the literature. Darlington (1968) gives an overview of the older methods, Lindeman, Merenda, and Gold (1980, p. 119 ff.) propose averaging sequential sums of squares over all orderings of regressors, Pratt (1987) yields a justification for an earlier proposal by Hoffman (1960) that had already been rejected by Darlington (1968) and others, and Feldman (2005)  Examination has a relatively high positive correlation with Education, and both these variables have a relatively high negative correlation with Agriculture, Examination is also negatively correlated with Catholic. This structure leads to the strong dependence of allocation of relative importance on the way of looking at the matter.

The linear model and the relative importance metrics
A linear model with an intercept can be written as The response of object i is modelled as a linear function of regressor values x i1 , . . . , x ip , with unknown coefficients β 1 , . . . , β p , and e i represents the unexplained part. For the example, p = 5 regressors are available for explaining the response Fertility.
In linear regression, the coefficients β k , k = 0, . . . , p, are estimated by minimizing the sum of squared unexplained parts. If we denote the estimated coefficients asβ k and the fitted response values asŷ i =β 0 + x i1β1 + · · · + x ipβp , the coefficient of determination R 2 can be written as (2) R 2 measures the proportion of variation in y that is explained by the p regressors in the model.
The following output shows the linear regression results for the example data: > linmod <-lm(Fertility~., data = swiss) > summary(linmod) We see that R 2 is 70.67% and that all regressors except Examination are significant in this model, with Fertility increasing for higher Infant.Mortality and higher proportion of Catholics and Fertility decreasing for higher values for Agriculture, Education and Examination. This is somewhat in line with expectations, though the direction of the effect of Agriculture might come as a surprise.
Note that the stored linear model object linmod will be used in many subsequent calculations.

Simple relative importance metrics
The metric first One way of looking at relative importance is to compare, what each regressor alone is able to explain, i.e., to compare the R 2 -values from p regression models with one regressor only. These univariate R 2 -values are identical to the squared correlations of the regressors with the response. They are available in relaimpo under the name first. If regressors are correlated, the sum of these individual contributions is often far higher than the overall R 2 of the model with all regressors together, i.e., the overall model explains less than the sum of all individual models. This is also the case for the example data:  Johnson and Lebreton (2004), in that it only uses the direct effect. Also, it is not suitable for decomposing R 2 into contributions from individual regressors, since the contributions do not naturally add up to R 2 .

The metric last
Another way of looking at relative importance is to compare, what each regressor is able to explain in addition to all other regressors that are available. Here, we ascribe to each regressor the increase in R 2 when including this regressor as the last of the p regressors. This approach is implemented in relaimpo under the name last. If regressors are correlated, these contributions again do not add up to the overall R 2 , but typically add up to far less than the overall R 2 . A direct calculation from a linear model in R can be obtained using the function drop1. ( For the metric last, like in significance testing, Examination comes out particularly low. This is not surprising, since the relation between regressors in terms of last is identical to the relation between regressors in terms of t-test statistics (cf. e.g., Bring 1996). The metric last has also been called "usefulness" by Darlington (1968), who also notes that basing relative importance considerations on this metric is close to using siginificance testing for assessing importance. Again, the contributions do not naturally decompose R 2 . Also, the metric does again not comply with the definition of relative importance by Johnson and Lebreton (2004), in that it does not at all use the direct effect.

The metric betasq
A further approach to assessing individual contributions consists in considering standardized coefficients. The regressors in data set swiss are all percentages scaled from 0 to 100 (though most of them do not use the full scale). Imagine that one regressor is rescaled from the 0-100 scale to a 0-1 scale. Obviously, if nothing else changes, the respective coefficient has to be multiplied by the factor 100 in order to adjust for the change. Likewise, if the response Fertility were rescaled from the scale 0 to 100 to the scale 0 to 1, all β's would have to be divided by the factor 100 in order to adjust for this rescaling. Standardized coefficients are introduced as scale-invariant versions of the coefficients, by adjusting with estimated standard deviations:β where s kk and s yy denote the empirical variances of regressor x k and the response y respectively. (As long as one only compares regressors within models for the same response y, division by (s yy ) is irrelevant.) The squared standardized coefficient has been proposed as a metric for relative importance. It is available in relaimpo as betasq.
> sx <-as.numeric(lapply(swiss[, 2:6], "sd")) > (linmod$coefficients[2:6] * sx/sd(swiss$Fertility))^2  Bring (1996) cites an ongoing debate about whether or not this metric is appropriate for relative importance considerations. Johnson and Lebreton (2004) criticize standardized coefficients, because they take too little acount of the direct effect. Whatever stand one takes in the debate, if interest is in decomposing R 2 , betasq is not an appropriate metric, since it does not provide a natural decomposition of R 2 .
The metric pratt Hoffman (1960) proposed to multiply the standardized coefficient with the marginal correlation. Since the sum of these products over all regressors is just the overall R 2 , he considered this metric a natural decomposition of R 2 . This proposal has been criticized and even ridiculed, but has found a powerful defender in Pratt (1987) and is therefore included in relaimpo under the name pratt. Although Pratt (1987) found some interesting properties of the metric (cf. also Johnson and Lebreton 2004), several researchers have brought forward quite counterintuitive examples (e.g., Bring 1996), and the main criticism in the context of decomposing R 2 is that the metric can allocate negative contributions (as for Agriculture in the example data). Even Pratt says that the metric is not usable in this case, so that this metric is only applicable in some but not all situations. Because of this property, the present author does not recommend this metric.
Thus, the chapter of simple metrics has not offered anything that yields a useful (=at least nonnegative) natural decomposition of R 2 , although some of the metrics -especially combinations of several metrics -can provide insights regarding the contributions of regressors.

Computer-intensive relative importance metrics
The following metrics, lmg and pmvd, require far more computational effort. Both these metrics decompose R 2 into non-negative contributions that automatically sum to the total R 2 . This is an advantage they have over all simple metrics.
The difficulty in decomposing R 2 for regression models with correlated regressors lies in the fact that each order of regressors yields a different decomposition of the model sum of squares, which is illustrated below: > anova(linmod) In R, the command anova provides sequential sums of squares. "Sequential" means that the regressors are entered into the model in the order they are listed. The sequential sums of squares of all regressors do sum to the model sum of squares (in this example 5072.91 or 70.67% of the total response sum of squares of n i=1 (y i −ȳ) 2 =7177.96). Division of sequential sums of squares by the total response sum of squares yields sequential R 2 contributions. The example above shows that the order of regressors can have a very strong impact on the relative importance assessment: for example, Agriculture and Infant.Mortality receive about three times the share when entered first in comparison to when entered last into the model. The discrepancy between the two orders is particularly striking for Examination which receives the largest share for one of the orders and the smallest share for the other one. Obviously, unless there is a meaningful prior knowledge that prompts a natural order among variables, an approach based on the sequential R 2 of one fixed order of regressors is not appropriate for judging relative importance. Sometimes, applied researchers apply stepwise regression and decompose R 2 based on the order obtained by this automatic approach. Even though this approach relieves the researcher from the burden to decide on an arbitrary ordering, the automatic decision is not so much less arbitrary and is therefore not recommended (cf. also Bring 1996, and references therein).

Analysis of Variance
The approach taken by the metrics lmg and pmvd is based on sequential R 2 s, but takes care of the dependence on orderings by averaging over orderings, either using simple unweighted averages (lmg) or weighted averages with data-dependent weights (pmvd). First, the approaches are formally described giving the formulae for their calculation, then the functioning of both approaches is illustrated using the swiss data.
For describing the metrics, the following notations are useful: The R 2 for a model with regressors in set S is given as The additional R 2 when adding the regressors in set M to a model with the regressors in set S is given as The order of the regressors in any model is a permutation of the available regressors x 1 , . . . , x p and is denoted by the tuple of indices r = (r 1 , . . . , r p ). Let S k (r) denote the set of regressors entered into the model before regressor x k in the order r. Then the portion of R 2 allocated to regressor x k in the order r can be written as The metric lmg With (6), the metric lmg (in formulae denoted as LMG) can be written as Orders with the same S k (r) = S can be summarized into one summand, which simplifies the formula into Christensen (1992) has shown that this metric can also be written in a different way that some people consider to be more intuitive: This formula shows lmg as the average over average contributions in models of different sizes.
From a heuristic and ad-hoc perspective, unweighted averaging is a natural thing to do, so that lmg has been adopted or reinvented by many researchers. The very popular paper by Chevan and Sutherland (1991) has coined the expression "hierarchical partitioning" and has extended the idea to other regression models, and the R package hier.part implements this proposal (cf. also Section 6). The metric lmg has nevertheless not made it into the mainstream of applied statistical analysis (with possible exceptions in some special fields of application), which may be due to both computational burden and lack of understanding what is really calculated. Johnson and Lebreton (2004) recommend lmg, since it clearly uses both direct effects (orders with x k first) and effects adjusted for other regressors in the model (orders with x k last). Given the current choices of metrics for relative importance, the present author also recommends lmg, particularly if interest is in decomposing R 2 .
The metric pmvd 1 It is known and not surprising (cf. e.g., Feldman 2005) that the lmg contribution of a regressor with coefficient 0 can be positive, if this regressor is correlated with one or more strong contributors. Feldman considers this as a disadvantage of the lmg allocations. He introduces the metric PMVD (denoted throughout the text of this paper with its relaimpo name pmvd, in formulae as PMVD) in order to cure this disadvantage: Per construction, pmvd guarantees that a regressor with 0 estimated coefficient is assigned a relative importance of 0, and -more important -that the allocated contribution asymptotically approaches 0 if the true coefficient is 0. As mentioned before, pmvd can be seen as an average over orderings as well, but with data-dependent weights for each order: where p(r) denotes the data-dependent pmvd weights that will be discussed in the following. Note that formula (8) is not used for computation; there is a more efficient computation approach that exploits a game-theoretic background of pmvd (cf. Feldman 2005). Regarding the weights in formula (8), Feldman's (2005) idea was to use the sequences seqR 2 ({x r k+1 , . . . , x rp }|{x r 1 , . . . , x r k }), k = 1, . . . , p − 1 for determining the weights of all orders r. The weights have been derived from a set of axioms the most important of which is the axiom of proper exclusion that requires that a regressor with actual coefficient 0 should also be allocated an R 2 share of 0. The weights obtained are proportional to i.e., the weights are .
A careful look at the L(r) reveals that the factors increase with increasing i. L(r) will be particularly large, if the first regressor already captures a very substantial amount of the variance, or if the last regressor captures only a very small amount of the variance. (If the last regressor(s) do(es) not capture any variance (estimated coefficient(s) 0), L(r) is infinite, but the situation is nevertheless well-behaved: Feldman shows via limiting considerations that the weights are only positive for orders with all 0 coefficient regressors last, which leads to a share of 0 for these regressors, as desired.) Note that pmvd's property of assigning a zero share to regressors with coefficient 0 comes at the price of a large variability in allocated contributions for many applications (exemplified e.g. in Figure 2). This is due to basing the weights on the potentially very variable order-dependent empirical sequential R 2 allocations. pmvd weights can be seen as a compromise between the equal weights of lmg and the (inappropriate) approach of basing relative importance assessments on one order determined by an automatic variable selection procedure (datadependent weight 1 on one order). Given the currently available choices, pmvd is one of the two metrics that come closest to Johnson and Lebreton's (2004) reasonable definition of relative importance so that its use can be recommended, perhaps in comparison with lmg (but see also the subsection on the exclusion property on page 11). Table 1 shows a comparison of pmvd and lmg for two different four-regressor models for the swiss data: One model omits the regressor Agriculture, the other the regressor Examination.

Comparison of lmg and pmvd on two examples
Hence, there are 24 orders of the four regressors in each of the models, so that it is feasible to show all the allocated sequential R 2 s and the pmvd weights. In the example with Agriculture omitted, the pmvd weights are very concentrated on a few orders: the orders with Examination last are the only ones that receive a weight of more than 1% (bold-face weights). This is the reason that the pmvd share for Examination is so much lower than the lmg share. In the example with Examination omitted the weights are far more balanced, orders with Agriculture or Infant.Mortality last are the ones that tend to receive the higher weights, orders with low allocations for Education receive a low weight.
Is pmvd's property "exclusion" desirable? Feldman (2005) specifically constructs pmvd to allocate a share of zero to any regressor with coefficient zero (called "exclusion"). However, lmg's property of allowing correlated regressors to benefit from each other's shares does have a plausible background, at least for some purposes. Consider the linear model based on the causal chain (assuming linear relations) In this linear model, β 1 is zero. For prediction purposes, x 1 does not contribute anything, once x 2 is known. Thus, the request for a zero allocated share for x 1 seems reasonable. However, if one is interested in relative importance from a causal perspective, x 1 's contribution should not be zero in spite of its zero coefficient. The linear model equation (9)  increasing correlation among regressors, so that lmg's property of allowing correlated regressors to benefit from each other's shares can be seen as a way to take care about uncertainty of information regarding the true underlying structure. Hence, particularly when the focus of the research is more on causal than on predictive importance, lmg's behaviour appears to be more appropriate than pmvd's.

Calculating relative importances based on the covariance matrix
In the previous section, computer-intensive methods have been discussed. If computing time is an issue, it can be helpful to reconsider calculation methods developed when computers were not widely available. Old textbooks on linear regression provide methods to calculate most regression results from the (p + 1) × (p + 1) covariance matrix S of the response y and the p regressors that are in this section denoted by a 1 × p-row vector x. Following e.g., Lindeman, Merenda, and Gold (1980, p. 103 ff), the relevant formulae are outlined below.
This sub section is meant for readers who want to dive into the calculations done in relaimpo.

Defining variances and covariances as
wherex .j denotes the arithmetic mean of the j-th regressor, the corresponding variancecovariance matrices can be defined as S xx = (s jk ) j=1,...,p k=1,...,p , S xy = (s ky ) k=1,...,p , S yx = S T xy , and can be combined into the overall variance-covariance matrix as The estimated coefficients for regressors x 1 , . . . , x p in model (1) can be calculated aŝ β 1,...,p = S −1 xx S xy (the coefficient β 0 is irrelevant for relative importance), and the unexplained variance of the response is obtained as s yy − S yx S −1 xx S xy .
The coefficient of determination R 2 from regressing y on x can thus be obtained as If the regressors in x are partitioned into two groups x 1 and x 2 , i.e., the variance covariance matrix is partitioned as the unexplained variance-covariance matrix of y and x 2 after adjusting for x 1 can be calculated as Note that (14) is the inverse of the appropriately-sized top left block of S −1 with S partitioned as shown in (13). The top left element of (14) is the unexplained variance of the response y after regressing y on x 1 . Hence, the difference between the top left element of (14) and the unexplained variance (11) from the full model divided by s yy is the sequential R 2 when adding the regressors in x 2 to the model in addition to the regressors in x 1 (seqR 2 (x 2 |x 1 ) when identifying x 1 and x 2 with the sets of regressors contained in them, cf. (5)). Note that all relevant calculations for sequential linear models can be based on the variance-covariance matrix and its conditional forms according to the formulae given in this section, i.e., after calculation of the variance-covariance matrix, no further calculations with all observations are needed. Thus, calc.relimp can also be applied to a variance-covariance matrix directly, as documented with package relaimpo.

Bootstrapping regression models
So far, point estimates for R 2 decompositions have been obtained. In order to assess, which regressors are clearly different and which are similar in terms of relative importance, variability of the estimates needs to be estimated. In particular the computer-intensive metrics lmg and pmvd are not easily tackled by proper distributional results (even if there is a distributional result on lmg for multivariate normal data, cf. e.g., Budescu (1993)). Therefore, relaimpo provides a bootstrapping facility for assessing the variability of all metrics.
When bootstrapping regression models, there are two principally different reasonable approaches (cf. e.g., Davison and Hinkley 1997;Fox 2002): The regressors can be considered fixed in some situations, e.g., for experimental data. In our data example, where the fixed set of all French-speaking swiss provinces is investigated, it can also be justified to work with fixed regressors. If regressors are considered fixed, only the error terms are random. Contrary, in most relative importance investigations, it makes far more sense to consider also the regressors as random, since the observations are a random sample from a larger population. These two scenarii prompt two different approaches for bootstrapping.
Bootstrapping in relaimpo is done using the functions boot and boot.ci from R package boot. R package boot is based on the book by Davison and Hinkley (1997). Percentile intervals, BCa intervals, normal intervals and basic intervals are supported (default: BCa intervals, except for ranks for which intervals are always percentile intervals).

Features of the package relaimpo
5.1. Forced to percentages: rela=TRUE All metrics come in two different versions: so far, we have discussed the natural scale of each metric. relaimpo allows to force metrics to sum to 100% (instead of R 2 or no meaningful sum). This is achieved by the option rela=TRUE. The default is rela=FALSE.

Adjustment: Some regressors always stay in the model
In many applications, some influences have to be included into the model and are not meant to compete with the other influences, e.g., gender in epidemiological or marketing applications. If one wants to include these influences first and only decompose the remaining R 2 among the remaining regressors, this can be achieved by the option always. In the swiss example, there is no natural variable one would want to treat like this. Purely for demonstration purposes, Catholic is now forced to be always first: > calc.relimp(linmod, type = c("lmg", "pmvd"), always = "Catholic")

Treatment of missing values
If functions calc.relimp or boot.relimp are applied to a formula object, the option na.action is available and will work like usual in R. If nothing is specified in the function call, the overall setting of na.action (that can be viewed by options("na.action")) will be applied. For objects other than formula objects, relaimpo bases all analyses on complete cases only. Whenever observations are deleted because of missing values, a warning is printed.

Bootstrapping
A call to the function boot.relimp 2 requests bootstrap runs the results of which are stored in an object of class relimplmboot. Afterwards, the result object can be (repeatedly) evaluated with the function booteval.relimp. booteval.relimp works on the output from boot.relimp and allows among other things selection of a subset of the metrics, selection of one or several confidence levels and suppression of confidence intervals for differences or ranks.
> bootresult <-boot.relimp(linmod, b = 1000, type = c("lmg", "pmvd", "last", + "first"), fixed = FALSE) Bootstrapping makes computation times a real issue, if many bootstrap runs are required (cf. also Section 7). The recommended BCa bootstrap intervals require a substantial number of bootstrap runs (default b=1000 should not be decreased, cf. e.g., Hesterberg, Moore, Monaghan, Clipson, and Epstein 2005) and are themselves slow to calculate. It may be an alternative to work with percentile confidence intervals (always used for ranks) or normal distribution based confidence intervals in order to get at least an indication of variability.
Coverage probabilities for percentile confidence intervals with b=1000 and normal confidence intervals with b=200 have been investigated in some simulations and have proven to be somewhat liberal (non-coverage up to twice nominal level). For this reason, the output contains a warning that bootstrap confidence intervals can be somewhat liberal. Performance of BCa intervals has not been simulated (since they take so much longer); they might well perform better, if calculated from a large enough number of bootstrap resamples.

Bootstrap confidence intervals for ranks and their visualization
In many applications, the main interest is a ranking of regressors. Rankings of regressors can obviously be obtained by comparing the allocated shares. In order to assess stability of the ranking, bootstrap confidence intervals for ranks are provided (always percentile intervals). Rank bootstrap intervals are visualized by a lettering system. > booteval.relimp(bootresult, typesel = c("lmg", "pmvd"), level = 0.9, + bty = "perc", nodiff = TRUE) Lettering shows that Education cannot be in the last position, while Agriculture can be in the last three positions only for both metrics. Furthermore, Examination cannot be in the last position for lmg (no restriction for pmvd), while Infant.Mortality cannot be in the first position for pmvd (no restriction for lmg). Note that lmg confidence intervals are contained in pmvd confidence intervals for all variables but Agriculture in this example, which exemplifies the fact that pmvd is a more variable metric than lmg under many circumstances, as was already mentioned earlier.

Bootstrap confidence intervals for differences
If there is a detailed interest in the comparison of relative importances for pairs of regressors, confidence intervals for differences may be of interest.
> eval <-booteval.relimp(bootresult, typesel = c("lmg", "pmvd"), level = 0.9, + bty = "perc", norank = TRUE) > eval The confidence intervals for differences show which differences in contributions can be considered statistically significant (in an exploratory sense). Stars mark those contributions for which the confidence interval does not contain the equality case. Note that the procedure generates a substantial amount of output, if confidence intervals for differences are not suppressed, since confidence intervals for the metrics themselves are also always shown. (The norank=TRUE-option suppresses the lettering for rank confidence intervals.)

Plotting results
Barplots of results can be obtained using the plot-methods on the output objects of functions calc.relimp and booteval.relimp. Figure 1 shows barplots of all metrics for the swiss example (metrics calculated on page 4) and can be created by the R statements > par(cex.axis = 0.8) > plot(metrics, names.abbrev = 3) Since bootstrapping has taken place, barplots with variability lines can also be created (Figure 2).
> par(cex.axis = 0.9) > plot(booteval.relimp(bootresult, typesel = c("lmg", "pmvd"), level = 0.9), + names.abbrev = 2, bty = "perc") 6. Comparison to related R packages R has two further packages that are devoted to relative importance. Package relimp by Firth (2006) implements a method of comparing the contributions of two groups of regressors that has been proposed by Silber, Rosenbaum, and Ross (1995). This method does something quite special that is not very closely related to the scope of relaimpo. Hence, it is not further considered here. Package hier.part by Walsh and Mac Nally (2005) is more closely related and is therefore discussed in this section in comparison to relaimpo. (For some of the code in this section, hier.part needs to be installed.)  Note that hier.part is more general than relaimpo in that it covers more general regression models and more goodness-of-fit statistics, while relaimpo is restricted to the linear model with goodness-of-fit statistic R 2 . On the other hand, relaimpo is more general for the linear model in that it covers more metrics for relative importance. Furthermore, relaimpo provides appropriate bootstrap confidence intervals and makes use of the specifics of linear models for being faster in computing and thus allowing more regressors.

Method LMG
For comparing hier.part to relaimpo, the goodness-of-fit choice for the former has to be chosen as R 2 (gof=Rsqu). Then the standard output of hier.part for the example data (barplot turned off, since almost identical to that in relaimpo) is shown below: The first bit of output (gfs) lists the R 2 values for all sub models. Then, IJ shows the individual and joint contributions of each regressor, and I.perc shows a percentage rescaling of the individual contributions. In fact, I.perc from hier.part coincides with relaimpo's lmg for rela=TRUE, I from hier.part coincides with relaimpo's lmg for rela=FALSE, and J from hier.part is the difference between first and lmg for rela=FALSE. The following program illustrates how the relevant portion of the output from hier.part can be reproduced using relaimpo (metrics calculated on page 4): > IJ <-cbind(I = metrics$lmg, J = metrics$first -metrics$lmg, + Total = metrics$first) > I.perc <-as.matrix(100 * calc.relimp(linmod, type = "lmg", rela = T)$lmg) > colnames(I.perc) = "" > list(IJ = IJ, I.perc = I.perc)  Since the example serves the purpose of underscoring the connection between results from hier.part and relaimpo, the more complicated second call to function calc.relimp has been used.

Computation times
The metrics lmg and pmvd require a lot of computation in case of many regressors. If one wants to apply these for many regressors and potentially even in connection with a bootstrap analysis, it is helpful to know in advance how much computing time will be needed. Table 2 shows computing times for 3 to 12 regressors for both lmg and pmvd and for comparison also for hier.part (barplot turned off). All times are averages over 100 runs on a Windows XP Professional system, AMD Athlon XP 1700+, 1.47GHz, 256MB RAM. We see that relaimpo's CPU times are virtually unaffected by the change in sample size, while hier.part times do change significantly. This is due to the fact that calculation of metrics in relaimpo is based on the covariance matrix (as explained in Section 3.3) which is only calculated once while hier.part calculates 2 p − 1 regression models using all observations.  Table 2: CPU times in seconds (average of 100 runs each) for p equi-correlated regressors with variances 1 and pairwise correlations 0.5.
For relaimpo, we see that pmvd takes longer than lmg for large numbers of regressors p. This is due to the fact that formula (8) cannot be simplified as much as formula (7). (Nevertheless, as mentioned before, formula (8) is not directly used for computations.) In fact, the time for lmg roughly doubles when adding a regressor, while the growth factor for times for pmvd increases with increasing number of regressors, so that the time difference between the two methods increases quite dramatically with increasing numbers of regressors (for 15 regressors, for example, pmvd needs about 525 seconds CPU, while lmg needs about 43 seconds).

Final remarks
The functionality of R package relaimpo has been explained and illustrated in this paper, using the data set swiss that is available in R. This dataset has a complicated correlation structure among regressors which makes assessment of relative importances somewhat ambiguous. R package relaimpo broadens R's possibilities of assessing relative importances in linear models by offering a choice of six metrics two of whichlmg and pmvd -are recommended if interest is in decomposing R 2 . pmvd is newly provided, and for lmg computing times are substantially improved vs. the package hier.part. For all metrics, relaimpo offers bootstrap confidence intervals for the estimated relative importances themselves as well as for pairwise differences of relative contributions and for regressors' ranks in terms of relative importance. These help preventing the analyst from over-interpreting differences.
The availability of six metrics -which are already a selection from even more candidates -emphasizes the fact that there is no unique unchallenged metric for relative importance in case of correlated regressors. The recommended metrics lmg and pmvd are also not entirely satisfactory, but are so far the metrics that come closest to Johnson and Lebreton's (2004) definition of relative importance, as cited in the introduction. There is a substantial amount of relatively recent literature on relative importance. A (certainly incomplete) list of further references on relative importance can be found on the author's website http://www.tfh-berlin.de/~groemp/. Regarding future developments, it is intended to allow treating regressors in groups in order to support applications with large numbers of regressors that are not unusual e.g., in marketing applications. If there are e.g., 30 regressors, a complete analysis including bootstrap confidence intervals will be prohibitive even on modern computers. Often, regressors can be grouped, and an analysis of twelve groups, say, will again be feasible. Being able to handle grouped regressors will also make it possible to support factors with more than two levels. As a further step, factors with interaction terms will require the procedure to be able to respect a pre-defined hierarchy, i.e., some orders of regressors (interaction before main effect) must be excluded from the possibilities. This would also be a desirable extension of relaimpo. Furthermore, it would be useful to be able to accomodate weights in the analysis, since often survey data come with observation weights that reflect sampling probabilities. Further suggestions from users are welcome.