Multiple regression can be a beguiling, temptation-filled analysis. Some of the predictors will be significant. Perhaps there is a relationship, or is it just by chance? You can add higher-order polynomials to bend and twist that fitted line as you like, but are you fitting real patterns or just connecting the dots?
All the while, the R-squared R 2 value increases, teasing you, and egging you on to add more variables! Previously, I showed how R-squared can be misleading when you assess the goodness-of-fit for linear regression analysis. In my last post, I showed how R-squared cannot determine whether the coefficient estimates and predictions are biased, which is why you must assess the residual plots.
However, R-squared has additional problems that the adjusted R-squared and predicted R-squared are designed to address. Problem 1: Every time you add a predictor to a model, the R-squared increases, even if due to chance alone.
It never decreases. Consequently, a model with more terms may appear to have a better fit simply because it has more terms. Problem 2: If a model has too many predictors and higher order polynomials, it begins to model the random noise in the data.
This condition is known as overfitting the model and it produces misleadingly high R-squared values and a lessened ability to make predictions. The adjusted R-squared compares the explanatory power of regression models that contain different numbers of predictors. Suppose you compare a five-predictor model with a higher R-squared to a one-predictor model. Or is the R-squared higher because it has more predictors? Simply compare the adjusted R-squared values to find out! To nevertheless obtain an overall best estimator, I also calculated the average MSE across all conditions for each estimator.
The result is displayed in Table 3. Importantly, the positive-part Ezekiel estimator again was best with an average MSE of 0. However, the average MSE of the second-best estimator maximum likelihood was 0. Since the positive-part Ezekiel estimator was best both in terms of average as well as maximum MSE, I investigated the impact of always using it.
The full results are displayed in Table 5. Thus, the increase in MSE when always using the positive-part Ezekiel estimator was relatively mild.
Note : The increase is relative to the minimum average and maximum mean squared error respectively and expressed as a percentage. In this paper, I compared the novel exact Olkin-Pratt estimator and 19 additional estimators of the squared multiple correlation using different perspectives. These different perspectives all follow directly from optimality concepts established in theoretical statistics and are based on bias, MSE, or a combination thereof.
Regarding the most prevalent uniformly minimum MSE unbiased perspective, the results are unambiguous and in line with the theoretical optimality property established for the Olkin-Pratt estimator. The exact Olkin-Pratt estimator was optimal. It was the only estimator that was empirically unbiased across all conditions. Consequently, based on this perspective, the exact Olkin-Pratt estimator should always be used. Regarding the perspectives that consider only MSE, the results are more ambiguous.
No estimator had uniformly lowest MSE across all conditions. Even more importantly, no estimator was uniformly best according to the maximum or average MSE perspectives.
However, across all conditions, the positive-part version of the most widely used Ezekiel adjusted R 2 estimator performed best both according to the maximum MSE as well as average MSE perspective. To choose the best estimator under the MSE-only perspective, two cases have to be distinguished.
Second, this is not the case. If this is not the case, then one first has to decide for the maximum or the average MSE perspective. The maximum MSE perspective matches well with frequentist principles, whereas the average MSE perspective matches better with Bayesian principles. After this choice has been made, one can select the best estimator using Tables 2 or 4. In case constraints do not allow such an individualized choice, I recommend using the positive-part Ezekiel estimator. This choice is especially defendable in situations where the sample size N is large compared to the number of predictors p , as here the difference between estimators is small.
Thus, choosing the default estimator, which most readers know, is a sound strategy. If it is not possible to determine what is more important — unbiasedness or minimization of MSE — I recommend using the unbiased exact Olkin-Pratt estimator. There are three reasons for this. The first one is consistency. The uniformly lowest MSE unbiased perspective is the standard in regression analysis. Third, and in relation to this, the fact that an estimator is unbiased guarantees that when estimates for the same property of interest from multiple studies are aggregated in a meta-analysis, this aggregation eventually leads to the true value, which is not the case when using a biased estimator, even if it has lower MSE.
On the surface, these recommendations conflict with previous recommendations based on simulation studies with similar designs. Yin and Fan recommended using the Pratt estimator and Shieh the positive-part version of the Pratt estimator. The conflict with Yin and Fan is quickly resolved.
They considered only the Pratt and the Claudy approximation of the Olkin-Pratt estimator but not the more elaborate versions. Additionally, they used bias as the only metric for comparison. Thus, were they to repeat their study with the estimator included in this comparison, they should conclude that the exact Olkin-Pratt estimator should always be used.
The conflict with Shieh can also be resolved. While Shieh considered bias and MSE, he eventually based his conclusions almost exclusively on bias.
Additionally, he considered computational complexity and the estimator not returning impossible negative values as factors. Balancing all these factors lead to the positive-part Pratt estimator. Using this same balance of factors, I would expect that the results presented here would not change the recommendation by Shieh While the rationale presented in Shieh is sound, the rationale given here has several advantages.
First, computational complexity becomes an irrelevant factor due to the R package provided, which can compute all estimators within milliseconds. Second, while ignoring estimators that do not return impossible negative values is intuitively appealing and beneficial from a pure MSE-based perspective, it is detrimental from the more prevalent unbiasedness perspective. As I already mentioned, the fact that an estimator is unbiased guarantees that when estimates for the same property of interest are obtained based on multiple studies, then the average of these estimates converges to the true value.
Consequently, from this perspective, returning impossible values on one sample is less detrimental than converging on the wrong value when averaging across many samples see, Okada, , for a paper-length elaboration of this argument.
All results presented here rely on the assumption of a multivariate normal distribution of the predictor variables. This limitation is shared with all previous comparisons. As such, repeating this study with different distributions for the predictor variables to investigate the robustness of the results with regard to this assumption is recommended for future work. Assessing bias and MSE through a simulation study has several disadvantages.
First, the values are estimated and not computed exactly. I mitigated this issue by employing a much larger number of replications than previous comparisons , and hypothesis tests to account for the remaining small uncertainty of the estimates. I did not use more precise alternatives such as analytical derivations or numerical methods Shieh, because both approaches would not allow a direct assessment of the provided R package.
A second disadvantage is that the conclusions from a simulation study often do not generalize beyond the considered design. I diminished this issue by augmenting the results of the simulation study with theoretical results. As such, the central finding that the Olkin-Pratt estimator is uniformly minimum MSE unbiased generalizes beyond the design considered.
Whether the outcome that overall, the Ezekiel estimator performs best in terms of MSE also generalizes to other designs remains to be investigated. Also, selecting the MSE optimal estimator is only possible if the parameters of a data set sample size, number of predictors lie within the range considered here. For this reason, I carefully selected the parameter ranges such that they cover the majority of parameter values reported in psychology. Nevertheless, providing a table for all relevant parameter combinations is impossible.
Instead, I advise researchers to run their own small simulation studies to determine the MSE optimal estimator for their particular situation. In conclusion, I recommend using the exact Olkin-Pratt estimator by default.
However, if the researcher is confident that minimizing MSE is more critical than unbiasedness, then a different estimator should be used. In this case, I recommend an individualized choice based on the strategy described at the beginning of this discussion, and if this is not feasible or the sample size N is large compared to the number of predictors p , the positive-part version of the Ezekiel estimator.
In general, the hypergeometric function is difficult to compute and for some inputs it is even not defined since it does not converge. Thus, there must be at least 4 samples more than there are predictors. Unknown 10 October at Bilal 7 January at Unknown 11 June at Unknown 22 March at Newer Post Older Post Home. Subscribe to: Post Comments Atom. Collabra: Psychology 6 1 : That is true but incomplete. First, almost no variable is totally useless. In fact, in some cases see below it works very badly for noise variables.
Note also that I didn't have to "fish around" for an example like this, this was the first one I tried. It's a way of adjusting for the complexity of a model. Sign up to join this community. The best answers are voted up and rise to the top. Stack Overflow for Teams — Collaborate and share knowledge with a private group. Create a free Team What is Teams? Learn more. Which is better: r-squared or adjusted r-squared? Ask Question.
Asked 3 years, 4 months ago. Active 7 months ago. Viewed 11k times. Improve this question. Ronith Ronith 1 1 gold badge 1 1 silver badge 6 6 bronze badges. My answer here may prove of interest: stats.
0コメント