James–Stein estimator

James–Stein Estimator

Comparison of Mean Squared Error (MSE) between Maximum Likelihood (ML) and James–Stein (JS) estimators

The James–Stein estimator is a statistical estimator used in the context of multivariate statistics. It is particularly known for its application in shrinkage estimation, where it provides a method to improve the estimation of multiple parameters simultaneously. The James–Stein estimator is a type of biased estimator that can outperform the traditional maximum likelihood estimator (MLE) in terms of mean squared error (MSE) when estimating the mean of a multivariate normal distribution.

Background

The James–Stein estimator was introduced by Charles Stein in 1961, and it challenged the conventional wisdom of the time by demonstrating that the MLE is not always the best estimator in terms of MSE. The estimator is particularly effective when dealing with three or more parameters, where it "shrinks" the MLE towards a central point, often the overall mean, thereby reducing the overall estimation error.

Mathematical Formulation

Consider a vector of observations \( \mathbf{X} = (X_1, X_2, \ldots, X_p) \) from a multivariate normal distribution with unknown mean \( \mathbf{\mu} \) and known variance \( \sigma^2 \). The MLE of \( \mathbf{\mu} \) is simply \( \mathbf{X} \). The James–Stein estimator \( \hat{\mathbf{\mu}}_{JS} \) is given by:

\[ \hat{\mathbf{\mu}}_{JS} = \left(1 - \frac{(p-2)\sigma^2}{\|\mathbf{X}\|^2}\right) \mathbf{X} \]

where \( p \) is the number of parameters, and \( \|\mathbf{X}\|^2 \) is the squared Euclidean norm of \( \mathbf{X} \).

Properties

The James–Stein estimator is a shrinkage estimator, meaning it "shrinks" the MLE towards a central point, which reduces the variance of the estimator at the cost of introducing some bias. This trade-off results in a lower MSE compared to the MLE, especially when \( p \geq 3 \). The estimator is inadmissible for \( p = 1 \) or \( p = 2 \), meaning that there are other estimators with uniformly lower MSE.

Applications

The James–Stein estimator is widely used in econometrics, biostatistics, and machine learning for improving the estimation of multiple parameters. It is particularly useful in situations where the number of parameters is large relative to the number of observations, a common scenario in modern data analysis.

Related Pages

• Shrinkage estimator
• Maximum likelihood estimation
• Multivariate normal distribution
• Biased estimator