WLS - Weighted Least Squares

Overview

The Weighted Least Squares (WLS) estimator provides efficient estimation for linear regression models with known heteroskedastic error structures. By incorporating knowledge of the error variance pattern through optimal weighting, WLS achieves Best Linear Unbiased Estimator (BLUE) properties under heteroskedasticity.

Mathematical Foundation

Heteroskedastic Model

\[\mathbf{y} = \mathbf{X}\mathbf{\beta} + \mathbf{\epsilon}\]

where \(\text{Var}(\epsilon_i) = \sigma^2/w_i\) and \(w_i\) are known weights.

WLS Estimator

The WLS estimator minimizes the weighted sum of squared residuals:

\[\hat{\mathbf{\beta}}_{WLS} = (\mathbf{X}'\mathbf{W}\mathbf{X})^{-1}\mathbf{X}'\mathbf{W}\mathbf{y}\]

where \(\mathbf{W} = \text{diag}(w_1, w_2, \ldots, w_n)\).

Implementation

Uses \(\sqrt{w_i}\) transformation for numerical stability: - Transform: \(\tilde{y}_i = \sqrt{w_i} y_i\), \(\tilde{\mathbf{x}}_i = \sqrt{w_i} \mathbf{x}_i\) - Apply OLS to transformed data

API Reference

Constructor

WLS(fit_intercept=True)

Parameters: - fit_intercept (bool): Whether to calculate the intercept

Methods

fit()

fit(X, y, weights) -> None

Fit the WLS model using weighted transformation.

Parameters: - X (np.ndarray): Training data matrix - y (np.ndarray): Target values - weights (np.ndarray): Sample weights (must be positive)

Requirements: - Weights typically represent \(w_i = 1/\text{Var}(\epsilon_i)\) - Higher weights indicate lower variance (higher precision)

predict()

predict(X) -> np.ndarray

Generate predictions using the fitted WLS model.

Statistical Methods

standard_errors() → Standard errors from weighted covariance matrix
t_statistics() → T-statistics for significance tests
p_values() → P-values for coefficient tests
confidence_intervals(alpha=0.05) → Confidence intervals
summary() → Comprehensive regression summary

Properties

Core Results: - coefficients: WLS regression coefficients - intercept: WLS regression intercept - r_squared: Weighted R-squared - r_squared_adj: Adjusted R-squared - mse: Mean squared error (original scale) - residuals: Original scale residuals

Model Information: - n_samples: Number of observations - n_features: Number of features - fit_intercept: Whether intercept is included

Implementation Details

Numerical Stability

\(\sqrt{w_i}\) transformation prevents numerical issues with extreme weights
Automatic algorithm selection (Normal equations vs SVD)
Robust handling of singular systems

Performance Features

Memory-efficient: weights not stored after fitting
Vectorized operations for optimal speed
Minimal memory allocation

Usage Guidelines

Use WLS when: - Error variance structure is known - Observations have different precisions - Want efficient estimation under heteroskedasticity - Weights can be estimated consistently

Common Weight Specifications: - Inverse variance: \(w_i = 1/\text{Var}(\epsilon_i)\) - Precision weights: \(w_i = 1/\sigma_i^2\) - Frequency weights: \(w_i = n_i\) (grouped data)

Consider alternatives when: - Unknown heteroskedasticity → OLS with robust standard errors - Complex error correlation → GLS - No prior knowledge of weights → OLS