GLS - Generalized Least Squares

Overview

The Generalized Least Squares (GLS) estimator provides efficient estimation for linear regression models with known error covariance structures. It handles both heteroskedasticity and serial correlation by transforming the model to achieve the Best Linear Unbiased Estimator (BLUE) properties.

Mathematical Foundation

Correlated Error Model

\[\mathbf{y} = \mathbf{X}\mathbf{\beta} + \mathbf{\epsilon}\]

where \(\text{Var}(\mathbf{\epsilon}) = \sigma^2\mathbf{\Omega}\) and \(\mathbf{\Omega}\) is a known positive definite matrix.

GLS Estimator

\[\hat{\mathbf{\beta}}_{GLS} = (\mathbf{X}'\mathbf{\Omega}^{-1}\mathbf{X})^{-1}\mathbf{X}'\mathbf{\Omega}^{-1}\mathbf{y}\]

Cholesky Whitening Implementation

Uses Cholesky decomposition \(\mathbf{\Omega} = \mathbf{L}\mathbf{L}'\) where \(\mathbf{L}\) is lower triangular: - Transform: \(\tilde{\mathbf{y}} = \mathbf{L}^{-1}\mathbf{y}\), \(\tilde{\mathbf{X}} = \mathbf{L}^{-1}\mathbf{X}\)
- Apply OLS to whitened data

API Reference

Constructor

GLS(fit_intercept=True)

Parameters: - fit_intercept (bool): Whether to calculate the intercept

Methods

fit()

fit(X, y, sigma) -> None

Fit the GLS model using Cholesky whitening transformation.

Parameters: - X (np.ndarray): Training data matrix - y (np.ndarray): Target values - sigma (np.ndarray): Error covariance matrix (must be positive definite)

Requirements: - sigma must be symmetric and positive definite - Represents \(\text{Var}(\mathbf{\epsilon})\) up to scale \(\sigma^2\)

predict()

predict(X) -> np.ndarray

Generate predictions using the fitted GLS model.

Statistical Methods

standard_errors() → Standard errors from GLS covariance matrix
t_statistics() → T-statistics for significance tests
p_values() → P-values for coefficient tests
confidence_intervals(alpha=0.05) → Confidence intervals
summary() → Comprehensive regression summary

Properties

Core Results: - coefficients: GLS regression coefficients - intercept: GLS regression intercept - r_squared: R-squared from whitened regression - r_squared_adj: Adjusted R-squared - mse: Mean squared error of whitened residuals - residuals: Whitened residuals

Model Information: - n_samples: Number of observations - n_features: Number of features - fit_intercept: Whether intercept is included

Implementation Details

Numerical Stability

Cholesky decomposition for efficient whitening
Algorithm selection: Cholesky vs SVD based on problem size
Robust handling of ill-conditioned covariance matrices

Performance Features

Memory-efficient: covariance matrix not stored after decomposition
Vectorized operations using optimized linear algebra
In-place triangular solvers for whitening

Usage Guidelines

Use GLS when: - Error covariance structure is known - Presence of heteroskedasticity and/or serial correlation - Want efficient estimation under general correlation patterns - Working with time series or spatial data

Common Covariance Structures: - AR(1): \(\Omega_{ij} = \rho^{|i-j|}\) (autoregressive) - Heteroskedastic: \(\Omega = \text{diag}(\sigma_1^2, \ldots, \sigma_n^2)\) - Compound Symmetry: \(\Omega_{ij} = \sigma^2\) for \(i \neq j\)

Consider alternatives when: - Covariance structure is unknown → OLS with robust standard errors - Only diagonal heteroskedasticity → WLS - Endogenous regressors → IV or TSLS