OLS - Ordinary Least Squares

Overview

The Ordinary Least Squares (OLS) estimator provides linear regression analysis with comprehensive statistical inference capabilities. It includes heteroskedasticity-robust standard errors and intelligent numerical algorithm selection for optimal performance across diverse problem specifications.

Mathematical Foundation

The Linear Model

\[\mathbf{y} = \mathbf{X}\mathbf{\beta} + \mathbf{\epsilon}\]

where \(\mathbf{y}\) is the dependent variable vector, \(\mathbf{X}\) is the design matrix, \(\mathbf{\beta}\) is the parameter vector, and \(\mathbf{\epsilon}\) is the error term.

OLS Estimator

\[\hat{\mathbf{\beta}} = (\mathbf{X}'\mathbf{X})^{-1}\mathbf{X}'\mathbf{y}\]

Statistical Properties

Finite Sample Properties (under Gauss-Markov assumptions): - Unbiased: \(E[\hat{\mathbf{\beta}}|\mathbf{X}] = \mathbf{\beta}\) - Efficient: Best Linear Unbiased Estimator (BLUE) - Covariance: \(\text{Var}(\hat{\mathbf{\beta}}|\mathbf{X}) = \sigma^2(\mathbf{X}'\mathbf{X})^{-1}\)

Robust Standard Errors: White (1980) heteroskedasticity-consistent estimator: \(\(\hat{\mathbf{V}}_{HC0} = (\mathbf{X}'\mathbf{X})^{-1}\mathbf{X}'\mathbf{\Omega}\mathbf{X}(\mathbf{X}'\mathbf{X})^{-1}\)\)

API Reference

Constructor

OLS(fit_intercept=True, robust=False)

Parameters: - fit_intercept (bool): Whether to calculate the intercept - robust (bool): Whether to use heteroskedasticity-robust standard errors

Methods

fit()

fit(X, y) -> None

Fit the OLS model using optimized algorithms.

Parameters: - X (np.ndarray): Training data matrix - y (np.ndarray): Target values

predict()

predict(X) -> np.ndarray

Generate predictions using the fitted model.

Statistical Methods

standard_errors() → Standard errors of coefficients
t_statistics() → T-statistics for significance tests
p_values() → P-values for coefficient tests
confidence_intervals(alpha=0.05) → Confidence intervals
summary() → Comprehensive regression summary
covariance_matrix() → Coefficient covariance matrix

Properties

Core Results: - coefficients: Regression coefficients - intercept: Regression intercept - fit_intercept: Whether intercept is included - robust: Whether robust standard errors are used

Statistical Measures: - r_squared: Coefficient of determination - r_squared_adj: Adjusted R-squared - mse: Mean squared error - residuals: Regression residuals

Model Information: - n_samples: Number of observations - n_features: Number of features

Implementation Details

Numerical Algorithms

Cholesky Decomposition: Primary method for well-conditioned problems
SVD: Fallback for ill-conditioned or rank-deficient matrices
Automatic Selection: Based on condition number and problem structure

Performance Features

Memory-efficient computation
Vectorized operations using optimized BLAS
Cache-optimized memory access patterns

Usage Guidelines

Use OLS when: - Linear relationship between variables - Exogenous regressors (no endogeneity) - Need for computational efficiency - Establishing baseline results

Consider alternatives when: - Heteroskedasticity → WLS - Error correlation → GLS - Endogeneity → IV or TSLS - Panel data → FE