Ridge - Ridge Regression (L2 Regularization)

Overview

The Ridge regression estimator provides L2 regularized linear regression with comprehensive statistical inference capabilities. Ridge regression addresses multicollinearity and overfitting by adding a penalty term proportional to the sum of squares of coefficients, shrinking them toward zero while maintaining all features in the model.

Mathematical Foundation

The Regularized Linear Model

\(\mathbf{y} = \mathbf{X}\mathbf{\beta} + \mathbf{\epsilon}\)

where \(\mathbf{y}\) is the dependent variable vector, \(\mathbf{X}\) is the design matrix, \(\mathbf{\beta}\) is the parameter vector, and \(\mathbf{\epsilon}\) is the error term.

Ridge Objective Function

Ridge regression minimizes the penalized sum of squares:

\(\min_{\mathbf{\beta}} \|\mathbf{y} - \mathbf{X}\mathbf{\beta}\|^2 + \alpha\|\mathbf{\beta}\|^2\)

where \(\alpha \geq 0\) is the regularization parameter controlling the strength of the penalty.

Ridge Estimator

\(\hat{\mathbf{\beta}}_{Ridge} = (\mathbf{X}'\mathbf{X} + \alpha\mathbf{I})^{-1}\mathbf{X}'\mathbf{y}\)

Note: When fit_intercept=True, the intercept is not regularized, requiring special matrix construction.

Statistical Properties

Bias-Variance Tradeoff: - Bias: \(E[\hat{\mathbf{\beta}}_{Ridge}] - \mathbf{\beta} = -\alpha(\mathbf{X}'\mathbf{X} + \alpha\mathbf{I})^{-1}\mathbf{\beta}\) - Variance: \(\text{Var}(\hat{\mathbf{\beta}}_{Ridge}) = \sigma^2(\mathbf{X}'\mathbf{X} + \alpha\mathbf{I})^{-1}\mathbf{X}'\mathbf{X}(\mathbf{X}'\mathbf{X} + \alpha\mathbf{I})^{-1}\)

Key Properties: - Coefficients are shrunk toward zero (biased but lower variance) - All features remain in the model (no feature selection) - Numerical stability improved for ill-conditioned problems - When \(\alpha = 0\), Ridge reduces to OLS

API Reference

Constructor

Ridge(alpha=1.0, fit_intercept=True)

Parameters: - alpha (float): Regularization strength. Must be non-negative. Higher values create more regularization - fit_intercept (bool): Whether to calculate the intercept (intercept is not regularized)

Methods

fit()

fit(X, y) -> None

Fit the Ridge regression model using optimized algorithms.

Parameters: - X (np.ndarray): Training data matrix - y (np.ndarray): Target values

predict()

predict(X) -> np.ndarray

Generate predictions using the fitted regularized coefficients.

Statistical Methods

standard_errors() → Standard errors of regularized coefficients
t_statistics() → T-statistics for significance tests
p_values() → P-values for coefficient tests
confidence_intervals(alpha=0.05) → Confidence intervals
summary() → Comprehensive regression summary with regularization info
covariance_matrix() → Regularized coefficient covariance matrix

Properties

Core Results: - coefficients: Regularized regression coefficients - intercept: Regression intercept (unregularized) - alpha: Regularization parameter - fit_intercept: Whether intercept is included

Statistical Measures: - r_squared: Coefficient of determination - mse: Mean squared error - residuals: Regression residuals

Model Information: - n_samples: Number of observations - n_features: Number of features

Implementation Details

Numerical Algorithms

Cholesky Decomposition: Primary method when \(\alpha > 0\) (ensures positive definiteness)
SVD: Fallback for edge cases or when \(\alpha = 0\)
Automatic Selection: Based on regularization strength and matrix conditioning

Regularization Matrix Construction

With Intercept: Intercept column excluded from regularization
Without Intercept: All coefficients regularized equally
Efficient Implementation: Direct matrix operations without explicit regularization matrix

Performance Features

Memory-efficient regularized matrix computation
Vectorized operations using optimized BLAS
Cached computations for statistical inference
Smart algorithm selection based on problem characteristics

Parameter Selection Guidelines

Choosing Alpha

Small Alpha (α ≈ 0): - Approaches OLS behavior - Less bias, more variance - May not address multicollinearity effectively

Moderate Alpha (α ≈ 0.1 - 10): - Balanced bias-variance tradeoff - Good starting point for most problems - Effective multicollinearity handling

Large Alpha (α >> 10): - Heavy regularization - High bias, low variance - Coefficients shrunk close to zero

Cross-Validation for Alpha Selection

from sklearn.model_selection import cross_val_score
import numpy as np

# Test multiple alpha values
alphas = np.logspace(-3, 3, 50)
cv_scores = []

for alpha in alphas:
    ridge = Ridge(alpha=alpha)
    scores = cross_val_score(ridge, X, y, cv=5, scoring='neg_mean_squared_error')
    cv_scores.append(-scores.mean())

optimal_alpha = alphas[np.argmin(cv_scores)]

Usage Examples

Basic Usage

import numpy as np
from econometrust import Ridge

# Generate sample data with multicollinearity
np.random.seed(42)
n_samples, n_features = 100, 5
X = np.random.randn(n_samples, n_features)
X[:, 4] = X[:, 0] + 0.1 * np.random.randn(n_samples)  # Correlated feature

# True coefficients
beta_true = np.array([1.5, -2.0, 0.5, 1.0, 0.0])
y = X @ beta_true + 0.1 * np.random.randn(n_samples)

# Fit Ridge regression
ridge = Ridge(alpha=1.0, fit_intercept=True)
ridge.fit(X, y)

print(f"Coefficients: {ridge.coefficients}")
print(f"R-squared: {ridge.r_squared:.4f}")
print(f"MSE: {ridge.mse:.4f}")

Statistical Inference

# Examine coefficient significance
std_errors = ridge.standard_errors()
t_stats = ridge.t_statistics()
p_values = ridge.p_values()

print("Coefficient Analysis:")
for i, (coef, se, t, p) in enumerate(zip(ridge.coefficients, std_errors, t_stats, p_values)):
    print(f"β{i}: {coef:.4f} (SE: {se:.4f}, t: {t:.2f}, p: {p:.4f})")

# 95% confidence intervals
ci = ridge.confidence_intervals(alpha=0.05)
print(f"\n95% Confidence Intervals:")
for i, (lower, upper) in enumerate(ci):
    print(f"β{i}: [{lower:.4f}, {upper:.4f}]")

Comprehensive Summary

# Full regression summary
print(ridge.summary())

Prediction

# Generate predictions for new data
X_new = np.random.randn(10, n_features)
predictions = ridge.predict(X_new)
print(f"Predictions: {predictions}")

Comparing Different Alpha Values

# Compare Ridge with different regularization strengths
alphas = [0.0, 0.1, 1.0, 10.0]
results = {}

for alpha in alphas:
    ridge = Ridge(alpha=alpha)
    ridge.fit(X, y)
    results[alpha] = {
        'coefficients': ridge.coefficients.copy(),
        'r_squared': ridge.r_squared,
        'mse': ridge.mse
    }

# Display comparison
print("Alpha Comparison:")
print("Alpha\t\tR²\t\tMSE\t\tCoeff Norm")
for alpha in alphas:
    r = results[alpha]
    coeff_norm = np.linalg.norm(r['coefficients'])
    print(f"{alpha:.1f}\t\t{r['r_squared']:.4f}\t\t{r['mse']:.4f}\t\t{coeff_norm:.4f}")

Real-World Applications

Economic Forecasting

# Economic indicators with potential multicollinearity
# GDP growth prediction using multiple correlated indicators

import pandas as pd

# Load economic data (example structure)
# data = pd.read_csv('economic_indicators.csv')
# Features: inflation, unemployment, interest_rates, consumer_confidence, etc.

# Ridge regression handles correlated economic indicators well
ridge_econ = Ridge(alpha=2.0, fit_intercept=True)
# ridge_econ.fit(X_economic, gdp_growth)

# Interpret regularized coefficients
# Coefficients show relative importance while handling multicollinearity

Financial Risk Modeling

# Portfolio risk factors with Ridge regression
# Handles correlated risk factors in financial models

# Multiple risk factors (market, size, value, momentum, etc.)
ridge_risk = Ridge(alpha=0.5, fit_intercept=True)
# ridge_risk.fit(risk_factors, portfolio_returns)

# Ridge provides stable coefficient estimates
# Less sensitive to small changes in risk factor correlations

Marketing Mix Modeling

# Marketing channel attribution with correlated channels
# TV, Radio, Digital channels often have complex interactions

ridge_marketing = Ridge(alpha=1.5, fit_intercept=True)
# ridge_marketing.fit(marketing_spend, sales)

# Ridge maintains all channels in the model
# Provides interpretable coefficients for budget allocation

Usage Guidelines

Use Ridge when: - Multicollinearity among predictors - Overfitting concerns with many features - Need to retain all features in the model - Seeking stable coefficient estimates - Working with correlated economic/financial variables

Consider alternatives when: - Feature selection is desired → Lasso regression - No multicollinearity issues → OLS - Heteroskedasticity → WLS with Ridge-like penalties - Endogeneity → IV or TSLS - Panel data → FE with regularization

Ridge vs. Other Methods: - vs. OLS: Ridge handles multicollinearity better, provides more stable estimates - vs. Lasso: Ridge keeps all features, provides smoother coefficient paths - vs. Elastic Net: Ridge is simpler, no feature selection, purely L2 penalty