Data preprocessing modules

Kenneth French 49 Industry Portfolios

S&P 500 CRSP data

Preprocessing utilities for S&P 500 CRSP monthly data.

This module loads the cleaned CRSP/WRDS monthly file used in the experiments and converts monthly stock returns into synthetic price indices.

tda_finance.data_preprocessing.preprocess_sp500_crsp.load_sp500_prices_from_monthly_returns(path: str = 'data/sp500_crsp_monthly_clean.parquet', file_format: str = 'parquet', start_price: float = 100.0, min_price: float = 1e-06) → DataFrame[source]

Load monthly S&P 500 returns and convert them into synthetic prices.

The expected input file contains at least the following columns: PERMNO, Month and MonthlyRet. PERMNO is used as the asset identifier because it is more stable than the ticker.

Parameters:

  • path (str, default="data/sp500_crsp_monthly_clean.parquet") – Path to the cleaned CRSP monthly file.

  • file_format ({"parquet", "csv"}, default="parquet") – File format used to read the input data.

  • start_price (float, default=100.0) – Initial value used for each synthetic price index.

  • min_price (float, default=1e-6) – Lower bound applied to gross returns to avoid zero synthetic prices.

Returns:

Synthetic price matrix indexed by month-end date, with one column per PERMNO.

Return type:

pandas.DataFrame

Raises:

ValueError – If the file format is unsupported, required columns are missing, or duplicated PERMNO-month observations are found.

tda_finance.data_preprocessing.preprocess_sp500_crsp.load_sp500_returns_matrix(path: str = 'data/sp500_crsp_monthly_clean.parquet', file_format: str = 'parquet') → DataFrame[source]

Load monthly S&P 500 returns as a Month x PERMNO matrix.

Parameters:

  • path (str, default="data/sp500_crsp_monthly_clean.parquet") – Path to the cleaned CRSP monthly file.

  • file_format ({"parquet", "csv"}, default="parquet") – File format used to read the input data.

Returns:

Monthly return matrix indexed by month-end date, with one column per PERMNO.

Return type:

pandas.DataFrame

Raises:

ValueError – If the file format is unsupported.