Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/ecosystem.yml
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ jobs:
qualification:
permissions:
contents: read
uses: ml4t/ecosystem/.github/workflows/qualify-library.yml@a3080f8e09b197098c9d796240e746aca516492e # 2026-09-23 policy snapshot
uses: ml4t/ecosystem/.github/workflows/qualify-library.yml@066dc737a879faea51a5b8a900315441712ea89a # 2026-09-25 policy snapshot
with:
import-package: ml4t.data
prerelease-exception: python-315-polars
2 changes: 1 addition & 1 deletion .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ jobs:
needs: preflight
permissions:
contents: read
uses: ml4t/ecosystem/.github/workflows/qualify-library.yml@a3080f8e09b197098c9d796240e746aca516492e # 2026-09-23 policy snapshot
uses: ml4t/ecosystem/.github/workflows/qualify-library.yml@066dc737a879faea51a5b8a900315441712ea89a # 2026-09-25 policy snapshot
with:
import-package: ml4t.data
prerelease-exception: python-315-polars
Expand Down
6 changes: 3 additions & 3 deletions docs/getting-started/provider-selection.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,9 +24,9 @@ data = SyntheticProvider(seed=42).fetch_ohlcv(
| Equity bars | Yahoo Finance, Alpaca, EODHD, Tiingo, Twelve Data, Massive, Finnhub | Some adapters require credentials, accounts, or paid history |
| Foreign exchange | OANDA, Twelve Data, FXMacroData | OANDA and Twelve Data require credentials |
| Futures and options | Databento, Binance, OKX | Historical or licensed data may be metered |
| Macroeconomic series | FRED, FXMacroData | FRED requires an API key for normal use |
| Research factors | Fama-French, AQR | Network access and source-specific terms |
| Prediction markets | Kalshi, Polymarket | Network access and changing public endpoints |
| [Macroeconomic series](../providers/macro.md) | FRED, FXMacroData | FRED requires an API key for normal use |
| [Research factors](../providers/factors.md) | Fama-French, AQR | Network access and source-specific terms |
| [Prediction markets](../providers/prediction_markets.md) | Kalshi, Polymarket | Network access and changing public endpoints |
| Frozen equity history | Wiki Prices | Local historical dataset ending in 2018 |
| CFTC positioning | COT | Install the `cot` extra |

Expand Down
1 change: 1 addition & 0 deletions docs/providers/alpaca.md
Original file line number Diff line number Diff line change
Expand Up @@ -146,5 +146,6 @@ headers, and retries transient failures per pagination page.

## See Also

- [Equity](equities.md) and [ETF](etfs.md) source references
- [Alpaca Market Data docs](https://docs.alpaca.markets/us/docs/about-market-data-api)
- [Provider README](index.md)
243 changes: 39 additions & 204 deletions docs/providers/alternative_data.md

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions docs/providers/aqr.md
Original file line number Diff line number Diff line change
Expand Up @@ -91,6 +91,7 @@ When using AQR data, cite the relevant papers:

## See Also

- [Research factor data sources](factors.md)
- [AQR Datasets](https://www.aqr.com/Insights/Datasets)
- [Fama-French Provider](fama_french.md)
- [Provider reference](index.md)
1 change: 1 addition & 0 deletions docs/providers/binance.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,7 @@ Binance may block access from certain countries. Consider using `BinancePublicPr

## See Also

- [Cryptocurrency data sources](crypto.md)
- [Binance API](https://www.binance.com/en/binance-api)
- [BinancePublic Provider](binance_public.md)
- [Provider reference](index.md)
1 change: 1 addition & 0 deletions docs/providers/binance_public.md
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,7 @@ Same as Binance:

## See Also

- [Cryptocurrency data sources](crypto.md)
- [Binance Data Portal](https://data.binance.vision)
- [Binance Provider](binance.md)
- [Provider reference](index.md)
1 change: 1 addition & 0 deletions docs/providers/coingecko.md
Original file line number Diff line number Diff line change
Expand Up @@ -74,5 +74,6 @@ be converted into correct daily OHLC values.

## See Also

- [Cryptocurrency data sources](crypto.md)
- [CoinGecko API](https://www.coingecko.com/en/api/pricing)
- [Provider reference](index.md)
30 changes: 30 additions & 0 deletions docs/providers/crypto.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
# Cryptocurrency Data Sources

Cryptocurrency data is fragmented across venues, instrument types, and chains. Preserve exchange,
pair, quote currency, contract type, and symbol history. Aggregated prices can conceal venue
closures, wash trading, and differences between spot, futures, perpetuals, and options.

!!! note "Verified September 2026"
Entries were checked against the linked official product documentation. Coverage and access
depend on the vendor contract and can change.

| Source and product | Coverage or history | Acquisition | Access | Material research caveat | ml4t-data provider |
|--------------------|---------------------|-------------|--------|--------------------------|--------------------|
| [Binance Spot API](https://github.com/binance/binance-spot-api-docs) | Binance spot pairs, trades, order books, and bars | REST and WebSocket APIs | Public market endpoints; credentials for private or higher-limit use | Venue availability and product access vary by jurisdiction | `BinanceProvider` |
| [Binance Public Data](https://data.binance.vision/) | Bulk spot and futures market files | Public archive download | Free | Files are venue-specific and symbol coverage changes over time | `BinancePublicProvider` |
| [OKX Market Data](https://www.okx.com/docs-v5/en/#rest-api-market-data) | Spot and derivatives, including perpetual funding and premium data | REST and WebSocket APIs | Public market endpoints | Instrument and jurisdiction availability change; retain instrument metadata | `OKXProvider` |
| [CoinGecko API](https://docs.coingecko.com/) | Aggregated asset, exchange, market, and on-chain DEX data | REST API | Demo and paid API plans | Aggregated asset identity and market prices are not a substitute for venue-level execution data | `CoinGeckoProvider` |
| [CryptoCompare API](https://developers.cryptocompare.com/documentation) | Aggregated and exchange-specific crypto market data | REST and streaming APIs | API key for supported usage | Release qualification depends on live credential validation; aggregation methodology matters | `CryptoCompareProvider` |
| [Kaiko Market Data](https://www.kaiko.com/products/market-data) | Centralized and decentralized venues, spot and derivatives, trades and order books | API, streaming, and cloud delivery | Institutional license | Venue and instrument history depend on the contracted product | No |
| [Tardis.dev historical data](https://docs.tardis.dev/historical-data-details/overview) | Raw and normalized messages for centralized crypto exchanges, including closed venues | API and downloadable files | Commercial plan; limited samples | Reconstruction requires exchange-specific message semantics and snapshot handling | No |
| [CoinAPI Market Data](https://www.coinapi.io/products/market-data-api) | Multi-exchange spot and derivatives market data | REST, WebSocket, FIX, and files | API key; plan-dependent | Normalized symbols and aggregate feeds can hide exchange-specific contract details | No |

On-chain metrics, developer activity, and social signals belong in the
[alternative-data reference](alternative_data.md). This page covers tradable market data.

## Related References

- [Alternative data sources](alternative_data.md)
- [Futures data sources](futures.md)
- [Options data sources](options.md)
- [Market data selection](market_data.md)
1 change: 1 addition & 0 deletions docs/providers/cryptocompare.md
Original file line number Diff line number Diff line change
Expand Up @@ -68,5 +68,6 @@ Consult CryptoCompare's current terms before use. Access and limits were not ver

## See Also

- [Cryptocurrency data sources](crypto.md)
- [CryptoCompare Pricing](https://min-api.cryptocompare.com/pricing)
- [Provider reference](index.md)
1 change: 1 addition & 0 deletions docs/providers/databento.md
Original file line number Diff line number Diff line change
Expand Up @@ -181,6 +181,7 @@ streaming, use `provider.client` directly.

## See Also

- [Equity](equities.md), [futures](futures.md), and [options](options.md) source references
- [Databento Pricing](https://databento.com/pricing)
- [Databento Reference](databento_reference.md) - Detailed schema guide
- [Provider reference](index.md)
1 change: 1 addition & 0 deletions docs/providers/eodhd.md
Original file line number Diff line number Diff line change
Expand Up @@ -112,5 +112,6 @@ Get your API key at [eodhd.com/register](https://eodhd.com/register).

## See Also

- [Equity](equities.md), [ETF](etfs.md), and [fundamental](fundamentals.md) source references
- [EODHD Pricing](https://eodhd.com/pricing)
- [Provider reference](index.md)
37 changes: 37 additions & 0 deletions docs/providers/equities.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
# Equity Data Sources

This reference compares reproducible equity price and microstructure products. For cross-sectional
research, prefer sources that retain delisted securities and expose corporate-action inputs. For
intraday research, distinguish consolidated US feeds from a single exchange or venue.

!!! note "Verified September 2026"
Entries were checked against the linked official product documentation. Coverage and access
depend on the vendor contract and can change.

| Source and product | Coverage or history | Acquisition | Access | Material research caveat | ml4t-data provider |
|--------------------|---------------------|-------------|--------|--------------------------|--------------------|
| [Yahoo Finance](https://finance.yahoo.com/) | Global listed equities; vendor-defined history | Web-backed API through the provider | No credential | Not a survivorship-free research panel; adjustment and retention rules are vendor controlled | `YahooFinanceProvider` |
| [Alpaca Market Data](https://docs.alpaca.markets/docs/about-market-data-api) | US equities and ETFs from 2016 | REST and WebSocket APIs | Account; feed entitlement depends on plan | The basic feed is IEX rather than the consolidated SIP; delisted coverage is not documented | `AlpacaDataProvider` |
| [Tiingo End-of-Day](https://www.tiingo.com/documentation/end-of-day) | US equities, funds, and ETFs with raw and adjusted prices | REST API | API key; plan-dependent limits | Not documented as a survivorship-free universe | `TiingoProvider` |
| [EODHD](https://eodhd.com/financial-apis/api-for-historical-data-and-volumes/) | Global end-of-day and intraday securities data | REST API | API key; coverage depends on plan | Exchange, adjustment, and delisted coverage vary by market and subscription | `EODHDProvider` |
| [Finnhub](https://finnhub.io/docs/api) | US quotes plus plan-dependent global and historical data | REST and WebSocket APIs | API key; endpoint entitlement depends on plan | Free access does not establish a complete historical universe | `FinnhubProvider` |
| [Twelve Data](https://twelvedata.com/docs) | Multi-exchange equities and other assets | REST and WebSocket APIs | API key; plan and exchange entitlements apply | Coverage and delays differ across exchanges | `TwelveDataProvider` |
| [Massive Stocks](https://massive.com/stocks) | US stock trades, quotes, aggregates, and reference data | REST, WebSocket, and flat files | API key; history and feed depend on plan | Confirm SIP versus exchange coverage and corporate-action treatment | `MassiveProvider` |
| [CRSP US Stock Databases](https://www.crsp.org/research/crsp-us-stock-databases/) | Active and inactive US securities, monthly from 1925 and daily from 1962 | Institutional files, commonly through WRDS | Academic or institutional license | Identifier and delisting-return conventions require deliberate handling | No |
| [Norgate Data](https://norgatedata.com/data-content-tables.php) | US, Australian, and Canadian stocks with plan-dependent delisted history | Local database and client integrations | Paid subscription | Daily data only; survivorship and index-membership features depend on tier | No |
| [Sharadar Equity Prices](https://data.nasdaq.com/databases/SEP) | Active and delisted US equities from 1998 | API and bulk delivery | Paid subscription | Daily bars; adjustment fields must be selected consistently | No |
| [NYSE TAQ](https://www.nyse.com/market-data/historical/daily-taq) | Trades and quotes reported by US exchanges | Daily institutional files | Exchange data license | Raw records need symbol-history, correction, and session processing | No |
| [AlgoSeek US Equities](https://www.algoseek.com/products.html#us_equity) | US trades, quotes, bars, reference data, and historical constituents | Cloud delivery and bulk files | Commercial license; samples available | Product schemas and SIP-derived fields differ; confirm the exact package | No |
| [Databento Nasdaq TotalView-ITCH](https://databento.com/datasets/XNAS.ITCH) | Nasdaq order-level data from 2018 | API and batch download | Usage-based account | Raw exchange messages are not adjusted and do not represent the full US consolidated market | `DataBentoProvider` |
| [Nasdaq TotalView-ITCH samples](https://emi.nasdaq.com/ITCH/Nasdaq%20ITCH/) | Selected full trading days for Nasdaq-listed securities | Public binary files | Free samples | Samples are sparse and unsuitable for a broad historical panel | `ITCHSampleProvider` |
| [LSEG Tick History](https://www.lseg.com/en/data-analytics/market-data/data-feeds/tick-history) | Cross-asset trades, quotes, and depth from global venues | API and managed file delivery | Institutional license | Venue, contributor, and field coverage depend on the licensed package | No |

`DataBentoProvider` and `MassiveProvider` are multi-asset classes. The mappings above describe only
their equity capabilities. See the [provider comparison](index.md) for package-level credentials
and interfaces.

## Related References

- [ETF data sources](etfs.md)
- [Fundamental data sources](fundamentals.md)
- [Market data selection](market_data.md)
31 changes: 31 additions & 0 deletions docs/providers/etfs.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
# ETF Data Sources

ETF research often needs more than traded prices. Fund identity, liquidation history, holdings,
classifications, fees, distributions, and net asset value can each come from a different product.
Do not treat an active-fund screener as a historical universe.

!!! note "Verified September 2026"
Entries were checked against the linked official product documentation. Coverage and access
depend on the vendor contract and can change.

| Source and product | Coverage or history | Acquisition | Access | Material research caveat | ml4t-data provider |
|--------------------|---------------------|-------------|--------|--------------------------|--------------------|
| [Yahoo Finance](https://finance.yahoo.com/markets/etfs/) | Listed ETF prices and current descriptive data | Web-backed API through the provider | No credential | Current listings and vendor-adjusted prices do not form a survivorship-free fund panel | `YahooFinanceProvider` |
| [Alpaca Market Data](https://docs.alpaca.markets/docs/about-market-data-api) | US-listed ETF trades, quotes, and bars from 2016 | REST and WebSocket APIs | Account; feed entitlement depends on plan | Provides market data, not a historical holdings or classification database | `AlpacaDataProvider` |
| [Tiingo End-of-Day](https://www.tiingo.com/documentation/end-of-day) | US equity, mutual-fund, and ETF prices | REST API | API key; plan-dependent limits | Not documented as a complete dead-fund universe | `TiingoProvider` |
| [Tiingo Fund and ETF Fees](https://www.tiingo.com/documentation/mutual-fund-and-etf-fees) | Current and historical fee records for funds and ETFs | Enterprise data delivery | Institutional agreement | Fee history is a separate product from prices and holdings | `TiingoProvider` for prices only |
| [EODHD](https://eodhd.com/financial-apis/stock-etfs-fundamental-data-feeds) | ETF prices plus plan-dependent holdings and fund fields | REST API | API key; plan-dependent | Holdings dates, classifications, and constituent weights require point-in-time checks | `EODHDProvider` |
| [Massive Stocks](https://massive.com/stocks) | US-listed ETF trades, quotes, aggregates, and reference data | REST, WebSocket, and flat files | API key; plan-dependent | ETF-specific holdings and net asset value are not the same as exchange market data | `MassiveProvider` |
| [Sharadar Fund Prices](https://sharadar.com/docs/funds) | Active and delisted US-listed funds, ETFs, closed-end funds, ETNs, and ETDs from December 1997 | API and bulk delivery | Paid subscription | Daily price product; verify whether a separate dataset is needed for holdings or classifications | No |
| [Norgate Data](https://norgatedata.com/data-package-faq.php) | US ETF and ETN history from 1993, with plan-dependent delisted coverage | Local database and client integrations | Paid subscription | Daily frequency; constituent and classification features depend on package | No |
| [CRSP Survivor-Bias-Free US Mutual Fund Database](https://www.crsp.org/research/) | Active and inactive US open-end mutual funds with fund characteristics and returns | Institutional files, commonly through WRDS | Academic or institutional license | It is a mutual-fund research database, not a complete ETF holdings product | No |

The same ticker can represent different share classes or change its fund objective. Preserve stable
identifiers and effective dates for classifications, fees, and holdings rather than joining only on
the current ticker.

## Related References

- [Equity data sources](equities.md)
- [Fixed-income data sources](fixed_income.md)
- [Market data selection](market_data.md)
27 changes: 27 additions & 0 deletions docs/providers/factors.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# Research Factor Data Sources

Factor libraries publish returns from research portfolios, not returns from investable products.
Choose a source by universe, formation method, weighting, frequency, and revision policy. Do not
combine similarly named factors without reconciling their construction.

!!! note "Verified September 2026"
This category has enough distinct, maintained sources for a standalone reference. Entries were
checked against the linked official or author-maintained data page.

| Source and product | Coverage or history | Acquisition | Access | Material research caveat | ml4t-data provider |
|--------------------|---------------------|-------------|--------|--------------------------|--------------------|
| [Kenneth French Data Library](https://mba.tuck.dartmouth.edu/pages/faculty/ken.french/data_library.html) | US and international factors, sorted portfolios, industries, and breakpoints; core US monthly series begin in 1926 | Downloadable ZIP, CSV, and text files | Free | Histories can be reconstructed after upstream CRSP revisions; archive vintages when reproducibility matters | `FamaFrenchProvider` |
| [AQR Data Library](https://www.aqr.com/Insights/Datasets) | Equity and cross-asset value, momentum, quality, low-beta, trend, and long-history research portfolios | Downloadable spreadsheets | Free under dataset terms | Series are hypothetical research portfolios, not AQR product returns; methodology differs by paper | `AQRFactorProvider` |
| [Global Factor Data](https://jkpfactors.com/data) | 153 characteristics across 93 countries and four regions, with portfolio sorts and reference files | Configurable file downloads; stock-level data through WRDS | Factor returns free under CC BY-NC 4.0; stock-level access requires WRDS | Noncommercial data license and global data-screening choices constrain reuse and comparison | No |

`FamaFrenchProvider.fetch()` selects supported factor, portfolio, industry, international, or
breakpoint datasets. `AQRFactorProvider.download()` acquires the spreadsheets and `fetch()` reads a
supported local dataset. Neither class computes live portfolio returns.

## Selection Notes

- Use Fama-French for canonical academic benchmarks and portfolio sorts.
- Use AQR for paper-specific equity and cross-asset premia such as QMJ, BAB, VME, and TSMOM.
- Use Global Factor Data for broad characteristic definitions and consistent country coverage.
- Store the downloaded file and retrieval date because published research histories may be
revised.
1 change: 1 addition & 0 deletions docs/providers/fama_french.md
Original file line number Diff line number Diff line change
Expand Up @@ -110,6 +110,7 @@ ff5_mom = provider.fetch_combined(["ff5", "mom"])

## See Also

- [Research factor data sources](factors.md)
- [Ken French Data Library](https://mba.tuck.dartmouth.edu/pages/faculty/ken.french/data_library.html)
- [AQR Provider](aqr.md)
- [Provider reference](index.md)
1 change: 1 addition & 0 deletions docs/providers/finnhub.md
Original file line number Diff line number Diff line change
Expand Up @@ -74,5 +74,6 @@ Get your API key at [finnhub.io/register](https://finnhub.io/register).

## See Also

- [Equity](equities.md) and [fundamental](fundamentals.md) source references
- [Finnhub Pricing](https://finnhub.io/pricing)
- [Provider reference](index.md)
Loading
Loading