openbb_sec company facts fetcher never actually caches: re-downloads full companyfacts.json for every statement type
Summary
openbb_sec.utils.company_facts.get_standardized_financials() fetches SEC's full companyfacts JSON per call via a fresh aiohttp_client_cache.CachedSession(expire_after=3600 * 6), but since no cache= argument is passed, aiohttp_client_cache defaults to a brand-new, empty, in-process CacheBackend()/DictCache() on every single call. The 6-hour cache therefore never actually caches anything across calls -- each call gets its own throwaway cache that is discarded when the function returns. Calling obb.equity.fundamental.income(), .balance(), and .cash() for the same symbol -- which all resolve to this same function and the same underlying SEC endpoint -- downloads the identical multi-megabyte payload three separate times instead of once.
Environment
openbb4.7.2openbb-core1.6.13 (latest available on PyPI at time of filing)openbb-sec1.6.7 (latest available on PyPI at time of filing)- Python 3.12, Linux
Root cause
openbb_sec/utils/company_facts.py, get_standardized_financials() -> _fetch():
async def _fetch(cik_str: str) -> dict:
url = f"https://data.sec.gov/api/xbrl/companyfacts/CIK{cik_str}.json"
if use_cache:
from aiohttp_client_cache.session import CachedSession
async with CachedSession(expire_after=3600 * 6) as session:
try:
resp = await amake_request(url, headers=HEADERS, session=session, timeout=300)
finally:
await session.close()
else:
resp = await amake_request(url, headers=HEADERS, timeout=300)
...
CachedSession.__init__ (in aiohttp_client_cache/session.py) does:
self.cache = cache or CacheBackend()
No cache= is ever passed at the call site above, so every invocation gets a fresh CacheBackend(), whose responses store is a plain in-process DictCache() (see aiohttp_client_cache/backends/base.py). That cache is scoped to the CachedSession instance, which is created and thrown away inside _fetch() on every call -- nothing persists between calls, so the 6-hour expire_after never gets a chance to produce a cache hit.
income_statement.py, balance_sheet.py, and cash_flow.py (same directory) each independently call get_standardized_financials() for their own statement type, with no shared cache or shared fetch between them, even though all three read the same underlying companyfacts JSON and simply project different fields from it.
Reproduction / evidence
The endpoint itself, fetched directly, confirms the payload size involved:
$ curl -sS -H "User-Agent: test [email protected]" \
"https://data.sec.gov/api/xbrl/companyfacts/CIK0000320193.json" | wc -c
3789099
$ curl -sS -H "User-Agent: test [email protected]" \
"https://data.sec.gov/api/xbrl/companyfacts/CIK0000019617.json" | wc -c
7927352
~3.8MB and ~7.9MB respectively for these two example companies. Calling obb.equity.fundamental.income(symbol, provider="sec"), then .balance(symbol, provider="sec"), then .cash(symbol, provider="sec") for the same symbol in the same process triggers three independent full downloads of that same payload rather than one -- confirmed by timing each call and observing three separate multi-second-to-multi-minute fetches of comparable duration, rather than two fast cache hits following one real fetch. Larger/longer-filing-history companies (correspondingly larger companyfacts.json payloads) show proportionally worse cumulative cost, since the redundancy multiplies whatever the single-fetch cost already is.
Suggested direction
Either:
- Pass a shared, persistent
cache=(e.g. a module-level or request-scopedCacheBackendinstance, or aSQLiteBackendlike the Form 4 fetcher's own on-disk cache) intoCachedSession(...)in_fetch(), so the 6-hour TTL actually spans multiple calls, not just the lifetime of oneasync withblock; or - Fetch
companyfactsonce per (symbol, session) and letincome_statement.py/balance_sheet.py/cash_flow.pyshare that single result rather than each independently callingget_standardized_financials()end-to-end.
Either would cut real SEC bandwidth/time cost for a common usage pattern (fetching all three statements for one company) by roughly two-thirds.
Happy to provide more detail if useful.
Source: OpenBB-finance/OpenBB