openbb_sec Form 4 (insider trading) fetcher leaks sockets in CLOSE_WAIT under concurrent downloads

Summary

openbb_core.provider.utils.helpers.amake_request() opens a brand-new aiohttp.ClientSession for every individual HTTP request rather than reusing one across a batch. openbb_sec's Form 4 (insider trading) fetcher (openbb_sec/utils/form4.py::download_data()) calls this once per filing URL, in concurrent batches of 8 (asyncio.Semaphore(8) + asyncio.gather). Under that concurrent load, sockets accumulate in CLOSE_WAIT with unread response data still sitting in the kernel receive buffer, and per-call latency degrades noticeably as a run progresses.

Environment

• openbb 4.7.2 (via pip install openbb)
• openbb-core 1.6.13 (latest available on PyPI at time of filing)
• openbb-sec 1.6.7 (latest available on PyPI at time of filing)
• Python 3.12, Linux

Root cause

openbb_core/provider/utils/helpers.py, amake_request():

with_session = kwargs.pop("with_session", "session" in kwargs)
session = kwargs.pop("session", await get_async_requests_session(**kwargs))

try:
    response = await session.request(method, url, **kwargs)
    return await response_callback(response, session)
finally:
    if not with_session:
        await session.close()

Every call that doesn't explicitly pass session=/with_session=True gets a fresh ClientSession via get_async_requests_session(), uses it for exactly one request, then closes it. openbb_sec/utils/form4.py::get_form_4_data() calls amake_request() this way — no shared session — once per filing URL:

response = await amake_request(
    url,
    headers=SEC_HEADERS,
    response_callback=response_callback,
    timeout=30,
)

download_data() (same file) drives this concurrently:

async with asyncio.Semaphore(8):
    for url_chunk in [non_cached_urls[i:i+8] for i in range(0, len(non_cached_urls), 8)]:
        await asyncio.gather(*[get_one(url) for url in url_chunk])
        await asyncio.sleep(1.125)

Creating and tearing down a TLS session per single request is an anti-pattern aiohttp's own docs warn against explicitly ("Don't create a session per request. Most likely you need a session per application which performs all requests together") — see https://docs.aiohttp.org/en/stable/client_reference.html#client-session. Under 8-way concurrent bursts, this repeated session churn appears to leave sockets in CLOSE_WAIT rather than being cleanly reclaimed.

Reproduction / evidence

Running fundamentals + insider-trading ingestion for a list of ~20 tickers (via obb.equity.ownership.insider_trading(ticker, provider="sec"), which drives the Form 4 path above) and inspecting the process's sockets mid-run:

$ ss -tnp | grep <pid>
CLOSE-WAIT 35362  0  <local>:45316   <sec-edge-ip>:443  users:(("python3",pid=...))
CLOSE-WAIT 9318   0  <local>:45330   <sec-edge-ip>:443  users:(("python3",pid=...))
CLOSE-WAIT 50450  0  <local>:45714   <sec-edge-ip>:443  users:(("python3",pid=...))
... (40+ more CLOSE-WAIT entries)

The non-zero Recv-Q values (up to ~50KB) indicate response data was sent by the server but never fully read/drained by the client before the session was closed. A direct curl to the same SEC endpoints during the same window returned in under a second, ruling out a network-path or server-side latency explanation — the client's own connection handling is the bottleneck. Process CPU time stayed low relative to wall-clock time throughout (consistent with time spent waiting on I/O rather than computing), and per-request latency visibly worsened as more CLOSE_WAIT sockets accumulated over the course of the run.

Suggested direction

Reuse a single ClientSession across the batch of concurrent requests in download_data() (pass session=/with_session=True into amake_request()/get_form_4_data() from a session opened once for the whole download_data() call, closed once at the end) rather than opening/closing one per URL. This is the pattern amake_requests() (plural, in the same helpers.py file) appears to already support for other callers — download_data() doesn't seem to use it.

Happy to provide more detail or test a fix if useful.