Order tracking - v2 strategy orders accepted by the exchange are dropped during a network outage, leaving an untracked position

Describe the bug

While monitoring a long-running bot, six sell orders were accepted by the exchange and received exchange order IDs before a DNS outage interrupted status, trade-update, and cancellation requests. Hummingbot later marked these orders as OrderFailure and removed them from lost-order tracking. The removal happened immediately after another DNS error, not after the exchange confirmed that the orders were cancelled, filled, or not found.

After the bot was stopped, the exchange account showed an accumulated 0.0006 BTC short position that remained open. This exactly matches the combined size of the six lost sell orders (6 x 0.0001 BTC), while SQLite contains no TradeFill for any of them. This is strong evidence that the lost orders created untracked exchange exposure, although the exchange order and trade history is still needed to map each fill conclusively to the six order IDs.

The behavior appears to come from shared Hummingbot order-tracking code rather than the kalshi_perpetual connector or the V2 controller. The active-order status handler sends generic exceptions, including network errors, to the order-not-found counter. If another polling task has already moved the order into lost-order tracking, a later network exception can remove it as though the exchange had confirmed it was missing. The same affected code is present on the tested branch, origin/development, and origin/master; Kalshi was only the connector where the behavior was observed.

The following sequence shows one order being accepted with an exchange order ID, receiving repeated DNS errors, being marked failed locally, and then being removed as “not found” immediately after another DNS error:

2026-09-12 11:14:05,411 - 1891380 - hummingbot.core.event.event_reporter - EVENT_LOG - {"timestamp": 1789182845.0, "type": "OrderType.LIMIT_MAKER", "trading_pair": "BTC-USD", "amount": "0.000100", "price": "77328", "order_id": "HBOTSBCUD65b4097ec9796097bcd04a7b4df", "creation_timestamp": 1789182845.0, "exchange_order_id": "01a0939b-a048-7f47-b45a-0bf74612a485", "leverage": 3, "position": "OPEN", "event_name": "SellOrderCreatedEvent", "event_source": "kalshi_perpetual"}
2026-09-12 11:17:20,538 - 1891380 - hummingbot.connector.derivative.kalshi_perpetual.kalshi_perpetual_derivative.KalshiPerpetualDerivative - WARNING - Error fetching status update for the active order HBOTSBCUD65b4097ec9796097bcd04a7b4df: Cannot connect to host external-api.kalshi.com:443 ssl:default [Temporary failure in name resolution].
2026-09-12 11:17:20,538 - 1891380 - hummingbot.connector.client_order_tracker - WARNING - The order HBOTSBCUD65b4097ec9796097bcd04a7b4df(01a0939b-a048-7f47-b45a-0bf74612a485) will be considered lost. Please check its status in the exchange.
2026-09-12 11:17:20,546 - 1891380 - hummingbot.core.event.event_reporter - EVENT_LOG - {"timestamp": 1789183040.0, "order_id": "HBOTSBCUD65b4097ec9796097bcd04a7b4df", "order_type": "OrderType.LIMIT_MAKER", "error_message": null, "error_type": null, "event_name": "MarketOrderFailureEvent", "event_source": "kalshi_perpetual"}
2026-09-12 11:17:45,116 - 1891380 - hummingbot.connector.derivative.kalshi_perpetual.kalshi_perpetual_derivative.KalshiPerpetualDerivative - WARNING - Error fetching status update for the active order HBOTSBCUD65b4097ec9796097bcd04a7b4df: Cannot connect to host external-api.kalshi.com:443 ssl:default [Temporary failure in name resolution].
2026-09-12 11:17:45,116 - 1891380 - hummingbot.connector.client_order_tracker - INFO - The lost order HBOTSBCUD65b4097ec9796097bcd04a7b4df(01a0939b-a048-7f47-b45a-0bf74612a485) was not found and will be removed

The same DNS-error-to-removal sequence occurred for five more exchange-acknowledged orders in the same log:

• HBOTSBCUD65b40adf9b869097bcd04a7b4df at 11:23:48.635, lines 16941-16942
• HBOTSBCUD65b41192bc107097bcd04a7b4df at 11:53:29.881, lines 28598-28599
• HBOTSBCUD65b417078fc24097bcd04a7b4df at 12:17:46.013, lines 37849-37850
• HBOTSBCUD65b419b4787f3097bcd04a7b4df at 12:29:43.459, lines 47117-47118
• HBOTSBCUD65b45b5cc0cd8097bcd04a7b4df at 17:23:25.216, lines 57837-57838

None of these removal sequences contains an exchange 404 or not_found response. Each removal is immediately preceded by Temporary failure in name resolution.

The rate-limit budget drains while offline ❗

This looks like the same underlying pattern as the main issue where the polling loop keeps firing requests without knowing the bot is offline. One consequence is orders being dropped from tracking. Another is the rate limit.

• While the network was blocked, the throttler kept counting requests that could not leave the machine. An attempt costs its tokens whether or not it is actually sent, so the read budget drained with nothing reaching the exchange: API rate limit on PerpsReadBucket (200 calls per 1s) has almost reached. Limits used is 160 in the last 1 seconds
• That line appeared 20 times, and twice the budget was fully used at 200 of 200, once at the exact moment connectivity returned.

The bot retries about twice a second because _status_polling_loop (exchange_py_base.py ~line 814) waits half a second after a failure and tries again, without ever checking whether the network is back. The rate-limit tokens are taken before the request is sent, in AsyncRequestContextBase.acquire() (async_request_context_base.py:68), so a request that never leaves the machine still costs its tokens and nothing is given back.

Steps to reproduce

1. Pick a connector that does not override _handle_update_error_for_active_order. Connectors that do override it are not affected.
2. Place a limit order and confirm the exchange gave it an order ID.
3. Cut the network for that bot only, leaving it running and not restarted. Blocking by cgroup works well and leaves the rest of the machine online.
4. Keep the network down long enough for the bot to give up on the order.
5. Look for The lost order ... was not found and will be removed right after a network error, with no 404 or not_found from the exchange.
6. Restore the network and compare the local Order, OrderStatus and TradeFill records against the exchange's order history.
7. Stop the bot and check whether the exchange still holds a position the local records do not explain.

Release version

dev-2.17

Type of installation

Source

Attach required files

logs_11092026_conf_generic_pmm_mister_pmmmister01.log.2026-09-11.log 11092026_pmmmister01.yml