#588·Bindu

[Bug]: manifest_worker fails closed on transient facilitator settle timeouts instead of flagging payment-orphaned

Author: MeenbudhaCreated Jun 25, 2026Updated Jun 25, 2026
Labelsbug

Describe the bug clearly

During periods of network congestion on EVM chains (e.g., Base mainnet/Sepolia), block confirmation times can exceed the facilitator client's internal timeout (typically 5–10 seconds). When the facilitator's /settle endpoint times out or fails during transmission, the _settle_payment method in manifest_worker.py logs a settlement failure and returns metadata marking the task payment status as payment-failed.

However, since the signed EIP-3009 authorization payload was already broadcasted, the transaction can still successfully confirm on-chain shortly after the timeout. When this happens:

  1. The client's wallet is debited.
  2. The agent's wallet is credited.
  3. The agent's task record is permanently stuck as failed (with status payment-failed rather than payment-orphaned), meaning the user paid for a service that will never execute, and no alert is triggered for the operator.

To Reproduce

  1. Configure a Bindu agent with x402 payments enabled on a live network (e.g., Base Sepolia).
  2. Initiate a task request to the agent.
  3. Simulating network congestion, introduce a mock delay or force the HTTP facilitator client to return a timeout exception during /settle.
  4. Observe the console log registering the settlement error and transitioning the task state to failed with payment-failed metadata.
  5. Check the target EVM block explorer or the agent's wallet to observe that the transaction succeeded on-chain, indicating a successful credit transfer.

Expected behavior

  • The system should distinguish between a facilitator network timeout (transient) and a hard API rejection (permanent).
  • If a facilitator /settle call fails or times out, the task should eventually be picked up by a background reconciliation worker. This worker should check the blockchain for the AuthorizationUsed event matching the saved EIP-3009 nonce, updating the status to payment-orphaned or triggering an automated/manual refund.

Screenshots or logs

log
[21:30:40] INFO     Starting deployment for agent: 438b4815-7ebe-d853-b95d-48b32b68fa3a
...
[21:30:52] ERROR    Error settling payment: HTTPFacilitatorClient.settle timed out after 10.0s
[21:30:52] INFO     Payment settlement failed; task not executed.

Environment

  • OS: Windows 10 / POSIX
  • Python version: Python 3.12+ (tested on Python 3.14.0)
  • Package version: bindu core v0.3.15.dev1519

Additional context The root of the issue is in bindu/server/workers/manifest_worker.py (inside the _settle_payment and _handle_settlement_failure methods). When _settle_payment encounters an exception (such as a timeout), it falls back to:

python
        except Exception as e:
            logger.opt(exception=True).error("Error settling payment: {}", e)
            return _failure_metadata(str(e))

This returns _failure_metadata, which tags the status as payment-failed instead of payment-orphaned. The worker has no secondary mechanism to check if the transaction is pending or eventually confirmed.


Suggested Fixes & Improvements

  1. Reconciliation Loop (Detection): Implement a background reconciliation worker (similar to ManifestWorker but scheduled to run every few minutes). This worker should:

    • Query the task storage for tasks marked payment-failed within the last 24 hours.
    • Parse the stored EIP-3009 metadata (x402_nonce, x402_authorization.from, and x402_network).
    • Perform an eth_getLogs RPC query on the target USDC contract to search for the corresponding AuthorizationUsed event.
    • If the transaction was completed, transition the task's payment status to payment-orphaned to signal the operator.
  2. Facilitator Polling / Increased Timeout: Introduce a configurable retry or validation step in _settle_payment for timeouts. Before classifying the payment as failed, the client could poll the chain's state to confirm if the signature/nonce was already consumed on-chain.