CCH restart does not recover LND outgoing success/preimage after downtime
Summary
A CCH send_btc order can remain stuck in OutgoingInFlight if the LND outgoing payment succeeds while CCH is down.
In this case LND has already completed the outgoing payment and learned the preimage, but after CCH restarts the order does not recover the terminal LND payment state, does not transition to OutgoingSuccess, and never settles the Fiber incoming invoice.
Impact
The cross-chain swap can get stuck after the BTC/Lightning outgoing leg has succeeded. The Fiber incoming invoice remains unsettled even though the preimage is available in LND. The payer-side Fiber payment stays inflight and can eventually fail/stall while CCH has already paid the outgoing BTC invoice.
Reproducer
The following regression test waits until the order reaches OutgoingInFlight, stops CCH/Fiber node1, settles the destination LND hold invoice while CCH is down, verifies the LND invoice is SETTLED, then restarts CCH and waits for the order to become Success.
def test_cch_r004_send_btc_recovers_lnd_success_during_cch_downtime(self):
"""CCH-R004 / CCH-T006.
If LND outgoing payment succeeds while CCH is down, restart must recover
the terminal LND payment/preimage and settle the Fiber incoming invoice.
"""
self.open_wrapped_btc_channel_to_cch()
preimage = self.generate_random_preimage()
payment_hash = sha256_hex(preimage)
lnd_invoice = self.LNDs[1].addholdinvoice(
payment_hash.replace("0x", ""),
1000,
"CCH-R004 send_btc restart recovery",
)
order = self.fiber1.get_client().send_btc(
{
"btc_pay_req": lnd_invoice["payment_request"],
"currency": "Fibd",
}
)
assert order["payment_hash"] == payment_hash
payment = self.fiber2.get_client().send_payment(
{"invoice": order["incoming_invoice"]["Fiber"]}
)
self.wait_cch_order_state(
self.fiber1, payment_hash, "OutgoingInFlight", timeout=120
)
self.fiber1.stop()
self.LNDs[1].ln_cli_with_cmd(f"settleinvoice {preimage.replace('0x', '')}")
wait_lnd_invoice_state(self.LNDs[1], payment_hash, "SETTLED")
self.restart_cch()
self.wait_cch_order_state(self.fiber1, payment_hash, "Success", timeout=180)
self.wait_payment_state(self.fiber2, payment["payment_hash"], "Success")
incoming_invoice = self.fiber1.get_client().get_invoice(
{"payment_hash": payment_hash}
)
assert incoming_invoice["status"] == "Paid"Expected behavior
After restart, CCH should recover the terminal LND outgoing payment state and preimage for the active order, transition:
OutgoingInFlight -> OutgoingSuccess -> SuccessThen it should settle the Fiber incoming invoice, so the Fiber payer payment reaches Success and the incoming invoice becomes Paid.
Actual behavior
The test times out waiting for CCH order Success. The order remains stuck in OutgoingInFlight after restart.
Relevant log timeline from one failing run:
2026-06-29T08:03:57.713774Z DEBUG SendLightningOutgoingPaymentExecutor request payment_hash=c41be710f7f5d9ebbf6301908f0d61cb950d587e9abfec52a6a44bcfe0c611e1 timeout_seconds=60 cltv_limit=90 fee_limit_sat=0
2026-06-29T08:03:57.744549Z DEBUG SendLightningOutgoingPaymentExecutor response payment_hash=c41be710f7f5d9ebbf6301908f0d61cb950d587e9abfec52a6a44bcfe0c611e1 status=InFlight has_payment_preimage=false
2026-06-29T08:03:57.744810Z DEBUG tracking event PaymentChanged { payment_hash: Hash256(0xc41be710f7f5d9ebbf6301908f0d61cb950d587e9abfec52a6a44bcfe0c611e1), status: Inflight, failure_reason: None, has_payment_preimage: false }CCH is then stopped. While CCH is down, LND completes the payment and learns the preimage:
2026-06-29 16:03:59.509 [INF] SRVR: Adding preimage=c4027bea755fbc03ab1410fd04db02ad152e04d46f6c9082ac287873cabb5fd1 to witness cache for c41be710f7f5d9ebbf6301908f0d61cb950d587e9abfec52a6a44bcfe0c611e1After CCH restart, it resumes the active order, but no PaymentChanged Success event is observed and no Fiber incoming settlement is dispatched:
2026-06-29T08:04:00.727795Z DEBUG PaymentTracker: will connect https://localhost:10009/
2026-06-29T08:04:00.728145Z DEBUG Resumed tracking for active order c41be710f7f5d9ebbf6301908f0d61cb950d587e9abfec52a6a44bcfe0c611e1The Fiber payer side later remains inflight and stops:
2026-06-29T08:04:57.708959Z WARN Payment Hash256(0xc41be710f7f5d9ebbf6301908f0d61cb950d587e9abfec52a6a44bcfe0c611e1) is still not final after periodic check, maybe the channel is down. Status: Inflight, Active attempts: 1, Inflight: 1, Failed: 0, Total: 1, Retry count: 0, Last error: NoneLikely root cause
On CCH startup, active orders are resumed via ActionDispatcher::on_starting. For OutgoingInFlight, this dispatches TrackOutgoingPayment.
However, TrackOutgoingPaymentDispatcher currently does nothing:
// `CchActor` will track all fiber payments, so there's nothing to do here to track a single payment.
NoneFor LND outgoing payments, CCH relies on the global PaymentTracker, which calls LND TrackPayments(no_inflight_updates=true). LND's global TrackPayments / SubscribeAllPayments only replays currently inflight payments when a new subscription starts. It does not replay payments that already reached a terminal state while CCH was down.
LND's per-payment subscription (TrackPaymentV2 / SubscribePayment(payment_hash)) does return the current payment state immediately, including terminal succeeded payments and their preimage. CCH appears to need that per-payment recovery path for active Lightning outgoing orders.
Suggested fix direction
When resuming an active CCH order with a Lightning outgoing leg, especially in OutgoingInFlight, TrackOutgoingPayment should query/subscribe to LND by the order's payment_hash using TrackPaymentV2 or equivalent. If LND returns Succeeded with a preimage, CCH should emit:
PaymentChanged { status: Success, payment_preimage: Some(preimage) }The existing CCH state machine can then move the order to OutgoingSuccess and dispatch SettleIncomingInvoice to settle the Fiber incoming invoice.
Version observed
fnn git version 15399ac6 (15399ac 2026-06-26)
version: 0.9.0-rc4Source: nervosnetwork/fiber