[CHIA-4352] [Bug] Crawler: a gossiped peer timestamp at or above 2**63 aborts load_to_db and silently stops the good_peers refresh
What happened?
Crawler.save_to_db can stop refreshing the good_peers table for the rest of the process
lifetime, leaving the DNS server answering from a snapshot that never updates. The operator sees
one log line per attempt and nothing else.
Four things combine.
1. The gossiped timestamp is not range checked. RespondPeers carries a bare uint64 per
entry, and the crawler keeps the maximum without a bound:
# chia/seeder/crawler.py:233-236
if response_peer.host not in self.best_timestamp_per_peer:
self.best_timestamp_per_peer[response_peer.host] = response_peer.timestamp
self.best_timestamp_per_peer[response_peer.host] = max(
self.best_timestamp_per_peer[response_peer.host], response_peer.timestamp
)The value then becomes the record's best_timestamp at chia/seeder/crawler.py:252. The full
node's handler for the same message does clamp it:
# chia/server/node_discovery.py:476
if peer.timestamp < 100000000 or peer.timestamp > time.time() + 10 * 60:
# Invalid timestamp, predefine a bad one.
current_peer = TimestampedPeerInfo(peer.host, peer.port, uint64(time.time() - 5 * 24 * 60 * 60))2. SQLite cannot bind the value. add_peer binds best_timestamp as a parameter:
uint64(2**63 - 1) binds OK
uint64(2**63) OverflowError: Python int too large to convert to SQLite INTEGER
uint64(2**64 - 1) OverflowError: Python int too large to convert to SQLite INTEGER3. One unusable row ends the whole batch. chia/seeder/crawl_store.py load_to_db has no
per-row guard, so the first record that cannot bind ends the function and its commit() is never
reached.
4. The raise skips the next line. In chia/seeder/crawler.py save_to_db:
for i in range(1, 5):
try:
await self.crawl_store.load_to_db()
await self.crawl_store.load_reliable_peers_to_db()
return
except Exception as e:
self.log.error(f"Exception while saving to DB: {e}.")
await asyncio.sleep(5)
continueload_reliable_peers_to_db (chia/seeder/crawl_store.py:317) is the only writer of the
good_peers table, and chia/seeder/dns_server.py reads that table for every answer it gives.
Because the two calls sit on consecutive lines inside one try, the throw on the first skips the
second. All four retries hit the same record and the batch is abandoned.
The record holding the value stays in host_to_records, so every later save_to_db fails the
same way.
Reproduction
No network needed. Save as repro.py at the repo root and run it with the venv python:
import asyncio, os, tempfile, time
import aiosqlite
from chia_rs.sized_ints import uint32, uint64
from chia.seeder.crawl_store import CrawlStore
from chia.seeder.peer_record import PeerRecord, PeerReliability
def rec(host, best_ts):
return PeerRecord(host, host, uint32(8444), False, uint64(0), uint32(0), uint64(0),
uint64(int(time.time())), uint64(best_ts), "undefined", uint64(0),
tls_version="unknown")
async def main():
db = os.path.join(tempfile.mkdtemp(), "crawler.db")
conn = await aiosqlite.connect(db)
store = await CrawlStore.create(conn)
now = int(time.time())
for i in range(3):
h = f"10.0.0.{i}"
r = PeerReliability(h); r.update(True, 1)
store.maybe_add_peer(rec(h, now), r)
await store.load_to_db()
await store.load_reliable_peers_to_db()
print(f"before: good_peers = {len(await store.get_good_peers())}")
# one gossiped entry, timestamp straight off the wire
h = "10.9.9.9"
r = PeerReliability(h); r.update(True, 1)
store.maybe_add_peer(rec(h, 2**64 - 1), r)
try:
await store.load_to_db()
except Exception as e:
print(f"load_to_db: {type(e).__name__}: {e}")
print(f"after: good_peers = {len(await store.get_good_peers())} (load_reliable_peers_to_db never ran)")
await conn.close()
asyncio.run(main())Output:
before: good_peers = 3
load_to_db: OverflowError: Python int too large to convert to SQLite INTEGER
after: good_peers = 3 (load_reliable_peers_to_db never ran)The count staying at 3 is the point: good_peers keeps whatever it last held and is never
rewritten, so the DNS server keeps answering from that snapshot.
Suggested fix
- Clamp the gossiped timestamp where it enters
chia/seeder/crawler.py, the waychia/server/node_discovery.py:476already does for the full node. - Guard the row loop in
load_to_dbso one unusable record cannot end the batch, and log which record it was. Without this, any future field that can exceed SQLite's range has the same effect. - Give
load_reliable_peers_to_dbits owntryinsave_to_db, or move it out from behindload_to_db. Thegood_peerstable going stale should not be a silent side effect of a failed write to a different table.
Version
2.7.3, and current main (checked at 67cdfcda4). chia/seeder/crawler.py,
chia/seeder/crawl_store.py and chia/server/node_discovery.py are identical in both for the
lines above.
What platform are you using?
Linux
What ui mode are you using?
CLI
Relevant log output
chia.seeder.crawl_store: WARNING Saving peers to DB...
chia.seeder.crawler : ERROR Exception while saving to DB: Python int too large to convert to SQLite INTEGER.
chia.seeder.crawler : ERROR Waiting 5 seconds before retry...
chia.seeder.crawl_store: WARNING Saving peers to DB...
chia.seeder.crawler : ERROR Exception while saving to DB: Python int too large to convert to SQLite INTEGER.
chia.seeder.crawler : ERROR Waiting 5 seconds before retry...Source: Chia-Network/chia-blockchain