panic: pgid (6348659008741242810) above high water mark (236003)
Author: vansanteCreated May 23, 2017Updated Mar 14, 2018
Hi,
We are in the process of replacing a MySQL database with a bolt database, and after inserting a lot of records (we are estimating in the range of 10 million) into a single boltdb file we ran into the following panic:
panic: pgid (6348659008741242810) above high water mark (236003)
goroutine 6170 [running]:
project/vendor/github.com/boltdb/bolt.(*node).spill(0xc42021bd50, 0xc420309168, 0x10)
/opt/go/src/project/vendor/github.com/boltdb/bolt/node.go:375 +0x631
project/vendor/github.com/boltdb/bolt.(*Bucket).spill(0xc4204c9a00, 0xc4203d6000, 0xc420309538)
/opt/go/src/project/vendor/github.com/boltdb/bolt/bucket.go:570 +0x17b
project/vendor/github.com/boltdb/bolt.(*Bucket).spill(0xc420200718, 0x145ddfdb, 0x100a9a0)
/opt/go/src/project/vendor/github.com/boltdb/bolt/bucket.go:537 +0x8f8
project/vendor/github.com/boltdb/bolt.(*Tx).Commit(0xc420200700, 0x0, 0x0)
/opt/go/src/project/vendor/github.com/boltdb/bolt/tx.go:163 +0x125
project/vendor/github.com/boltdb/bolt.(*DB).Update(0xc4201c83c0, 0xc420309770, 0x0, 0x0)
/opt/go/src/project/vendor/github.com/boltdb/bolt/db.go:605 +0xf7
project/vendor/github.com/asdine/storm.(*node).readWriteTx(0xc4204c9780, 0xc4205a7360, 0xc42017da50, 0xc420832a70)
/opt/go/src/project/vendor/github.com/asdine/storm/node.go:113 +0x107
project/vendor/github.com/asdine/storm.(*node).Set(0xc4204c9780, 0xad93e8, 0xa, 0x9f5b00, 0xc42017da40, 0x9f5b00, 0xc42017da50, 0xc4203098e8, 0xc42017da50)
/opt/go/src/project/vendor/github.com/asdine/storm/kv.go:72 +0x19d
project/vendor/github.com/asdine/storm.(*DB).Set(0xc42021bb20, 0xad93e8, 0xa, 0x9f5b00, 0xc42017da40, 0x9f5b00, 0xc42017da50, 0xfd9440, 0xc42012e360)
/opt/go/src/project/vendor/github.com/asdine/storm/storm.go:147 +0x75
project/vendor/github.com/asdine/storm.(*DB).checkVersion(0xc42021bb20, 0x3e, 0x180)
/opt/go/src/project/vendor/github.com/asdine/storm/storm.go:276 +0x236
project/vendor/github.com/asdine/storm.Open(0xc4204c9740, 0x3e, 0xc420309a10, 0x1, 0x1, 0x0, 0xc420309a18, 0x58b6b3)
/opt/go/src/project/vendor/github.com/asdine/storm/storm.go:56 +0x261
project/database/boltdb.ConstructDB(0xc4204c9740, 0x3e, 0xc4205a7310, 0x9f5b00, 0xc42017d930, 0xc4205a7310)
/opt/go/src/project/database/boltdb/db.go:25 +0x8dWe make use of the Storm ORM (https://github.com/asdine/storm) and this panic happened when attempting to reopen the database we just migrated and wrote the above number of records into. Storm then tried to set its version into a bucket, triggering this panic.
Do you have any idea what could cause this and how we can prevent it in the future? (I am posting this here because after tracing the error it seems to occur deep in boltdb and it seems unrelated to Storm)
Thanks
Source: boltdb/bolt