Running a second instance against the same database breaks the job scheduler

app/jobs registers the enabled jobs into an in-process cron.Cron and stores each job's scheduler handle in sys_job.entry_id. The scheduler is per process; entry_id is one shared column. A second instance pointed at the same database therefore does not divide the work, it overwrites it.

The mechanism

app/jobs/jobbase.go:110, on every startup:

err := sysJob.GetList(db, &jobList)     // WHERE status = 2
...
_, err = sysJob.RemoveAllEntryID(db)    // UPDATE sys_job SET entry_id = 0 WHERE entry_id > 0
...
for i := 0; i < len(jobList); i++ {
    sysJob.EntryId, err = AddJob(crontab, j)
    err = sysJob.Update(db, jobList[i].JobId)
}

RemoveAllEntryID (app/jobs/models/sys_job.go:56) is scoped to nothing. Instance B's startup zeroes every id instance A wrote, then writes its own over the top.

The ids collide by construction. cron.EntryID is a counter local to each Cron (c.nextID++, robfig/cron/v3 cron.go:161), so both instances hand out 1, 2, 3… to their own copy of the list — and GetList has no ORDER BY, so the same number can name a different job on each one.

Two symptoms, neither of which logs anything

An enabled job fires once per instance. Every instance registers the whole status = 2 list in its own scheduler. There is no lock and no leader election, so N instances is N executions per tick.

"Stop" in the UI reports success and the job keeps running. SysJob.RemoveJob (app/jobs/service/sys_job.go:21) reads entry_id from the database and calls cron.Remove against the local scheduler. Cron.removeEntry filters the entry list by id and returns nothing (cron.go:347) — an id from another process either matches nothing, or matches the wrong job. The handler then writes entry_id = 0 and returns 200 either way.

That handler has a separate reporting defect at #890; this one is upstream of it and survives fixing it.

Where it bites

Not only Kubernetes. scripts/k8s/deploy.yml pins replicas: 1, but nothing stops a second process under docker-compose, systemd, or a hand-rolled rolling deploy — including, briefly, any deploy that starts the new process before stopping the old one. The overlap is enough: the new process's startup wipes the old one's ids on the way in.

Direction

Leader election on the database — a lease row that one instance holds and heartbeats, with only the holder calling jobs.Setup — is the smallest change that makes replicas > 1 safe. It does not distribute the jobs, and it is not meant to: the HTTP side scales, the scheduler stays single-writer, and entry_id keeps meaning what it means today.

Scoping entry_id per instance and distributing the jobs is the larger version of this, and it needs a migration plus an answer for what happens to a job whose owner disappears mid-run. Worth doing after the first, not instead of it.

Until either lands the constraint should be stated where someone raising a replica count will read it, which is scripts/k8s/deploy.yml.