ALTER DATABASE ... READ ONLY finishes synced without publishing a schema version when the owner's value is unchanged, leaving other TiDB nodes permanently divergent

Bug Report

1. Minimal reproduce step (Required)

Branch feature/release-8.5-read_only @ 48e340f5bf7fe4a19c2763fa2bf41df3c013536a (tracking issue #62122).

A. Unit assertion

Call onModifySchemaReadOnly with dbInfo.ReadOnly == args.ReadOnly:

observed: ver == 0, no schema diff written, job still reports State:synced expected: ver != 0, a diff is published

B. Pure SQL, 2 TiDB nodes

  CREATE DATABASE d;
  CREATE TABLE d.t(id INT PRIMARY KEY, v VARCHAR(16));
  ALTER DATABASE d READ ONLY = 1;      -- both nodes now cache READ ONLY=1

Then fire ALTER DATABASE d READ ONLY = 0 concurrently from two different TiDB nodes. Two jobs are created, only one allocates a schema version:

  JOB_ID  JOB_TYPE                   STATE
  116     modify schema read only    synced   <- published NOTHING
  115     modify schema read only    synced   <- real work, ver 58 -> 59

[job_worker.go:1124] "schema version doesn't change" jobID=116

N jobs, N-1 schema versions. A serial redundant ALTER creates no job at all, because the executor-side check (pkg/ddl/executor.go:428) short-circuits against the submitting node's cached InfoSchema before a job is ever submitted.

C. End-to-end — how it becomes a permanent outage

Reproduced twice on independent clusters, most recently on a freshly built one with no prior history: PD 1 + TiKV 1 (stock pingcap/{pd,tikv}:v8.5.8, max-replicas=1) + TiDB ×2 running the branch binary (Git Branch: feature/release-8.5-read_only, Git Commit Hash: 48e340f5bf).

1. CREATE DATABASE d + table; ALTER DATABASE d READ ONLY = 1. Both nodes report READ ONLY=1.
2. Start a log backup, take a snapshot while still locked (snapshot carries ReadOnly=1).
3. ALTER DATABASE d READ ONLY = 0 — captured in the log. Wait for the log checkpoint to pass it.
4. Drop the database, then PiTR restore.

The restore rewrites the database meta to ReadOnly=0 out of band — no DDL job, no diff, no per-database version bump. If nothing forces the nodes to re-read that database's meta, they keep the cached ReadOnly=1 indefinitely (see §3 for why that happens in production). The corrective unlock is then a silent no-op:

after restore both nodes READ ONLY=1, INSERT -> ERROR 3989, schema version 104 ALTER DATABASE d READ ONLY = 0; -- no error, no warning schema version 104 -> 104 <- NO VERSION ALLOCATED

Retrying does not help. Four unlock attempts produced four jobs, four log lines, zero version increments:

  [job_worker.go:1124] "schema version doesn't change" jobID=159
  [job_worker.go:1124] "schema version doesn't change" jobID=160
  [job_worker.go:1124] "schema version doesn't change" jobID=161
  [job_worker.go:1124] "schema version doesn't change" jobID=162

Full lifecycle of one of them — note State:synced, Err::

  [job_worker.go:830]  "run DDL job"                   jobID=159
  [job_worker.go:1124] "schema version doesn't change" jobID=159
  [job_worker.go:423]  "finish DDL job"                jobID=159 job="ID:159, Type:modify schema read only, State:synced, SchemaState:public, SchemaID:126, TableID:0, RowCount:0, ArgLen:0, Err:<nil>, ErrCount:0, Version: v2"
  [executor.go:6795]   "DDL job is finished"           jobID=159

Starting a fresh process for one node heals that node and nothing else:

tidb-0 (never restarted) READ ONLY=1 INSERT -> ERROR 3989 tidb-1 (fresh process) writable INSERT -> OK both at schema version 104

The node that executed the job was itself the DDL owner (OWNER_ID == SELF_ID, and "run DDL job" appears in its log) and the node that stayed divergent — it read the meta, published nothing, and remained inconsistent with what it had just read. Owner identity is not a factor in this bug.

Alternate Repro: BR's log restore replays ALTER DATABASE ... READ ONLY by writing the DB meta directly. On a base carrying the filtered refresh-meta path (RefreshMetaForTables → applyRefreshMeta), a filtered restore emits per-object refresh diffs instead of the full-reload marker, and applyRefreshMeta in pkg/infoschema/builder.go reconciles only charset/collate and placement for a schema that already exists:

  needsCharsetUpdate   := currentSchema.Charset != dbInfo.Charset || currentSchema.Collate != dbInfo.Collate
  needsPlacementUpdate := !equalPlacementPolicy(currentSchema.PlacementPolicyRef, dbInfo.PlacementPolicyRef)

ReadOnly appears nowhere in that file, so the flag is never refreshed. Nodes keep the cached value and the corrective ALTER then no-ops per the mechanism above.

Scope note. This branch predates applyRefreshMeta / RefreshMetaForTables (zero hits at 48e340f5bf), and RestoreMetaKVFiles is called unconditionally here (br/pkg/task/stream.go:1433), so the full-reload marker always fires on this branch and the out-of-band window must be created another way — repro B, or repro C with the marker suppressed. The applyRefreshMeta gap is an integration issue for whichever base this feature merges onto, and should be fixed alongside.

Customer impact. ~6 hour outage on two production clusters: 2 of 3 TiDB nodes rejected all writes to the affected databases with ERROR 3989 while the meta said writable. Four unlock statements produced two schema-version increments; the two that no-op'd are exactly the two databases that stayed read-only. Their one healthy node was the one that had independently restarted.

2. What did you expect to see? (Required)

ALTER DATABASE ... READ ONLY = should leave every TiDB node agreeing with the persisted meta, whether or not the owner's own value changed. A node whose cached InfoSchema disagrees with the meta should be corrected by the next ALTER, and an operation that reports State:synced should have published its effect cluster-wide.

3. What did you see instead (Required)

The statement reports success while publishing nothing, so divergent nodes are never corrected and stay divergent until the process restarts.

4. What is your TiDB version? (Required)

  Release Version: v8.5.0-feature
  Edition: Community
  Git Commit Hash: 48e340f5bf
  Git Branch: feature/release-8.5-read_only
  GoVersion: go1.23.12