A start that dies inside `ArchiveMarkFile` / `ClusterMarkFile` after activation strands `ACTIVATION_IN_PROGRESS_TIMESTAMP` in the mark file — every later start fails with `active mark file detected`, and nothing expires it

Summary

ArchiveMarkFile and ClusterMarkFile activate an existing mark file first (Agrona's MarkFile swaps the activity timestamp to ACTIVATION_IN_PROGRESS_TIMESTAMP, Long.MAX_VALUE) and only then save the previous run's error buffer to a new <component>-<date>-error.log beside it. That save rethrows any I/O failure. When it throws, the constructor has not finished, the owning Context has no markFile reference to reset, and the file keeps Long.MAX_VALUE. Every later start then computes now − Long.MAX_VALUE — negative, so below any liveness timeout — and throws active mark file detected. The marker never expires; only editing or deleting the mark file recovers the component.

The ordinary trigger is a full disk: a component that died of a full disk still has errors in its error buffer, so its first restart has to write an error log onto that same full disk. Freeing or expanding the disk afterwards does not help, because the marker is already stranded.

Versions

Observed on Aeron 1.51.0 / Agrona 2.4.1. The constructor order is unchanged on master (links below are pinned to dad28d2c; 1.53.2 has the same shape).

Mechanism

1. ArchiveMarkFile(Archive.Context) activates the existing file — ArchiveMarkFile.java#L122 — which compare-and-sets the activity timestamp to ACTIVATION_IN_PROGRESS_TIMESTAMP (MarkFile.java#L698).
2. Then it saves the previous run's errors — ArchiveMarkFile.java#L151 — through CommonContext.saveExistingErrors, which rethrows any exception (CommonContext.java#L1267-L1298).
3. If that throws, the constructor throws before Archive.Context.conclude() assigns markFile (Archive.java#L1323), so the failure path's if (null != markFile) markFile.signalReady(NULL_VALUE) (Archive.java#L170) has nothing to reset.
4. The next start reads Long.MAX_VALUE, computes a negative age, and throws (MarkFile.java#L693).

ClusterMarkFile has the same shape — activation at ClusterMarkFile.java#L153, saveExistingErrors at #L201 — and its component-type check between the two also throws after activating, with the same result. The consensus module's and service container's signalFailedStart() paths likewise only run once the constructor has returned.

(On 1.51.0 / Agrona 2.4.1 the same lines are ArchiveMarkFile L122 → L151, Archive L1311 and L156–159, ClusterMarkFile L175 → L223, CommonContext L1188–1217, MarkFile L55 / L692 / L697.)

Reproduction

Standalone against 1.51.0, using the public ClusterMarkFile constructor (on master, the variant that also takes filePageSize). The error-log write is made to fail with chattr +i on the directory, standing in for a full disk:

final File dir = Files.createTempDirectory("stranded-mark").toFile();
final File file = new File(dir, ClusterMarkFile.FILENAME);
final EpochClock clock = SystemEpochClock.INSTANCE;

// 1. A run that logged an error and stopped (the mark stamped NULL_VALUE on the way out).
try (ClusterMarkFile mark = new ClusterMarkFile(file, ClusterComponentType.CONSENSUS_MODULE, 1024 * 1024, clock, 10_000))
{
    mark.signalReady(clock.time());
    new DistinctErrorLog(mark.errorBuffer(), clock).record(new RuntimeException("boom"));
    mark.signalReady(Aeron.NULL_VALUE);
}

// 2. The next start dies after activating: the directory refuses the new error-log file.
new ProcessBuilder("chattr", "+i", dir.getPath()).inheritIO().start().waitFor();
try
{
    new ClusterMarkFile(file, ClusterComponentType.CONSENSUS_MODULE, 1024 * 1024, clock, 10_000);
}
catch (final Exception ex)
{
    System.out.println("run 2: " + ex);
}
finally
{
    new ProcessBuilder("chattr", "-i", dir.getPath()).inheritIO().start().waitFor();
}

// 3. The directory is writable again — and every later start is refused, forever.
new ClusterMarkFile(file, ClusterComponentType.CONSENSUS_MODULE, 1024 * 1024, clock, 10_000);

Output:

run 1: mark closed, timestamp=-1
WARNING: existing errors saved to: /tmp/stranded-mark.../CONSENSUS_MODULE-2026-09-18-23-08-34-372+0000-error.log
run 2: java.io.FileNotFoundException: /tmp/stranded-mark.../CONSENSUS_MODULE-2026-09-18-23-08-34-372+0000-error.log (Operation not permitted)
after run 2: activityTimestamp=9223372036854775807   (ACTIVATION_IN_PROGRESS_TIMESTAMP)
run 3: java.lang.IllegalStateException: active mark file detected: /tmp/stranded-mark.../cluster-mark.dat
run 4: java.lang.IllegalStateException: active mark file detected: /tmp/stranded-mark.../cluster-mark.dat

(activityTimestamp read at offset 16: the 8-byte SBE message header, then the field at 8 in the block.)

Impact observed

Three cluster members died together when their archive volume hit the low-storage threshold. On restart, two members failed at Archive.launch on archive-mark.dat — 29 restarts in two hours — and the third, on a fresh pod over the same volume, got its archive and consensus module up and then failed in ClusteredServiceContainer on cluster-mark-service-0.dat. The volume had ~2 GB free by then. Recovery needed a byte-level reset of the marker in each stranded file (checking the value is exactly 7fffffffffffffff first, then writing NULL_VALUE at offset 16).

Suggested fix

Either would do; the first is the smaller change:

1. Roll the activation back when the constructor fails after activating. In both constructors, wrap the work after new MarkFile(file, true, …) and, on failure, restore the timestamp before rethrowing — the same posture Archive's failure path already takes once it holds a reference:

   java复制

   final MarkFile existingMarkFile = new MarkFile(file, true, …);   // timestamp := ACTIVATION_IN_PROGRESS_TIMESTAMP
   try
   {
       // decode header, saveExistingErrors, zero the error buffer, type check …
   }
   catch (final RuntimeException ex)
   {
       existingMarkFile.timestampRelease(NULL_VALUE);   // let the next start activate the file
       CloseHelper.quietClose(existingMarkFile);
       throw ex;
   }

2. Save the existing errors before activating — or make the save best-effort (print the observations through the fallback logger when the file cannot be written), since the error buffer is zeroed right after regardless and a failed archive of old errors should not stop the component from starting.

A regression test for either: an existing mark file with a non-empty error buffer whose directory refuses new files; construct, expect the exception, then assert the activity timestamp is NULL_VALUE rather than ACTIVATION_IN_PROGRESS_TIMESTAMP (and that a second construction succeeds once the directory is writable). ClusterMarkFileTest currently asserts a freshly opened file holds the marker until ready, but nothing covers a failure inside that window.

Workaround

We now run a boot preflight before the archive, consensus module and service container launch: a mark file whose activity timestamp is exactly ACTIVATION_IN_PROGRESS_TIMESTAMP is reset to NULL_VALUE with an error log line. It works because a live owner never stamps Long.MAX_VALUE (it stamps the epoch time, or NULL_VALUE on close), but it is a workaround for a window that belongs to the constructors.