[chore]: Add backward compatibility integration tests for backends

Requirement

Some feature additions and other changes require the contributor and reviewer to test backward compatibility. Currently that process is manual, we should automate it. An overview of that flow might look like:

Build Jaeger against main branch ---> Write traces to backend ---> Build Jaeger against PR branch ---> Read and assert traces

This flow is inspired by the manual workflow.

FOR THE INTERESTED CONTRIBUTORS: Don't raise a PR directly, first lets discuss the approach and then we might go to the implementation phase

Progress

• framework - #9437
• es/os - #9437
• cassandra #9441
• clickhouse #9441

Updated on 2026-08-29: the design, as built in #9437

Two phases over one corpus. The corpus is fixed before either Jaeger starts, so the reader asserts against exactly what the writer wrote. The write phase runs a Jaeger built from main and only writes; the read phase runs the Jaeger built from the pull request, only reads, and purges when it is done.

The framework is backend-agnostic. A backend joins by adding a BackwardCompatibility test that describes the run exactly as it describes its ordinary e2e suite. The framework derives the write phase from that description, so the backend test stays the only place that knows anything about the backend.

The write phase runs the earlier revision's own configuration file, because a configuration written for the pull request may name settings that older binary does not have.

A missing reference binary fails the run rather than skipping it. The tests skip themselves when JAEGER_OLD_BINARY and JAEGER_OLD_CONFIG_DIR are unset, so the make target refuses to run without them — a workflow that forgot to set them fails instead of going green on tests that never ran.

CI is opt-in per pull request. ci-backward-compatibility.yml is one workflow that sits outside the CI Orchestrator, gates nothing, and runs only on a pull request labelled ci:backward-compat; backends are matrix entries. Gating it through an input on the existing ci-e2e-* workflows does not work, because a job-level if cannot read the matrix context and so collapses to a constant that skips the job outright.

Adding cassandra or clickhouse

1. A compat mode in that backend's scripts/e2e script, alongside its existing direct and e2e modes.
2. A BackwardCompatibility test in that backend's cmd/jaeger/internal/integration test file.
3. A matrix entry in .github/workflows/ci-backward-compatibility.yml.

The two TODO(#8691) markers show where.