bug: /array/<token>/config can 404 permanently after a project API token reset

Bug Description

Some projects serve a permanent 404 from /array/<token>/config and /array/<token>/config.js on the assets host, while their previous, revoked token still serves 200 from the same host. The current token is valid: it resolves on token-authenticated endpoints, and the project ingests normally.

Every affected project I have found had its project API token reset at some point. A reset on its own does not reproduce it, so a reset looks necessary but not sufficient.

Confirmed on several unrelated production projects in the US region, months after their reset, still 404 today.

Scope

This is not a feature flags problem. The file is the SDK bootstrap config for the whole project: session recording, surveys, heatmaps, error tracking, product tours, site apps and site functions, push, logs, autocapture and dead clicks, the analytics endpoint, and the requested SDK snippet version. Feature flags appear in it only as a hasFeatureFlags boolean.

The team label is for routing, because the delivery path sits there: HyperCache, the dedicated flags Redis, the FEATURE_FLAGS_LONG_RUNNING Celery queue, and rust/hypercache-server. .github/CODEOWNERS declares no owner for posthog/models/remote_config.py, posthog/storage/hypercache.py, or rust/hypercache-server/.

What an affected project looks like

1. GET /array/<current token>/config returns 404, and /config.js returns 404.
2. The same path with the project's previous token returns 200.
3. The current token resolves: GET /api/early_access_features/?token=<current token> returns 200.
4. The state persists for months. The nightly sync_all_remote_configs run does not fix it.

Not reproducible on demand: on a healthy project with no traffic, resetting the token republishes within about a minute and both paths return 200, immediately and on re-check.

When testing, note the endpoint answers HEAD with 200 without a cache lookup, so curl -I hides the failure. Use curl -s -o /dev/null -w '%{http_code}'.

Root cause

The 404 comes from the origin, not the CDN. The response carries cf-cache-status: EXPIRED, so Cloudflare revalidated against the origin, and a never-before-seen cache-busting query string on the same path returns 404 as well.

The two readers disagree for the same token. For an affected project, POST /flags?v=2&config=true returns that project's real config (full sessionRecording object, autocaptureExceptions: true, capturePerformance enabled) at the same moment the assets host returns 404. So the config exists in at least one tier and the assets reader cannot see it.

That points at the unchanged fast path in sync():

if not force and config == self.config:
    RemoteConfig.get_hypercache().set_cache_value_redis_only(self.team, config, track_expiry=True)
    return

RemoteConfig.build_config() does not include the project token, so after a reset the rebuilt config is byte-identical and this branch is taken. set_cache_value_redis_only writes the primary cache and best-effort mirrors to the secondary, and skips both the durable write and the CDN purge. The cache key is token-derived (cache/team_tokens/<token>/array/config.json), so the token change created a new key that never receives a durable copy. The old key keeps the only durable copy, which is exactly why the revoked token still serves 200. Once the assets reader's tier no longer holds the new key, it falls through to durable storage, finds nothing, and 404s for good.

Nothing repairs it:

• Hourly refresh_expiring_caches re-stamps cache entries only, never durable storage.
• Nightly sync_all_remote_configs calls sync() without force, so it takes the same fast path.
• warm_remote_configs_cache is cache-only by design.
• The hypercache verifier covers CacheType = Literal["flags", "team_metadata"] plus flag definitions. array/config is not verified, so a missing entry is never detected.

A project escapes only by accident, when an unrelated edit changes the config contents and triggers a full write.

Impact

Measured on a production project: at the moment its token was reset, exception autocapture and web vitals capture both fell by roughly an order of magnitude per pageview and stayed down, while pageview volume carried on unchanged. Both had been steady for weeks beforehand. The small tail afterwards is consistent with clients still holding a cached config.

Nothing surfaces the failure: no error, no UI warning, events keep flowing, and feature flags keep evaluating. The only visible symptom is a 404 in the browser console, which reads as cosmetic. A project in this state runs without session replay, surveys, heatmaps, web vitals and exception autocapture until someone notices.

Because projects escape only by accident, the ones that stay broken are the quiet ones that never change a setting.

Reproduction

Local stack. No production or customer data needed.

# python manage.py shell
from posthog.models.team import Team
from posthog.models.user import User
from posthog.models.remote_config import RemoteConfig
from posthog.storage import object_storage
from posthog.tasks.remote_config import update_team_remote_config

team = Team.objects.get(id=1)          # any local team
user = User.objects.first()
hc = RemoteConfig.get_hypercache()
old_key = hc.get_cache_key(team)

# Baseline: published, and durable.
print("cache tier:", hc.get_from_cache_with_source(team)[1])
print("durable:", object_storage.read(old_key, missing_ok=True) is not None)                    # True

team.reset_token_and_save(user=user, is_impersonated_session=False)
team.refresh_from_db()
update_team_remote_config(team.id)     # the sync the post_save signal queues

# 1. The new key is cache-only. The durable copy is still under the OLD key.
print("new key cache tier:", hc.get_from_cache_with_source(team)[1])                            # redis
print("new key durable:", object_storage.read(hc.get_cache_key(team), missing_ok=True))         # None  <- the defect
print("old key durable:", object_storage.read(old_key, missing_ok=True) is not None)            # True

# 2. Lose the cache copy, as an eviction, failover or TTL would.
hc.delete_cache_entry(team, kinds=["redis"])

# 3. Nothing repairs it. Re-syncing takes the same fast path.
update_team_remote_config(team.id)
print("after resync, durable:", object_storage.read(hc.get_cache_key(team), missing_ok=True))   # None
print("after resync, cache:", hc.get_from_cache_with_source(team))                              # (None, ...)

With rust/hypercache-server pointed at the same tiers, GET /array/<new token>/config returns 404 from step 2 on, and sync(force=True) is the only thing that fixes it.

Open questions for whoever owns the runtime

1. Which cache tier does rust/hypercache-server read in production, and is it the same one set_cache_value_redis_only mirrors to? The reader divergence above says no, or that the mirror write is failing or being evicted.
2. Is that tier subject to eviction under memory pressure? That would explain why only a minority of reset projects fail.
3. _mirror_to_secondary swallows exceptions, so a failing mirror is invisible. Should it be?
4. What do CELERY_TASK_REMOTE_CONFIG_SYNC{result=...} and the refresh task's failed counter show for affected teams?

Suggested fix

1. Write durably whenever the cache key changes: force a full sync from the token-change path, and delete the entry under the old key.
2. Add array/config to the hypercache verifier, so a missing entry is detected and repaired whatever the cause. This alone would have healed every affected project with no ticket.
3. Backfill the projects currently stranded. No existing tool does this: sync_all_remote_configs does not pass force, and warm_remote_configs_cache is cache-only.

Finding affected projects

Join posthog_activitylog (scope Team, a detail change on api_token) to posthog_remoteconfig and keep rows where synced_at is null or older than the token change. synced_at only moves on a content change, so those rows have had no full republish since their reset. Confirm each against the live endpoint before acting, since a project can be healthy despite matching.

Debug info

• PostHog Cloud, US region. Reproduced by observation on live projects, and deterministically on a local stack per the steps above.