Tracking issue: High Availability / horizontal scaling support

This is a tracking issue that includes things that need to be fixed/changed before Pocket ID can scale horizontally on multiple replicas.

A baked-in assumption is that we're going to leverage an actor framework for synchronizing state across multiple replicas and for orchestrating background jobs.

Core blocker

• Remove single-instance app lock — DB application_lock exits any second replica on startup.
  • Initial implementation #1624

In-memory state to share/coordinate

• App config cache — atomic.Pointer config only refreshes in the process that wrote it
  • #1604
• Instance ID — currently maintained inside App Config
  • #1579
• Rate limiter — per-process IP-based limiter
  • #1556
• Minor read caches — version check, app-images extension map
  • No need to address the external JWKS cache, can stay per-process
• JWT signing keys - coordinate creation of keys upon startup of a new cluster
  • Necessary fixes in #1733

Background jobs (run on exactly one instance)

• LDAP sync — must not run concurrently across replicas
  • #1651
• SCIM sync — must not run concurrently across replicas
  • #1680
• API key expiry job — single run cluster-wide
  • #1687
• Analytics job — single run to avoid duplicate pings
  • #1556
• GeoLite update job — single run, tied to shared GeoLite storage below
  • #1647
• Orphaned temp file cleanup — single run (filesystem backend only)
  • #1556

Local-filesystem dependencies

• Filesystem storage backend — local disk isn't shared, require DB/S3 backend for multi-instance
• Default profile picture caching — generated avatars written to local disk per-instance
• GeoLite DB on local disk — per-instance copy + download
  • #1647

See note: https://github.com/pocket-id/pocket-id/pull/1647#issuecomment-5186855645

Migrate to actor + alarm (and delete the cleanup job)

Each of these is currently expired by db_cleanup_job.go. Move expiry into the owning actor's alarm and remove the matching cleanup function.

• WebAuthn sessions — actor with expiry alarm, drop ClearWebauthnSessions
  • #1698
• One-time access tokens — actor with expiry alarm, drop ClearOneTimeAccessTokens
  • #1611
• Signup tokens — actor with expiry alarm, drop ClearSignupTokens
  • #1611
• Email verification tokens — actor with expiry alarm, drop ClearEmailVerificationTokens
  • #1625
• OAuth2 sessions — (unified fosite store: auth codes, access/refresh tokens, PAR) actor with expiry alarm; drop ClearOAuth2Sessions
  • #1698
• OAuth2 client-assertion JTIs — (replay-protection IDs) actor with expiry alarm, drop ClearOAuth2JTIs
  • #1698
• OIDC interaction sessions — (abandoned-session pruning) actor/alarm, drop ClearInteractionSessions
  • #1698
• Reauthentication tokens — actor with expiry alarm, drop ClearReauthenticationTokens
  • #1698
• Audit log retention — actor/alarm-driven pruning, drop ClearAuditLogs
  • #1698
• Unused default profile pictures — alarm-driven pruning, drop ClearUnusedDefaultProfilePictures
  • #1556

Residual TODOs

• Integrate Francis data into backup/restore
  • #1645
• Configure HA_ENABLED env var
• Allow connecting to an external Francis runtime
• Switch devicelogin actor to use signal - See https://github.com/pocket-id/pocket-id/pull/1594#issuecomment-5096418021 and docs for signal built-in actor @stonith404

For v3

• SQLite: use separate database for Francis #1695
• Update FS storage layout to have a separate directory for cache vs state - See https://github.com/pocket-id/pocket-id/pull/1647#issuecomment-5186855645

Supersedes #1351