providers/ldap: support fetching users/groups on-demand instead of caching the entire directory in "cached" search/bind mode

Is your feature request related to a problem?

The LDAP provider's "cached" bind_mode and search_mode load the entire directory (every user and every group) into outpost memory via MemorySearcher.fetch(), unconditionally, on startup and on every refresh_interval_s cycle. This happens regardless of how much of the directory a connecting client actually needs.

Our use case: a TAK Server instance authenticating ~13,000 users against an LDAP outpost. TAK Server's queries are always scoped to a single user's own bind DN (a direct object lookup + memberOf read) -- it never performs a directory-wide search. Despite that, the outpost still has to hold all ~13,000 users and ~1,600 groups in memory to answer any of these narrowly-scoped lookups.

This causes two concrete problems as the directory grows:

1. Memory footprint scales with total directory size, not with how many users are actually connecting/authenticating at a given time. We've had to size our outpost tasks up to 8 GiB of memory to keep headroom, for a directory that a per-user cache would only need a small fraction of at any given moment.

2. Because "cached" mode means each outpost replica independently builds its own full copy, running multiple replicas for HA multiplies memory by directory size again per replica, rather than sharing or splitting the load. Two replicas means two full in-memory copies of the same directory.

3. Startup/refresh time to rebuild the full cache scales with directory size, which made our outposts fail NLB/health checks and get killed by our orchestrator (ECS) before the initial sync finished, requiring us to add a large health check grace period as a workaround.

This is closely related to #25270 ("Authenticate against LDAP on demand without syncing the whole directory"), but that issue is about the LDAP source side (importing from a large external directory). This request is the mirror case on the provider/outpost side: authentik is itself the directory, and the LDAP provider is serving it out to LDAP clients, but still has to fully cache it to do so efficiently.

Describe the solution you'd like

Add a search/bind mode (or a variant of "cached" mode) where the outpost fetches and caches users/groups lazily, on first access, rather than eagerly fetching the entire directory upfront. Entries would be fetched via the existing direct/API path on a cache miss, then held with a TTL (similar to the existing bind session cache in SessionBinder, which already uses ttlcache with per-entry expiry) rather than being wiped and fully rebuilt on every refresh_interval_s cycle.

For clients like ours whose queries are always scoped to a single user's own DN, this would mean the outpost only ever holds the working set of recently-active users in memory, rather than the entire directory -- without giving up the performance benefit of caching for repeat lookups.

The current code already has some of the structural pieces for this: MemorySearcher already holds a DirectSearcher instance internally and delegates specific request types to it (SearchBase, SearchSubschema) even while operating in cached mode, so a hybrid "fetch-on-miss, cache the result" model would extend a pattern that's already partially present rather than requiring an entirely new architecture.

Full-directory search requests (from clients that do need to enumerate the whole directory) could still fall back to a full fetch, so this wouldn't need to break existing behavior for those use cases -- it would just avoid paying the full-directory cost for clients that never ask for more than their own entry.

Describe alternatives that you've considered

• Direct search/bind mode (no caching at all): every request becomes a live API call to the authentik core, going through Postgres for every bind/search. In testing, this moved the load rather than reducing it -- our authentik server/worker and database, which were already running close to capacity, became overloaded and the server tasks became unhealthy and were restarted by our orchestrator under normal login traffic. This isn't a viable substitute for caching at our scale.

• Sizing outposts larger and adding a long health check grace period: this is what we're doing today. It works, but cost and fragility both scale with directory size, and doesn't address the underlying per-replica duplication under HA.

• Splitting into multiple LDAP providers with different Base DNs, each with its own outpost, to shard the directory across smaller caches: possible in principle, but adds real operational complexity (multiple providers/outposts to manage) for a problem that's really about the caching strategy of a single provider, not about needing multiple providers.

• Disabling nested/hierarchical group lookups or restructuring group hierarchy: doesn't affect this, since authentik already flattens group membership before serving it over LDAP regardless of source structure.

Additional context

For context, we're already running a locally-patched build of the LDAP outpost to fix an unrelated but related-in-effect memory leak in MemorySearcher (pointers into superseded snapshots pinned via UserFlags.UserInfo, causing steady memory growth under normal login churn) -- the same fix proposed in #26028. That fix addresses a leak on top of the "cached" mode's baseline behavior; this feature request is about that baseline behavior itself (caching the entire directory regardless of what's actually being queried), which remains even once the leak is fixed.

Edit (after discussion with @cheesegrits below):

Mechanism behind the polling and latency concerns above: TAK Server runs a periodic re-check of group membership for every active client connection, on a timer sourced from the same <ldap updateinterval> config value discussed above. The timer resets on reconnect with no persistence, so events causing many clients to reconnect at once (restart, network blip, LB failover) produce a synchronized burst of lookups rather than a steady trickle.

The load shape differs depending on how clients authenticate:

• Username/password (LDAP) clients: each periodic re-check does a fresh password bind as that specific user, then a group search — exercising both bind_mode and search_mode.
• X.509 clients (our deployment): each periodic re-check binds as a single shared service account and only performs a group search — exercising search_mode almost exclusively, with negligible bind_mode load.

Separately, the existing direct-search path's group query (CoreGroupsList with IncludeUsers/IncludeChildren/IncludeParents, filtered by MembersByPk) hydrates full member/parent/child data for every matching group before trimming to the single querying user afterward — unnecessary cost for a simple "what groups is this user in" check, and a plausible explanation for the multi-second latency reported testing direct mode above.

Together, a TTL-based lazy cache built on the existing direct-search path would likely hit the same wall once TTLs expire under this per-connection polling rate (for either auth style), or after a mass-reconnect event — a lighter query path for the single-user-membership case looks like a prerequisite for lazy caching to pay off here, not an optional follow-on.