[Bug]: Wiki taxonomy embedding bypasses BATCH_EMBED_SIZE and exceeds provider input-count limits

Affected Component

Backend Service & API

Bug Description

Source-level bug report: Wiki taxonomy selection calls the embedding provider directly, bypassing the existing configurable batching path. Providers with a per-request input-count limit can therefore reject these requests even when the configured batch size is below that limit.

This report contains only public source references and hypothetical reproduction inputs. It does not include deployment details or production logs.

Public source references at revision 1edcd54b43606d9079bb36650efe3f68707a79ea:

1. batch.go:27–84: BatchEmbedWithPool reads BATCH_EMBED_SIZE, defaults to 5 when unset, and splits inputs before calling model.BatchEmbed.
2. wiki_ingest_taxonomy.go:195–200: selectRelevantFolders instead calls embedder.BatchEmbed(ctx, folderTexts) and embedder.BatchEmbed(ctx, itemTexts) directly.
3. openai.go:163: BatchEmbed puts the entire texts slice into the request's input array, without further splitting.

The folder and item calls consequently bypass the environment-controlled batch size. This is distinct from RPM/TPM rate limiting, single-text token limits, and vector-database write batching.

Steps to Reproduce

Proposed minimal synthetic reproduction, derived from the public code; this isolated fixture has not yet been executed:

1. Configure an OpenAI-compatible mock embedding endpoint that rejects more than 20 input texts per HTTP request and otherwise returns one vector per input.
2. Leave BATCH_EMBED_SIZE unset so the public code's default of 5 applies.
3. Exercise selectRelevantFolders with nonempty taxonomy items and a folder pool of 80 distinct two-level paths. The pool exceeds wikiTaxonomyFeedAllMaxFolders = 60, avoiding the early-return path.
4. Record the mock endpoint's request input count. The direct folder call submits all 80 texts instead of sub-batches of at most 5, so the mock rejects it.
5. Cover the item call separately with more than 20 items: ensure the folder request can succeed, then check that the item request is also batched.

The numbers 20 and 80 above are synthetic test parameters, not production measurements.

Expected Behavior

Both Wiki taxonomy embedding calls should respect the same batching setting as the normal document indexing path, preserving vector count and order.

A possible minimal change is to reuse the existing batching interface for both calls:

folderVecs, err := embedder.BatchEmbedWithPool(ctx, embedder, folderTexts)
itemVecs, err := embedder.BatchEmbedWithPool(ctx, embedder, itemTexts)

This is a proposed fix, not a tested patch. Regression coverage should check each provider request's input count, output ordering/count, both folder and item paths, and the existing fallback when a provider still fails.

Actual Behavior

As implemented in the linked source, the full folder or item slice is sent directly to BatchEmbed, and this call path does not consult BATCH_EMBED_SIZE.

When the provider returns an error, the folder branch logs a warning and returns capFolders(pool, wikiTaxonomyPromptMaxPaths). The item branch has the same fallback. Similarity-based selection is skipped; this does not imply that the entire document parse necessarily fails.

Reducing BATCH_EMBED_SIZE or worker concurrency cannot constrain the input count on a call that bypasses the batching helper.

WeKnora Version

Public source revision analyzed: 1edcd54b43606d9079bb36650efe3f68707a79ea. This is a backend source-level report; UI version is not applicable.

Deployment Method

Other — source-level analysis; the proposed reproduction uses a mock embedding endpoint and does not require a particular deployment method.

Operating System

Not applicable to the source-level finding; no OS-specific behavior is involved in the referenced call path.

Relevant Logs

No private deployment logs are included. The following warning text is quoted from the public source, not from a captured runtime log:

wiki ingest: taxonomy plan folder embed failed, feeding all folders: %v

A regression fixture should assert that the mock provider never receives more inputs than the effective batch size. No runtime test result is claimed here.

Related but different: #2533 and #2953 concern embedding rate pacing/retry recovery. This report concerns Wiki taxonomy bypassing input-count batching.

Confirmation

• I have searched existing issues and confirmed this is a new one