Cloud ingestion text-processor can hit 5-minute function timeout on very large documents

Summary

Large documents (e.g. a CSV with tens of thousands of rows) can push the cloud-ingestion text-processor custom skill past its 5-minute function timeout, causing the Azure AI Search indexer to record a transientFailure with:

Enrichment.WebApiSkill.<index>-cloud-text-processor-skill
Web Api response status: 'InternalServerError', Web Api response details: ''

App Insights on the function app shows:

Microsoft.Azure.WebJobs.Host.FunctionTimeoutException
Timeout value of 00:05:00 was exceeded by function: Functions.process_text

Root cause

app/functions/text_processor/host.json pins functionTimeout: "00:05:00". The process_text function loads all pages for a document into memory and computes embeddings + writes chunks sequentially per invocation. On a document that produces many chunks (large CSVs, very long PDFs, etc.), the wall-clock time exceeds that cap even after #3070's page-grouping fix reduces memory pressure.

Repro

1. Enable cloud ingestion (azd env set USE_CLOUD_INGESTION true) and deploy.
2. Upload a CSV with ~10,000+ rows into the content container.
3. Trigger the indexer.
4. Watch indexer status — big CSV blob fails with Web Api response status: 'InternalServerError'. App Insights shows the FunctionTimeoutException above.

Options

Trade-offs, roughly in increasing effort:

1. Short-term: bump functionTimeout to 00:10:00 in app/functions/text_processor/host.json. That's the max allowed on the Consumption plan we currently deploy — buys ~2× headroom with zero infra change.
2. Move text-processor to Flex Consumption or Premium. Flex Consumption caps at ~60 min; Premium/Dedicated allows unlimited (functionTimeout: -1). Requires infra plan change and a cost tradeoff.
3. Refactor the skill for smaller units of work. Split large documents into batches that fit inside a single skill invocation and let Search retry per-batch. Also lets us parallelize embeddings across invocations.
4. Parallelize embeddings inside a single invocation (larger batch size / concurrent requests to Azure OpenAI). Cheapest per-doc speedup but bounded by embedding TPM.

Context

Surfaced while validating #3070 (CSV row grouping). #3070 reduces memory pressure but doesn't shrink per-invocation wall-clock time — the timeout is now the next bottleneck for very large inputs. Filing this so the pipeline scaling work is tracked separately from the CSV grouping bug fix.

References

• Function timeouts by plan: https://learn.microsoft.com/azure/azure-functions/functions-scale#timeout
• app/functions/text_processor/host.json (current functionTimeout: "00:05:00")
• Related: #2878 (original OOM bug), #3070 (CSV row grouping)