[Enhancement]: Preflight summarize and clamp max_tokens when a call would exceed the model window

Target Component

• AI Agents (Researcher/Developer/Executor)
• External Integrations (LLM/Search APIs)

Enhancement Description

Problem Statement

PentAGI sends a fixed per-agent max_tokens on every LLM call. OpenAI-compatible backends such as vLLM reject the request when:

prompt_tokens + tool_schema_tokens + max_tokens > max_model_len

Typical failure: Generator (or another tool-using agent) returns HTTP 400 with an empty body. The agent chain retries a few times and then stops. The UI looks stuck.

This is easy to hit on local / custom models with a 32k window:

• Provider Settings often persist max_tokens near the full window (we have seen 26214 and 32700).
• Generator already sends a large system prompt plus several tool schemas on the first call, before any chain history exists.
• Existing csum summarization runs after a tool round, is byte-budgeted (default last section 50KB, QA up to 64KB), and is not compared to max_model_len.
• If the summarizer LLM call itself 400s, the error is swallowed and the uncompressed chain is kept.

Workarounds (raise --max-model-len, manually keep max_tokens at 1k–4k, start a new Flow) shrink the blast radius but do not close the gap.

Use Case

Automation flows against a custom OpenAI-compatible server (vLLM) with max_model_len=32768. Creating or continuing a task fails in the Generator with empty HTTP 400 even when the model and /v1/chat/completions work for short chat prompts.

Proposed Solution

Before CallWithTools / CallEx:

1. Estimate prompt size (messages + tool schemas). A conservative bytes/4 estimate with slack is enough; /tokenize is optional.
2. Read the model window (max_model_len from /models, or an explicit config/env).
3. If estimate + configured_max_tokens > window:
   • If the chain has older turns, run a tighter csum pass (smaller LastSecBytes / MaxQABytes, KeepQASections=1) and persist the compressed chain.
   • Clamp this call's max_tokens to window - estimate - safety_margin (keep at least ~256 completion tokens).
4. If the prompt still cannot fit a minimum completion budget, hard-truncate older body pairs. Do not call the summarizer LLM when that call would also overflow.
5. Apply the same clamp to performSimpleChain (Summarizer / Adviser). Otherwise “compress first” can 400 before the real agent call.
6. On empty-body 400 / context-overflow, run the same pipeline once more and retry (the existing retry loop in callWithRetries is the natural hook).

Expected Benefits

• Custom / vLLM / Ollama-compatible backends stop dying on the first oversized Generator call.
• Long Flows degrade by losing old detail, not by hard-failing.
• Users can keep a reasonable completion budget without having to guess a number that stays safe for every prompt.

Example Scenario

1. Custom provider, model window 32768, Generator max_tokens=4096 (or larger).
2. New task → Generator builds system + human + tools.
3. Today: prompt + 4096 > 32768 → empty 400 → task creation fails.
4. After this change: preflight clamps max_tokens (and summarizes if history exists) so the request fits; the task proceeds.

Technical Details

Implementation Approach

• Hook flowProvider.callWithRetries immediately before CallWithTools (backend/pkg/providers/performer.go).
• Hook performSimpleChain immediately before CallEx (backend/pkg/providers/performers.go).
• Reuse csum.SummarizeChain with a tighter SummarizerConfig for the preflight pass; write the result back to msg_chain.
• Persist max_model_len (vLLM already returns it on /models, but LoadModelsFromHTTP currently drops extra fields) and/or add something like LLM_SERVER_MAX_MODEL_LEN.
• Optional: clamp reasoning max_tokens the same way when modern reasoning format is enabled.

Integration Points

• AI Agents: Generator, Pentester, Coder, Assistant, and Simple (summarizer) all share these call sites.
• No new services. No new dependencies required if we estimate tokens from bytes.
• Monitoring: log estimated_prompt_tokens, window, clamped_max_tokens, and whether preflight summarization ran (Langfuse event is enough).

Security Considerations

• Compression only rewrites the existing in-flow message chain; it does not send data to a new service.
• Hard-truncate must keep the last body pair (reasoning / tool-call signatures) as csum already does.

Related

• #5 reported the same empty Generator 400 on vLLM; it was closed as parameter / tool-call incompatibility and token-count warnings (PR #17). The overflow inequality was not addressed.
• #57 is about raising max_tokens because replies were cut short — the opposite problem.
• #344 / #347 document long Qwen flows looping; they do not change summarization timing.

I searched issues, PRs, and discussions for max_tokens, max_model_len, context window, summarizer, overflow, clamp, vllm, and prompt too long. I did not find an open proposal for preflight summarize + clamp.

Alternative Solutions

1. Operators only — keep max_tokens at 1k–4k and/or raise --max-model-len.

   • Pros: no code change.
   • Cons: first-call overflow and summarizer-failure paths still exist; Settings “Save” can write a huge max_tokens again.

2. Gateway that rewrites max_tokens to window - prompt.

   • Pros: works for any client.
   • Cons: outside PentAGI; does not compress history when the prompt itself is over the window.

3. Preferred: in-process preflight (this issue).

   • Handles both oversized completion budgets and oversized history at the only two LLM call sites.

Verification

• I have checked that this enhancement hasn't been already proposed
• This enhancement aligns with PentAGI's goal of autonomous penetration testing
• I have considered the security implications of this enhancement
• I have provided clear use cases and benefits