[Feature] Add statistical input token admission for AI functions
Feature request
Is your feature request related to a problem? Please describe.
AI SQL functions can invoke an external model once per input row, so an unexpectedly large query can consume significant input tokens. StarRocks currently has no pre-execution budget for this estimated work.
Describe the solution you'd like
Add an optional statistical input-token admission check using the existing physical AIProject plan and optimizer statistics:
- Show the plan's estimate, or an explicit unknown result, in EXPLAIN VERBOSE/COSTS.
- Keep admission disabled by default; when enabled, reject unknown or over-budget estimates before deployment.
- Preserve the existing AI Provider and SYSTEM execution routes and query/transaction lifecycle.
- Document supported estimates and their limitations: this is not exact tokenization, a billing quota, or runtime token accounting.
Describe alternatives you've considered
Query queue CPU/memory accounting and runtime usage reporting serve different purposes: neither prevents a plan from starting based on estimated external-model input tokens. This feature should reuse existing statistics and deployment boundaries without changing those mechanisms.
Additional context
Related AI function integration: #78807.
Source: StarRocks/starrocks