Community Case Study: Structured constraints improve code generation stability by 10x
Hi Baidu/ERNIE team,
Summary
A community developer (non-professional programmer) ran a controlled A/B test showing that structured output constraints ("Code Bagu" / 代码八股) can improve LLM code generation stability by an order of magnitude — with domestic Chinese models performing exceptionally well under constraint.
Key Metrics
| Scenario | Violations | Latency | Rounds |
|---|---|---|---|
| Unconstrained general LLM | 34 structural violations | 8000-word thought chain, then dead loop | N/A |
| With structured constraints | 0 violations | 3.2s | 2 dialogue rounds |
Test Details
- Task: HTTP health checker (multi-threaded + caching + exception handling)
- Modules tested: 3 Python modules of increasing complexity
- Simple: Config hot-reloader (passed with 1 retry)
- Medium: HTTP health checker (passed first try)
- Complex: Student data reporter (passed first try)
- Constraint system: Code Bagu — enforces fixed structural format (八股 paragraph structure, symbolic duality checks, format consistency)
Core Insight
Not a model competition, but an engineering methodology competition.
Domestic models (tested with DeepSeek v4-pro) under structured constraints achieved stable, fast, compliant code generation that outperformed unconstrained general LLMs.
Links
- Full article (Chinese): https://zhuanlan.zhihu.com/p/2038534207488648441
- Open source tool: https://github.com/linyy750911/bagu-coding
- Compliance test report: Code with constraints passes all validation; unconstrained code fails with 34 violations
Suggestion for Baidu/ERNIE
If Baidu/ERNIE considers adding structured output constraint options to the API/SDK layer, it could become a significant differentiator for enterprise developers seeking deterministic, auditable code generation.
Submitted as a community case study. No collaboration or resources requested — just sharing data that may be useful for your developer relations or technical evangelism.
Source: PaddlePaddle/ERNIE