evals: add RTL language coverage

Author: MHAlikhaniCreated Sep 11, 2026Updated Sep 11, 2026

Add RTL and multilingual evaluation coverage

I think it would be useful to add a few evaluation cases for RTL languages, especially Persian and Arabic.

The current evals do a good job of measuring response structure and actionability, but they are mostly English-focused. Some of the existing rules can behave differently when RTL text is mixed with LTR content such as commands, file paths, URLs, and code.

For example:

src/auth.ts
npm test -- auth.spec.ts

can become harder to scan when embedded in RTL text. Numbered steps and punctuation can also be affected.

Proposal

Add a small set of language-aware cases covering:

  • en as the control
  • fa (Persian)
  • ar (Arabic)

The cases could cover:

  • action-first responses
  • multi-step tasks
  • mixed RTL/LTR technical content
  • multi-turn state
  • verbosity vs. actionability

Something like:

json
{
  "id": "rtl-debug-fa-001",
  "language": "fa",
  "direction": "rtl",
  "rule": "action-first"
}

The goal isn't to add language-specific rules to the skill. This should stay in the eval layer and simply verify that the existing behavior holds across different languages and writing directions.

Acceptance criteria

  • Add representative Persian cases
  • Add representative Arabic cases
  • Keep English control cases where appropriate
  • Cover mixed RTL/LTR content
  • Extend the rubric for RTL/LTR readability
  • Document how to run the new cases
  • Ensure existing English evals are unaffected

I think this would be a useful extension of the current evaluation system and a good way to catch issues that aren't visible in English-only testing.