[Feature] DIK-structured Turkish syntactic dependency annotation dataset for structured linguistic evaluation

Describe the feature

Current LLM evaluation benchmarks in OpenCompass focus heavily on English-centric tasks or surface-level semantic understanding, while fine-grained syntactic structure evaluation for morphologically rich languages (such as Turkish) remains limited.

In particular, existing benchmarks do not adequately evaluate a model’s ability to:

represent embedded clause structures track nominalization (-DIK structures) resolve pro-drop subjects maintain cross-clause dependency relations preserve syntactic function mapping in interrogative constructions

To address this gap, I propose a structured DIK-style syntactic annotation dataset for Turkish, designed as a pilot benchmark that can be integrated into OpenCompass evaluation pipelines.

The dataset consists of fully structured JSON annotations of Turkish sentences, where each sentence encodes:

sentence type (interrogative / declarative / etc.) main clause decomposition embedded DIK-complement clause structure verb morphology (tense, mood, person, lemma) syntactic roles (subject, object, adverbials) pro-drop resolution case marking and nominalization tracking

Example structure: { "text": "...", "sentence_type": "...", "main_clause": { "subject": "...", "verb": {...}, "object": { "type": "DIK_complement_clause", "embedded_clause": {...} } } }

A pilot set of 20 fully standardized sentences has been completed, maintaining a consistent annotation schema throughout, with manually validated syntactic decomposition and a scalable design that does not require schema modification.

If this pilot is positively evaluated, the dataset will be expanded to:

100–300 high-quality sentences (Phase 1) optional extension to multi-language morphologically rich benchmarks (Phase 2) integration into structured reasoning / syntax evaluation tasks

This feature enables OpenCompass to:

evaluate syntactic reasoning beyond surface semantics benchmark LLM performance on morphologically rich languages support structured linguistic evaluation tasks extend evaluation diversity beyond English-centered datasets

Will you implement it?

• I would like to implement this feature and create a PR!