How are C source functions extracted for dataset construction?
Thank you for the great work on Decompile-Bench and the LLM4Decompile project. I’ve been reading the paper "Decompile-Bench: Million-Scale Binary-Source Function Pairs for Real-World Binary Decompilation" and exploring the code, and I have a question regarding how source functions are extracted from C code during the dataset construction (the CTF pipeline).
I noticed that in some evaluation scripts (e.g., embedding_gte.py), tree-sitter-c is used to parse C code and extract identifiers. While this is fine for identifier-based rewards, I’m curious about how function boundaries are determined when you build the binary–source function pairs. Specifically, when you need to map a DWARF function trace back to the exact source code region (in the Trace step), do you need to parse the C source to extract function bodys?
The reason I’m asking is that tree-sitter (and many other C parsers) can produce incorrect or incomplete ASTs when the code contains #ifdef / #if preprocessor directives. This makes it challenging to robustly extract function boundaries from real-world C projects, which often have a lot of conditional compilation.
Could you share how you handled this issue in your pipeline? Understanding this would be very helpful for my own work and for anyone trying to replicate or build upon your dataset. Thank you for your time and for making the code and data publicly available.
Source: albertan017/LLM4Decompile