Whole-Program Reverse Engineering with GPT-3
This is a little toy prototype of a tool that attempts to summarize a whole binary using GPT-3 (specifically the text-davinci-003 model), based on decompiled code provided by Ghidra. However, today's language models can only fit a small amount of text into their context window at once (4096 tokens for text-davinci-003, a couple hundred lines of code at most) -- most programs (and even some functions) are too big to fit all at once.
GPT-WPRE attempts to work around this by recursively creating natural language summaries of a function's dependencies and then providing those as context for the function itself. It's pretty neat when it works! I have tested it on exactly one program, so YMMV.
You need:
pip install -r requirements.txtMake sure the program you want to analyze is open and that the Ghidra bridge server is running. Then extract the control flow graph and decompiled functions with:
$ python extract_ghidra_decomp.py
Building call graph: 100%|██████████████████████████| 651/651 [02:21 B -------> C
`--> D --^Then we would start by summarizing C, then D, then B, and finally A. The sequence of prompts looks like:
Describe what this function does in a single sentence:
```
int C(double param1) {
// ...
}
```Giving perhaps a summary like "This function frobnicates the parameter and returns the result." Then we would prompt:
Given the following summaries:
C: This function frobnicates the parameter and returns the result.
Describe what this function does in a single sentence:
```
void D(int param1) {
// Some code that calls C
}
```And so on until we have a summary for A.
Unfortunately, sometimes even a single function can be too big to summarize in a single prompt when context is included. In this case we need to split up the individual function, summarize its parts, and recombine those summaries.
The right way to do this might be do the same topological sort strategy as before but on the function's control flow graph rather than the program's call graph. But a) topological sort isn't defined for graphs with cycles, and whereas mutual recursion is rare at the call graph level, cycles in a control flow graph (aka "loops") are very common; and more prosaically, b) I don't know if Ghidra's decompiler exposes a source-level CFG.
Instead, when we encounter a function that's too big, we split it into sequential chunks of up to 100 lines, summarize each chunk as a paragraph, and then recombine the summaries. If 100 lines is too many, we try 80, 60, ..., down to 20 lines. If this is still too big, we instead summarize the chunks as single sentences.
This all happens in recursive_summarize.py's summarize_long_code function.
No open issues yet, or sync has not completed.