ghidriff

Ghidriff - Ghidra Binary Diffing Engine

ghidriff provides a command-line binary diffing capability with a fresh take on diffing workflow and results.

It leverages the power of Ghidra's ProgramAPI and FlatProgramAPI to find the added, deleted, and modified functions of two arbitrary binaries. It is written in Python3 using pyghidra to orchestrate Ghidra and jpype as the Python to Java interface to Ghidra.

Its primary use case is patch diffing. Its ability to perform a patch diff with a single command makes it ideal for automated analysis. The diffing results are stored in JSON and rendered in markdown (optionally side-by-side HTML). The markdown output promotes "social" diffing, as results are easy to publish in a gist or include in your next writeup or blog post.

High Level

flowchart LR

a(old binary - rpcrt4.dll-v1) --> b[GhidraDiffEngine]
c(new binary - rpcrt4.dll-v2) --> b

b --> e(Ghidra Project Files)
b --> diffs_output_dir

subgraph diffs_output_dir
    direction LR
    i(rpcrt4.dll-v1-v2.diff.md)
    h(rpcrt4.dll-v1-v2.diff.json)
    j(rpcrt4.dll-v1-v2.diff.side-by-side.html)
end

Sample Diffs

Features

• Command Line (patch diffing workflow reduced to a single step)
• Highlights important changes in the TOC
• Fast - Can diff the full Windows kernel in less than a minute (after Ghidra analysis is complete)
• Enables Social Diffing
  • Beautiful Markdown Output
  • Easily hosted in a GitHub or GitLab gist, blog, or anywhere markdown is supported
  • Visual Diff Graph Results
• Supports both unified and side by side diff results (unified is default)
• Provides unique Meta Diffs:
  • Binary Strings
  • Called
  • Calling
  • Binary Metadata
• Batteries Included
  • Docker support
  • Automated Testing
  • Ghidra (No license required)

See below for CVE diffs and sample usage

Design Goals

• Find all added, deleted, and modified functions
• Provide foundation for automation
• Simple, Fast, Accurate
• Resilient
• Extendable
• Easy sharing of results
• Social Diffing

Powered by Ghidra

The heavy lifting of the binary analysis is done by Ghidra and the diffing is possible via Ghidra's Program API. ghidriff provides a diffing workflow, function matching, and resulting markdown and HTML diff output.

Docs

• ghidriff official Docs
• Blog Post

Engine

An "engine" is a self-contained, but externally-controllable, piece of code that encapsulates powerful logic designed to perform a specific type of work.

ghidriff provides a core base class GhidraDiffEngine that can be extended to create your own binary diffing implementations.

The base class implements the first 3 steps of the Ghidra headless workflow:

1. Create Ghidra Project - Directory and collection of Ghidra project files and data
2. Import Binary to project - Import one or more binaries to the project for analysis
3. Analyze Binary - Ghidra will perform default binary analysis on each binary

The base class provides the abstract method find_matches where the actual diffing (function matching) takes place.

Extending ghidriff

ghidriff can be used as is, but it offers developers the ability to extend the tool by implementing their own differ. The basic idea is create new diffing tools by implementing the find_matches method from the base class.

class NewDiffTool(GhidraDiffEngine):

    def __init__(self,verbose=False) -> None:
        super().__init__(verbose)

    @abstractmethod
    def find_matches(
            self,            
            old: Union[str, pathlib.Path],
            new: Union[str, pathlib.Path]
    ) -> dict:
        """My amazing differ"""

        # find added, deleted, and modified functions
        # 

        return [unmatched, matched]

Implementations

There are currently 3 diffing implementations, which also display the evolution of diffing for the project.

1. SimpleDiff - A simple diff implementation. "Simple" as in it relies mostly on known symbol names for matching.
2. StructualGraphDiff - A slightly more advanced differ, beginning to perform some more advanced hashing (such as Halvar's Structural Graph Comparison)
3. VersionTrackingDiff - The latest differ, with several correlators (an algorithm used to score specific associations based on code, program flow, or any observable aspect of comparison) for function matching. This one is fast.

Each implementation leverages the base class, and implements find_changes.

Usage

…

Extendend Usage

There are quite a few options here, and some complexity. Generally you can succeed with the defaults, but you can override the defaults as needed. One example might be to increase the JVM RAM used to run Ghidra to enable faster analysis of large binaries (--max-ram-percent 80). See help for details of other options.

Show Extended Usage

…

Using Custom Analyzer Settings

If you want to configure specific analyzers for your Ghidra binary analysis, set a custom program_options.json with --program-options.

ghidriff --prog-options prog_options.json tapisrv.dll.x64.10.0.10240.20708 tapisrv.dll.x64.10.0.10240.20708

The program_options.json would need to look something like this:

…

The custom settings will then be used for your binary analysis.

Setting a Custom Image Base Address (Bootloaders, etc.)

If you are reverse engineering firmware or other fun binary and want to change the base address for the binary, use the --base-address parameter to change the base address.

$ ghidriff --base-address 0x80000 STM32F103C-firmware.bin STM32F103Ca-firmware.bin

Quick Start Environment Setup

1. Download and install Ghidra. The current development target is Ghidra 12.0.4 with pyghidra 3.x.
2. Set Ghidra Environment Variable GHIDRA_INSTALL_DIR to Ghidra install location.
3. Pip install ghidriff

Windows

PS C:\Users\\user> [System.Environment]::SetEnvironmentVariable('GHIDRA_INSTALL_DIR','C:\ghidra_10.2.3_PUBLIC_20230208\ghidra_10.2.3_PUBLIC')
PS C:\Users\\user> pip install ghidriff

Linux / Mac

export GHIDRA_INSTALL_DIR="/path/to/ghidra/"
pip install ghidriff

On macOS, install a JDK supported by your Ghidra release first, then set GHIDRA_INSTALL_DIR to the unpacked Ghidra application directory. If macOS Gatekeeper quarantines the downloaded Ghidra archive, remove the quarantine attribute before first launch:

xattr -dr com.apple.quarantine /path/to/ghidra_12.0.4_PUBLIC
export GHIDRA_INSTALL_DIR="/path/to/ghidra_12.0.4_PUBLIC"
pip install ghidriff

UV

export GHIDRA_INSTALL_DIR="/path/to/ghidra/"
uvx ghidriff

Ghidriff in a Box

Don't want to install Ghidra and Java on your host? Try "Ghidriff in a box". It supports multiple-platforms (x64 and arm64).

Docker

docker pull ghcr.io/clearbluejar/ghidriff:latest

This is a docker container with the latest PyPi version of Ghidriff installed. You can check the latest container here.

For Docker command-line diffing

You will need to map the binaries you want to compare into the container. See below for an example.

mkdir -p ghidriffs
wget https://msdl.microsoft.com/download/symbols/clfs.sys/9848245C6f000/clfs.sys -O ghidriffs/clfs.sys.x64.10.0.22621.2506
wget https://msdl.microsoft.com/download/symbols/clfs.sys/D929C6E56f000/clfs.sys -O ghidriffs/clfs.sys.x64.10.0.22621.2715
docker run -it --rm -v $(pwd)/ghidriffs:/ghidriffs ghcr.io/clearbluejar/ghidriff:latest  ghidriffs/clfs.sys.x64.10.0.22621.2506 ghidriffs/clfs.sys.x64.10.0.22621.2715

The result will produce the following.

…

Devcontainer - For Ghidriff development

Use the .devcontainer in this repo. If you don't know how, follow the detailed instructions here: ghidra-python-vscode-devcontainer-skeleton quick setup.

The devcontainer targets ghcr.io/clearbluejar/ghidra-python:12.0.4ghidra3.13python-bookworm. After rebuilding it, the post-create step installs ghidriff with test and dev extras.

Useful development commands:

make install-dev
make test
make test-fast
make test-integration
make lint
make check

make test runs the full suite. make test-fast runs tests that do not launch Ghidra. make test-integration runs the Ghidra-backed tests and requires tests/data, GHIDRA_INSTALL_DIR, and a compatible pyghidra/Ghidra runtime. JVM arguments that begin with - should be passed with equals syntax, for example:

ghidriff --jvm-args=-Xmx8G --decompiler-timeout 120 old.bin new.bin

Deferred design items are tracked in docs/deferred-issues.md.

Use Cases

Diffing a full Windows Kernel

Download two versions of the kernel (older and latest binary):

wget https://msdl.microsoft.com/download/symbols/ntoskrnl.exe/F7E31BA91047000/ntoskrnl.exe -O ntoskrnl.exe.10.0.22621.1344
wget https://msdl.microsoft.com/download/symbols/ntoskrnl.exe/17B6B7221047000/ntoskrnl.exe -O ntoskrnl.exe.10.0.22621.1413

Console Output:

…

Run ghidriff:

ghidriff ntoskrnl.exe.10.0.22621.1344 ntoskrnl.exe.10.0.22621.1413

Console Output

…

Analyze the Diff

Results in this beatiful markdown: ntoskrnl.exe.10.0.22621.1344-ntoskrnl.exe.10.0.22621.1413.diff.md

See if you can figure out what function was patched for CVE-2023-2342.

• Details of CVE-2023-2342 can be found here: https://bugs.chromium.org/p/project-zero/issues/detail?id=2392

Prefer a side by side diff? Try out ghidriff's custom html viewer. https://diffpreview.github.io/?b95ae854a92ee917cd0b5c7055b60282

Results stored in ghidriffs folder

…

Diffing CVE-2023-21768

Details of the CVE-2023-21768 (detailed in this blog post). What if you wanted to repeat this patch diff with ghidriff?

1. Download two versions of AFD.sys (vulnerable and patched):

wget https://msdl.microsoft.com/download/symbols/afd.sys/0C5C6994A8000/afd.sys -O afd.sys.x64.10.0.22621.1028
wget https://msdl.microsoft.com/download/symbols/afd.sys/50989142A9000/afd.sys -O afd.sys.x64.10.0.22621.1415

1. Run ghidriff:

ghidriff afd.sys.x64.10.0.22621.1028 afd.sys.x64.10.0.22621.1415

1. Review results

The diff results are posted in this GitHub gist. The vulnerable function AfdNotifyRemoveIoCompletion was identified here with a single line change