crash

CORE ANALYSIS SUITE

The core analysis suite is a self-contained tool that can be used to investigate either live systems, kernel core dumps created from dump creation facilities such as kdump, kvmdump, xendump, the netdump and diskdump packages offered by Red Hat, the LKCD kernel patch, the mcore kernel patch created by Mission Critical Linux, as well as other formats created by manufacturer-specific firmware.

o The tool is loosely based on the SVR4 crash command, but has been completely integrated with gdb in order to be able to display formatted kernel data structures, disassemble source code, etc.

o The current set of available commands consist of common kernel core analysis tools such as a context-specific stack traces, source code disassembly, kernel variable displays, memory display, dumps of linked-lists, etc. In addition, any gdb command may be entered, which in turn will be passed onto the gdb module for execution.

o There are several commands that delve deeper into specific kernel subsystems, which also serve as templates for kernel developers to create new commands for analysis of a specific area of interest. Adding a new command is a simple affair, and a quick recompile adds it to the command menu.

o The intent is to make the tool independent of Linux version dependencies, building in recognition of major kernel code changes so as to adapt to new kernel versions, while maintaining backwards compatibility.

A whitepaper with complete documentation concerning the use of this utility can be found here:

     https://crash-utility.github.io/crash_whitepaper.html

These are the current prerequisites:

o At this point, x86, ia64, x86_64, ppc64, ppc, arm, arm64, alpha, mips, mips64, loongarch64, riscv64, s390 and s390x-based kernels are supported. Other architectures may be addressed in the future.

o One size fits all -- the utility can be run on any Linux kernel version version dating back to 2.2.5-15. A primary design goal is to always maintain backwards-compatibility.

o In order to contain debugging data, the top-level kernel Makefile's CFLAGS definition must contain the -g flag. Typically distributions will contain a package containing a vmlinux file with full debuginfo data. If not, the kernel must be rebuilt:

 For 2.2 kernels that are not built with -g, change the following line:

    CFLAGS = -Wall -Wstrict-prototypes -O2 -fomit-frame-pointer

 to:

    CFLAGS = -g -Wall -Wstrict-prototypes -O2 -fomit-frame-pointer

 For 2.4 kernels that are not built with -g, change the following line:

    CFLAGS := $(CPPFLAGS) -Wall -Wstrict-prototypes -O2 -fomit-frame-pointer -fno-strict-aliasing

 to:

    CFLAGS := -g $(CPPFLAGS) -Wall -Wstrict-prototypes -O2 -fomit-frame-pointer -fno-strict-aliasing

 For 2.6 and later kernels that are not built with -g, the kernel should
 be configured with CONFIG_DEBUG_INFO enabled, which in turn will add
 the -g flag to the CFLAGS setting in the kernel Makefile.

 After the kernel is re-compiled, the uncompressed "vmlinux" kernel
 that is created in the top-level kernel build directory must be saved.

o Requirements for building: Fedora: make gcc gcc-c++ ncurses-devel zlib-devel lzo-devel snappy-devel bison wget patch texinfo libzstd-devel gmp-devel mpfr-devel Ubuntu/Debian: make gcc g++ libncurses-dev zlib1g-dev liblzo2-dev libsnappy-dev bison wget patch texinfo libzstd-dev Arch Linux: make gcc ncurses zlib lzo snappy bison wget patch texinfo zstd openSUSE: make gcc gcc-c++ ncurses-devel zlib-devel lzo-devel snappy-devel bison wget patch texinfo libzstd-devel

To build the crash utility:

$ tar -xf crash-9.0.x.tar.gz
$ cd crash-9.0.x
$ make

To cross compile the crash utility for aarch64 on x86_64: $ make CROSS_COMPILE=aarch64-linux-gnu- -jnproc

Supported arches for cross compilation: x86_64, x86, aarch64, s390x, powerpc64, alpha, sparc64, mips, riscv64

The initial build will take several minutes because the embedded gdb module must be configured and built. Alternatively, the crash source RPM file may be installed and built, and the resultant crash binary RPM file installed.

The crash binary can only be used on systems of the same architecture as the host build system. There are a few optional manners of building the crash binary:

o On an x86_64 host, a 32-bit x86 binary that can be used to analyze 32-bit x86 dumpfiles may be built by typing "make target=X86". o On an x86 or x86_64 host, a 32-bit x86 binary that can be used to analyze 32-bit arm dumpfiles may be built by typing "make target=ARM". o On an x86 or x86_64 host, a 32-bit x86 binary that can be used to analyze 32-bit mips dumpfiles may be built by typing "make target=MIPS". o On an ppc64 host, a 32-bit ppc binary that can be used to analyze 32-bit ppc dumpfiles may be built by typing "make target=PPC". o On an x86_64 host, an x86_64 binary that can be used to analyze arm64 dumpfiles may be built by typing "make target=ARM64". o On an x86_64 host, an x86_64 binary that can be used to analyze ppc64le dumpfiles may be built by typing "make target=PPC64". o On an x86_64 host, an x86_64 binary that can be used to analyze riscv64 dumpfiles may be built by typing "make target=RISCV64". o On an x86_64 host, an x86_64 binary that can be used to analyze loongarch64 dumpfiles may be built by typing "make target=LOONGARCH64".

Traditionally when vmcores are compressed via the makedumpfile(8) facility the libz compression library is used, and by default the crash utility only supports libz. Recently makedumpfile has been enhanced to optionally use the LZO, snappy or zstd compression libraries. To build crash with any or all of those libraries, type "make lzo", "make snappy" or "make zstd".

crash supports valgrind Memcheck tool on the crash's custom memory allocator. To build crash with this feature enabled, type "make valgrind" and then run crash with valgrind as "valgrind crash vmlinux vmcore".

All of the alternate build commands above are "sticky" in that the special "make" targets only have to be entered one time; all subsequent builds will follow suit.

If the tool is run against a kernel dumpfile, two arguments are required, the uncompressed kernel name and the kernel dumpfile name.

If run on a live system, only the kernel name is required, because /dev/mem will be used as the "dumpfile". On Red Hat or Fedora kernels where the /dev/mem device is restricted, the /dev/crash memory driver will be used. If neither /dev/mem or /dev/crash are available, then /proc/kcore will be be used as the live memory source. If /proc/kcore is also restricted, then the Red Hat /dev/crash driver may be compiled and installed; its source is included in the crash-9.0.x/memory_driver subdirectory.

If the kernel file is stored in /boot, /, /boot/efi, or in any /usr/src or /usr/lib/debug/lib/modules subdirectory, then no command line arguments are required -- the first kernel found that matches /proc/version will be used as the namelist.

For example, invoking crash on a live system would look like this:

$ crash

crash 9.0.x
Copyright (C) 2002-2026  Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
Copyright (C) 1999-2006  Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011, 2020-2022  NEC Corporation
Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
Copyright (C) 2015, 2021  VMware, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter "help copying" to see the conditions.
This program has absolutely no warranty.  Enter "help warranty" for details.
 
GNU gdb 16.2
Copyright 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "i686-pc-linux-gnu"...
 
      KERNEL: /boot/vmlinux
    DUMPFILE: /dev/mem
        CPUS: 1
        DATE: Fri Apr 24 10:38:29 CST 2026
      UPTIME: 10 days, 22:55:18
LOAD AVERAGE: 0.08, 0.03, 0.01
       TASKS: 42
    NODENAME: ha2.mclinux.com
     RELEASE: 2.4.0-test10
     VERSION: #11 SMP Thu Nov 4 15:09:25 EST 2000
     MACHINE: i686  (447 MHz)
  MEMORY: 128 MB
         PID: 3621                                  
     COMMAND: "crash"
        TASK: c463c000  
         CPU: 0
       STATE: TASK_RUNNING (ACTIVE)

crash> help

*              files          mod            sbitmapq       union          
alias          foreach        mount          search         vm             
ascii          fuser          net            set            vtop           
bpf            gdb            p              sig            waitq          
bt             help           ps             struct         whatis         
btop           ipcs           pte            swap           wr             
dev            irq            ptob           sym            q              
dis            kmem           ptov           sys            
eval           list           rd             task           
exit           log            repeat         timer          
extend         mach           runq           tree           

crash version: 9.0.0    gdb version: 16.2
For help on any command above, enter "help ".
For help on input options, enter "help input".
For help on output options, enter "help output".

crash> 

When run on a dumpfile, both the kernel namelist and dumpfile must be entered on the command line. For example, when run on a core dump created by the Red Hat netdump or diskdump facilities:

$ crash vmlinux vmcore

crash 9.0.x
Copyright (C) 2002-2026  Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
Copyright (C) 1999-2006  Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011, 2020-2022  NEC Corporation
Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
Copyright (C) 2015, 2021  VMware, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter "help copying" to see the conditions.
This program has absolutely no warranty.  Enter "help warranty" for details.
 
GNU gdb 16.2
Copyright 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "i686-pc-linux-gnu"...

      KERNEL: vmlinux
    DUMPFILE: vmcore
        CPUS: 4
        DATE: Fri Apr 24 10:38:29 CST 2026
      UPTIME: 00:02:40
LOAD AVERAGE: 2.24, 0.96, 0.37
       TASKS: 70
    NODENAME: pro1.lab.boston.redhat.com
     RELEASE: 2.6.3-2.1.214.11smp
     VERSION: #1 SMP Tue Mar 2 10:58:27 EST 2004
     MACHINE: