x86-64 MOV r64, m64 reload is lifted as sign-extended low i32 pointer

Summary

RetDec v5.0 lifts a 64-bit x86-64 pointer reload in sqlite's balance function as a sign-extension of the low 32 bits. The truncated value is then used as a pointer base and passed to decodeFlags.

The machine instruction is:

9658b: 48 8b 9d 58 fe ff ff    movq -424(%rbp), %rbx

This is a REX.W 64-bit load. The value loaded into rbx is used immediately:

96592: 4c 8b 63 48              movq 72(%rbx), %r12
96596: 4c 8b 7b 50              movq 80(%rbx), %r15
...
965fb: 48 89 df                 movq %rbx, %rdi
965fe: e8 ed bf fd ff           callq 0x725f0 <decodeFlags>

Reproducer

Input binary and observed output are available here:

https://github.com/x14ngch3n/retdec/tree/poc-issue-1249/poc/issue-1249

Files:

• sqlite3.debug.xz: x86-64 PIE sqlite binary with debug info.
  • uncompressed SHA-256: 1f9dbde8d6d6f445f31169b391e8a9994f359ed00110efc79ad4ba6af6d699f0
  • compressed SHA-256: ff6cde7166d5a5d4a5a58903aceace345f14b947384cc0a5d9aa0929ecb146ef
• retdec.up.ll.xz: RetDec v5.0 output, disassembled/upgraded to textual LLVM IR for easier inspection.
  • compressed SHA-256: 75eb6c1d231acd39e3de92914b954c9fd573f3e36faf80693f528b14416bdc93
• balance-0x96560-0x96610.objdump.txt: disassembly window around the problematic instruction.

Commands:

xz -d -k sqlite3.debug.xz
retdec-decompiler -s -k -o sqlite3.c sqlite3.debug

Then inspect RetDec's LLVM output for function balance, around the code corresponding to machine address 0x9658b.

Expected behavior

The lifted value used as rbx should preserve the full 64-bit value loaded by:

movq -424(%rbp), %rbx

Observed behavior

The observed RetDec IR does not contain a literal trunc i32, but the mul i64 %1597, 4294967296 followed by ashr exact i64 ..., 32 is equivalent to taking the low 32 bits and sign-extending them back to i64:

%1596 = load ptr, ptr %stack_var_-432, align 8, !asm !60166
%1597 = ptrtoint ptr %1596 to i64
%sext50 = mul i64 %1597, 4294967296
%1598 = ashr exact i64 %sext50, 32, !asm !60166

For example, for an i64 value x, this computes:

(int64_t)(int32_t)(x & 0xffffffff)

That %1598 value is then used as the pointer base and as the first argument to decodeFlags:

%1599 = add nsw i64 %1598, 72, !asm !60167
%1602 = add nsw i64 %1598, 80, !asm !60168
...
%1644 = call i64 @decodeFlags(i64 %1598, i8 %1570), !asm !60187

For any valid pointer whose upper 32 bits are non-zero, this does not preserve the semantics of the machine instruction.

Patch

I opened a separate PR for a candidate fix and regression test:

https://github.com/avast/retdec/pull/1250