ClamAV misreads PDF encryption values that use short octal escapes

Describe the bug

When ClamAV reads the encoded strings in /O, /U, /OE, and /UE, it appears to assume that a backslash followed by a digit always starts a three-digit octal value.

But those values can also be written with one or two digits, without leading zeros. For example, \23\213 and \023\213 represent the same bytes. They can also be written in hexadecimal as <138B>, which ClamAV handles correctly.

For \23\213, ClamAV takes 2, 3, and the next backslash as the first three digits. This gives it incorrect encryption values.

In the files we tested, the decoded /O value then has the wrong length. ClamAV rejects it before trying the empty password and reports Heuristics.Encrypted.PDF, even though the document opens without a password prompt.

We reproduced this on ClamAV 1.5.4 using three synthetic PDFs with identical decoded encryption values and document text. The examples below illustrate three ways to represent the same two bytes, 13 8B:

All three files use AES-256/R6 encryption with an empty opening password. Both octal variants pass qpdf --check without warnings.

Relevant code

The apparent cause is in pdf_readstring’s octal-escape handling. It reads q[0], q[1], and q[2] as a three-digit octal value. It checks that three characters remain, but does not check whether all three are octal digits. For \23\213, this consumes the next backslash as the third digit.

The resulting /O value then fails the length validation in pdf_handle_enc. This exits before reaching the owner/user password checks.

Because the PDF is marked encrypted but never marked decryptable, this condition reports Heuristics.Encrypted.PDF.

How to reproduce the problem

1. Extract the attached clamav-short-octal-reproducer.zip.
2. Run a ClamAV daemon with AlertEncryptedDoc yes.
3. From the extracted directory, run:

clamdscan --stream synthetic.pdf synthetic-short-octal.pdf synthetic-padded-octal.pdf

Observed output:

synthetic.pdf: OK
synthetic-short-octal.pdf: Heuristics.Encrypted.PDF FOUND
synthetic-padded-octal.pdf: OK

Expected: all three synthetic PDFs should scan without the encrypted-document alert because all three can be decrypted with an empty password.

clamav-short-octal-reproducer.zip