Hyperlink URI contains raw non-UTF-8 path bytes on Unix

What version of ripgrep are you using?

Reproduced with ripgrep 14.1.0 (Ubuntu 24.04, apt). The relevant code path looks unchanged on master as of today — see the two links below.

How did you install ripgrep?

apt install ripgrep inside ubuntu:24.04. Also checked 15.1.0 (Homebrew) on macOS, where the case cannot be constructed because APFS rejects filenames that are not valid UTF-8 (EILSEQ). So this is Unix-with-a-permissive-filesystem only.

What operating system are you using ripgrep on?

Linux (any filesystem that allows arbitrary bytes in a filename).

Describe your bug.

On Unix, path bytes that are not valid UTF-8 are copied verbatim into the hyperlink URI. The result is neither a URI (RFC 3986 is ASCII-only) nor an IRI (RFC 3987 needs decodable characters), and it puts a byte outside 32–126 into an OSC 8 sequence, which the OSC 8 specification linked from --hyperlink-format's own man page entry says is undefined behaviour: "the parameters and the URI must not contain any bytes outside of the 32–126 range. If they do, the behavior is undefined. Bytes outside of this range in the URI must be URI-encoded."

I want to be clear that I am not reporting the general non-ASCII behaviour. HyperlinkPath::encode leaves 128.. unencoded on purpose and the comment above it explains why — RFC 8089 §4 does not mandate an encoding, and UrlCreateFromPathW does not encode non-ASCII on Windows. For a valid UTF-8 path that reasoning holds and produces a usable IRI. This report is about the narrower case that reasoning does not cover: bytes that are not part of any valid UTF-8 sequence, where there is nothing to be lenient about.

What are the steps to reproduce the behavior?

docker run --rm ubuntu:24.04 bash -c '
  apt-get update -qq && apt-get install -y -qq ripgrep
  cd "$(mktemp -d)"
  printf "hit\n" > "$(printf "bad\xff.txt")"
  rg --color=always --hyperlink-format=default -H hit . | od -An -c | head -3'

What is the actual behavior?

 033   ]   8   ;   ;   f   i   l   e   :   /   /   4   1   1   f
   b   8   2   5   a   6   0   1   /   t   m   p   /   t   m   p
   .   l   Q   D   o   a   c   c   x   q   n   /   b   a   d 377

The 377 at the end is the raw 0xFF from the filename, inside the URI.

What is the expected behavior?

Either of these would resolve it, and both are consistent with the existing design:

1. Percent-encode only the bytes that cannot be part of valid UTF-8. %FF is lossless, unambiguous, round-trips exactly, and is explicitly permitted by RFC 8089. It does not affect the documented non-ASCII behaviour, because a valid UTF-8 path contains no such byte — so the Windows rationale in the comment is untouched.
2. Refuse to build a hyperlink, which is what the Windows branch already does for the analogous case: from_path bails with "path is not valid UTF-8" when to_str() returns None. The Unix branch takes path.as_os_str().as_bytes() with no equivalent check, so the two platforms disagree about a case they both can describe.

Relevant code, for reference:

• HyperlinkPath::encode — the 128.. arm of the match
• the #[cfg(unix)] from_path just above it, versus the #[cfg(windows)] one that gives up on non-UTF-8

One small documentation point

The --hyperlink-format man page entry says of {path}:

The path is guaranteed to be absolute and percent encoded such that it is valid to put into a URI.

For any non-ASCII path that is not accurate — the result is an IRI, not a URI — and the guarantee is what led me to check this in the first place. The doc comment on encode is precise about what actually happens; the user-facing text is not. Softening it (or pointing at RFC 8089 the way the code comment does) would help, independently of whatever you decide about the invalid-UTF-8 case.

Thanks for ripgrep — this came out of writing up how rg handles text encodings, and the rest of what I measured was exactly as documented.