textdecodebase64 rejects any string containing U+FFFD, so it cannot decode what textencodebase64 produced

Terraform Version

Terraform v1.18.0-dev
on darwin_arm64

(Built from main at f8e7458f5. The code path is unchanged since the function was introduced, so released versions behave the same.)

Terraform Configuration Files

output "encoded" {
  # A string that contains U+FFFD REPLACEMENT CHARACTER.
  value = textencodebase64("caf�", "UTF-8")
}

output "round_trip" {
  value = textdecodebase64(textencodebase64("caf�", "UTF-8"), "UTF-8")
}

Debug Output

Reproduced in terraform console, one expression per invocation:

$ echo 'textencodebase64("caf�", "UTF-8")' | terraform console
"Y2Fm77+9"

$ echo 'textdecodebase64("Y2Fm77+9", "UTF-8")' | terraform console
╷
│ Error: Invalid function argument
│
│   on <console-input> line 1:
│   (source code not available)
│
│ Invalid value for "source" parameter: the given string contains symbols
│ that are not defined for UTF-8.
╵

$ echo 'base64decode("Y2Fm77+9")' | terraform console
"caf�"

Expected Behavior

textdecodebase64("Y2Fm77+9", "UTF-8") returns "caf�".

Y2Fm77+9 is the base64 of 63 61 66 EF BF BD — four well-formed UTF-8 characters, the last being U+FFFD REPLACEMENT CHARACTER. Nothing about it is undefined for UTF-8, and textencodebase64 produced that exact string one line earlier. base64decode, which takes the same bytes and also requires valid UTF-8, returns the string without complaint.

Actual Behavior

The call fails with "the given string contains symbols that are not defined for UTF-8", so:

• textencodebase64 and textdecodebase64 do not round-trip for any string that contains U+FFFD;
• the two sibling functions disagree on the same input — base64decode accepts it, textdecodebase64 rejects it;
• the error message is inaccurate: U+FFFD is defined for UTF-8, and it names the wrong cause, so a config author has nothing to act on.

Steps to Reproduce

1. echo 'textdecodebase64("Y2Fm77+9", "UTF-8")' | terraform console

Additional Context

The cause is in internal/lang/funcs/encoding.go:

decoder := encoding.NewDecoder()
decoded, err := decoder.Bytes(sDec)
if err != nil || bytes.ContainsRune(decoded, '�') {
    return cty.UnknownVal(cty.String), function.NewArgErrorf(0, "the given string contains symbols that are not defined for %s", encName)
}

The ContainsRune check exists for a good reason: golang.org/x/text decoders substitute U+FFFD for bytes they cannot decode rather than returning an error, so its presence is the only signal that something was lost. But the check cannot tell a substituted U+FFFD from one the source actually contained, and rejects both.

Where this shows up in practice is text that has already been through a lossy decode once — a file read with the wrong encoding, a column of mojibake, captured log output. Passing such a value through textencodebase64/textdecodebase64 fails, and the message points at the encoding rather than at the character.

The existing test at internal/lang/funcs/encoding_test.go for gQ== with windows-1250 (byte 0x81, genuinely undefined there) is the case the check is meant to catch, and it should keep failing.

A decode is faithful exactly when encoding the result again reproduces the source bytes: a substituted U+FFFD does not reproduce the byte it replaced, while one that was really in the source does. That distinguishes the two without weakening the existing rejection. I have that change working with tests and will open a PR referencing this issue.

References

• internal/lang/funcs/encoding.go — TextDecodeBase64Func
• internal/lang/funcs/encoding_test.go — TestBase64TextDecode

---

Transparency, per the AI Usage section of CONTRIBUTING.md: this report was prepared with AI assistance. The reproduction above was run against a binary built from main, and the output is copied verbatim from that run.