#4604·fiber

Consolidate sanitization/normalization helpers and remove double processing

Author: ReneWerner87Created Aug 11, 2026Updated Sep 3, 2026
Labels🧹 Updatesv3

Core and middleware carry around 40 duplicated or divergent transform sites (origin/host normalization, comma-list scanning, ETag parsing, control-char handling, quoted-string production, URL composition guards), grouped into 13 clusters below. Goal: one implementation per concern, at most one pass per value, and the fasthttp trust boundary written down at the call sites that depend on it. Suggested order: clusters 1, 2 and 5 first (most call sites).

New helpers go to internal/ packages (no new public API); only generic byte primitives belong in gofiber/utils.

Fix first

Three small bug-ish fixes that can land as standalone PRs before any consolidation:

  • router.go buildRouteURL substitutes path params with no escaping; urlnorm.RootedPath afterwards defuses a leading //, but a param containing ? or # still restructures the URL, while Redirect().Route() escapes query values via utils.AppendQueryEscape. Escape substituted params with path-segment rules (or reject structure-altering values).
  • client/hooks.go parserRequestURL substitutes path params via unanchored strings.ReplaceAll(uri, ":"+key, val): :id also matches inside :idx, and unescaped values containing /, ? or # restructure the request target. Anchor at segment boundaries and percent-encode values.
  • extractors/extractors.go FromParam runs url.PathUnescape on c.Params(). With UnescapePath: false (default) that is the single decode; with UnescapePath: true the path was already decoded once in configDependentPaths, so params get decoded twice (%2520 arrives as a space). Decide which layer decodes and make the count independent of the config flag.

What fasthttp v1.73.0 already guarantees (server path)

  • Every exported header-value write (Set, Add, SetCanonical, SetCookie, SetContentType, special setters) funnels through initHeaderValueBytes -> removeNewLines: CR and LF are replaced with a space, silently; header keys are CR/LF-filtered too; trailer keys are the one different shape (validated and rejected instead). No exported API bypasses the value filter. Fiber-side CRLF stripping before a header write is therefore browser-parity normalization at most, never injection defense.
  • Parsed request-header values reject C0 bytes except HTAB, and reject DEL; bytes >= 0x80 pass. A violating request is answered 400 and the connection closed; obs-folds are collapsed to one space. A request target containing any CTL byte is rejected too.
  • Server-side URI().Path() is always percent-decoded and normalized (rooted, // dedup, /./ and /../ resolved). Decoding runs before the collapsing, so an encoded %2F becomes a real separator in Path(). PathOriginal() keeps the raw bytes. No server-side switch disables this; the exported URI.DisablePathNormalizing field only changes RequestURI() output, never Path().
  • URI().Host() is lowercased, unescaped and validated at parse time; URI.SetHost only lowercases; Request.Header.Host() stays raw.
  • Cookie writes strip CR/LF and replace ; with a space (SetPath additionally normalizes); cookie values are never percent-coded by fasthttp; invalid request-cookie values are silently dropped at parse.

Gaps fiber must keep covering itself: NUL and other C0/DEL bytes pass the header write filter (values are mutated silently, never rejected); on Unix \ is not a path separator for fasthttp's normalization, so %5C sequences survive into Path(); RequestCtx.Redirect appends fragments raw.

Clusters

  • 1. Origin toolkit (cors/csrf): normalizeOrigin is near-identical in middleware/cors/utils.go and middleware/csrf/helpers.go (sole delta: csrf rejects non-http(s) schemes), the wildcard-subdomain matcher is character-identical in both, csrf additionally carries a test-only subdomain.match (dead production code), config-time TrimSpace and the redaction closure are duplicated too. The compare paths diverge: csrf uses internal/schemehost.Match (default-port folding), cors a lowercased map lookup without port folding, and the csrf referer check builds an origin string a third way. Target: one internal origin package consumed by both; today's deltas (scheme allowlist, port folding, idempotence check) become documented policy instead of accidents.
  • 2. Host normalization and IDNA: three comparators disagree (middleware/hostauthorization normalizeHost: lowercase, port strip, bracket strip, trailing-dot strip, punycode; internal/schemehost normalizeHostPort: lowercase plus default-port folding; proxy DomainForward: bare EqualFold on the raw Host header, no port strip), plus the fold pairs in domain.go/Subdomains and three ToLower(host) copies in client/cookiejar. IDNA alone runs with three profiles in three directions: hostauthorization idna.Lookup.ToASCII, req.go Subdomains idna.Lookup.ToUnicode, middleware/redirect a custom lenient idna.New profile. Target: one tiered helper (Fold / Strip / Canonical) with IDNA as a named policy; prefer fasthttp's already-clean URI().Host() over re-folding the raw header (note: only the parse path validates, SetHost does not).
  • 3. Route pattern preprocessing: the trio "lowercase when !CaseSensitive, trim trailing / when !StrictRouting, RemoveEscapeChar" exists three times in router.go (addPrefixToRoute, normalizePath, register) and again in path.go RoutePatternMatch; RemoveEscapeChar has string and bytes twins. getGroupPath trims the prefix and mount.go, domain.go and router.go trim its return value again. Target: one NormalizePattern helper; configDependentPaths stays the single request-time pass, with a comment stating that fiber routes on PathOriginal() (raw) by design, that UnescapePath is the single path decode (see fix-first item 3 for the extractor interplay), and that fasthttp's normalized Path() is only consumed by the fasthttp.FS flows.
  • 4. Location composition: the guard level depends on the entry point. Back() normalizes via inline copies in redirect.go normalizeRefererURL (verbatim re-implementation of urlnorm.AsBrowserReads/StripTabCRLF; middleware/redirect and router.go already use the shared helpers, redirect.go is the lone holdout) plus backslash->slash and a leading-slash-run collapse to two slashes, where urlnorm.RootedPath collapses to one - a policy split to reconcile. To() and Res.Location() write raw; Redirect().Route() runs RootedPath but not the strip pair. middleware/redirect normalizes rule targets at config time and again per request (the second pass is needed for substituted request data; split the passes instead of deleting one). middleware/rewrite composes with no guards; that is safe because fasthttp's URI().SetPath re-runs normalizePath, and that mechanism deserves the comment. The two glob->regex helpers anchor differently ("^"+k+"$" vs "^(?:"+k+")$", so alternation rules behave differently) and the two captureTokens diverge on trailing-slash trimming. Cleanup: a redirect.go comment still references asRoutePath, which no longer exists. Target: internal/urlnorm becomes the single pipeline with one policy table; one shared glob->regex + captureTokens helper; document the strip-vs-reject split against the client side (cluster 13).
  • 5. Comma-list header scanning: three real scanners in core (helpers.go getSplicedStrList, ctx.go headerListContainsToken, res.go headerContainsValue) plus the join/dedupe wrappers around them (joinHeaderValues, peekJoinedResponseHeader, appendUniqueValues), and four more scanners in middleware (adaptor hasCloseToken, proxy delConnectionListedHeaders, compress hasToken for Cache-Control no-transform and Vary, cache parseVary) plus cache joinedHeader. Target: internal/headerlist with Contains/ForEach/Join/AppendUnique plus a quote-aware iterator for the consumers that need it.
  • 6. ETag: middleware/etag isNoneMatch splits If-None-Match naively on commas (its own comment admits commas inside opaque-tags mis-parse) while core isEtagStale is quote-aware; etagWeakMatch duplicates core matchEtag/matchEtagStrong, which additionally factor through normalizeEtag. Target: internal/etag Parse/Match/Format used by both Fresh() and the middleware.
  • 7. Media-type parameters: three quote/escape-aware scanners (helpers.go unescapeHeaderValue + forEachMediaRange/getOffer, req.go Charset with its own state machine, the param-strip+fold pair in MediaType and etag isEventStream), plus middleware/cache unquoteCacheDirective as an orphan third quoted-pair unescaper. internal/mediatype NormalizeRequestContentType folds fasthttp's header buffer in place; the aliasing contract is documented at the function, the risk is call-site discipline, so keep one entry point. Target: internal/mediatype owns param iteration, quoted-pair unescape and param-strip, including the Cache-Control consumer.
  • 8. Forwarded headers: the per-element trim for X-Forwarded-For exists three times (extractIPsFromHeader, extractIPFromHeader forward and reverse), X-Forwarded-Host/-Proto parse separately; middleware/limiter's default key is a raw, unbounded c.IP() even when ProxyHeader-derived (contrast cache's boundKeySegment, which bounds and escapes). Target: one forwarded-header iterator; bounded-key guidance for storage keys built from client-controlled values.
  • 9. Control-char handling: six SWAR masks, not two: helpers.go quoteEscapeMask, internal/logtemplate controlScrubMask, proxy ctlOrDELMask, basicauth asciiCTLMask (byte-identical to proxy's), basicauth validHeaderMask, requestid visibleASCIIMask (a strict subset of the previous). Beyond the mask primitive, the policies disagree: the same class of user-controlled value is deleted (res.go sanitizeFilename, via unicode.IsControl, so C1 too), replaced with a space (logtemplate), rejected (basicauth, requestid, proxy, sse) or percent-encoded (quoteRawString), with HTAB exempt in some and not others. Target: one parameterized mask primitive plus an explicit policy table (which value class gets which treatment); logtemplate stays the single log scrubber.
  • 10. Quoted-string production: basicauth Go-quotes Realm and Charset via strconv.Quote, keyauth uses %q for five params (realm, error, error_description, error_uri, scope); both emit Go string syntax (\\\uXXXX), not RFC 9110 quoted-string. helpers.go quoteRawString is the correct dialect with two callers (Content-Disposition and the Link rel= value in res.go), beside res.go encodeExtValue for RFC 8187. Layering note: for Attachment/Download the CR/LF/C0 branches of quoteRawString are dead code, sanitizeFilename already deleted every control rune, and fasthttp scrubs a third time; the branches are live only for Link rel=. Target: one shared producer with one documented layering.
  • 11. SameSite and cookie layering: res.go Cookie holds two mappings (string -> http.SameSite -> fasthttp) and validates via http.Cookie.Valid() before fasthttp's silent CR/LF+semicolon mutation could trigger - that ordering is the contract, currently undocumented. middleware/session's own mapping recognizes only Strict and None and silently maps everything else to Lax, including the valid CookieSameSiteDisabled that res.go honors; it also overwrites CookieSecure in the non-None branch and never validates the config value at init. Target: one mapper honoring all four values; session validates at init; layering comment at res.Cookie.
  • 12. Filesystem path guards: middleware/static's stack in order: bytesToPathString folds literal \ to / before decoding, so the later backslash reject only fires on %5C-revealed ones; fixpoint url.PathUnescape loop (double-encoding defense on an already-decoded input); segment-wise .. scan; NUL reject; Windows volume/UNC checks; fs.ValidPath; the /__fiber_invalid__ sentinel. res.go SendFile is smaller: path.Clean only in the fs.FS branch of sendFileContentLength, the os.Stat branch runs filepath.FromSlash with no clean. Percent-decode currently has four dialects in the tree (static's fixpoint loop, middleware/redirect's lenient percentDecode, extractors' strict url.PathUnescape, client/cookiejar's escapePercent on the encode side); the shared guards should settle the stray-% policy. **Target:** shared guard helpers plus a layer table documenting which check exists because of which fasthttp behavior (the backslash guard stays because \ is not a separator on Unix).
  • 13. Client URL building: fix-first item 2 covers the substitution bug; the query side already delegates to fasthttp.Args and stays. Document (or align, see cluster 4) the philosophy split: the client rejects CTL bytes in upstream Locations while the server-side composer strips them. Related, same shape: middleware/paginate encodeQueryValues re-implements url.Values.Encode allocation-free on top of the shared utils.AppendQueryEscape; fold or document. Target: anchored, escaped substitution; one documented client/server policy split.

Acceptance

  • Each cluster ends with exactly one implementation; the duplicates are deleted, not deprecated.
  • No value is sanitized/normalized/escaped twice on any request or response path; every call site that relies on a fasthttp guarantee carries a comment naming it.
  • Outputs stay byte-identical unless a divergence was a bug; behavior changes are listed per PR.
  • The three fix-first items land as standalone PRs ahead of the clusters.

Non-issue, noted to prevent re-filing: the append on the package-level weakPrefix in middleware/etag/etag.go GenerateWeak looks racy but is not; a static []byte("W/") literal is initialized with cap == len, so every append reallocates.