#1312·memcached

proxy: `delete <key> noreply` rejected as malformed and silently dropped; error-path replies sent for noreply commands (1.6.45)

Author: blidecCreated Aug 21, 2026Updated Aug 24, 2026

Describe the bug

In proxy mode (1.6.45) two related noreply problems occur on the ASCII protocol:

  1. delete <key> noreply is rejected as malformed and the delete is silently dropped. The proxy answers with CLIENT_ERROR bad command line format. Usage: delete <key> [noreply] — note the usage string permits exactly the syntax it just refused. The key is not deleted.
  2. Error-path replies are sent for noreply commands. Demonstrated with incr against a non-numeric value: the proxy returns CLIENT_ERROR cannot increment or decrement non-numeric value, where a non-proxy memcached stays silent.

Both matter because a client that sends noreply does not read a reply. The unread response stays in the socket, so every subsequent response on that connection is shifted by one, and the visible failure surfaces on some later unrelated command.

To Reproduce

memcached 1.6.45 in proxy mode, minimal routelib config (one pool, route_direct), one backend. Raw socket, reading with a short timeout so "no reply" is distinguishable from a reply:

set K 0 60 3      -> STORED
abc
delete K noreply  -> CLIENT_ERROR bad command line format.  Usage: delete <key> [noreply]
get K             -> VALUE K 0 3 / abc / END      <-- delete was dropped
delete K          -> DELETED                      <-- same proxy, same connection, works
get K             -> END

Per-verb comparison, each on a fresh connection, value abc (non-numeric) pre-set:

noreply command non-proxy memcached proxy 1.6.45
set K 0 60 3 noreply silent (correct) silent (correct)
touch K 60 noreply silent (correct) silent (correct)
incr K 1 noreply silent (correct) CLIENT_ERROR cannot increment or decrement non-numeric value
delete K noreply silent, key deleted CLIENT_ERROR bad command line format, key NOT deleted

So set/touch with noreply behave correctly in proxy mode on the happy path; delete fails unconditionally even against a healthy backend, and error paths reply regardless of noreply.

The rejection appears to happen at command-parse time, before routing: route functions configured for delete in the routelib config never run for these requests, and the response prefix is CLIENT_ERROR (proxy rejecting the client's command) rather than SERVER_ERROR (backend failure).

Why this is impactful

pymemcache — and I expect other clients — defaults to default_noreply=True, and its delete() returns True unconditionally without reading a reply. So through a proxy, every cache invalidation is silently dropped while the application is told it succeeded, leaving stale entries until TTL. The queued CLIENT_ERROR then surfaces on a later get/set, which makes the tracebacks look unrelated to deletes.

The 1.6.15 release notes include "proxy: hacky method of supporting noreply/quiet", so noreply support in proxy mode appears intended rather than absent.

I'm aware the protocol docs advise against noreply on the ASCII protocol because errors cannot be aligned with requests, and we are disabling it client-side regardless. Filing this because the delete rejection specifically looks like a parser bug — the proxy refuses a syntax its own error message documents as valid — and because it fails on a healthy backend where a direct connection succeeds.

System Information

  • OS/Distro: Alpine Linux (musl)
  • Version of OS/distro: 3.20, Linux kernel 6.12 (aarch64)
  • Version of memcached: 1.6.45, built from upstream source, running with -o proxy_config (routelib)
  • Hardware detail: aarch64 container, 2 CPU, 768Mi
  • Control host: non-proxy memcached 1.6.22 (managed service), same commands, TLS

Caveat on the control: it is 1.6.22 rather than 1.6.45, so the comparison is proxy-vs-non-proxy across two versions. I was not able to test a non-proxy 1.6.45. Happy to re-run against any build you suggest.

Detail

stats settings (proxy), relevant excerpt:

STAT maxconns 65536
STAT tcpport 11211
STAT num_threads 2
STAT binding_protocol proxy
STAT auth_enabled_sasl no
STAT auth_enabled_ascii no
STAT item_size_max 1048576
STAT hash_algorithm murmur3
STAT verbosity 0

Routelib config is a single pools{} entry with route_direct to it, settings{ backend_connect_timeout = 2, backend_retry_waittime = 5, tcp_keepalive = true }, and verbose/debug/trace all false. Full stats / stats settings output available on request.