grpcproxy: range cache treats an open-ended range end ("\x00") as its start key, serializable reads through the proxy go stale after writes
Bug report criteria
- This bug report is not security related, security issues should be disclosed privately via the report form.
- This is not a support request or question, support requests or questions should be raised in the etcd discussion forums.
- You have read the etcd bug reporting guidelines.
- Existing open issues along with etcd frequently asked questions have been checked and this is not a duplicate.
What happened?
Through a grpc-proxy, a serializable get --from-key keeps returning its cached result after a put above the start key, and del --from-key leaves cached reads above the start key in place. Linearizable reads through the same proxy are correct.
The range cache builds its interval-tree entries from the raw wire RangeEnd (server/proxy/grpcproxy/cache/store.go:97 in Add, :145 in Invalidate). A RangeEnd of a single zero byte, which the server executes as "every key >= key" (mkGteRange in server/etcdserver/txn/delete.go:63, api/etcdserverpb/rpc.proto:561), becomes a StringAffineInterval whose end sorts below its begin, and such an interval only ever intersects its own start key.
Both directions on main at d848b43ae, one member, a grpc-proxy on 127.0.0.1:23790:
$ etcdctl --endpoints http://127.0.0.1:23790 put a 1
OK
$ etcdctl --endpoints http://127.0.0.1:23790 put z 1
OK
$ etcdctl --endpoints http://127.0.0.1:23790 get a --from-key --consistency=s --keys-only
a
z
$ etcdctl --endpoints http://127.0.0.1:23790 put b 2
OK
$ etcdctl --endpoints http://127.0.0.1:23790 get a --from-key --consistency=s --keys-only
a
z
$ etcdctl --endpoints http://127.0.0.1:23790 get a --from-key --consistency=l --keys-only
a
b
z
$ etcdctl --endpoints http://127.0.0.1:23790 get z --consistency=s --keys-only
z
$ etcdctl --endpoints http://127.0.0.1:23790 del a --from-key
3
$ etcdctl --endpoints http://127.0.0.1:23790 get z --consistency=s --keys-only
z
$ etcdctl --endpoints http://127.0.0.1:23790 get z --consistency=l --keys-only
WithPrefix() on an empty key sends the same wire shape (key and range end both "\x00"), so a cached whole-keyspace serializable range behaves the same way. The two lines are unchanged on release-3.6 and release-3.5.
What did you expect to happen?
The cache treats a single zero byte as an open-ended range end the way watch.go:325 and the auth range cache (server/auth/range_perm_cache.go:185) already do: the second serializable get --from-key a returns a, b, z and the last get z returns nothing.
How can we reproduce it (as minimally and precisely as possible)?
etcd with defaults, etcd grpc-proxy start --endpoints http://127.0.0.1:12379 --listen-addr 127.0.0.1:23790 (the range cache is always on), then the commands above against the proxy. A table test on cache.Cache that pins both directions with finite-range controls is in the linked PR; its six open-ended rows fail on main.
Anything else we need to know?
Same shape as the txn overlap guard fixed in #22395, which pointed at these two sites and left them out. Not security related, auth checks run on the server; this is stale serializable data from the proxy cache.
Etcd version (please run commands below)
main at d848b43ae (etcd Version: 3.8.0-alpha.0, built with scripts/build.sh); the cache code is the same on release-3.6 and release-3.5.
Etcd configuration (command line flags or environment variables)
Defaults, single member.
Etcd debug information (please run commands below, feel free to obfuscate the IP address or FQDN in the output)
N/A
Relevant log output
N/A
Source: etcd-io/etcd