bug: retain Nacos service cache when a per-service refresh query fails
Current Behavior
In the Nacos discovery refresh path, a per-service Nacos query failure can be treated as if the service were no longer referenced by APISIX when another service refresh succeeds.
In APISIX 3.13.0, fetch_full_registry builds service_names only after get_url succeeds. If service B fails to refresh, B is absent from service_names. The cleanup loop then deletes keys present in curr_service_in_use but absent from service_names.
This conflates whether a service remains referenced by APISIX configuration with whether its Nacos query succeeded. A transient failure for B can therefore delete B's last successful registry snapshot from the nacos shared dict. Subsequent requests for B can fail with no valid upstream node, although a usable cached snapshot existed before the refresh.
This is disruptive during a partial Nacos/API/network failure: service A refreshes successfully, service B times out or returns an error, and only B loses its cache.
Expected Behavior
A Nacos query failure should mean that the latest state is unknown, not that the service has been removed from APISIX configuration.
For a service still referenced by an APISIX route, upstream, service, or stream route, a failed refresh should retain its last successful cache entry. A successful refresh should update that entry normally. A cache entry should be deleted only after the service is no longer referenced by APISIX configuration.
Please also confirm the desired semantics for a successful but empty hosts response. This proposal intentionally does not change the 3.13.0 behavior for that case; it only changes the error path.
Error Logs
No production log is attached. The relevant failure is logged as:
get_url: <instance-list-path> err: <error>
The refresh then continues and the cleanup phase can delete the failed service's prior cache entry.
Steps to Reproduce
- Configure two Nacos-discovered services, A and B, and allow an initial refresh to populate both entries in the
nacosshared dict. - On a later refresh, make the instance-list query for A succeed.
- Make the instance-list query for B fail, for example with a timeout, network failure, or HTTP error.
- In 3.13.0, B is not added to
service_names, because that assignment happens only afterget_urlsucceeds. - The cleanup loop sees B in
curr_service_in_usebut not inservice_namesand deletes B's cached entry. - Send a request to an APISIX upstream using B. It may fail because discovery no longer finds B's nodes.
Proposed fix:
- Rename
curr_service_in_usetocurr_configured_services. - Build
configured_servicesfrom APISIX configuration before each Nacos query. - Calculate the cache key and record
configured_services[key] = truebefore callingget_url. - On a query error, log and skip only that service's cache update.
- Delete a cached key only when it was configured in the preceding refresh but is absent from the current
configured_servicesset.
This keeps a failed service's previous cache entry available, while a service removed from APISIX configuration is still cleaned up on the next refresh.
Environment
- APISIX version: 3.13.0
- Discovery type: Nacos
- Relevant source file:
apisix/discovery/nacos/init.lua - Cache storage:
lua_shared_dict nacos 10m - Reproduced by code-path analysis on the 3.13.0 branch; no operating-system-specific behavior is required.
Source: apache/apisix