`/ready` reports connected=true while Kafka brokers are unreachable
/ready reports connected=true while Kafka brokers are unreachable
Description
Environment
- Redpanda Connect: 4.103.1
- Previous version tested: 4.98.0
- Kubernetes
- Calico CNI
I am testing the behavior of the Redpanda Connect /ready endpoint when the Kafka brokers become unreachable.
According to the Redpanda Connect documentation:
/pingcan be used as a liveness probe as it always returns a 200.
/readycan be used as a readiness probe as it serves a 200 only when both the input and output are connected, otherwise a 503 is returned.
Documentation: Redpanda Connect Monitoring / Health Checks
Steps to Reproduce
Deploy Redpanda Connect 4.103.1 with a Kafka/Redpanda input and output.
Configure short TCP connection timeouts:
tcp:
connect_timeout: 2s
tcp_user_timeout: 2sVerify that
/readyinitially reports both input and output as connected.Use a network policy to intentionally block traffic between the Redpanda Connect pod and the Kafka brokers.
Observe that Redpanda Connect repeatedly reports broker connection failures:
level=warning msg="unable to open connection to broker"
err="dial tcp <broker-address>:<port>: i/o timeout"
path=root.input
level=warning msg="unable to open connection to broker"
err="dial tcp <broker-address>:<port>: i/o timeout"
path=root.output- Continue querying
/readywhile the brokers remain unreachable.
Actual Result
Despite continuous connection timeout errors, /ready continues to return:
{
"statuses": [
{
"path": "input",
"connected": true
},
{
"path": "output",
"connected": true
}
]
}Expected Result
Based on the documented behavior of /ready, when the input and/or output cannot establish connections, I would expect /ready to return HTTP 503 and report the affected component as disconnected.
Question
Is this behavior expected in Redpanda Connect 4.103.1?
If not, could this be a bug or regression in the /ready connection-status handling?
Source: redpanda-data/connect