vmauth logs errors for empty AWS NLB connectivity-test connections when PROXY protocol is enabled
Describe the bug
vmauth logs an error when an accepted connection sends no bytes while -httpListenAddr.useProxyProtocol=true is enabled.
This happens for us with an AWS Network Load Balancer TLS listener. AWS performs listener connectivity tests by opening TCP connections from the NLB address to registered targets without sending any data:
https://docs.aws.amazon.com/elasticloadbalancing/latest/network/target-group-health-checks.html
These expected connections eventually produce errors such as:
cannot read proxy proto conn for TCP addr "<nlb-private-ip>:<port>": cannot read proxy protocol header: read tcp4 <vmauth-pod-ip>:8427-><nlb-private-ip>:<port>: read: connection timed outThe source address was confirmed to be an ENI belonging to the NLB.
The NLB target group has proxy_protocol_v2.enabled=true. Regular NLB health checks and customer connections provide valid PROXY protocol headers and are working correctly. All targets remain healthy.
Could vmauth avoid to log an error when no bytes were received before the PROXY protocol read timed out?
To Reproduce
Run
vmauthwith:-httpListenAddr=0.0.0.0:8427 -httpListenAddr.useProxyProtocol=trueRegister it as a target of an AWS Network Load Balancer target group.
Enable PROXY protocol v2 on the target group.
Add a TLS listener to the NLB.
Wait for an AWS TLS listener connectivity test that opens a connection to the target without sending data.
Observe the PROXY protocol read timeout in the
vmauthlogs.
The AWS-generated connectivity tests are not always easy to reproduce on low-traffic load balancers. I observed them continuously on a busy production NLB, but did not see them on otherwise identical low traffic test NLB.
Version
vmauth v1.148.1-enterpriseThe relevant behavior is implemented in the open-source lib/netutil/proxyprotocol.go code.
Logs
{"level":"error","caller":"VictoriaMetrics/lib/netutil/proxyprotocol.go:32","msg":"cannot read proxy proto conn for TCP addr \"<nlb-private-ip>:<port>\": cannot read proxy protocol header: read tcp4 <vmauth-pod-ip>:8427-><nlb-private-ip>:<port>: read: connection timed out"}Screenshots
No response
Used command-line flags
-httpListenAddr=0.0.0.0:8427,0.0.0.0:8426
-httpListenAddr.useProxyProtocol=true,falseAdditional information
AWS states that PROXY protocol headers are included in normal NLB health-check connections:
AWS separately documents TLS listener connectivity tests that originate from the NLB address and contain no data packets:
https://docs.aws.amazon.com/elasticloadbalancing/latest/network/target-group-health-checks.html
Source: VictoriaMetrics/VictoriaMetrics