[ Bug]: selenium/node-kubernetes pod restarts automatically

What happened?

Summary

The test-node-kubernetes pod is restarting automatically without any manual intervention. The restart is observed even after increasing the liveness probe timeoutSeconds from 1 second to 10 seconds.

Environment

• Namespace: spr-ops
• Deployment: test-node-kubernetes
• Component: Selenium Grid Kubernetes Node

Current Status

kubectl get pods -n spr-ops | grep test

test-node-kubernetes-85485bd7fc-82sx6    1/1 Running   4 (118m ago)   15h
test-selenium-hub-84f5f85bfb-7pdtb       1/1 Running   0              9h

The pod has restarted 4 times within the last 15 hours.

Findings

1. Container Exit Code

Last State:
  Terminated
  Reason: Error
  Exit Code: 143

Exit Code 143 indicates the container received a SIGTERM signal (128 + 15), which means Kubernetes gracefully terminated the container instead of the application crashing on its own.

2. Container Logs

The logs consistently show a graceful shutdown sequence.

WARN received SIGTERM indicating exit request
INFO waiting for selenium-grid-kubernetes to die
WARN stopped: selenium-grid-kubernetes (terminated by SIGTERM)
Shutdown complete

This indicates:

• The application is not crashing unexpectedly.
• Kubernetes is explicitly sending a SIGTERM signal.
• The application performs a graceful shutdown after receiving the termination signal.

3. Liveness Probe Configuration

Current configuration:

livenessProbe:
  httpGet:
    path: /readyz
    port: 5555
  timeoutSeconds: 1
  periodSeconds: 10
  failureThreshold: 3

This configuration means Kubernetes:

• Performs a health check every 10 seconds.
• Waits only 1 second for the /readyz endpoint to respond.
• Restarts the container after 3 consecutive failures (approximately 30 seconds).

This behavior is consistent with the observed SIGTERM-based shutdown.

Change Attempted

To verify whether the aggressive timeout was causing false liveness failures, the following change was made:

timeoutSeconds: 10

However, the pod continued to restart, indicating that increasing the timeout alone did not resolve the issue.

Expected Behavior

The test-node-kubernetes pod should remain healthy and continue running unless:

• the application becomes unhealthy,
• the node is drained,
• a deployment rollout occurs, or
• another Kubernetes lifecycle event intentionally restarts the pod.

Actual Behavior

The pod receives a SIGTERM from Kubernetes and restarts automatically, despite increasing the liveness probe timeout.

Impact

• Unexpected Selenium node restarts.
• Existing browser sessions may be interrupted.
• Potential instability in Selenium Grid during test execution.

Investigation Completed

• Verified restart count using kubectl get pods.
• Verified container exit code (143 / SIGTERM).
• Reviewed container logs confirming graceful shutdown.
• Reviewed liveness probe configuration.
• Increased timeoutSeconds from 1 to 10.
• Restart issue still persists.

Next Steps / Assistance Required

Further investigation is required to determine what is triggering the SIGTERM.

Command used to start Selenium Grid with Docker (or Kubernetes)

kubectl deploy test-selenium-hub -n spr-ops

Relevant log output

WARN received SIGTERM indicating exit request
INFO waiting for selenium-grid-kubernetes to die
WARN stopped: selenium-grid-kubernetes (terminated by SIGTERM)
Shutdown complete

Operating System

mac

Docker Selenium version (image tag)

selenium/node-kubernetes:4.46.0-20260707

Selenium Grid chart version (chart version)

No response