Datahub system update fails with timeout on large Elasticsearch index reindexing

Describe the bug When upgrading to DataHub version v1.3, the datahub system update process fails at the BuildIndicesStep during Elasticsearch index reindexing. The operation times out after 30 seconds, causing the entire upgrade process to fail. This issue only occurs for indices containing a large amount of data.

To Reproduce

1. Upgrade an existing DataHub Helm release in a Kubernetes environment with large Elasticsearch indices.
2. The upgrade process will trigger the datahub-system-update Kubernetes Job (defined in templates/datahub-upgrade/datahub-system-update-job.yml).
3. The job will start, and the pod logs will show it progressing through the update steps.
4. The job will fail and the pod will enter an Error state when it reaches the BuildIndicesStep (Step 4/14). The pod logs will show the timeout error.

Expected Behavior The datahub-system-update job should complete successfully as part of the Helm upgrade, even with large indices. The reindexing operation should finish within the allocated time, or the timeout should be configurable to handle large indices.

Log Excerpt with Error

2026-01-28 12:56:12,612 [main] ERROR c.l.m.s.e.i.ESIndexBuilder:865 - Failed to reindex dh_queryindex_v2 to dh_queryindex_v2_1769604941951: Exception java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-3 [ACTIVE]
...
java.lang.RuntimeException: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-3 [ACTIVE]
...
2026-01-28 12:56:12,710 [main] INFO  c.l.d.u.impl.DefaultUpgradeReport:15 - Failed Step 4/14: BuildIndicesStep. Failed after 3 retries.

Environment:

• Elasticsearch Version: 8.17.4
• DataHub Version: v1.3.0.1
• Deployment: Official DataHub helm chart

Additional Context The failure occurs within the datahub-system-update job, which is part of the Helm release upgrade process. The root cause is a java.net.SocketTimeoutException that occurs during the Elasticsearch reindexing task. This indicates that the reindexing operation for the dh_queryindex_v2 index takes longer than the hardcoded 30-second timeout. The problem is specific to large installations with a lot of metadata. The process fails and rolls back by deleting the newly created index.

Proposed Solution To avoid the hardcoded HTTP timeout, the reindexing process should be made asynchronous. This can be achieved by using Elasticsearch's async reindex feature (_reindex API with wait_for_completion=false).

The implementation would involve:

1. Initiating the reindex operation with the wait_for_completion=false parameter. The API will immediately return a task ID for the long-running reindex job.
2. The datahub-system-update job will then use this task ID to poll Elasticsearch's Task API to monitor the progress of the reindexing task.
3. The job will wait until the task is reported as complete before proceeding to the next step of the upgrade.

This approach decouples the upgrade process from the duration of the reindexing itself, making the system more robust and scalable for large-scale deployments. It replaces a short-lived synchronous HTTP request with a long-running background task that can be monitored without blocking the main upgrade thread and causing timeouts.