Chunk job failure race can return HTTP 500 instead of the original 404

Description

A failed dynamic-cache chunk job can intermittently return HTTP 500 instead of preserving the original API exception. This was reproduced with a cloud-backed task after removing an image referenced by its manifest.

The chunk worker consistently raises the expected rest_framework.exceptions.NotFound, but the request thread can observe the RQ job's failed status before rq_exception_handler has saved RQMetaWithFailureInfo. In that window, wait_for_rq_job() constructs the fallback ChunkCreationError("Cannot create chunk"), which becomes HTTP 500.

Steps to reproduce

Run the existing regression test repeatedly:

pytest   tests/python/rest_api/test_tasks.py::TestWorkWithTask::test_work_with_task_containing_non_stable_cloud_storage_files   --count=10 -x

The test:

1. Creates a cloud-backed task with use_cache=True.
2. Removes images_with_manifest/image_case_65_1.png after task creation.
3. Requests the original frame.
4. Expects HTTP 404 containing the missing object name.

In one observed run, repetitions 1–7 passed and repetition 8 returned HTTP 500. The failing test call took 17.41 seconds, below its 25-second timeout.

Expected behavior

The frame request returns HTTP 404 with:

The file 'images_with_manifest/image_case_65_1.png' not found on the cloud storage 'public'

Actual behavior

The request intermittently returns HTTP 500:

cvat.apps.engine.cache.ChunkCreationError: Cannot create chunk

The chunk worker still contains the correct underlying exception:

rest_framework.exceptions.NotFound:
The file 'images_with_manifest/image_case_65_1.png' not found on the cloud storage 'public'

Observed timing

For the failing job chunks:prepare-item-segment_58_chunk_0_100:

11:57:59.047  Chunk job raises NotFound
11:57:59.054  RQ starts handling the failed execution
11:57:59.063  rq_exception_handler is invoked
11:57:59.069  Request path raises ChunkCreationError
11:57:59.073  GET /api/tasks/37/data/ returns 500

Successful repetitions had a larger interval between exception-handler invocation and the HTTP response.

Suspected cause

wait_for_rq_job() treats the RQ failed status as meaning that CVAT's exception metadata is already available:

elif job_status in ("failed",):
    rq_job.get_meta()
    job_meta = RQMetaWithFailureInfo.for_job(rq_job)
    raise _build_chunk_job_failure_exception(rq_job, job_meta)

RQ publishes the failed status before CVAT's rq_exception_handler finishes saving exc_type and exc_args. When the request thread polls during that interval, _build_chunk_job_failure_exception() falls back to the generic ChunkCreationError.

Possible fix

When a chunk job is failed but RQMetaWithFailureInfo.exc_type is absent, retry metadata refresh for a short bounded interval before using the fallback exception. If metadata remains unavailable, log the job ID, raw rq_job.exc_info, and job metadata to preserve the underlying failure for diagnosis.

A focused unit test could simulate failed becoming visible one metadata refresh before exc_type and exc_args.