Retrying one failed RabbitMQ message can report "not found" even though it exists

Hi, I have an issue:

The iii::queue::redrive_message function can return redriven: 0 for a message that is still present in the failed-message queue.

"Redrive" means moving one failed message back to its normal queue so iii can try it again.

The problem happens when the selected message is behind another message in the failed-message queue. The iii RabbitMQ queue worker reads failed messages one at a time to find the requested ID. When a message does not match, iii immediately returns it to the same queue. RabbitMQ may place it near the front again, causing iii to read the same message repeatedly and never reach the requested one.

Versions

• iii engine: 0.22.1
• queue worker: 0.21.3
• queue storage: RabbitMQ
• queue type: named function queue

The current code on main appears to have the same problem:

https://github.com/iii-hq/workers/blob/main/queue/src/adapters/rabbitmq/adapter.rs#L153-L227

Steps to reproduce

1. Configure a named iii queue backed by RabbitMQ.
2. Send at least two messages that fail and move into the failed-message queue.
3. Use engine::queue::dlq_messages to list the failed messages.
4. Copy the ID of a message that is not first in the queue.
5. Call iii::queue::redrive_message with that message ID.

Example request:

{
  "queue": "__fn_queue::test-jobs",
  "message_id": "message-2"
}

The result is:

{
  "queue": "__fn_queue::test-jobs",
  "message_id": "message-2",
  "redriven": 0
}

The message still appears in engine::queue::dlq_messages, so it exists but was not found.

Expected result

iii should find the selected message, move it back to the normal queue, and return:

{
  "queue": "__fn_queue::test-jobs",
  "message_id": "message-2",
  "redriven": 1
}

Other failed messages should remain in the failed-message queue.

What appears to be happening

The RabbitMQ code checks one failed message at a time.

When the message is not the requested one, iii puts it back immediately. RabbitMQ can put it back in the same position, near the front of the queue.

The next check can therefore receive the same message again. The search stops after a fixed number of checks and reports that the requested message was not found, even though it was never reached.

Impact

Retrying one failed message is unreliable when the failed-message queue contains more than one message.

The only available workaround may be retrying every failed message at once. That can repeat work that should remain stopped.

Any ideas?