[BUG] Failures with No Logs

Mage version

OSS 9.77

Describe the bug

We are observing job pod failures that produce little to no actionable logs.
Based on the attached dashboard and queries, there have been 4 such failures in the past 30 days.

These queries are intended to surface pipeline runs that failed at the orchestration / pod level, but do not emit expected application-level logs, making root-cause analysis difficult from the Mage UI alone.

What this shows (high‑level)

The dashboard titled “Failures with no Logs by Workspace, Job, and Pipeline ID” identifies:

• Workspaces (orchestration pods)
• Jobs (pipeline UUIDs)
• Pipeline runs (pipeline run IDs)

where the job failed, but:

• No meaningful Mage logs were produced or
• The failure surfaced only as infrastructure‑level errors (timeouts, exceptions, or generic errors)

These runs appear to fail before or during pipeline initialization, rather than inside a task with normal logging.

Why this matters

For these failures:

• The Mage UI often shows a failed run without useful step‑level logs
• There’s not enough information to determine whether the failure is:
  • Pod startup related
  • Infrastructure-related (node, scheduling, image pull, etc.)
  • A Mage control‑plane issue
• This makes troubleshooting difficult without:
  • Loki queries
  • Kubernetes pod inspection
  • Internal Mage context

How the queries work (conceptually)

All three queries follow the same structure:

1. Scan logs for orchestration job pods

   • Filter to known Mage orchestration pods
   • Scope to the production cluster and Mage namespace

2. Look for failure signals

   • Error-level logs
   • Exception-level structured logs
   • Pipeline timeout messages

3. Extract pipeline identifiers from logs

   • pipeline_uuid
   • pipeline_run_id

4. Aggregate failures

   • Grouped by:
     • Workspace (orch_pod)
     • Job (pipeline_uuid)
     • Pipeline run (pipeline_run_id)

Each query captures a different failure mode, but they all roll up into the same “failure with no logs” category.

<ORCH_GROUP> =
(orchestrator-a|orchestrator-b|orchestrator-c|orchestrator-d|orchestrator-e|orchestrator-f|orchestrator-g|orchestrator-h|orchestrator-i|orchestrator-j|orchestrator-k|orchestrator-l|orchestrator-m|orchestrator-n|orchestrator-o|orchestrator-p|orchestrator-q)

Query 1 — Error-level failures (excluding known noise)

This query identifies error-level log entries from orchestration pods, while explicitly excluding known non-actionable messages.

What it catches

• Generic errors emitted by the pod
• Failures that do not cleanly map to a Mage task
• Errors that occur before standard pipeline logging begins

Sanitized query

sum by (orch_pod, pipeline_uuid, pipeline_run_id)(
  label_replace(
    count_over_time(
      {
        namespace=~"<NAMESPACE>",
        pod=~".*(<ORCH_GROUP>).*-[0-9]+",
        cluster="<CLUSTER_NAME>"
      }
      | detected_level="error"
      != "does not exist in <REDACTED_PATH>"
      != "Pipeline run timed out after "
      | regexp `pipeline_run_id[^0-9]*(?P<pipeline_run_id>[0-9]+)`
      | regexp `pipeline_uuid[^a-zA-Z0-9_]*(?P<pipeline_uuid>[a-zA-Z0-9_]+)`
      [$__auto]
    ),
    "orch_pod",
    "$1",
    "pod",
    ".*(<ORCH_GROUP>).*-[0-9]+"
  )
)

Query 2 — Structured EXCEPTION events

This query looks for explicit exception-level events in structured (JSON-style) logs.

What it catches

• Hard failures emitted as "level": "EXCEPTION"
• Failures that stop execution before normal step logging begins
• Control-plane or initialization exceptions

Sanitized query

sum by (orch_pod, pipeline_uuid, pipeline_run_id)(
  label_replace(
    count_over_time(
      {
        namespace=~"<NAMESPACE>",
        pod=~"<POD_PREFIX>-(<ORCH_GROUP>)-.*",
        cluster=~"<CLUSTER_NAME>"
      }
      |= "\"level\": \"EXCEPTION\""
      | regexp "\"pipeline_run_id\": (?P<pipeline_run_id>\\d+)"
      | regexp "\"pipeline_uuid\": \"(?P<pipeline_uuid>[^\"]+)\""
      [$__range]
    ),
    "orch_pod",
    "$1",
    "pod",
    "<POD_PREFIX>-(<ORCH_GROUP>)-.*"
  )
)

Query 3 — Pipeline timeout failures

This query isolates pipeline timeouts, which are a common source of failed runs with minimal logging.

What it catches

• Pipelines that exceed execution or startup thresholds
• Runs that terminate before step-level logs are flushed or indexed

Sanitized query

sum by (orch_pod, pipeline_uuid, pipeline_run_id)(
  label_replace(
    count_over_time(
      {
        namespace=~"<NAMESPACE>",
        pod=~".*(<ORCH_GROUP>).*-[0-9]+",
        cluster=~"<CLUSTER_NAME>"
      }
      |= "Pipeline run timed out after"
      | regexp `pipeline_run_id[^0-9]*(?P<pipeline_run_id>[0-9]+)`
      | regexp `pipeline_uuid[^a-zA-Z0-9_]*(?P<pipeline_uuid>[a-zA-Z0-9_]+)`
      [$__range]
    ),
    "orch_pod",
    "$1",
    "pod",
    ".*(<ORCH_GROUP>).*-[0-9]+"
  )
)

##We’re looking for help understanding:

• Why some Mage job pods appear to fail without emitting standard pipeline logs

To reproduce

No response

Expected behavior

No response

Screenshots

No response

Operating system

No response

Additional context

No response