/workflowDefs aborts while serializing a 100+ MB /metadata/workflow response (Broken pipe)

Related to #781.

Describe the bug

The legacy Workflow Definitions page (/workflowDefs) calls GET /metadata/workflow, which loads every version of every workflow, including complete task blueprints. In a real PostgreSQL-backed deployment the response grew large enough that the server was still serializing it when the client disconnected, and Tomcat failed the socket write with Broken pipe, surfaced as AsyncRequestNotUsableException.

This is not a metadata read or workflow execution failure. Jackson serialization runs successfully and fails only when Tomcat writes to a socket the client has already closed.

Details

• Conductor version: main
• Persistence implementation: Postgres
• Affected endpoint: GET /metadata/workflow (used by the Workflow Definitions UI)
• Deployment scale when it broke:
  • 3,914 workflow definition versions
  • 1,950 distinct workflow names
  • 104,358,911-byte /metadata/workflow response
  • 16.65-second direct response time

GET /metadata/workflow/latest-versions reduces this to 1,950 definitions, but the response is still 56,587,879 bytes and takes 8.67 seconds because each item still contains the complete task blueprint.

Actual error

2026-09-08 01:04:02.007
[TID: d0382df2211145c39ffd9c9bc5d4a300.172.17888294407267261]
[http-nio-8080-exec-43]
ERROR com.netflix.conductor.rest.controllers.ApplicationExceptionMapper
- Error AsyncRequestNotUsableException url: '/api/metadata/workflow'

org.springframework.web.context.request.async.AsyncRequestNotUsableException:
ServletOutputStream failed to write: java.io.IOException: Broken pipe
    at org.springframework.web.context.request.async.StandardServletAsyncWebRequest
        $LifecycleHttpServletResponse.handleIOException(StandardServletAsyncWebRequest.java:343)
    at org.springframework.web.context.request.async.StandardServletAsyncWebRequest
        $LifecycleServletOutputStream.write(StandardServletAsyncWebRequest.java:401)
    at org.springframework.util.StreamUtils$NonClosingOutputStream.write(StreamUtils.java:261)
    at com.fasterxml.jackson.core.json.UTF8JsonGenerator._flushBuffer(UTF8JsonGenerator.java:2210)
    at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:183)
    at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serializeContents(
        IndexedListSerializer.java:119)
    at org.springframework.http.converter.json.AbstractJackson2HttpMessageConverter.writeInternal(
        AbstractJackson2HttpMessageConverter.java:483)
    at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:903)
    ... servlet filter and Tomcat request frames omitted ...

Caused by: org.apache.catalina.connector.ClientAbortException:
java.io.IOException: Broken pipe
    at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:340)
    at org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:99)
    ... 87 more

Caused by: java.io.IOException: Broken pipe
    at java.base/sun.nio.ch.SocketDispatcher.write0(Native Method)
    at java.base/sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:542)
    at org.apache.tomcat.util.net.NioChannel.write(NioChannel.java:129)
    ... 87 more

To Reproduce

1. Register enough workflow definitions/versions that /metadata/workflow serializes to tens of MB.
2. Open the Workflow Definitions page (/workflowDefs).
3. The browser issues GET /metadata/workflow?short=true&metadata=true&classifier=workflow.
4. On a slow/interrupted client, the server logs AsyncRequestNotUsableException / Broken pipe while still writing the response.

Expected behavior

The Workflow Definitions page loads its table without transferring complete task blueprints for every version, and a client disconnect is not counted as a server-side 5xx business error.

Proposed fix (preferred): a lightweight list projection

latest-versions alone is not enough — switching the page to complete latest definitions preserves its fields but the response is still ~57 MB. The definitions table needs name, description, owner, timeout settings, task types, and task count; it does not need every task's full configuration body.

A projection containing every field the table uses is ~844 KB for the same 1,950 workflows (~70 KB gzip), while preserving the current visible columns, client-side filters, and sorting.

• Keep GET /metadata/workflow unchanged for API compatibility.
• Keep GET /metadata/workflow/latest-versions response type unchanged.
• Add a latest-workflow list projection with all fields the definitions table uses, but no complete task blueprints (task blueprints reduced to distinct taskTypes + taskCount).
• Let the UI fetch the complete definition only after a user selects a workflow and version.
• Classify AsyncRequestNotUsableException as a client disconnect: do not write a second error body and do not count it as a Conductor business 5xx.
• PostgreSQL should avoid deserializing complete task blueprints for the list response; other backends keep a behaviorally equivalent fallback.

Alternative considered

Migrate the legacy UI to a paginated /latest-versions endpoint (as proposed in #843/#934). This changes the /latest-versions return type from List<WorkflowDef> to a paginated result — a breaking response-type change — and still returns complete task blueprints. The additive list endpoint avoids both problems and retains the legacy UI's complete client-side filtering and sorting.

Acceptance criteria

• /workflowDefs no longer calls the all-versions /metadata/workflow GET endpoint.
• Every currently displayed or selectable table field remains available.
• Opening a workflow still loads its complete selected version.
• Existing metadata endpoint contract tests remain unchanged and pass.
• With ~1,950 workflow names, the list response stays below 2 MB.
• Client disconnects are not counted as Conductor business 5xx errors.

Verified fix (internal dev deployment)

Measured on the deployed build, comparing the exact requests the Workflow Definitions page issues before and after the change (uncompressed, 5 runs each, warm cache):

• Response time: 5.7x faster (82.4% reduction, 7.40 s → 1.30 s).
• Payload: 52.7x smaller (98.1% reduction, 45.5 MiB → 0.86 MiB) — below the 2 MB threshold.
• Per-row weight: 32.7x lighter (~16 KB → ~0.5 KB), from dropping the full tasks blueprint in favour of taskCount + taskTypes, plus returning latest-version-per-name only.

The old request already used short=true&metadata=true (the UI's real call) yet still returned 45.5 MiB because complete task blueprints were included — confirming the projection, not just version dedup, removes the bulk of the payload.

I have this implemented with tests and can open a PR immediately if the approach is agreed.