用 JSON 线条来实施 AI 流化 响应

Background When streaming AI chat responses, Server-Sent Events (SSE) are commonly used. They are also adopted by APIs from OpenAI and Anthropic, as well as by MCP server responses. In fact, I implemented several AI chat projects that modified responses from AI platforms while streaming them to the browser. In doing so, I encountered an issue where SSE did not work because of certain intermediary proxies and load balancers, such as AWS App Runner. After taking a closer look at the SSE specification, I no longer felt that using SSE was right when the purpose was not actually event notification. The API's block data itself is JSON. This is also the same format as the structured logging sent to services such as CloudWatch Logs today (I had already been working on structuring application logs as JSON). Moving from SSE to JSON Lines Chunked Communication What I came up with was a combination of and (which is not defined by the IAEA). With this approach, even if a proxy or load balancer buffers the response and returns it as a single body rather than chunks, only streaming is lost; the final complete data remains unchanged. Because it is JSON Lines (NDJSON), all you need to do is split on line feeds (LF) and JSON-parse each line. It is also easy to inspect in browser developer tools. However, implementing this from scratch every time is a bit of work, so I implemented and published jsonl-webstream, an npm library of stream utilities for browsers and servers (Node.js). The library has zero dependencies. tilfin / jsonl-webstream Lightweight library for JSON Lines web stream between browsers and Node.js environments jsonl-webstream Lightweight library for JSON Lines web stream between browsers and Node.js environments Overview This library provides utilities for processing JSON Lines formatted data through the Web Streams API It enables efficient streaming of JSON Lines data with minimal memory overhead across browsers and Node.js environments. Installation Usage Reading JSON Lines Writing JSON Lines … View on GitHub I named the library webstream to emphasize that it uses the Web Streams API, rather than Node.js's traditional Stream. At present, the two inevitably tend to coexist because of library support, but I would like to move clearly toward Web Streams. How to Use jsonl-webstream Server utility function: When you call with no arguments, it returns a stream and a writer. The stream is a ReadableStream itself and can be used as an API response. The writer instance lets you send data with , mainly while processing responses from an AI platform. Finally, just call . You can set a callback handler with for when the connection is closed on the client side (in the browser). Client utility function: Simply call it as after . This stream is also a ReadableStream, so it handles the server response transparently. Inside a loop, append the contents of each data object to the UI. Implementation Server Immediately return the from as a , then call from the separate AI invocation process. Convert each AI response chunk into the data you need and send it. Client Pass to . When you read the returned with , each JSONL line is returned as an object, so you can add the received delta directly to the UI. When the stop button calls on the reader, is called on the server. Supplementary Material Comparing Raw Data: SSE vs. JSON Lines Chunked Communication The key-value structure is removed, making the data one level flatter, more compact, and free of blank lines in between. text/event-Stream application/jsonl