out_gelf: UDP socket is never recreated after send() fails, all subsequent records are lost

Bug Report

Describe the bug

out_gelf in UDP mode creates its socket once in cb_gelf_init() and never recreates it. If that socket becomes unusable at runtime, every subsequent send() fails with the same error forever and all records are lost — the plugin keeps writing to the same broken fd until the process is restarted.

To be clear about what is and isn't being reported: a datagram lost in the network is expected for UDP and is not a bug. Here the process gets a synchronous, deterministic answer from its own kernel saying "this socket can no longer send", and discards that information.

All line references are permalinks to tag v4.2.2 (ddfef36), the version we run.

The socket is created once, in the init callback: https://github.com/fluent/fluent-bit/blob/ddfef360d7f3ac5268942c47ccc9b01864424a05/plugins/out_gelf/gelf.c#L478-L492

There are three send() sites on the UDP path and none of them touches the socket:

• gelf_send_udp_pckt(), L189-L196 — the non-chunked branch calls flb_errno() and returns -1, which does reach the engine as a retry.
• gelf_send_udp_chunked(), L170-L179 — logs via flb_errno() but has no return in the error path and ends with return 0, so it reports success even when every chunk failed. Its caller (L187-L189) discards the return value anyway. With the defaults compress true and packet_size 1420, any message that doesn't fit in one packet takes this path, so a fully failed send is reported to the engine as delivered — no retry, no dropped-records metric.
• gelf_send_udp(), L221-L226 — returns the error without logging it.

close(ctx->fd) appears only in the init error path and in cb_gelf_exit(). Nothing reconnects or recreates the socket after a send error, so FLB_RETRY just repeats send() on the same dead fd.

To Reproduce

We hit this on Kubernetes with Cilium socket load balancing, but the root cause is generic: any event that leaves a connected UDP socket disconnected reproduces it.

1. out_gelf with Mode udp pointing at a Kubernetes ClusterIP service.
2. Cilium with kube-proxy replacement and socketLB rewrites the service IP to a backend pod IP inside connect().
3. Delete that backend pod. Cilium terminates sockets pinned to it (cilium/cilium#25169, extended to pod netns in cilium/cilium#33459). For UDP the kernel path is udp_abort() → __udp_disconnect(), which disconnects the socket instead of closing the fd.
4. The socket is now unconnected, so send() — which carries no destination address — fails permanently:

[error] [/src/fluent-bit/plugins/out_gelf/gelf.c:193 errno=89] Destination address required
[error] [engine] chunk '1-1789021941.897344143.flb' cannot be retried: task_id=6, input=tail.0 > output=gelf.0

errno=89 is EDESTADDRREQ. In our case: 3800 such errors and 1885 dropped chunks in 40 minutes on a single node, and delivery never recovered on its own — only a DaemonSet restart fixed it, which matches the code above, since a new process means a new cb_gelf_init(). The real loss is likely higher, because messages taking the chunked path fail without being counted at all.

Note that this is specific to disconnection. An ordinary ICMP port-unreachable sets sk_err but leaves sk_state and the destination intact, so the next send() returns the pending ECONNREFUSED once and then recovers by itself. That case self-heals; this one does not.

Expected behavior

The UDP path should recreate the socket before retrying when send() reports that the socket itself is no longer usable.

A fix already exists in the codebase and doesn't need to be invented: out_syslog moved its UDP mode from a raw fd to flb_upstream with FLB_IO_UDP in #11728. With that, flb_upstream_conn_get() checks flb_socket_error() (SO_ERROR) before handing a pooled connection back, and flb_io_net_connect() re-runs flb_net_udp_connect() for a new one — so this class of failure is handled generically. The TCP/TLS path of out_gelf itself already works this way (L286-L292); only the UDP path holds a permanent raw fd.

Your Environment

• Version used: 4.2.2. The UDP path is unchanged on current master (f784724, 2026-09-07): plugins/out_gelf/gelf.c is byte-identical to v4.2.2 except for the copyright year. The last commit touching the file is a981f66 "plugins: update copyright year to 2026" (2026-01-01); the last functional change is 876d6e8 (2025-03-17), unrelated to the send path.
• Configuration: [OUTPUT] Name gelf / Mode udp / Compress true / Host <clusterip-service> / Port 12201, with storage.type filesystem on the input
• Environment name and version: Kubernetes 1.28.3, Cilium 1.16.19 (kube-proxy replacement, socketLB)
• Operating System and version: Ubuntu, kernel 6.8.0-138
• Filters and plugins: tail input, kubernetes filter, out_gelf output

Additional context

The same "connect once in init, never reconnect" pattern is in the other two users of flb_net_udp_connect() — out_udp (plugins/out_udp/udp_conf.c) and out_syslog before #11728 — so this looks like a shared pattern rather than something specific to GELF. Both of those at least return the error to the engine instead of reporting success.

storage.keep_rejected preserves the rejected chunks on disk, but it doesn't restore delivery, so it isn't a workaround for this.

Logs lost this way are unrecoverable for us: the pods producing them are deleted once their job finishes, so the log server is the only archive. The failure is also invisible from the outside — the server stays healthy and simply receives nothing, which makes it easy to misdiagnose as a server-side problem.