Google Cloud DNS provider: changes.create carries no context or timeout, so one slow response stalls the reconcile loop indefinitely

What happened:

In a production cluster, ExternalDNS stopped applying DNS record changes for ~30 minutes. No records were created or updated during the window, and the process never logged an error or restarted.

The cause was a single changes.create call to the Cloud DNS API that did not return for roughly 30 minutes, at which point the API responded 502. For that entire period the controller was blocked inside that one HTTP call. Once it returned, ExternalDNS recovered on its own and applied a catch-up batch covering everything that had backlogged.

Seven downstream workloads were waiting on records that never appeared. Pods that depend on DNS resolution at startup restarted or crash-looped for the duration (one up to 13 times).

There are four places where something could have bounded this, and none of them do. Line references are against master (b6a23ab9).

1. No timeout on the HTTP client. google.DefaultClient returns a client whose Timeout is the zero value, and NewInstrumentedClient replaces Transport while leaving Timeout untouched:

// provider/google/google.go:135-142
gcloud, err := google.DefaultClient(ctx, dns.NdevClouddnsReadwriteScope)
gcloud = extdnshttp.NewInstrumentedClient(gcloud)
dnsClient, err := dns.NewService(ctx, option.WithHTTPClient(gcloud))

NewInstrumentedClient (pkg/http/http.go:77-84) mutates and returns the same *http.Client, assigning only Transport. grep -n "Timeout:" provider/google/ pkg/http/ returns no matches.

2. No context on the mutation call. This is specific to changes.create, and it is worth being precise about the scope, because the read paths are fine:

• Zones and Records use Pages(ctx, f) (provider/google/google.go:195, :231), and Pages sets c.ctx_ = ctx internally (dns/v1/dns-gen.go:4338, :6153). Those calls do carry the reconcile context and are cancellable.
• The write path does not. submitChange issues the call bare, despite already having a live ctx in scope from its own signature:

// provider/google/google.go:305
if _, err := p.changesClient.Create(p.project, zone, c).Do(); err != nil {

ChangesService.Create never sets ctx_ (dns-gen.go:2791-2797), so .Do() passes nil into gensupport.SendRequest, which has an explicit nil branch:

// google.golang.org/api/internal/gensupport/send.go:93-95
if ctx == nil {
    return client.Do(req)
}
return send(ctx, client, req)

So this one request runs with no context at all. ChangesCreateCall also uses SendRequest rather than SendRequestWithRetry, so there is no library-level retry either. A consequence: this call is not cancellable on shutdown — the SIGTERM context (controller/execute.go, contextWithSigtermHandler) has no path to it, so SIGTERM cannot interrupt it. Deleting the pod was the only way out. (time.Sleep(p.batchChangeInterval) at google.go:309 is uninterruptible for the same reason, which compounds this across many batches.)

Note also that the ctx given to dns.NewService(ctx, ...) is used for credential setup and does not propagate to requests — option.WithHTTPClient short-circuits the auth/transport wrapping entirely (api/transport/http/dial.go), and generated calls read only c.ctx_.

3. No transport-level bound. The base is http.DefaultTransport, which sets DialContext 30s, TLSHandshakeTimeout 10s, IdleConnTimeout 90s, ExpectContinueTimeout 1s — and leaves ResponseHeaderTimeout at 0. The chain is CustomRoundTripper → oauth2.Transport (with Base == nil, falling back to http.DefaultTransport). Nothing bounds time-to-first-byte on a connection that is already established. This matches the incident: the TCP connection stayed up, 30s keepalives held the socket open, and the API eventually answered after 30 minutes.

4. No liveness rescue. The reconcile loop in controller/controller.go:181 is single-goroutine and fully synchronous (no go func in the file):

for {
    if c.ShouldRunOnce(time.Now()) {
        if err := c.RunOnce(ctx); err != nil { ... }
    }
    select {
    case <-ticker.C:
    case <-ctx.Done():

RunOnce runs Registry.Records (:76) → Source.Endpoints (:89) → plan.Calculate (:118) → Registry.ApplyChanges (:121) straight-line, so a blocked ApplyChanges blocks everything, and lastSyncTimestamp (:134) stops advancing. Meanwhile /healthz is a static literal served from a separate goroutine (go serveMetrics(...), controller/execute.go:92):

// controller/execute.go:239-242
http.HandleFunc("/healthz", func(w http.ResponseWriter, _ *http.Request) {
    w.WriteHeader(http.StatusOK)
    _, _ = w.Write([]byte("OK"))
})

It reported healthy for the whole 30 minutes, so the Kubernetes liveness probe never restarted the wedged pod. The stall was therefore unbounded in time rather than capped at a restart interval.

What you expected to happen:

A slow or hung Cloud DNS API response should fail the request after a bounded interval, so the reconcile loop can log the error, return, and retry on the next iteration. A single unlucky API call should not be able to halt reconciliation for an arbitrary length of time, and should not survive SIGTERM.

For reference, Google Cloud support guidance we received on this incident states that their API frontends enforce a maximum active request processing time of 60 seconds for standard API calls, which would make a client-side timeout of ~60-65s a safe bound that does not introduce false failures. The same guidance recommends truncated exponential backoff with jitter for retries (1s initial, 2.0 multiplier, capped at 30-60s). I do not have a public documentation link for those figures, so please treat them as indicative rather than authoritative.

How to reproduce it (as minimally and precisely as possible):

The failure is in the client plumbing rather than in any particular Kubernetes resource, so it reproduces with any Source that yields at least one pending change. The stall must be induced on the changes.create request specifically — a blanket network blackhole is not a valid reproduction, because the first API call in RunOnce is Registry.Records → Zones → Pages(ctx, ...), which carries the reconcile context and therefore aborts correctly on SIGTERM. Blackholing everything demonstrates a different, non-buggy code path.

Two ways to target the write path:

1. Stub changesServiceInterface (provider/google/google.go:61) with a Create whose Do() blocks indefinitely, and drive submitChange with a non-empty change. This is the minimal unit-level reproduction.
2. For an end-to-end reproduction, place a method- and path-selective proxy in front of the API that accepts and then never answers only POST /dns/v1/projects/*/managedZones/*/changes, passing all other requests through. http.DefaultTransport honours ProxyFromEnvironment, so HTTPS_PROXY is a viable injection point, but the proxy must be selective — and note that with an http:// proxy the hang is on the CONNECT response, before any TLS handshake with the origin.

Expected observation either way: the process blocks in submitChange indefinitely, lastSyncTimestamp stops advancing, no error is logged, /healthz continues to return 200, and SIGTERM does not terminate the process.

Suggested fix, in rough order of value:

1. Set an explicit Timeout on the *http.Client before passing it to option.WithHTTPClient. This alone bounds the hang, because client.Do(req) honours client.Timeout even on the nil-context path (net/http/client.go, setRequestCancel installs a deadline context when the request has none; the no-op CustomRoundTripper.CancelRequest at pkg/http/http.go:54 does not defeat this, and oauth2.Transport clones with req.Context(), preserving it). Ideally configurable — a provider-scoped flag such as --google-request-timeout would sit naturally alongside the existing --google-batch-change-interval and --google-batch-change-size (pkg/apis/externaldns/types.go:566-567). Note that --request-timeout is not the right home for this: it is documented as Kubernetes-API-only and is already deprecated in favour of --kube-api-request-timeout (types.go:713-714). One caveat worth stating: http.Client.Timeout does not bound the oauth2 token refresh that happens inside oauth2.Transport.RoundTrip, which uses http.DefaultClient internally — a hung token endpoint would remain unbounded, though it is at least cancellable.
2. Pass the context on the create call — .Context(ctx).Do() at google.go:305, using the ctx already threaded into submitChange. This fixes the un-cancellable-on-SIGTERM behaviour and is arguably a separate defect. While there: GoogleProvider.ctx (google.go:125) is dead code — grep -c 'p\.ctx' provider/google/google.go returns 0, so its doc comment ("The context parameter to be passed for gcloud API calls") is aspirational and the field can be removed. The time.Sleep at :309 should likewise become a select on ctx.Done().
3. Optionally, retry with truncated exponential backoff and jitter per the guidance above.

I am happy to open a PR for (1) and (2) if the approach sounds right.

Anything else we need to know?:

This was previously reported in #1264 (November 2019), which described the absence of a timeout on the Cloud DNS client. That issue is closed as completed, by PR #1271 ("Use non-deprecated initializer with context"), which moved construction from the deprecated dns.New() to dns.NewService() — the change #1264 asked for. But that move did not by itself introduce any timeout, a point raised in the #1264 thread at the time and never followed up: dns.NewService() made a context possible to plumb, and the create call still does not plumb one. So the substantive gap #1264 described is still fully present six years later, which is why I am filing fresh rather than reopening.

The /healthz observation in point 4 is a distinct concern — a liveness endpoint decoupled from loop progress means no provider can be restarted out of a wedged state, and lastSyncTimestamp already exists as a signal that could inform it. Happy to file that separately if you would prefer to keep this issue scoped to the Google provider. (#6670 appears to describe this same class of unbounded-provider-call problem for PowerDNS.)

Environment:

• External-DNS version: observed in production on v0.19.0. I re-checked the relevant code at v0.22.0 (latest release) and on master (b6a23ab9) and it is unchanged: no Timeout: in provider/google/ or pkg/http/, no context on the changes.create call site, and /healthz still a static 200.
• DNS provider: Google Cloud DNS
• Others: Go client google.golang.org/api v0.297.0, golang.org/x/oauth2 v0.37.0; relevant flags --google-batch-change-interval=3s, --google-batch-change-size=500, --google-zone-visibility=public, --policy=sync, --txt-prefix, sources service and ingress.

Checklist

• I have searched existing issues and tried to find a fix myself
• I am using the latest release, or have checked the staging image to confirm the bug is still reproducible — code path verified unchanged at v0.22.0 and master
• I have provided the actual process flags (not Helm values)
• I have provided kubectl get <resource> -o yaml output including status — not applicable; the defect is in provider HTTP client construction and is independent of which resource produced the change
• I have provided full external-dns debug logs — the pathological window produced no log output at all, which is part of the report; happy to supply surrounding logs if useful
• I have described what DNS records exist and what I expected