Answers from the incident where every dashboard looked politely wrong.
The deployment had already been marked complete when the invoice page began returning 503s.
The new container was serving traffic, the PostgreSQL migration had committed, and the feature flag was on.
A added in another repository prevented the new pod from reaching .
The application team saw errors, the database team saw a clean migration, and Platform saw green nodes.
By the time we put all three facts in one incident channel, 63 deployment messages had buried the one that mattered. “Is This Actually A DevOps Failure Or Just One Bad Deploy?” It was a DevOps failure because four teams completed valid local work and nobody owned the release handoff between them.
Calling it “just a bad deploy” would have been convenient.
We could have fixed the policy, replayed the release, written a short incident note, and carried on pretending that a green Argo CD application means a service is ready for users.
The pod was healthy according to Kubernetes.
It was also unable to call a dependency required to render an invoice.
Both things can be true, which is why a deployment status alone is a fairly poor witness.
Our old release process had hidden contracts in too many places: The service repository declared its image and Helm values.
The infrastructure repository held network rules.
Database migrations ran from a separate GitHub Actions workflow.
Feature flags lived in LaunchDarkly, owned by whoever had last touched the feature.
The runbook lived in Confluence, where it had last been edited in February.
We’ve started putting the release dependencies in the service repository, close to the code that needs them.
It is not a clever system.
It is a file that a human can read during an incident and a pipeline can check before promotion.
The pipeline does not yet validate every field.
It checks that the named NetworkPolicy exists in the target cluster and runs the smoke test after the migration.
That alone would have caught Monday’s problem before 11% of invoice-page requests failed.
We dislike using “DevOps” as a department name.
In this case, it means the work of making handoffs visible before production discovers them for us. “Do Developers Really Need Production Access?” Yes, but they need narrow production access tied to the service they own, not a shared credential passed around like an office spare key.
For too long, our answer was to give engineers log access and tell them to ask Platform for anything else.
That worked while we had six services and one cluster.
We now have 34 services, three production clusters, and an on-call rota where the person holding the pager may know the application code better than the person who built the cluster.
The useful access is usually boring: Read logs, events, and deployment state in the team namespace.
View configured secrets by name, never by value.
Restart a failed deployment through a recorded workflow.
Fetch an approved diagnostic profile during an incident.
Read relevant cloud metrics and traces.
The access we do not hand out includes cluster-wide secret reads, direct production database writes, and unrestricted into pods.
There are rare reasons for each.
Those reasons deserve an incident ticket and a second human looking at the command.
Our Kubernetes groups now map to service ownership.
The team cannot casually inspect , and Platform cannot claim every namespace is somebody else’s concern.
The empty permission is deliberate.
We had one incident in May where an engineer fixed a live configuration file inside a container, then forgot it existed.
The next rollout erased it, as containers are designed to do.
We spent forty minutes rediscovering the same failure with more coffee involved.
Kubernetes has plenty of sharp edges here, so we keep the policies close to the upstream RBAC model rather than inventing an internal permissions language. “Why Can’t Platform Just Own Every Deployment?” Platform can own the deployment path; it cannot own the operational knowledge of every service.
We tested the “Platform approves production” model last autumn.
It looked safe on a process diagram.
In practice, it put two platform engineers between every product team and a release, including changes that only altered invoice wording or a CSS bundle.
The queue grew, people started sending urgent messages, and approval became a ritual performed after somebody had already decided to deploy.
That is gatekeeping with better YAML.
The service team now owns whether its code should go live.
Platform owns the tooling that makes the decision visible, repeatable, and reversible: the GitHub Actions templates, the Argo CD projects, the cluster policies, and the audit trail.
Security owns the controls that must be true before the workflow can issue production credentials.
Those boundaries are written down because informal agreement lasts exactly until the person who remembers it is on holiday.
We use for files that change shared deployment behavior.
A service team can change its replica count or an application environment variable.
A change to a shared ingress class, an organisation-level action, or a production NetworkPolicy requires Platform review.
GitHub documents the awkward parts of CODEOWNERS matching and branch protection, including the fact that a file can have one visible owner line while several teams assume they own it.
We found two of those in the first review.
The rule is not “Platform stays out of releases.” We still join risky database changes, regional failovers, and anything touching shared ingress.
We simply refuse to become the human merge queue for routine service work. “What Does ‘You Build It, You Run It’ Mean On Pager?” It means the team that changes a service takes first call for its user-facing failures, with Platform covering failures in the shared runtime.
We had to make that sentence more precise than it sounds. “Run it” had become a polite way to tell application engineers that they owned every alert, including node pressure, expired cluster certificates, and a broken Fluent Bit daemonset.
Nobody learns much from a pager at 03:00 that says “the whole platform is vaguely unhappy.” Our current split is based on the failing layer.
If returns 5xx responses because of a bad release, Catalog gets paged.
If every namespace loses DNS resolution, Platform gets paged.
If the application cannot reach Redis because its own connection pool is exhausted, Catalog gets paged; if the Redis service itself is unavailable, the data platform rotation gets paged and Catalog gets an incident notification.
That sounds obvious after writing it down.
It was not obvious in PagerDuty, where 19 services had a generic escalation policy called .
We changed the alert annotation format so it carries an owner, a runbook URL, and a statement of what user action is failing.
The last field has reduced some thoroughly unhelpful alerts.
Mina, who has been on our platform rota longer than the current Grafana folder structure, objected to putting ownership in annotations.
Her point was reasonable: labels rot.
She was right, so the alert rule now validates the owner against our service catalog during CI.
We still have stale runbooks, but now they fail a check instead of waiting quietly for a bad night.
For alert design, we keep returning to the Google SRE Workbook guidance on paging: page for urgent user impact, not every noisy internal symptom.
It has saved us from several clever alerts that nobody could act on. “Are DORA Metrics Going To Become Another Scorecard?” They will if we publish team rankings, so we do not.
We track deployment frequency, lead time, change failure rate, and time to restore because they help us locate friction in the delivery system.
The DORA research is useful as a vocabulary.
It does not grant permission to declare that the team with fewer deployments is failing.
Our billing team deploys three or four times a week.
The reporting team deploys its batch export service a