[Bug]: Outage & Permanent Data Loss Window during Container Updates
What's the bug?
Target file: backend/src/providers/compute/docker.provider.ts
Architectural Flaw
DockerProvider.updateMachine() uses a "Destroy-Before-Create" approach. It stops and deletes the running, healthy container before verifying if the replacement container can pull or start.
If the new build or image pull fails, the original container is already destroyed, causing immediate total service downtime with no rollback capability. Relying on MachineGoneError DB cleanup post-failure is reactive and does not prevent the outage.
Availability & Disaster Recovery Impact: If container creation or startup fails during updateMachine() (e.g., corrupt image tag, missing registry credentials, host port collision, or container exit code 1), the original container has ALREADY been deleted. The service drops completely offline without an active container, destroying availability promises.
Implemented (current code)
const name = (current.Name ?? '').replace(/^\//, '') || params.appId;
await dockerRequest('POST', `/containers/${encodeURIComponent(params.machineId)}/stop`).catch(
() => undefined
);
await dockerRequest(
'DELETE',
`/containers/${encodeURIComponent(params.machineId)}?force=true`
);
const launched = await this.launchMachine({
appId: name,
image: params.image,
port: params.port,
cpu: params.cpu,
memory: params.memory,
envVars: params.envVars,
region: 'local',
ingress: params.ingress,
protocol: params.protocol,
scaleToZero: params.scaleToZero,
});Proposed Fix (Staged Blue-Green Strategy)
Instead of deleting the active workload upfront to overcome Docker container naming constraints, the deployment workflow should adopt a Create-Before-Destroy (Staged Blue-Green) pattern:
Staged Provisioning: Launch the replacement container under a temporary staging identifier (e.g.,
appname-stage-<timestamp>) alongside the existing workload. The original container continues to run and handle live user traffic undisturbed.Failure Isolation (Zero Downtime): If image pulling, container creation, environment loading, or port binding fails during the staging phase, the deployment operation immediately aborts. The broken staging container is discarded, while the original healthy container remains online without experiencing any service interruption or downtime.
Atomic Promotion & Teardown: Once the staging container boots successfully, safely stop and remove the superseded container, then perform an atomic Docker API rename operation (
POST /containers/{id}/rename) to promote the staging container to the primary application name.
Source: InsForge/InsForge