Executor deletes the context directory in multi-stage builds resulting in loss of command pathing

Author: rcolletteCreated Feb 6, 2021Updated Jun 2, 2025

Labelsarea/multi-stage buildswork-around-availablepriority/p1works-with-dockerdiffers-from-dockerissue/missing-filescategorized

Actual behavior The context directory is deleted when targeting a second stage of a multi-stage build when the second stage relies on a prior stage.

Expected behavior The context directory should not be deleted.

To Reproduce Steps to reproduce the behavior:

Start an interactive session of the debug executor with: docker run -it --entrypoint sh gcr.io/kaniko-project/executor:debug-v1.3.0
Run the following commands to create a builds directory and Dockerfile

bash

mkdir builds
cd builds
vi Dockerfile

Insert the following into the Dockerfile and save

dockerfile

FROM node:14.15-alpine3.12 AS base
LABEL type="build"

FROM base as build-and-test
LABEL type="build-and-test"

/kaniko/executor --context /builds --no-push --target build-and-test

Observe the following

INFO[0000] Resolved base name node:14.15-alpine3.12 to base 
INFO[0000] Resolved base name base to build-and-test    
INFO[0000] Retrieving image manifest node:14.15-alpine3.12 
INFO[0000] Retrieving image node:14.15-alpine3.12       
INFO[0000] Retrieving image manifest node:14.15-alpine3.12 
INFO[0000] Retrieving image node:14.15-alpine3.12       
INFO[0001] Built cross stage deps: map[]                
INFO[0001] Retrieving image manifest node:14.15-alpine3.12 
INFO[0001] Retrieving image node:14.15-alpine3.12       
INFO[0002] Retrieving image manifest node:14.15-alpine3.12 
INFO[0002] Retrieving image node:14.15-alpine3.12       
INFO[0002] Executing 0 build triggers                   
INFO[0002] Skipping unpacking as no commands require it. 
INFO[0002] LABEL type="build"                           
INFO[0002] Applying label type=build                    
INFO[0002] Storing source image from stage 0 at path /kaniko/stages/0 
INFO[0006] Deleting filesystem...                       
INFO[0006] Base image from previous stage 0 found, using saved tar at path /kaniko/stages/0 
INFO[0006] Executing 0 build triggers                   
INFO[0006] Skipping unpacking as no commands require it. 
INFO[0006] LABEL type="build-and-test"                  
INFO[0006] Applying label type=build-and-test           
INFO[0006] Skipping push to container registry due to --no-push flag 
sh: getcwd: No such file or directory
(unknown) #

The current working directory has been deleted.

Commands from the current working directory will not function

(unknown) # ls
sh: getcwd: No such file or directory
(unknown) # /busybox/ls
sh: getcwd: No such file or directory

When running with the Gitlab Runner Operator for Openshift, this behavior causes a loss of command pathing and the ability to locate the busybox directory itself, even if you first change directories to root (outside the current context)

.gitlab-ci.yaml

yaml

stages:
  - build

# TEMPLATES
.runner_tags_template: &runners
  tags:
    - pdx
    - dind

.except_master_and_prodfix_template: &except_master_and_prodfix
  except:
    - /^prodfix\/.*$/
    - master
    - tags

# BUILD
build:
  stage: build
  <<: *runners
  <<: *except_master_and_prodfix
  image:
    name: gcr.io/kaniko-project/executor:debug-v1.3.0
    entrypoint: ["sh"]
  script:
    - cd /
    - pwd
    - /kaniko/executor
      --context $CI_PROJECT_DIR
      --no-push
      --target build-and-test
    - cd /
    - pwd
    - cd /busybox
    - ls

gitlab build log (note the directory not found message appears in the log before the command that caused it)

[0KRunning with gitlab-runner 12.9.0 (4c96e5ad)
[0;m[0K  on pdx-gitlab-3-runner-d7f85cf7f-stxzl _TrpUuzy
[0;msection_start:1612649568:prepare_executor
[0K[0K[36;1mPreparing the "kubernetes" executor[0;m
[0;m[0KUsing Kubernetes namespace: gitlab-runners
[0;m[0KUsing Kubernetes executor with image gcr.io/kaniko-project/executor:debug-v1.3.0 ...
[0;msection_end:1612649568:prepare_executor
[0Ksection_start:1612649568:prepare_script
[0K[0K[36;1mPreparing environment[0;m
[0;mWaiting for pod gitlab-runners/runner-trpuuzy-project-20222355-concurrent-0dhzst to be running, status is Pending
Waiting for pod gitlab-runners/runner-trpuuzy-project-20222355-concurrent-0dhzst to be running, status is Pending
Running on runner-trpuuzy-project-20222355-concurrent-0dhzst via pdx-gitlab-3-runner-d7f85cf7f-stxzl...
section_end:1612649574:prepare_script
[0Ksection_start:1612649574:get_sources
[0K[0K[36;1mGetting source from Git repository[0;m
[0;m[32;1mFetching changes with git depth set to 50...[0;m
Initialized empty Git repository in /builds/preciselydata/cloud/pdx/product_definition_tool/.git/
[32;1mCreated fresh repository.[0;m
From https://gitlab.com/preciselydata/cloud/pdx/product_definition_tool
 * [new ref]         7d969d80cff9d699e51b126e0359937c2cb5a526 -> refs/pipelines/252507426
 * [new branch]      testBusyBox -> origin/testBusyBox
 * [new tag]         1.0.1468.f3c0870e -> 1.0.1468.f3c0870e
 * [new tag]         1.0.1469.7169ceb0 -> 1.0.1469.7169ceb0
 * [new tag]         1.0.1470.111f6d88 -> 1.0.1470.111f6d88
 * [new tag]         1.0.1471.98614d1c -> 1.0.1471.98614d1c
 * [new tag]         1.0.1473.e1ce3ccd -> 1.0.1473.e1ce3ccd
 * [new tag]         1.0.1474.1087dd94 -> 1.0.1474.1087dd94
 * [new tag]         1.0.1475.3c04842a -> 1.0.1475.3c04842a
[32;1mChecking out 7d969d80 as testBusyBox...[0;m

[32;1mSkipping Git submodules setup[0;m
section_end:1612649580:get_sources
[0Ksection_start:1612649580:restore_cache
[0K[0K[36;1mRestoring cache[0;m
[0;msection_end:1612649580:restore_cache
[0Ksection_start:1612649580:download_artifacts
[0K[0K[36;1mDownloading artifacts[0;m
[0;msection_end:1612649580:download_artifacts
[0Ksection_start:1612649580:build_script
[0K[0K[36;1mRunning before_script and script[0;m
[0;m[32;1m$ cd /[0;m
[32;1m$ pwd[0;m
/
[32;1m$ /kaniko/executor --context $CI_PROJECT_DIR --no-push --target build-and-test[0;m
[36mINFO[0m[0000] Resolved base name node:14.15-alpine3.12 to base 
[36mINFO[0m[0000] Resolved base name base to build-and-test    
[36mINFO[0m[0000] Using dockerignore file: /builds/preciselydata/cloud/pdx/product_definition_tool/.dockerignore 
[36mINFO[0m[0000] Retrieving image manifest node:14.15-alpine3.12 
[36mINFO[0m[0000] Retrieving image node:14.15-alpine3.12       
E0206 22:12:25.176806      18 aws_credentials.go:77] while getting AWS credentials NoCredentialProviders: no valid providers in chain. Deprecated.
	For verbose messaging see aws.Config.CredentialsChainVerboseErrors
[36mINFO[0m[0006] Retrieving image manifest node:14.15-alpine3.12 
[36mINFO[0m[0006] Retrieving image node:14.15-alpine3.12       
[36mINFO[0m[0007] Built cross stage deps: map[]                
[36mINFO[0m[0007] Retrieving image manifest node:14.15-alpine3.12 
[36mINFO[0m[0007] Retrieving image node:14.15-alpine3.12       
[36mINFO[0m[0007] Retrieving image manifest node:14.15-alpine3.12 
[36mINFO[0m[0007] Retrieving image node:14.15-alpine3.12       
[36mINFO[0m[0007] Executing 0 build triggers                   
[36mINFO[0m[0007] Skipping unpacking as no commands require it. 
[36mINFO[0m[0007] LABEL type="build"                           
[36mINFO[0m[0007] Applying label type=build                    
[36mINFO[0m[0007] Storing source image from stage 0 at path /kaniko/stages/0 
[36mINFO[0m[0008] Deleting filesystem...                       
[36mINFO[0m[0008] Base image from previous stage 0 found, using saved tar at path /kaniko/stages/0 
[36mINFO[0m[0008] Executing 0 build triggers                   
[36mINFO[0m[0009] Skipping unpacking as no commands require it. 
[36mINFO[0m[0009] LABEL type="build-and-test"                  
[36mINFO[0m[0009] Applying label type=build-and-test           
[36mINFO[0m[0009] Skipping push to container registry due to --no-push flag 
/busybox/sh: cd: line 145: can't cd to /busybox: No such file or directory
[32;1m$ cd /[0;m
[32;1m$ pwd[0;m
/
[32;1m$ cd /busybox[0;m
section_end:1612649590:build_script
[0Ksection_start:1612649590:after_script
[0K[0K[36;1mRunning after_script[0;m
[0;mtime="2021-02-06T22:12:28Z" level=error msg="exec failed: container_linux.go:349: starting container process caused \"exec: \\\"sh\\\": executable file not found in $PATH\""
exec failed: container_linux.go:349: starting container process caused "exec: \"sh\": executable file not found in $PATH"
section_end:1612649590:after_script
[0Ksection_start:1612649590:upload_artifacts_on_failure
[0K[0K[36;1mUploading artifacts for failed job[0;m
[0;msection_end:1612649590:upload_artifacts_on_failure
[0K[31;1mERROR: Job failed: command terminated with exit code 2
[0;m

With commands like ls and cp no longer functioning due to the missing busybox directory, build artifacts cannot be copied back to the context directory path, which is where they must be located for Gitlab to be able to store them using its artifacts functionality.

Additional Information

Dockerfile

dockerfile

FROM node:14.15-alpine3.12 AS base
LABEL type="build"

FROM base as build-and-test
LABEL type="build-and-test"

Build Context Nothing local required.
Kaniko Image (fully qualified with digest) gcr.io/kaniko-project/executor@sha256:473d6dfb011c69f32192e668d86a47c0235791e7e857c870ad70c5e86ec07e8c

Triage Notes for the Maintainers

Description	Yes/No
Please check if this a new feature you are proposing	- [ ]
Please check if the build works in docker but not in kaniko	- [X]
Please check if this error is seen when you use `--cache` flag	- [X]
Please check if your dockerfile is a multistage dockerfile	- [X]

Source: GoogleContainerTools/kaniko

View original on GitHub View discussion on GitHub