Skip to content

Commit f0f69ba

Browse files
omeheganomehegan
authored
Merge pull request #38 from buildkite-plugins/SUP-6559-sparse-checkout-cleanup-option
Add `cleanup_sparse_state` option to prevent sparse-checkout state leaking between jobs
2 parents 840d142 + 7e51df7 commit f0f69ba

8 files changed

Lines changed: 287 additions & 10 deletions

File tree

‎README.md‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -38,6 +38,20 @@ Whether to perform aggressive repository cleanup before checkout. This option ha
3838

3939
Use this option for pipeline upload jobs that don't need to preserve local changes.
4040

41+
#### `cleanup_sparse_state` ('true' or 'false')
42+
43+
Tear down sparse-checkout state after the job finishes, so that subsequent jobs on the same agent that do **not** use sparse checkout are not affected.
44+
45+
When `git sparse-checkout` runs, it writes `.git/config.worktree` and sets `extensions.worktreeConfig`, `core.sparseCheckout`, and `core.sparseCheckoutCone` in git config. On agents with persistent build directories, this state persists across jobs — causing subsequent non-sparse jobs to silently inherit the sparse paths and fail to find files outside them.
46+
47+
In most setups you do not need this option. The plugin's `hooks/environment` already isolates sparse checkouts into a `-sparse`-suffixed build directory, so non-sparse jobs on the same agent land elsewhere and never see sparse state. Enable `cleanup_sparse_state` only when your agent configuration overrides `BUILDKITE_BUILD_CHECKOUT_PATH` after the plugin's env hook runs, causing sparse and non-sparse checkouts to share the same directory.
48+
49+
Cleanup runs at `pre-exit` after the job's command finishes, so the next job on the same directory starts clean. This runs regardless of job success or failure. Cleanup runs at `pre-exit` rather than `post-checkout` so that sparse state stays active for the duration of the job command.
50+
51+
A second pass runs at `pre-checkout` as a safety net for cases where a previous job's `pre-exit` never fired (for example, agent crashes or `SIGKILL`).
52+
53+
The cleanup removes `.git/config.worktree` and unsets `extensions.worktreeConfig`, `core.sparseCheckout`, and `core.sparseCheckoutCone`. The working tree files are left intact. This deliberately avoids `git sparse-checkout disable`, which re-materialises the full working tree (expensive on large monorepos).
54+
4155
#### `verbose` ('true' or 'false')
4256

4357
Enable verbose logging with bash execution tracing (`set -x`). This shows each command being executed and can help debug issues with ssh-keyscan, git operations, or other checkout problems. When enabled, you'll see detailed output including command arguments and any error messages from underlying tools.
@@ -96,6 +110,23 @@ steps:
96110
clean_checkout: true
97111
```
98112

113+
### Cleaning up sparse-checkout state when the default path isolation is overridden
114+
115+
If your agent configuration causes sparse and non-sparse jobs to share the same checkout directory (overriding the plugin's `-sparse` path isolation), sparse-checkout state can leak between them. Enable `cleanup_sparse_state` to clean this up at `pre-exit` and `pre-checkout`:
116+
117+
```yaml
118+
steps:
119+
- label: "Sparse build"
120+
command: "make build"
121+
plugins:
122+
- sparse-checkout#v1.6.0:
123+
paths:
124+
- src
125+
cleanup_sparse_state: true
126+
```
127+
128+
The plugin will clean up stale sparse config in `pre-exit` (protecting the next job on the same directory from our own sparse state) and again in `pre-checkout` (protecting the current run from a previous interrupted job where `pre-exit` never fired).
129+
99130
## Testing
100131

101132
```bash

‎hooks/pre-checkout‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
#!/usr/bin/env bash
2+
3+
set -euo pipefail
4+
5+
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
6+
7+
# shellcheck source=lib/shared.bash
8+
. "$DIR/../lib/shared.bash"
9+
10+
# shellcheck source=lib/plugin.bash
11+
. "$DIR/../lib/plugin.bash"
12+
13+
setup_error_trap
14+
15+
VERBOSE_OPTION="$(plugin_read_config VERBOSE "false")"
16+
[[ "${VERBOSE_OPTION}" = "true" ]] && set -x
17+
18+
# Clean up any stale sparse-checkout config left by a previous run (including
19+
# interrupted jobs where pre-exit never fired) before we start our own checkout.
20+
if [[ "$(plugin_read_config CLEANUP_SPARSE_STATE "false")" = "true" ]]; then
21+
cleanup_sparse_checkout_config
22+
fi

‎hooks/pre-exit‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
#!/usr/bin/env bash
2+
3+
set -euo pipefail
4+
5+
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
6+
7+
# shellcheck source=lib/shared.bash
8+
. "$DIR/../lib/shared.bash"
9+
10+
# shellcheck source=lib/plugin.bash
11+
. "$DIR/../lib/plugin.bash"
12+
13+
setup_error_trap
14+
15+
VERBOSE_OPTION="$(plugin_read_config VERBOSE "false")"
16+
[[ "${VERBOSE_OPTION}" = "true" ]] && set -x
17+
18+
# Clean up sparse-checkout config before the job exits so subsequent non-sparse
19+
# jobs on the same agent directory are not affected.
20+
if [[ "$(plugin_read_config CLEANUP_SPARSE_STATE "false")" = "true" ]]; then
21+
cleanup_sparse_checkout_config
22+
fi

‎lib/shared.bash‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -173,6 +173,29 @@ string_strip_suffix() {
173173
echo "${1%"$2"}"
174174
}
175175

176+
# ============================================================================
177+
# Sparse checkout utilities
178+
# ============================================================================
179+
180+
# Tears down sparse-checkout state without re-materialising the working tree
181+
# (unlike `git sparse-checkout disable`, which is expensive on large monorepos).
182+
cleanup_sparse_checkout_config() {
183+
if [[ ! -d .git ]]; then
184+
log_info "No .git directory found, skipping sparse-checkout config cleanup"
185+
return 0
186+
fi
187+
log_info "Cleaning up sparse-checkout config"
188+
# Paths without skip-worktree set are no-ops, so feeding every tracked path
189+
# stays index-only and cheap even on large repos. -z handles special chars.
190+
git ls-files -z | git update-index -z --no-skip-worktree --stdin || true
191+
# Unset extension first so git ignores the worktree config we're about to delete.
192+
git config --unset extensions.worktreeConfig 2>/dev/null || true
193+
rm -f .git/config.worktree
194+
git config --unset core.sparseCheckout 2>/dev/null || true
195+
git config --unset core.sparseCheckoutCone 2>/dev/null || true
196+
log_success "Sparse-checkout config cleaned up"
197+
}
198+
176199
# ============================================================================
177200
# File utilities
178201
# ============================================================================

‎plugin.yml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,9 @@ configuration:
1616
clean_checkout:
1717
type: boolean
1818
default: false
19+
cleanup_sparse_state:
20+
type: boolean
21+
default: false
1922
verbose:
2023
type: boolean
2124
default: false

‎tests/post-checkout.bats‎

Lines changed: 10 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,12 @@
22

33
setup() {
44
load "${BATS_PLUGIN_PATH}/load.bash"
5+
HOOK_DIR="$PWD"
6+
}
7+
8+
teardown() {
9+
cd "$HOOK_DIR" 2>/dev/null || true
10+
unstub git 2>/dev/null || true
511
}
612

713
@test "Unshallow enabled and repo is shallow runs git fetch --unshallow" {
@@ -10,30 +16,26 @@ setup() {
1016
stub git "rev-parse --is-shallow-repository : echo 'true'" \
1117
"fetch --unshallow origin : echo 'git fetch unshallow'"
1218

13-
run "$PWD"/hooks/post-checkout
19+
run "$HOOK_DIR"/hooks/post-checkout
1420

1521
assert_success
1622
assert_output --partial 'Unshallowing repository'
1723
assert_output --partial 'Repository unshallowed successfully'
18-
19-
unstub git
2024
}
2125

2226
@test "Unshallow enabled and repo is not shallow skips unshallow" {
2327
export BUILDKITE_PLUGIN_SPARSE_CHECKOUT_POST_CHECKOUT_UNSHALLOW="true"
2428

2529
stub git "rev-parse --is-shallow-repository : echo 'false'"
2630

27-
run "$PWD"/hooks/post-checkout
31+
run "$HOOK_DIR"/hooks/post-checkout
2832

2933
assert_success
3034
assert_output --partial 'Repository is not shallow, skipping unshallow'
31-
32-
unstub git
3335
}
3436

3537
@test "Unshallow not configured skips post-checkout operations" {
36-
run "$PWD"/hooks/post-checkout
38+
run "$HOOK_DIR"/hooks/post-checkout
3739

3840
assert_success
3941
refute_output --partial 'Unshallowing repository'
@@ -45,10 +47,8 @@ setup() {
4547
stub git "rev-parse --is-shallow-repository : echo 'true'" \
4648
"fetch --unshallow origin : exit 1"
4749

48-
run "$PWD"/hooks/post-checkout
50+
run "$HOOK_DIR"/hooks/post-checkout
4951

5052
assert_failure
5153
assert_output --partial 'Failed to unshallow repository'
52-
53-
unstub git
5454
}

‎tests/pre-checkout.bats‎

Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,78 @@
1+
#!/usr/bin/env bats
2+
3+
setup() {
4+
load "${BATS_PLUGIN_PATH}/load.bash"
5+
HOOK_DIR="$PWD"
6+
}
7+
8+
teardown() {
9+
cd "$HOOK_DIR" 2>/dev/null || true
10+
if [[ -n "$WORK_DIR" && -d "$WORK_DIR" ]]; then
11+
rm -rf "$WORK_DIR"
12+
fi
13+
unstub git 2>/dev/null || true
14+
}
15+
16+
@test "Cleanup sparse state enabled - cleans up stale sparse config before checkout" {
17+
export BUILDKITE_PLUGIN_SPARSE_CHECKOUT_CLEANUP_SPARSE_STATE="true"
18+
19+
WORK_DIR="$(mktemp -d)"
20+
mkdir -p "$WORK_DIR/.git"
21+
echo '[core]' > "$WORK_DIR/.git/config.worktree"
22+
23+
stub git \
24+
"ls-files -z : true" \
25+
"update-index -z --no-skip-worktree --stdin : true" \
26+
"config --unset extensions.worktreeConfig : true" \
27+
"config --unset core.sparseCheckout : true" \
28+
"config --unset core.sparseCheckoutCone : true"
29+
30+
cd "$WORK_DIR"
31+
run "$HOOK_DIR/hooks/pre-checkout"
32+
33+
assert_success
34+
assert_output --partial 'Cleaning up sparse-checkout config'
35+
assert_output --partial 'Sparse-checkout config cleaned up'
36+
[[ ! -f "$WORK_DIR/.git/config.worktree" ]]
37+
}
38+
39+
@test "Cleanup sparse state enabled - succeeds even when no prior sparse config exists" {
40+
export BUILDKITE_PLUGIN_SPARSE_CHECKOUT_CLEANUP_SPARSE_STATE="true"
41+
42+
WORK_DIR="$(mktemp -d)"
43+
mkdir -p "$WORK_DIR/.git"
44+
45+
stub git \
46+
"ls-files -z : true" \
47+
"update-index -z --no-skip-worktree --stdin : true" \
48+
"config --unset extensions.worktreeConfig : true" \
49+
"config --unset core.sparseCheckout : true" \
50+
"config --unset core.sparseCheckoutCone : true"
51+
52+
cd "$WORK_DIR"
53+
run "$HOOK_DIR/hooks/pre-checkout"
54+
55+
assert_success
56+
assert_output --partial 'Cleaning up sparse-checkout config'
57+
assert_output --partial 'Sparse-checkout config cleaned up'
58+
}
59+
60+
@test "Cleanup sparse state enabled - skips when no .git directory exists yet" {
61+
export BUILDKITE_PLUGIN_SPARSE_CHECKOUT_CLEANUP_SPARSE_STATE="true"
62+
63+
WORK_DIR="$(mktemp -d)"
64+
65+
cd "$WORK_DIR"
66+
run "$HOOK_DIR/hooks/pre-checkout"
67+
68+
assert_success
69+
assert_output --partial 'No .git directory found, skipping sparse-checkout config cleanup'
70+
refute_output --partial 'Cleaning up sparse-checkout config'
71+
}
72+
73+
@test "Cleanup sparse state not configured - does nothing" {
74+
run "$HOOK_DIR/hooks/pre-checkout"
75+
76+
assert_success
77+
refute_output --partial 'sparse-checkout config'
78+
}

‎tests/pre-exit.bats‎

Lines changed: 98 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,98 @@
1+
#!/usr/bin/env bats
2+
3+
setup() {
4+
load "${BATS_PLUGIN_PATH}/load.bash"
5+
HOOK_DIR="$PWD"
6+
}
7+
8+
teardown() {
9+
cd "$HOOK_DIR" 2>/dev/null || true
10+
if [[ -n "$WORK_DIR" && -d "$WORK_DIR" ]]; then
11+
rm -rf "$WORK_DIR"
12+
fi
13+
unstub git 2>/dev/null || true
14+
}
15+
16+
@test "Cleanup sparse state enabled - removes config.worktree and unsets all sparse config keys" {
17+
export BUILDKITE_PLUGIN_SPARSE_CHECKOUT_CLEANUP_SPARSE_STATE="true"
18+
19+
WORK_DIR="$(mktemp -d)"
20+
mkdir -p "$WORK_DIR/.git"
21+
echo '[core]' > "$WORK_DIR/.git/config.worktree"
22+
23+
stub git \
24+
"ls-files -z : true" \
25+
"update-index -z --no-skip-worktree --stdin : true" \
26+
"config --unset extensions.worktreeConfig : true" \
27+
"config --unset core.sparseCheckout : true" \
28+
"config --unset core.sparseCheckoutCone : true"
29+
30+
cd "$WORK_DIR"
31+
run "$HOOK_DIR/hooks/pre-exit"
32+
33+
assert_success
34+
assert_output --partial 'Cleaning up sparse-checkout config'
35+
assert_output --partial 'Sparse-checkout config cleaned up'
36+
[[ ! -f "$WORK_DIR/.git/config.worktree" ]]
37+
}
38+
39+
@test "Cleanup sparse state enabled - succeeds even when no prior sparse config exists" {
40+
export BUILDKITE_PLUGIN_SPARSE_CHECKOUT_CLEANUP_SPARSE_STATE="true"
41+
42+
WORK_DIR="$(mktemp -d)"
43+
mkdir -p "$WORK_DIR/.git"
44+
45+
stub git \
46+
"ls-files -z : true" \
47+
"update-index -z --no-skip-worktree --stdin : true" \
48+
"config --unset extensions.worktreeConfig : true" \
49+
"config --unset core.sparseCheckout : true" \
50+
"config --unset core.sparseCheckoutCone : true"
51+
52+
cd "$WORK_DIR"
53+
run "$HOOK_DIR/hooks/pre-exit"
54+
55+
assert_success
56+
assert_output --partial 'Cleaning up sparse-checkout config'
57+
assert_output --partial 'Sparse-checkout config cleaned up'
58+
}
59+
60+
@test "Cleanup sparse state enabled - skips when no .git directory" {
61+
export BUILDKITE_PLUGIN_SPARSE_CHECKOUT_CLEANUP_SPARSE_STATE="true"
62+
63+
WORK_DIR="$(mktemp -d)"
64+
65+
cd "$WORK_DIR"
66+
run "$HOOK_DIR/hooks/pre-exit"
67+
68+
assert_success
69+
assert_output --partial 'No .git directory found, skipping sparse-checkout config cleanup'
70+
refute_output --partial 'Cleaning up sparse-checkout config'
71+
}
72+
73+
@test "Cleanup sparse state enabled - pipes tracked files to update-index" {
74+
export BUILDKITE_PLUGIN_SPARSE_CHECKOUT_CLEANUP_SPARSE_STATE="true"
75+
76+
WORK_DIR="$(mktemp -d)"
77+
mkdir -p "$WORK_DIR/.git"
78+
79+
stub git \
80+
"ls-files -z : echo 'lib/excluded.rb'" \
81+
"update-index -z --no-skip-worktree --stdin : true" \
82+
"config --unset extensions.worktreeConfig : true" \
83+
"config --unset core.sparseCheckout : true" \
84+
"config --unset core.sparseCheckoutCone : true"
85+
86+
cd "$WORK_DIR"
87+
run "$HOOK_DIR/hooks/pre-exit"
88+
89+
assert_success
90+
assert_output --partial 'Sparse-checkout config cleaned up'
91+
}
92+
93+
@test "Cleanup sparse state not configured - no cleanup runs" {
94+
run "$HOOK_DIR/hooks/pre-exit"
95+
96+
assert_success
97+
refute_output --partial 'sparse-checkout config'
98+
}

0 commit comments

Comments
 (0)