Skip to content

Commit fa33e01

Browse files
petetomasikpetetomasik
committed
Merge branch 'main' into SUP-5965-env-config-map-format
2 parents 264f105 + d550200 commit fa33e01

8 files changed

Lines changed: 863 additions & 9 deletions

File tree

‎README.md‎

Lines changed: 138 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -72,7 +72,79 @@ This is a sub-section that provides configuration for running commands or trigge
7272
- [Group](https://buildkite.com/docs/pipelines/configure/step-types/group-step)
7373
- [Conditionals](https://buildkite.com/docs/pipelines/conditionals)
7474

75-
:warning: This plugin may accept configurations that are not valid pipeline steps, this is a known issue to keep its code simple and flexible.
75+
#### Step Validation
76+
77+
The plugin validates all step configurations before uploading the pipeline. Invalid steps are automatically skipped with a warning logged to the build output.
78+
79+
**A valid step must have:**
80+
- A `command` or `commands` field (for command steps), OR
81+
- A `trigger` field (for trigger steps), OR
82+
- A `group` field with either:
83+
- An action (`command`, `commands`, or `trigger`) directly on the group, OR
84+
- Valid nested `steps`
85+
86+
**Invalid configurations that will be skipped:**
87+
88+
```yaml
89+
# ❌ Empty step - no action defined
90+
- path: "app/"
91+
config:
92+
label: "Deploy app" # Only has a label, no command/trigger
93+
94+
# ❌ Empty group - no action and no nested steps
95+
- path: "services/"
96+
config:
97+
group: "Deploy"
98+
# Missing: steps array or action
99+
```
100+
101+
**Valid configurations:**
102+
103+
```yaml
104+
# ✅ Valid - has command
105+
- path: "app/"
106+
config:
107+
label: "Deploy app"
108+
command: "echo deploying"
109+
110+
# ✅ Valid - group with nested steps
111+
- path: "services/"
112+
config:
113+
group: "Deploy"
114+
steps:
115+
- command: "deploy.sh"
116+
```
117+
118+
#### Plugins in Step Configurations
119+
120+
The plugin preserves `plugins:` blocks when specified in command step configurations. This allows you to use Buildkite plugins within your monorepo-watched steps.
121+
122+
**Example**
123+
124+
```yaml
125+
steps:
126+
- label: "Triggering pipelines"
127+
plugins:
128+
- monorepo-diff#v1.8.0:
129+
watch:
130+
- path: services/api/
131+
config:
132+
command: "npm test"
133+
plugins:
134+
- artifacts#v1.9.4:
135+
upload: "coverage/**/*"
136+
- docker-compose#v5.12.1:
137+
run: api
138+
- path: services/web/
139+
config:
140+
command: "yarn build"
141+
plugins:
142+
- docker#v5.13.0:
143+
image: "node:20"
144+
workdir: /app
145+
```
146+
147+
When changes are detected in the watched paths, the plugin generates steps that include the specified plugins. The `plugins:` blocks are preserved exactly as configured.
76148

77149
```yaml
78150
steps:
@@ -89,9 +161,9 @@ steps:
89161
- path: docker/
90162
config:
91163
group: docker/**
92-
steps:
164+
steps: # Required: groups must have either 'steps' or an action
93165
- plugins:
94-
- docker#latest:
166+
- docker#v5.13.0:
95167
build: service
96168
push: service
97169
- command: docker/run-e2e-tests.sh
@@ -388,6 +460,34 @@ steps:
388460

389461
The plugin automatically retries binary downloads up to 3 times with a 5-second delay between attempts. This handles transient network issues when downloading from GitHub.
390462

463+
### `verify_checksum` (optional)
464+
465+
Default: `false`
466+
467+
Enable SHA256 checksum verification for downloaded binaries to enhance security. When enabled, the plugin verifies checksums against those published in the GitHub release, providing protection against compromised artifacts, network attacks, and binary tampering.
468+
469+
Checksum verification is performed for:
470+
- Newly downloaded binaries (fails and deletes binary on mismatch)
471+
- Cached binaries before reuse (automatically re-downloads on mismatch)
472+
- Pre-installed binaries when `download: false` (best-effort, non-blocking)
473+
474+
To enable checksum verification:
475+
476+
```yaml
477+
steps:
478+
- label: "Triggering pipelines"
479+
plugins:
480+
- monorepo-diff#v1.8.0:
481+
verify_checksum: true # Recommended for enhanced security
482+
diff: "git diff --name-only HEAD~1"
483+
watch:
484+
- path: "foo-service/"
485+
config:
486+
trigger: "deploy-foo-service"
487+
```
488+
489+
If checksums are unavailable for a release or the SHA256 command is not found on the system, the plugin will warn but continue execution (graceful degradation).
490+
391491
### `hooks` (optional)
392492

393493
Currently supports a list of `commands` you wish to execute after the `watched` pipelines have been triggered
@@ -568,6 +668,41 @@ steps:
568668
command: "echo deploy-bar"
569669
```
570670

671+
## Troubleshooting
672+
673+
### "Skipping invalid step" warnings
674+
675+
If you see warnings like `Skipping invalid step: empty step configuration`, check that your step configuration includes:
676+
677+
1. For command steps: `command` or `commands` field
678+
2. For trigger steps: `trigger` field
679+
3. For group steps: `group` field with either `steps` array or an action
680+
681+
**Common issues:**
682+
683+
- Forgetting to add `command:` or `trigger:` inside the `config` block
684+
- Creating empty groups without nested steps
685+
- Using only metadata fields like `label`, `key`, or `env` without an action
686+
687+
**Example of fixing an invalid configuration:**
688+
689+
```yaml
690+
# ❌ Invalid - missing action
691+
- path: "app/"
692+
config:
693+
label: "Deploy app"
694+
env:
695+
- ENV=production
696+
697+
# ✅ Fixed - added command
698+
- path: "app/"
699+
config:
700+
label: "Deploy app"
701+
command: "deploy.sh"
702+
env:
703+
- ENV=production
704+
```
705+
571706
## Compatibility
572707

573708
| Elastic Stack | Agent Stack K8s | Hosted (Mac) | Hosted (Linux) | Notes |

‎hooks/command‎

Lines changed: 146 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -66,6 +66,23 @@ need_cmd() {
6666
fi
6767
}
6868

69+
# Detect platform-appropriate SHA256 command
70+
# Returns: Sets RETVAL to command string, returns 0 on success, 1 if no command found
71+
get_sha256_cmd() {
72+
if check_cmd sha256sum; then
73+
RETVAL="sha256sum"
74+
return 0
75+
elif check_cmd shasum; then
76+
RETVAL="shasum -a 256"
77+
return 0
78+
elif check_cmd sha256; then
79+
RETVAL="sha256"
80+
return 0
81+
else
82+
return 1
83+
fi
84+
}
85+
6986
# This wraps curl or wget.
7087
# Try curl first, if not installed, use wget instead.
7188
downloader() {
@@ -111,6 +128,92 @@ download_with_retry() {
111128
return 1
112129
}
113130

131+
# Download checksums.txt from GitHub release
132+
# Parameters: $1 = version (e.g., "v1.8.0"), $2 = destination file path
133+
# Returns: 0 on success, 1 on failure
134+
download_checksums() {
135+
local _version="$1"
136+
local _dest="$2"
137+
local _repo="https://github.com/buildkite-plugins/monorepo-diff-buildkite-plugin"
138+
local _url="${_repo}/releases/download/${_version}/checksums.txt"
139+
140+
if ! downloader "$_url" "$_dest"; then
141+
return 1
142+
fi
143+
144+
return 0
145+
}
146+
147+
# Verify binary against checksums.txt
148+
# Parameters: $1 = path to binary file, $2 = version (e.g., "v1.8.0"), $3 = architecture (e.g., "darwin_amd64")
149+
# Returns: 0 if valid, 1 if invalid
150+
verify_checksum() {
151+
local _binary_path="$1"
152+
local _version="$2"
153+
local _arch="$3"
154+
local _verify_enabled="${BUILDKITE_PLUGIN_MONOREPO_DIFF_VERIFY_CHECKSUM:-false}"
155+
156+
# Check if verification is disabled
157+
if [[ "$_verify_enabled" == "false" ]]; then
158+
return 0
159+
fi
160+
161+
# Detect SHA256 command
162+
if ! get_sha256_cmd; then
163+
say "Warning: No SHA256 command found (sha256sum, shasum, or sha256), skipping verification"
164+
return 0
165+
fi
166+
local _sha256_cmd="$RETVAL"
167+
168+
# Download checksums.txt to temporary file
169+
local _checksums_file
170+
_checksums_file=$(mktemp)
171+
172+
if ! download_checksums "$_version" "$_checksums_file"; then
173+
say "Warning: Could not download checksums.txt, skipping verification"
174+
rm -f "$_checksums_file"
175+
return 0
176+
fi
177+
178+
# Get expected checksum from checksums.txt
179+
local _binary_name="monorepo-diff-buildkite-plugin_${_arch}"
180+
local _expected_checksum
181+
_expected_checksum=$(grep "${_binary_name}" "$_checksums_file" | awk '{print $1}')
182+
183+
if [[ -z "$_expected_checksum" ]]; then
184+
say "Warning: Checksum not found in checksums.txt for ${_binary_name}, skipping verification"
185+
rm -f "$_checksums_file"
186+
return 0
187+
fi
188+
189+
# Calculate actual checksum
190+
local _actual_checksum
191+
if [[ "$_sha256_cmd" == "sha256" ]]; then
192+
# BSD format: SHA256 (file) = hash
193+
_actual_checksum=$($_sha256_cmd "$_binary_path" | awk '{print $4}')
194+
else
195+
# GNU/shasum format: hash file
196+
_actual_checksum=$($_sha256_cmd "$_binary_path" | awk '{print $1}')
197+
fi
198+
199+
# Clean up temporary file
200+
rm -f "$_checksums_file"
201+
202+
# Compare checksums
203+
if [[ "$_actual_checksum" != "$_expected_checksum" ]]; then
204+
red=$(tput setaf 1 2>/dev/null || echo '')
205+
reset=$(tput sgr0 2>/dev/null || echo '')
206+
say "${red}ERROR${reset}: Checksum verification failed for $_binary_path" >&2
207+
say "Expected: $_expected_checksum" >&2
208+
say "Actual: $_actual_checksum" >&2
209+
say "This may indicate a corrupted download or a security issue." >&2
210+
return 1
211+
fi
212+
213+
say "Checksum verification passed"
214+
return 0
215+
}
216+
114217
get_latest_version() {
115218
local _repo="https://api.github.com/repos/buildkite-plugins/monorepo-diff-buildkite-plugin"
116219
local _version=""
@@ -121,7 +224,7 @@ get_latest_version() {
121224
_version=$(wget -qO- "${_repo}/releases/latest" | grep -oE '"tag_name": "v[0-9]+\.[0-9]+\.[0-9]+"' | cut -d'"' -f4)
122225
fi
123226

124-
if [[ "$_version" =~ ^v[0-9]+\.[0-9]+\.[0-9]+ ]]; then
227+
if [[ "$_version" =~ ^v[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
125228
echo "${_version}"
126229
fi
127230
}
@@ -140,7 +243,7 @@ get_version() {
140243
_version=${BASH_REMATCH[1]}
141244
fi
142245

143-
if [[ "$_version" =~ ^v[0-9]+\.[0-9]+\.[0-9]+ ]]; then
246+
if [[ "$_version" =~ ^v[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
144247
true
145248
else
146249
_version=""
@@ -160,17 +263,27 @@ download_binary_and_run() {
160263
local _binary_version=""
161264
local test_mode="${BUILDKITE_PLUGIN_MONOREPO_DIFF_BUILDKITE_PLUGIN_TEST_MODE:-false}"
162265
local _url=""
266+
local _recovery_mode=false
163267

164268
if check_cmd "${executable}"; then
165269
_binary_version=$(get_binary_version)
166270
if [[ -z "$_binary_version" ]]; then
167271
say "Warning: Could not determine binary version, will download fresh copy"
272+
else
273+
# Before reusing cached binary, verify its checksum
274+
if ! verify_checksum "${executable}" "${_binary_version}" "${_arch}"; then
275+
say "Cached binary failed checksum verification, attempting recovery..."
276+
rm -f "${executable}"
277+
rm -f "${executable_version_file}"
278+
_binary_version=""
279+
_recovery_mode=true
280+
fi
168281
fi
169282
fi
170283

171284
if [[ "$test_mode" == "true" ]]; then
172285
true
173-
elif [[ "$_specified_version" =~ ^v[0-9]+\.[0-9]+\.[0-9]+ ]]; then
286+
elif [[ "$_specified_version" =~ ^v[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
174287
if [[ -z "$_binary_version" ]] || [[ "$_binary_version" != "$_specified_version" ]]; then
175288
_url=${_repo}/releases/download/${_specified_version}/monorepo-diff-buildkite-plugin_${_arch}
176289
_binary_version="${_specified_version}"
@@ -194,6 +307,21 @@ download_binary_and_run() {
194307
exit 1
195308
fi
196309
echo "${_binary_version}" > "${executable_version_file}"
310+
311+
# After successful download, verify checksum
312+
if ! verify_checksum "${executable}" "${_binary_version}" "${_arch}"; then
313+
rm -f "${executable}"
314+
if [[ "$_recovery_mode" == "true" ]]; then
315+
err "Recovery download also failed checksum verification. This may indicate a problem with the release artifacts."
316+
else
317+
err "Downloaded binary failed checksum verification and was deleted"
318+
fi
319+
fi
320+
321+
# If this was a recovery download, verify it succeeded
322+
if [[ "$_recovery_mode" == "true" ]]; then
323+
say "Binary recovery successful"
324+
fi
197325
fi
198326

199327
chmod +x "${executable}"
@@ -211,6 +339,21 @@ run_preinstalled_binary() {
211339
fi
212340
fi
213341

342+
# Best-effort verification for preinstalled binaries (non-blocking)
343+
# Try to detect version and verify, but don't fail if we can't
344+
local _binary_version
345+
_binary_version=$(get_binary_version)
346+
if [[ -n "$_binary_version" ]]; then
347+
get_architecture || true
348+
local _arch="$RETVAL"
349+
if [[ -n "$_arch" ]]; then
350+
# Attempt verification but don't fail if it doesn't work
351+
if ! verify_checksum "${_executable}" "${_binary_version}" "${_arch}" 2>/dev/null; then
352+
say "Warning: Could not verify checksum for preinstalled binary (version: ${_binary_version})"
353+
fi
354+
fi
355+
fi
356+
214357
${_executable} "$@"
215358
}
216359

0 commit comments

Comments
 (0)