Merge branch 'main' into SUP-5895-update-readme-plugins-in-steps

Author: petetomasik

README.md

@@ -72,6 +72,49 @@ This is a sub-section that provides configuration for running commands or trigge
 - [Group](https://buildkite.com/docs/pipelines/configure/step-types/group-step)
 - [Conditionals](https://buildkite.com/docs/pipelines/conditionals)
 
+#### Step Validation
+
+The plugin validates all step configurations before uploading the pipeline. Invalid steps are automatically skipped with a warning logged to the build output.
+
+**A valid step must have:**
+- A `command` or `commands` field (for command steps), OR
+- A `trigger` field (for trigger steps), OR
+- A `group` field with either:
+  - An action (`command`, `commands`, or `trigger`) directly on the group, OR
+  - Valid nested `steps`
+
+**Invalid configurations that will be skipped:**
+
+```yaml
+# ❌ Empty step - no action defined
+- path: "app/"
+  config:
+    label: "Deploy app"  # Only has a label, no command/trigger
+
+# ❌ Empty group - no action and no nested steps
+- path: "services/"
+  config:
+    group: "Deploy"
+    # Missing: steps array or action
+```
+
+**Valid configurations:**
+
+```yaml
+# ✅ Valid - has command
+- path: "app/"
+  config:
+    label: "Deploy app"
+    command: "echo deploying"
+
+# ✅ Valid - group with nested steps
+- path: "services/"
+  config:
+    group: "Deploy"
+    steps:
+      - command: "deploy.sh"
+```
+
 #### Plugins in Step Configurations
 
 The plugin preserves `plugins:` blocks when specified in command step configurations. This allows you to use Buildkite plugins within your monorepo-watched steps.
@@ -103,8 +146,6 @@ steps:
 
 When changes are detected in the watched paths, the plugin generates steps that include the specified plugins. The `plugins:` blocks are preserved exactly as configured.
 
-:warning: This plugin may accept configurations that are not valid pipeline steps, this is a known issue to keep its code simple and flexible.
-
 ```yaml
 steps:
   - label: "Triggering pipelines"
@@ -120,7 +161,7 @@ steps:
             - path: docker/
               config:
                 group: docker/**
-                steps:
+                steps:  # Required: groups must have either 'steps' or an action
                   - plugins:
                       - docker#v5.13.0:
                           build: service
@@ -340,6 +381,34 @@ steps:
 
 The plugin automatically retries binary downloads up to 3 times with a 5-second delay between attempts. This handles transient network issues when downloading from GitHub.
 
+### `verify_checksum` (optional)
+
+Default: `false`
+
+Enable SHA256 checksum verification for downloaded binaries to enhance security. When enabled, the plugin verifies checksums against those published in the GitHub release, providing protection against compromised artifacts, network attacks, and binary tampering.
+
+Checksum verification is performed for:
+- Newly downloaded binaries (fails and deletes binary on mismatch)
+- Cached binaries before reuse (automatically re-downloads on mismatch)
+- Pre-installed binaries when `download: false` (best-effort, non-blocking)
+
+To enable checksum verification:
+
+```yaml
+steps:
+  - label: "Triggering pipelines"
+    plugins:
+      - monorepo-diff#v1.8.0:
+          verify_checksum: true  # Recommended for enhanced security
+          diff: "git diff --name-only HEAD~1"
+          watch:
+            - path: "foo-service/"
+              config:
+                trigger: "deploy-foo-service"
+```
+
+If checksums are unavailable for a release or the SHA256 command is not found on the system, the plugin will warn but continue execution (graceful degradation).
+
 ### `hooks` (optional)
 
 Currently supports a list of `commands` you wish to execute after the `watched` pipelines have been triggered
@@ -520,6 +589,41 @@ steps:
                 command: "echo deploy-bar"
 ```
 
+## Troubleshooting
+
+### "Skipping invalid step" warnings
+
+If you see warnings like `Skipping invalid step: empty step configuration`, check that your step configuration includes:
+
+1. For command steps: `command` or `commands` field
+2. For trigger steps: `trigger` field
+3. For group steps: `group` field with either `steps` array or an action
+
+**Common issues:**
+
+- Forgetting to add `command:` or `trigger:` inside the `config` block
+- Creating empty groups without nested steps
+- Using only metadata fields like `label`, `key`, or `env` without an action
+
+**Example of fixing an invalid configuration:**
+
+```yaml
+# ❌ Invalid - missing action
+- path: "app/"
+  config:
+    label: "Deploy app"
+    env:
+      - ENV=production
+
+# ✅ Fixed - added command
+- path: "app/"
+  config:
+    label: "Deploy app"
+    command: "deploy.sh"
+    env:
+      - ENV=production
+```
+
 ## Compatibility
 
 | Elastic Stack | Agent Stack K8s | Hosted (Mac) | Hosted (Linux) | Notes |

hooks/command

@@ -66,6 +66,23 @@ need_cmd() {
   fi
 }
 
+# Detect platform-appropriate SHA256 command
+# Returns: Sets RETVAL to command string, returns 0 on success, 1 if no command found
+get_sha256_cmd() {
+  if check_cmd sha256sum; then
+    RETVAL="sha256sum"
+    return 0
+  elif check_cmd shasum; then
+    RETVAL="shasum -a 256"
+    return 0
+  elif check_cmd sha256; then
+    RETVAL="sha256"
+    return 0
+  else
+    return 1
+  fi
+}
+
 # This wraps curl or wget.
 # Try curl first, if not installed, use wget instead.
 downloader() {
@@ -111,6 +128,92 @@ download_with_retry() {
   return 1
 }
 
+# Download checksums.txt from GitHub release
+# Parameters: $1 = version (e.g., "v1.8.0"), $2 = destination file path
+# Returns: 0 on success, 1 on failure
+download_checksums() {
+  local _version="$1"
+  local _dest="$2"
+  local _repo="https://github.com/buildkite-plugins/monorepo-diff-buildkite-plugin"
+  local _url="${_repo}/releases/download/${_version}/checksums.txt"
+
+  if ! downloader "$_url" "$_dest"; then
+    return 1
+  fi
+
+  return 0
+}
+
+# Verify binary against checksums.txt
+# Parameters: $1 = path to binary file, $2 = version (e.g., "v1.8.0"), $3 = architecture (e.g., "darwin_amd64")
+# Returns: 0 if valid, 1 if invalid
+verify_checksum() {
+  local _binary_path="$1"
+  local _version="$2"
+  local _arch="$3"
+  local _verify_enabled="${BUILDKITE_PLUGIN_MONOREPO_DIFF_VERIFY_CHECKSUM:-false}"
+
+  # Check if verification is disabled
+  if [[ "$_verify_enabled" == "false" ]]; then
+    return 0
+  fi
+
+  # Detect SHA256 command
+  if ! get_sha256_cmd; then
+    say "Warning: No SHA256 command found (sha256sum, shasum, or sha256), skipping verification"
+    return 0
+  fi
+  local _sha256_cmd="$RETVAL"
+
+  # Download checksums.txt to temporary file
+  local _checksums_file
+  _checksums_file=$(mktemp)
+
+  if ! download_checksums "$_version" "$_checksums_file"; then
+    say "Warning: Could not download checksums.txt, skipping verification"
+    rm -f "$_checksums_file"
+    return 0
+  fi
+
+  # Get expected checksum from checksums.txt
+  local _binary_name="monorepo-diff-buildkite-plugin_${_arch}"
+  local _expected_checksum
+  _expected_checksum=$(grep "${_binary_name}" "$_checksums_file" | awk '{print $1}')
+
+  if [[ -z "$_expected_checksum" ]]; then
+    say "Warning: Checksum not found in checksums.txt for ${_binary_name}, skipping verification"
+    rm -f "$_checksums_file"
+    return 0
+  fi
+
+  # Calculate actual checksum
+  local _actual_checksum
+  if [[ "$_sha256_cmd" == "sha256" ]]; then
+    # BSD format: SHA256 (file) = hash
+    _actual_checksum=$($_sha256_cmd "$_binary_path" | awk '{print $4}')
+  else
+    # GNU/shasum format: hash  file
+    _actual_checksum=$($_sha256_cmd "$_binary_path" | awk '{print $1}')
+  fi
+
+  # Clean up temporary file
+  rm -f "$_checksums_file"
+
+  # Compare checksums
+  if [[ "$_actual_checksum" != "$_expected_checksum" ]]; then
+    red=$(tput setaf 1 2>/dev/null || echo '')
+    reset=$(tput sgr0 2>/dev/null || echo '')
+    say "${red}ERROR${reset}: Checksum verification failed for $_binary_path" >&2
+    say "Expected: $_expected_checksum" >&2
+    say "Actual:   $_actual_checksum" >&2
+    say "This may indicate a corrupted download or a security issue." >&2
+    return 1
+  fi
+
+  say "Checksum verification passed"
+  return 0
+}
+
 get_latest_version() {
   local _repo="https://api.github.com/repos/buildkite-plugins/monorepo-diff-buildkite-plugin"
   local _version=""
@@ -121,7 +224,7 @@ get_latest_version() {
     _version=$(wget -qO- "${_repo}/releases/latest" | grep -oE '"tag_name": "v[0-9]+\.[0-9]+\.[0-9]+"' | cut -d'"' -f4)
   fi
 
-  if [[ "$_version" =~ ^v[0-9]+\.[0-9]+\.[0-9]+ ]]; then
+  if [[ "$_version" =~ ^v[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
     echo "${_version}"
   fi
 }
@@ -140,7 +243,7 @@ get_version() {
     _version=${BASH_REMATCH[1]}
   fi
 
-  if [[ "$_version" =~ ^v[0-9]+\.[0-9]+\.[0-9]+ ]]; then
+  if [[ "$_version" =~ ^v[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
     true
   else
     _version=""
@@ -160,17 +263,27 @@ download_binary_and_run() {
   local _binary_version=""
   local test_mode="${BUILDKITE_PLUGIN_MONOREPO_DIFF_BUILDKITE_PLUGIN_TEST_MODE:-false}"
   local _url=""
+  local _recovery_mode=false
 
   if check_cmd "${executable}"; then
     _binary_version=$(get_binary_version)
     if [[ -z "$_binary_version" ]]; then
       say "Warning: Could not determine binary version, will download fresh copy"
+    else
+      # Before reusing cached binary, verify its checksum
+      if ! verify_checksum "${executable}" "${_binary_version}" "${_arch}"; then
+        say "Cached binary failed checksum verification, attempting recovery..."
+        rm -f "${executable}"
+        rm -f "${executable_version_file}"
+        _binary_version=""
+        _recovery_mode=true
+      fi
     fi
   fi
 
   if [[ "$test_mode" == "true" ]]; then
     true
-  elif [[ "$_specified_version" =~ ^v[0-9]+\.[0-9]+\.[0-9]+ ]]; then
+  elif [[ "$_specified_version" =~ ^v[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
     if [[ -z "$_binary_version" ]] || [[ "$_binary_version" != "$_specified_version" ]]; then
       _url=${_repo}/releases/download/${_specified_version}/monorepo-diff-buildkite-plugin_${_arch}
       _binary_version="${_specified_version}"
@@ -194,6 +307,21 @@ download_binary_and_run() {
       exit 1
     fi
     echo "${_binary_version}" > "${executable_version_file}"
+
+    # After successful download, verify checksum
+    if ! verify_checksum "${executable}" "${_binary_version}" "${_arch}"; then
+      rm -f "${executable}"
+      if [[ "$_recovery_mode" == "true" ]]; then
+        err "Recovery download also failed checksum verification. This may indicate a problem with the release artifacts."
+      else
+        err "Downloaded binary failed checksum verification and was deleted"
+      fi
+    fi
+
+    # If this was a recovery download, verify it succeeded
+    if [[ "$_recovery_mode" == "true" ]]; then
+      say "Binary recovery successful"
+    fi
   fi
 
   chmod +x "${executable}"
@@ -211,6 +339,21 @@ run_preinstalled_binary() {
     fi
   fi
 
+  # Best-effort verification for preinstalled binaries (non-blocking)
+  # Try to detect version and verify, but don't fail if we can't
+  local _binary_version
+  _binary_version=$(get_binary_version)
+  if [[ -n "$_binary_version" ]]; then
+    get_architecture || true
+    local _arch="$RETVAL"
+    if [[ -n "$_arch" ]]; then
+      # Attempt verification but don't fail if it doesn't work
+      if ! verify_checksum "${_executable}" "${_binary_version}" "${_arch}" 2>/dev/null; then
+        say "Warning: Could not verify checksum for preinstalled binary (version: ${_binary_version})"
+      fi
+    fi
+  fi
+
   ${_executable} "$@"
 }