Skip to content

fix: collapse breaking entrypoint invocations, tolerate oasdiff.yaml fail-on, test YAML support#112

Merged
reuvenharrison merged 5 commits intomainfrom
fix/breaking-entrypoint-and-yaml-tests
May 10, 2026
Merged

fix: collapse breaking entrypoint invocations, tolerate oasdiff.yaml fail-on, test YAML support#112
reuvenharrison merged 5 commits intomainfrom
fix/breaking-entrypoint-and-yaml-tests

Conversation

@reuvenharrison
Copy link
Copy Markdown
Contributor

@reuvenharrison reuvenharrison commented May 8, 2026

Summary

Two related fixes to entrypoint scripts plus tests for .oasdiff.yaml config-file support across all three free actions and exit-code tolerance in pr-comment.

What's wrong today

breaking/entrypoint.sh runs oasdiff breaking three times per workflow run

Invocation Purpose
Line 82/84 Capture default-format text → $GITHUB_OUTPUT
Line 90 --format githubactions → annotations on the PR
Line 124 --fail-on → exit code

That's two more spec parses + rule walks than necessary on every PR.

set -e + fail-on from .oasdiff.yaml aborts the script before rendering

With fail-on: ERR in the customer's .oasdiff.yaml and a real breaking change in the spec, the very first oasdiff breaking invocation exits 1, set -e aborts the entrypoint, and the customer sees an action that exits 1 with no rendered report, no annotations, and no notice link. Reproducible: drop .oasdiff.yaml with fail-on: ERR at the repo root, leave fail-on empty in the action with: block, push a PR with a breaking change.

The same shape of bug exists in pr-comment/entrypoint.sh: a Pro customer with fail-on: ERR in YAML loses the PR comment because the script aborts before posting.

What this PR does

breaking/entrypoint.sh: collapses runs 1+3 into a single invocation that captures the report and applies --fail-on, with exit-code tolerance via || exit_code=$? (mirrors the pattern already used in diff/entrypoint.sh). Run 2 (--format githubactions) stays as the annotations renderer, also tolerated. Final exit $exit_code preserves fail-on behaviour from either the input or .oasdiff.yaml. Result: 3 → 2 invocations; YAML fail-on now produces the same nicely-rendered output as input fail-on.

pr-comment/entrypoint.sh: captures oasdiff changelog's exit code and differentiates "fail-on triggered, we have the JSON" from "real failure, no output." Real failures still abort with the original exit code; fail-on-triggered exits proceed to post the PR comment, since pr-comment's exit code comes from the service response, not from oasdiff itself.

changelog/entrypoint.sh and diff/entrypoint.sh already run oasdiff once and don't need this fix.

Tests

End-to-end: .oasdiff.yaml config-file pickup

Four new jobs in .github/workflows/test.yaml exercise .oasdiff.yaml end-to-end through each action's Docker entrypoint:

Job Action YAML setting tested Assertion
oasdiff_breaking_yaml_config_fail_on breaking fail-on: ERR Action fails AND the report still renders to outputs.breaking (the entrypoint fix in this PR)
oasdiff_breaking_yaml_config_err_ignore breaking err-ignore: specs/err-ignore.txt Findings suppressed → outputs.breaking == "No breaking changes"
oasdiff_changelog_yaml_config_level changelog level: ERR WARN + INFO findings filtered out → 2 errors only (vs the 21 normally produced)
oasdiff_diff_yaml_config_exclude_elements diff exclude-elements: [description, title, summary] + format: text Diff comes back as "No changes"

Each job creates .oasdiff.yaml at the runner's repo root in a step before the action runs (separate runners, no cross-contamination). The legacy oasdiff.yaml filename still works as a back-compat fallback in oasdiff v1.15.3+; the test fixtures use the preferred .oasdiff.yaml name to match what we document.

Unit-level: pr-comment exit-code tolerance

Two new jobs run pr-comment/entrypoint.sh directly with a stubbed oasdiff on PATH (no Docker, no OASDIFF_TOKEN, no reachable service required), to exercise the entrypoint's branching logic in isolation:

Job Stub behaviour Assertion
pr_comment_tolerates_oasdiff_fail_on exit 1 with valid [] JSON output Script proceeds past the oasdiff invocation, emits the review-page notice, and exits 0 at the no-token skip. Verifies the new || oasdiff_exit=$? tolerance.
pr_comment_aborts_on_oasdiff_real_failure exit 2 with no stdout Script aborts with exit 2, prints ERROR: oasdiff exited 2 with no output, never reaches the notice line. Verifies the [ -z "$changelog" ] early-abort branch.

These cover the pr-comment fix without needing a token or a running service. The downstream service call is intentionally not exercised; what matters is that the script does (or does not) reach it.

Dependency on oasdiff v1.15.3

The four .oasdiff.yaml jobs require oasdiff v1.15.3 (which added .oasdiff.* as the preferred config filename). The Dockerfile bump landed in #114; this branch carries that bump via merge from main.

Backward compatibility

  • Existing tests pass unchanged.
  • Action input precedence and behaviour unchanged for customers who don't use a config file.
  • The breaking action's outputs.breaking content shape is unchanged.
  • Customers using the legacy oasdiff.yaml filename keep working: oasdiff v1.15.3 retains the legacy lookup as a fallback.

Test plan

  • CI green on all existing test jobs
  • CI green on the four new .oasdiff.yaml end-to-end jobs
  • CI green on the two new pr-comment tolerance jobs
  • Manually verify on a sample repo: .oasdiff.yaml with fail-on: ERR produces the rendered report alongside the failure

reuvenharrison and others added 5 commits May 8, 2026 22:28
…early-exit; test oasdiff.yaml across all three actions

The breaking action's entrypoint was running oasdiff three times per
invocation:
  1. default format → captured to $GITHUB_OUTPUT
  2. --format githubactions → annotations on the PR
  3. --fail-on → exit code

Plus a bug: with `set -e`, the first invocation would abort the script
if oasdiff exited non-zero — which happens when oasdiff.yaml sets
fail-on. Result: action exits silently with no rendered report.

Collapse runs 1+3 into a single invocation that captures the report
*and* applies --fail-on, with exit-code tolerance via `|| exit_code=$?`
(the same pattern already used in diff/entrypoint.sh). Run 2 stays as
the annotations renderer, also tolerated. Final `exit $exit_code`
preserves fail-on behaviour from either the input or oasdiff.yaml.

Net: one fewer oasdiff invocation per run (~1-3s saved on real specs),
and fail-on from oasdiff.yaml now produces the same nice rendered
output as fail-on from the action input.

changelog/entrypoint.sh and diff/entrypoint.sh already run oasdiff
once and don't need this fix.

Tests: four new jobs in .github/workflows/test.yaml verify oasdiff.yaml
support end-to-end:
- breaking: fail-on from YAML (also asserts the report still renders)
- breaking: err-ignore from YAML
- changelog: level from YAML
- diff: exclude-elements from YAML
…iff.yaml)

Same shape of bug as in breaking/entrypoint.sh: with `set -e`, the
oasdiff invocation aborts the script if the customer has fail-on set
in their oasdiff.yaml. For pr-comment specifically that means: customer
upgrades to Pro, has fail-on: ERR in YAML, breaking change appears →
script aborts before posting the PR comment. They paid for Pro and
got nothing.

Capture the exit code with `|| oasdiff_exit=$?`, then differentiate:

- non-zero exit + empty output → real failure (parse error, missing
  file, etc.), abort with the original exit code
- non-zero exit + non-empty output → fail-on triggered; we have the
  JSON we need; carry on and let the service decide the workflow's
  outcome

Note: pr-comment doesn't need the 3→2 invocation collapse from the
breaking/entrypoint.sh fix — it already runs oasdiff exactly once.
oasdiff v1.15.3 (just shipped) added .oasdiff.* as the preferred config
filename, with oasdiff.* kept as a back-compat fallback. The Docker
image referenced by these tests was bumped to v1.15.3 in #114, so the
new lookup is now in effect. Update the four config-file test jobs to
write the preferred filename so we canonicalise the recommended shape
in our own workflow examples. Behaviour unchanged: both filenames
continue to work; this just aligns the test fixtures with the
documentation we'll point customers at.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two new CI jobs run pr-comment/entrypoint.sh directly with a stubbed
oasdiff on PATH, no Docker / token / service required:

- pr_comment_tolerates_oasdiff_fail_on: stub exits 1 with valid JSON.
  Asserts the script reaches the no-token skip and exits 0. Without
  the fix, set -e would abort at the oasdiff invocation.
- pr_comment_aborts_on_oasdiff_real_failure: stub exits 2 with no
  output. Asserts the script aborts with exit 2 and the explicit
  'no output' error message.

Plain shell + GitHub-env scaffolding; no new test framework.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
EOF
@reuvenharrison reuvenharrison merged commit c364360 into main May 10, 2026
36 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant