PR feedback (feature overview)

[matejchalk]

User story

As a user, I want to get feedback on my pull requests, which tells me how the proposed changes will impact my Code PushUp reports. Specifically, I want a comment to be posts a report diff summary in Markdown. The diff should list which categories and audits have improved or regressed, and by how much.

As a paying customer, I also want to view the full diff in the portal UI. The PR comment should include a link to the portal diff page. I also want to be able to access this diff from the portal directly, by selecting source and target Git refs (branches/commits/tags).

Architecture

Free version

graph TD;
  user(["🤓 User"])
  pr(["📑 Pull Request"])
  comment(["🗨️ PR comment"])

  user -- creates --> pr
  user -- "👁️‍🗨️ views" --> comment
  pr -- triggers --> ghAction
  postComment -- creates --> comment

  subgraph ghAction ["🏗️ code-pushup/github-action"]

  collectSrc[["git checkout $GITHUB_HEAD_REF\ncode-pushup collect"]]
  collectTgt[["git checkout $GITHUB_BASE_REF\ncode-pushup collect"]]
  compare[["code-pushup compare source-report.json target-report.json"]]
  postComment[["github.issues.createComment({ ... })"]]

  srcReport("source-report.json")
  tgtReport("target-report.json")
  diffJson("reports-diff.json")
  diffMd("reports-diff.md")

  collectSrc --> srcReport
  collectTgt --> tgtReport
  srcReport & tgtReport --> compare
  compare --> diffJson & diffMd
  diffMd --> postComment
  
  end

Loading

Paid version

graph TD;
  user(["🤓 User"])
  pr(["📑 Pull Request"])
  comment(["🗨️ PR comment"])
  ghAction{{"🏗️ code-pushup/github-action"}}
  portal[("🌈 Portal")]

  user -. "💰 pays for" .-> portal
  user -- creates --> pr
  user -- "👁️‍🗨️ views" --> comment
  pr -- triggers --> ghAction
  ghAction -- "🗘 uploads to\n& downloads from" --> portal
  ghAction -- creates --> comment
  comment -- "🔗 links to" --> portal

Loading

The GitHub Action will accept optional parameters for interacting with portal. The flow will then differ from the free version in a few ways:

• Since it's likely that the target branch (e.g. main) already has an uploaded report, we can download it from the portal API instead of running collect again. This report caching should speed up CI runs.
• Because we want the reports to be accessible in the portal as well (primarily via link in PR comment), all reports collected by the GitHub Action are also uploaded to the portal (autorun instead of collect).
• The Markdown comment will have an additional link added, for accessing the full diff in the portal.

In order to compare any two reports by branch/commit/etc., the portal API will implement the diffing logic as well. In this case, the diff could be more complex, e.g. to compare which issues were introduced or fixed (would be too much detail to include in MD comment).

Acceptance criteria

Free features:

• CLI compare command (2 report.json files => reports-diff.json) #482
• CLI report diff MD formatting (reports-diff.json => reports-diff.md) #559
• GitHub Action for posting PR comments (collect source, collect target, compare) #575
• use GitHub Annotations API to link source file issues in PR #605

Paid features:

• portal
  • code-pushup/portal#268
  • code-pushup/portal#332
  • Report download in portal-client
  • code-pushup/portal#376
• CLI and other
  • CLI option to download cached report instead of collect
  • Link to portal comparison page in markdown comment #681
  • GitHub Action portal connection (download and upload reports)

Follow-up improvements

• use report artifact as cache

[matejchalk]

Implemented in original scope with exception of caching via portal downloads. Went with caching via artifacts instead, so that it's usable for the open-source version too.

Further improvements are planned for monorepo setups, see #731.