Add difference to CITGM output

[BridgeAR]

Right now CITGM shows +-n failures when finishing. The comparison is with the former run. Would it be possible to actually show the newly failed and newly passed entries? That would be an immense help while checking the results!

Another improvement would be to compare the diff with the former run that had the same base branch. That way major releases would not interfere. As an example: master has 30 failures. v10.x gets a new release and it has 10 failures. It now shows -20 failures. Now we run another CITGM against master and it has the same 30 failures as before but it highlights 20 new failures. At least that's the case as far as I know. Ideally that could be deactivated with a checkbox.

Ref: https://ci.nodejs.org/view/Node.js-citgm/job/citgm-smoker/2028/

[richardlau]

Right now CITGM shows +-n failures when finishing. The comparison is with the former run. Would it be possible to actually show the newly failed and newly passed entries? That would be an immense help while checking the results!

Sort of, but not in a particular nice way. And not as neatly summarised in one place as you might like.

For https://ci.nodejs.org/view/Node.js-citgm/job/citgm-smoker/2028/ for example (multiple images)

Also for an individual test you can view history:

but I don't know of a way of getting a summarised view.

Another improvement would be to compare the diff with the former run that had the same base branch. That way major releases would not interfere. As an example: master has 30 failures. v10.x gets a new release and it has 10 failures. It now shows -20 failures. Now we run another CITGM against master and it has the same 30 failures as before but it highlights 20 new failures. At least that's the case as far as I know. Ideally that could be deactivated with a checkbox.

AFAIK this isn't possible with the stuff built-in to Jenkins or with any of the available plugins (I'm happy to be corrected here) and we'd have to either write our own plugin or separate visualization tool.

[github-actions]

This issue is stale because it has been open many days with no activity. It will be closed soon unless the stale label is removed or a comment is made.

[mmarchini]

node-core-utils now have citgm comparison: nodejs/node-core-utils#455. Closing, but can be reopened if that's not what you're looking for.