[Bug]: Add-on flagged for ratings threshold keeps getting reflagged for review even after clearing needs-human-review for its older ratings

[abhn]

Is there an existing issue for this?

• I have searched the existing issues

What happened?

For an add-on that's flagged for ratings threshold

• the reviewer clears the needs human review flag as the flagging wasn't accurate
• the add-on review page is closed to let the add-on auto-approve
• the add-on got flagged again for review and ended up in manual review queue

What did you expect to happen?

• once an add-on review page is closed, the add-on is flagged again for review for Over ratings threshold for usage tier even though no new ratings were submitted for the add-on
• the ratings from the time of NHR-clearing should be excluded from re-flagging calculations, and the add-on should only be reflagged for this signal if it again gets enough ratings for it to exceed our thresholds

┆Issue is synchronized with this Jira Task

[diox]

@abhn can you share (privately in JIRA) review links of add-ons where that happened ?

[diox]

It's possible we are only checking for current (active) NHR when trying to find if the add-on has already been flagged?

[diox]

So, currently we skip only adding the NHR if the version has been explicitly reviewed by a human already. We do have code to avoid setting the same NHR... if there is already an active NHR for the same reason but 1) that would be for active NHR only and 2) we are not currently using it here (we don't pass unique_reason=True)

[abhn]

Shouldn't the add-on not get reflagged if no new ratings are submitted that'd prompt a recalculation, and thus, reflagging? What's the right way to tell the system "this is a false positive". I think the determination that thisi s a false positive should constitute the flag being explicitly reviewed by a human.

[diox]

I agree.

Currently we consider all ratings over the past 14 days, I think the solution is to start considering all ratings over the past 14 days or date of the last NHR for RATINGS_THRESHOLD on any version of the add-on, whichever is most recent.

It doesn't matter whether a human reviewed this or not, we don't need to reflag for the same set of ratings.

Side note: on top of this we should avoid flagging unlisted versions (only relevant for mixed channel add-ons: we currently flag the latest version in both channels, but that's pointless for unlisted).