JpegEncoderTests are not reliable
Author: antonfirsovCreated Feb 13, 2021Updated Feb 13, 2021
Labelsunit testsneeds triage
Prerequisites
- I have written a descriptive issue title
- I have verified that I am running the latest version of ImageSharp
- I have verified if the problem exist in both
DEBUGandRELEASEmode - I have searched open and closed issues to ensure it has not already been reported
Description
Our current validation logic in JpegEncoderTests is too tolerant, which caused #1549 to go through the quality gate completely unnoticed. The method for calculating the tolerance percentage is result of trial-and error experiments (dealing with platform differences if memory serves well):
The method above returns a comparer of 15% tolerance, for subsample=420, quality=100, which doesn't really make sense. The image in https://github.com/SixLabors/ImageSharp/issues/1549#issuecomment-778615704 has ~4% difference compared to the image before encoding.
Source: SixLabors/ImageSharp