Common Values incorrectly reporting (missing)

Current Behaviour

OS:Mac Python:3.11 Interface: Jupyter Lab pip: 22.3.1

dataset

It appears that when a column value_counts exceeds 200 within the common values section:

• when aggregations of a value exceed 200 the remaining is categorised as (missing)

It overall contradicts Missing and Missing(%) main statistics for a variable

Expected Behaviour

The (Missing) section within Common values should be removed or the difference to "other values"

Data Description

https://github.com/plotly/datasets/blob/master/2015_flights.parquet

Code that reproduces the bug

from pyspark.sql import SparkSession
from pyspark.sql import functions as F
from ydata_profiling import ProfileReport
import json

spark = SparkSession.builder.appName("ydata").getOrCreate()

spark_df = spark.read.parquet("ydata-test/2015_flights.parquet")

n_notnull = spark_df.filter(F.col("SCHEDULED_DEPARTURE").isNotNull()).count() 

profile = ProfileReport(spark_df, minimal=True)

value_counts_values = sum(json.loads(profile.to_json())["variables"]["SCHEDULED_DEPARTURE"]["value_counts_without_nan"].values())

missing_common_values = 1650418 # as per html report

assert missing_common_values ==  (n_notnull - value_counts_values)

pandas-profiling version

4.5.1

Dependencies

ydata-profiling==4.5.1
pyspark==3.4.1
pandas==2.0.3
numpy==1.23.5

OS

macos

Checklist

• There is not yet another bug report for this issue in the issue tracker
• The problem is reproducible from this bug report. This guide can help to craft a minimal bug report.
• The issue has not been resolved by the entries listed under Common Issues.