#4586·modin

BUG: read_sql produces duplicates

Author: JanPalasekCreated Jun 20, 2022Updated Sep 11, 2025
Labelsbug 🦗sqlExternalP3

System information

  • OS Platform and Distribution: CentOS Linux release 7.9.2009 (Core)
  • Modin version 0.15.1
  • Python version: 3.8
  • Code we can use to reproduce: Code is in the Source code but due to NDA I couldn't use the real code and used substitues instead.

Describe the problem

modin.read_sql function produces false duplicate entries.

How I found out:

  1. I ran the modin.read_sql, reading entries from MSSQL database (see Source code).
  2. Ran the group by query and it produced duplicate results.
  3. In order to validate this, I found the duplicate entries in the modin dataframe (with square brackets command) and found out that they are indeed present.

Then I did the same with pandas and there were no duplicate entries. I also ran validation query on the MSSQL database directly and no duplicates there either.

I tested it with both dask and ray backend, it works the same.

Source code / logs

python
import modin.config as cfg
cfg.Engine.put("dask")

import modin.pandas as pd
import modin

from dask.distributed import Client
client = Client()

# SELECT_QUERY is pretty simple select query with join and where, nothing more
# additional info: SELECT_QUERY was a string, not sqlalchemy expression
data = pd.read_sql(SELECT_QUERY, CONNECTION_STRING_MSSQL)

# this assert runs OK for pandas but not for modin
assert data .groupby(LIST_OF_UNIQUE_COLS).size()[lambda x: x > 1].pipe(len) == 0