BUG: read_csv() into FloatingArrays converts np.nan into pd.NA, even with distinguish_nan_and_na=True
Pandas version checks
I have checked that this issue has not already been reported.
I have confirmed this bug exists on the latest version of pandas.
I have confirmed this bug exists on the main branch of pandas.
Reproducible Example
import io
import pandas as pd
import numpy as np
csv = """,x
a,1.0
b,
c,nan"""
with pd.option_context("future.distinguish_nan_and_na", True):
s = pd.read_csv(io.StringIO(csv), index_col=0, dtype={0: str, 1: pd.Float64Dtype()},
keep_default_na=False, na_values=[""])["x"]
sIssue Description
Working with data having both pd.NA and np.nan is possible and, at least until pandas2, it was possible to save such dataframes as CSV in a format that keeps a distinction between the two (see #65227).
However, in the example above pandas fails with:
ValueError: Unable to parse string "nan" at position 2The current workaround consists in reading the CSV as string, and manually map its value to floats:
import io
import pandas as pd
import numpy as np
csv = """,x
a,1.0
b,
c,nan"""
def convert_nan(val: str):
match val:
case "":
return pd.NA
case _:
return np.float64(val) # includes np.nan
with pd.option_context("future.distinguish_nan_and_na", True):
s1 = pd.read_csv(io.StringIO(csv), index_col=0, dtype=str, keep_default_na=False)["x"]
s2 = s1.map(convert_nan)
s3 = s2.convert_dtypes()
s1,s2,s3returns:
(a 1.0
b
c nan
Name: x, dtype: str,
a 1.0
b <NA>
c NaN
Name: x, dtype: object,
a 1.0
b <NA>
c NaN
Name: x, dtype: Float64)Expected Behavior
I would expect read_csv() to distinguish between NA and NaN in the columns with dtype=Float64. Even more so if distinguish_nan_and_na=True.
a 1.0
b <NA>
c NaN
Name: x, dtype: Float64Installed Versions
INSTALLED VERSIONS
commit : ab90747e3dae0e69b1bdbf083820b8075689b34b python : 3.13.13 python-bits : 64 OS : Linux OS-release : 6.17.0-20-generic Version : #20~24.04.1-Ubuntu SMP PREEMPT_DYNAMIC Thu Mar 19 01:28:37 UTC 2 machine : x86_64 processor : x86_64 byteorder : little LC_ALL : None LANG : en_GB.UTF-8 LOCALE : en_GB.UTF-8
pandas : 3.0.2 numpy : 2.4.4 dateutil : 2.9.0.post0 pip : 26.0.1 Cython : None sphinx : None IPython : 9.5.0 adbc-driver-postgresql: None adbc-driver-sqlite : None bs4 : 4.13.5 bottleneck : None fastparquet : None fsspec : 2025.7.0 html5lib : None hypothesis : None gcsfs : None jinja2 : 3.1.6 lxml.etree : None matplotlib : 3.10.6 numba : 0.65.0 numexpr : 2.11.0 odfpy : None openpyxl : None psycopg2 : None pymysql : None pyarrow : 21.0.0 pyiceberg : None pyreadstat : None pytest : 8.4.1 python-calamine : None pytz : None pyxlsb : None s3fs : None scipy : 1.16.1 sqlalchemy : None tables : 3.10.2 tabulate : None xarray : None xlrd : None xlsxwriter : None zstandard : None qtpy : None pyqt5 : None
Source: pandas-dev/pandas