S3: "unsupported extension" warning for file names containing dots when default_extension is set
Describe the bug
S3Source.get_fields uses pathlib.Path(...).suffix, which is purely lexical and returns everything after the last dot. For S3 files whose names contain dots — e.g. events.account.update-2026-05-27-<hash>.gz — stripping the .gz compression suffix leaves .update-2026-05-27-<hash> as the apparent format extension. That is not a real format, so the inferrer emits an "unsupported extension" warning.
Because the leftover value is non-empty, the default_extension fallback on the path_spec is never reached — so even with default_extension: json set, every such file warns and is not resolved.
To Reproduce
- Configure an S3 path_spec with
default_extension: json. - Point it at files whose names contain dots before the compression suffix, e.g.
s3://.../events.account.update-2026-05-27-11-<hash>.gz. - Observe an
"unsupported extension"warning for every such file;default_extensionis not applied.
Expected behavior
When a file name has no recognised format extension (csv, tsv, json, parquet, avro), the configured default_extension should be used so the file is ingested, instead of warning. Files with a recognised extension (including compressed *.json.gz, *.parquet.gz) should behave as before.
Version
DataHub ingestion (S3 source), current master.
Source: datahub-project/datahub