#19690·datahub

S3: "unsupported extension" warning for file names containing dots when default_extension is set

Author: rospeCreated Sep 9, 2026Updated Sep 9, 2026
Labelsbug

Describe the bug

S3Source.get_fields uses pathlib.Path(...).suffix, which is purely lexical and returns everything after the last dot. For S3 files whose names contain dots — e.g. events.account.update-2026-05-27-<hash>.gz — stripping the .gz compression suffix leaves .update-2026-05-27-<hash> as the apparent format extension. That is not a real format, so the inferrer emits an "unsupported extension" warning.

Because the leftover value is non-empty, the default_extension fallback on the path_spec is never reached — so even with default_extension: json set, every such file warns and is not resolved.

To Reproduce

  1. Configure an S3 path_spec with default_extension: json.
  2. Point it at files whose names contain dots before the compression suffix, e.g. s3://.../events.account.update-2026-05-27-11-<hash>.gz.
  3. Observe an "unsupported extension" warning for every such file; default_extension is not applied.

Expected behavior

When a file name has no recognised format extension (csv, tsv, json, parquet, avro), the configured default_extension should be used so the file is ingested, instead of warning. Files with a recognised extension (including compressed *.json.gz, *.parquet.gz) should behave as before.

Version

DataHub ingestion (S3 source), current master.