Tableau: Virtual Connection Tables missing downstream lineage to Embedded Data Sources (field upstream re-fetch overwrites VC data)
Describe the bug
Virtual Connection Tables have no downstream lineage edges to the Embedded Data Sources that consume them. In one production instance, 40 of 41 sampled Virtual Connection Tables are lineage dead-ends: they have upstream edges (to Snowflake) but zero downstream edges to the Embedded Data Sources that connect through them.
Root cause
The Tableau connector fetches datasources in two phases:
- Initial fetch (
embedded_datasource_graphql_query/published_datasource_graphql_query) returns fields with fullupstreamColumnsdata, includingtable.nameand the... on VirtualConnectionTable { virtualConnection { id, name } }fragment. - Field upstream re-fetch (
datasource_upstream_fields_graphql_queryviaupdate_datasource_for_field_upstream) re-fetches fields to addupstreamFieldsfor column-level lineage, but only requeststable { __typename, id }.
The re-fetch merges back with field_dict.update(), which overwrites the original upstreamColumns and drops table.name and virtualConnection. Later, get_upstream_vc_tables sees __typename == "VirtualConnectionTable" but a null table.name and missing virtualConnection, so its guard silently skips the reference and no VC lineage edge is produced.
To Reproduce
- Ingest a Tableau source where Embedded Data Sources connect through Virtual Connection Tables.
- Inspect a Virtual Connection Table entity's lineage.
- Observe it has upstream edges but no downstream edges to the consuming Embedded Data Sources.
Expected behavior
Virtual Connection Tables should have downstream lineage edges to the Embedded Data Sources that consume them.
Version
DataHub ingestion (Tableau source), current master.
Source: datahub-project/datahub