The `mssql` source's query-lineage extraction fails silently
Describe the bug
The mssql source's query-lineage extraction (include_query_lineage: true) feeds raw text from sys.query_store_query_text / sys.dm_exec_sql_text directly into sqlglot.parse(sql, read="tsql") without stripping a SQL-Server-specific annotation these DMVs prepend to any statement executed inside a stored procedure that references a typed parameter or local variable, e.g.:
(@MinDate date, @Amount numeric(38,10))
insert into dbo.target_table (col1, col2)
select s.col1, s.col2
from dbo.source_table as s
where s.amount > @AmountThis annotation is not valid, executable T-SQL on its own — it's a metadata convention SQL Server's DMVs use to document which typed parameters a statement references (the same convention used for sp_executesql-style prepared-statement signatures), not part of the T-SQL grammar. sqlglot correctly rejects it as unparseable, since it isn't SQL. The bug is that the mssql source (datahub/ingestion/source/sql/mssql/query.py, get_query_history_from_query_store / get_query_history_from_dmv) doesn't strip this DMV-specific annotation before handing the text to sqlglot.parse(sql, read="tsql"), so every such statement fails to parse and the query is silently dropped from lineage extraction — with no corresponding increment to SqlAggregatorReport.num_queries_parse_failures in the ingestion report, making the failure invisible to users.
In practice, include_query_lineage on the mssql source produces zero lineage for any statement inside a stored procedure that references a typed parameter — the normal case for parameterized enterprise T-SQL ETL — while the ingestion report shows full success (num_queries_parse_failures: 0).
To Reproduce Steps to reproduce the behavior:
- Point the
mssqlsource at a SQL Server database with Query Store enabled, withinclude_query_lineage: true. - Ensure the target database has at least one stored procedure that executes a statement referencing one of the procedure's own typed parameters or local variables (e.g.
insert into dbo.target_table select ... where amount > @Amount, where@Amountis declared asnumeric(38,10)). - Run the procedure at least
min_query_callstimes so it's captured in Query Store. - Run ingestion and check the report: no downstream lineage appears for that table, despite
num_queries_extractedandnum_queries_parsedboth being non-zero andnum_queries_parse_failures: 0.
Minimal, dependency-free repro of the actual parse failure:
import sqlglot
# Fails: "Expecting )" — type with precision/scale arguments
sqlglot.parse("(@Amount numeric(38,10))select 1", read="tsql")
# Also fails: "Invalid expression / Unexpected token" — even a single no-arg type
sqlglot.parse("(@Id int)select 1", read="tsql")
# Also fails: multiple simple params
sqlglot.parse("(@Id int,@Name nvarchar(50))select 1", read="tsql")A closer-to-real-world example (as captured verbatim from sys.query_store_query_text for a stored-procedure statement referencing the procedure's @MinDate date and @Amount numeric(38,10) parameters):
import sqlglot
sql = (
"(@MinDate date,@Amount numeric(38,10))"
"insert into dbo.target_table (col1, col2) "
"select s.col1, s.col2 from dbo.source_table as s "
"where s.amount > @Amount"
)
sqlglot.parse(sql, read="tsql")
# sqlglot.errors.ParseError: Expecting ). Line 1, Col: 31.All raise sqlglot.errors.ParseError (or an equivalent "unexpected token" error). This is not limited to types with precision/scale arguments — every form of the header tested fails, so sqlglot's T-SQL grammar has no rule for this annotation at all, and the mssql source has no sanitizer for it either.
Expected behavior
The mssql source should strip this DMV parameter-annotation header before handing statement text to sqlglot.parse, so the actual SQL that follows gets parsed and contributes to lineage — mirroring the existing sanitization it already does for a different T-SQL/DMV quirk.
datahub/sql_parsing/sqlglot_utils.py, function parse_statement (lines ~164-181), already has precedent for this kind of pre-parse sanitization:
if isinstance(sql, str) and is_dialect_instance(dialect, "tsql"):
sanitized = _sanitize_tsql_temp_tables(sql)
...There is currently no equivalent sanitizer for the leading parameter-declaration header. Adding a _sanitize_tsql_param_header(sql) step here — stripping a leading (@name type[(args)], ...) block before the real statement — would fix this for all mssql/azuresqldb query-lineage consumers, without needing any change in sqlglot itself (the annotation isn't valid SQL, so sqlglot rejecting it isn't itself a bug).
Screenshots Not applicable — this is a headless ingestion/parsing bug, not a UI issue.
Desktop (please complete the following information):
acryl-datahubversion: 1.7.0.7sqlglotversion: 30.12.0- Source type:
mssql - Database: SQL Server 2022 (RTM-CU26)
- Relevant config:
include_query_lineage: true
Additional context
Impact: for the mssql source with include_query_lineage: true, this silently drops lineage for any query captured from inside a stored procedure that references a typed parameter or local variable. The ingestion report shows num_queries_parse_failures: 0, giving no indication that anything was skipped, which makes this very hard to detect without manually inspecting raw Query Store text and re-running the parser by hand — which is how this was found.
Source: datahub-project/datahub