#19861·datahub

The `mssql` source's query-lineage extraction fails silently

Author: erikmouritsCreated Sep 18, 2026Updated Sep 18, 2026
Labelsbug

Describe the bug The mssql source's query-lineage extraction (include_query_lineage: true) feeds raw text from sys.query_store_query_text / sys.dm_exec_sql_text directly into sqlglot.parse(sql, read="tsql") without stripping a SQL-Server-specific annotation these DMVs prepend to any statement executed inside a stored procedure that references a typed parameter or local variable, e.g.:

sql
(@MinDate date, @Amount numeric(38,10))
insert into dbo.target_table (col1, col2)
select s.col1, s.col2
from dbo.source_table as s
where s.amount > @Amount

This annotation is not valid, executable T-SQL on its own — it's a metadata convention SQL Server's DMVs use to document which typed parameters a statement references (the same convention used for sp_executesql-style prepared-statement signatures), not part of the T-SQL grammar. sqlglot correctly rejects it as unparseable, since it isn't SQL. The bug is that the mssql source (datahub/ingestion/source/sql/mssql/query.py, get_query_history_from_query_store / get_query_history_from_dmv) doesn't strip this DMV-specific annotation before handing the text to sqlglot.parse(sql, read="tsql"), so every such statement fails to parse and the query is silently dropped from lineage extraction — with no corresponding increment to SqlAggregatorReport.num_queries_parse_failures in the ingestion report, making the failure invisible to users.

In practice, include_query_lineage on the mssql source produces zero lineage for any statement inside a stored procedure that references a typed parameter — the normal case for parameterized enterprise T-SQL ETL — while the ingestion report shows full success (num_queries_parse_failures: 0).

To Reproduce Steps to reproduce the behavior:

  1. Point the mssql source at a SQL Server database with Query Store enabled, with include_query_lineage: true.
  2. Ensure the target database has at least one stored procedure that executes a statement referencing one of the procedure's own typed parameters or local variables (e.g. insert into dbo.target_table select ... where amount > @Amount, where @Amount is declared as numeric(38,10)).
  3. Run the procedure at least min_query_calls times so it's captured in Query Store.
  4. Run ingestion and check the report: no downstream lineage appears for that table, despite num_queries_extracted and num_queries_parsed both being non-zero and num_queries_parse_failures: 0.

Minimal, dependency-free repro of the actual parse failure:

python
import sqlglot

# Fails: "Expecting )" — type with precision/scale arguments
sqlglot.parse("(@Amount numeric(38,10))select 1", read="tsql")

# Also fails: "Invalid expression / Unexpected token" — even a single no-arg type
sqlglot.parse("(@Id int)select 1", read="tsql")

# Also fails: multiple simple params
sqlglot.parse("(@Id int,@Name nvarchar(50))select 1", read="tsql")

A closer-to-real-world example (as captured verbatim from sys.query_store_query_text for a stored-procedure statement referencing the procedure's @MinDate date and @Amount numeric(38,10) parameters):

python
import sqlglot

sql = (
    "(@MinDate date,@Amount numeric(38,10))"
    "insert into dbo.target_table (col1, col2) "
    "select s.col1, s.col2 from dbo.source_table as s "
    "where s.amount > @Amount"
)
sqlglot.parse(sql, read="tsql")
# sqlglot.errors.ParseError: Expecting ). Line 1, Col: 31.

All raise sqlglot.errors.ParseError (or an equivalent "unexpected token" error). This is not limited to types with precision/scale arguments — every form of the header tested fails, so sqlglot's T-SQL grammar has no rule for this annotation at all, and the mssql source has no sanitizer for it either.

Expected behavior The mssql source should strip this DMV parameter-annotation header before handing statement text to sqlglot.parse, so the actual SQL that follows gets parsed and contributes to lineage — mirroring the existing sanitization it already does for a different T-SQL/DMV quirk.

datahub/sql_parsing/sqlglot_utils.py, function parse_statement (lines ~164-181), already has precedent for this kind of pre-parse sanitization:

python
if isinstance(sql, str) and is_dialect_instance(dialect, "tsql"):
    sanitized = _sanitize_tsql_temp_tables(sql)
    ...

There is currently no equivalent sanitizer for the leading parameter-declaration header. Adding a _sanitize_tsql_param_header(sql) step here — stripping a leading (@name type[(args)], ...) block before the real statement — would fix this for all mssql/azuresqldb query-lineage consumers, without needing any change in sqlglot itself (the annotation isn't valid SQL, so sqlglot rejecting it isn't itself a bug).

Screenshots Not applicable — this is a headless ingestion/parsing bug, not a UI issue.

Desktop (please complete the following information):

  • acryl-datahub version: 1.7.0.7
  • sqlglot version: 30.12.0
  • Source type: mssql
  • Database: SQL Server 2022 (RTM-CU26)
  • Relevant config: include_query_lineage: true

Additional context Impact: for the mssql source with include_query_lineage: true, this silently drops lineage for any query captured from inside a stored procedure that references a typed parameter or local variable. The ingestion report shows num_queries_parse_failures: 0, giving no indication that anything was skipped, which makes this very hard to detect without manually inspecting raw Query Store text and re-running the parser by hand — which is how this was found.