Tests in Markdown files get line numbers relative to the code blocks, not to the file

When Robot Framework parses a Markdown file (*.robot.md, new in 7.5), it extracts the robotframework code blocks and parses their concatenation. All positions it reports are counted within that concatenation, not within the file: the lineno of tests and keywords, the line attribute in output.xml (lineno in output.json) and the token positions in the parsing model are off by the number of lines outside the code blocks, and the offset grows with every line of prose before or between them. IDEs, linters, analyzers, listeners, debuggers etc. that rely on these positions point to the wrong line.

Version information

Robot Framework 7.5 (Python 3.14.7 on linux)

Steps to reproduce

sample.robot.md:

# Sample suite

Some prose that explains the tests.

More prose.

```robotframework
*** Test Cases ***
First Test
    Log    hello

Second Test
    Log    world
```

Prose between the blocks.

```robotframework
*** Test Cases ***
Third Test
    Log    again
```

First Test is on line 9 of the file, Second Test on line 12, Third Test on line 20.

from robot.api import TestSuite

suite = TestSuite.from_file_system("sample.robot.md")
print([(t.name, t.lineno) for t in suite.tests])

Actual

[('First Test', 2), ('Second Test', 5), ('Third Test', 9)]

robot sample.robot.md writes the same numbers to output.xml: <test id="s1-t1" name="First Test" line="2">, line="5", line="9".

Expected

[('First Test', 9), ('Second Test', 12), ('Third Test', 20)] — the lines of the file, as with .robot files.

Impact

Every tool that takes positions from Robot Framework's models points to the wrong place in a Markdown file: a test explorer lists and opens tests at the wrong line, a linter or language server reports its findings on the wrong line, and a debugger cannot match a breakpoint set on a line of the file with the line Robot Framework reports for the running keyword. The only way around it for such a tool is to re-implement the extraction of the code blocks and map every position back to the file itself, which duplicates parsing logic that belongs to Robot Framework and breaks as soon as that logic changes.

What tools need

Positions in the parsing model (lineno, col_offset, end_col_offset of the tokens) that refer to the file. Everything else — lineno in the running and result models, line in output.xml and lineno in output.json, the lineno in listener attributes, error messages — follows from that.

This issue is about Markdown on purpose. reStructuredText suites have the same problem, but they are an old feature that hardly anyone uses, and no modern editor supports reStructuredText out of the box. Markdown is everywhere, editors such as VS Code render and edit it natively, and the feature is new in Robot Framework 7.5: if IDE support, debugging etc. work in Markdown files from the start, people are likely to use it. With file positions in the parsing model, tools could treat the code blocks like .robot files. In VS Code and PyCharm (I have not tried PyCharm yet), for example, RobotCode could let users run and debug the tests of a Markdown file directly from the editor — from the test explorer and the run buttons next to each test, with breakpoints on the lines of the file — and offer code completion, diagnostics, hover, navigation and highlighting inside the robotframework code blocks. That would be real added value: suites and resources could be documented and written in Markdown without giving up the editor support people are used to from .robot files. RobotCode has put its support for Markdown files on hold until the positions come from Robot Framework.

Cause

MarkdownParser._get_source (robot/running/builder/parsers.py) calls _read_markdown_data, which collects only the lines inside the robotframework/robot code blocks, removes the common indentation of each block with dedent and joins the blocks with "\n":

        return "\n".join(dedent("".join(blk)) for blk in blocks)

The lexer then numbers the lines of that string from 1: the first line of the first code block becomes line 1, and all lines outside the code blocks — headings, prose, blank lines, the fences themselves — are missing from the count. The code is the same on the current master.

Possible fix

Two variants, both in robot/running/builder/parsers.py and both tried out on Robot Framework 7.5 (see Verification below). Variant 1 fixes the line numbers with one added line. Variant 2 additionally fixes the columns of code blocks that are indented in the Markdown file, for example inside a list item, and needs a small, backwards compatible addition to RobotParser for that.

Variant 1 — line numbers, one added line in MarkdownParser. The idea is to keep the line count of the file in the text that is handed to the lexer. Every line outside a code block — prose, blank lines and the opening fence — contributes an empty block, and the existing "\n".join(...) then puts exactly one newline per such line into the text; the newline that the join places after a code block accounts for its closing fence. The lexer therefore sees the content of every code block on the same lines as in the file. The empty lines it sees in between are harmless: empty lines carry no meaning in the Robot Framework syntax, and with data_only=True they do not even reach the model. Nothing else changes: _get_source still returns a string, RobotParser is untouched, and the running and result models, output.xml/output.json, listener attributes and error messages get the file lines automatically, because they all come from the token positions.

The dedent of each block has to stay. A code block inside a list item is indented in the file, and Markdown treats that indentation as belonging to the list, not to the code. Without the dedent, the lines of such a block start with whitespace, the lexer reads *** Test Cases *** as an indented statement instead of a section header, and the whole block is lost (Non-existing setting ''.). This is the limitation of variant 1: in an indented code block the line numbers are right, but the columns are off by the removed indentation — a test name that is at column 2 in the file is reported at column 0, a keyword call below it at column 4 instead of 6. Code blocks that are not indented, the normal case, get exact lines and columns.

The complete MarkdownParser with variant 1; the only difference to 7.5 is the line marked new:

class MarkdownParser(RobotParser):
    extensions = (".robot.md", ".md", ".markdown")

    def _get_source(self, source: Path) -> str:
        with FileReader(source) as reader:
            return self._read_markdown_data(reader)

    def _read_markdown_data(self, mdfile: FileReader) -> str:
        fence_open = re.compile(r"\s*(`{3,}|~{3,})\s*(\S+)")
        fence_close = None
        blocks = []
        block = None
        for line in mdfile.readlines():
            if block is None:
                blocks.append([])  # new: keep the line count of the file
                match = fence_open.match(line)
                if match:
                    fence, lang = match.groups()
                    if lang.lower() in ("robotframework", "robot"):
                        fence_close = re.compile(rf"\s*{fence[0]}{{{len(fence)},}}\s*")
                        block = []
            elif fence_close.fullmatch(line):
                blocks.append(block)
                block = None
            else:
                block.append(line)
        if block:
            blocks.append(block)
        return "\n".join(dedent("".join(blk)) for blk in blocks)

Variant 2 — line numbers and columns, with a small hook in RobotParser. Variant 2 contains variant 1 and restores the columns of indented code blocks. While reading, MarkdownParser computes for every block the margin that dedent removes (the smallest indentation of its non-blank lines) and records it for each line of the block in a table line number in the file -> removed columns. After the model is built, a small ModelVisitor walks every statement and adds the recorded columns back to the col_offset of every token on such a line; end_col_offset follows, because it is derived from col_offset and the length of the token. Lines of code blocks that were not indented have no entry in the table and are not touched. The visitor works for the suite, init and resource models alike, as they are all File models.

To get the table from the reader to the model, the three parse methods of RobotParser (parse_suite_file, parse_init_file, parse_resource_file) keep their structure and get one additional call each, self._prepare_model(model), right after the model is built and model.source is set. In RobotParser this new hook does nothing, so the behaviour of .robot, .resource and all other parsers stays exactly as it is, and other parsers that read Robot Framework data out of another format could use the same hook later. MarkdownParser._read_markdown_data fills the table into self._indents while reading (the dedent moves into a small helper _dedent that also records the margin), and MarkdownParser._prepare_model runs RestoreColumns over the model when the table is not empty. _get_source still returns a string. The table is kept on the parser instance between the read and the hook; Robot Framework creates one MarkdownParser and reuses it for every file, which works because the table is reset at the start of every read and consumed right after by the same parse call. The column correction assumes a consistently indented code block, which dedent assumes as well.

The changed parts of RobotParser (unchanged parts shown as ...) and the complete MarkdownParser with variant 2:

from robot.parsing import File, ModelVisitor, get_init_model, get_model, get_resource_model


class RobotParser(Parser):
    ...

    def parse_suite_file(self, source: Path, defaults: TestDefaults) -> TestSuite:
        model = get_model(
            self._get_source(source),
            data_only=True,
            curdir=self._get_curdir(source),
            lang=self.lang,
        )
        model.source = source
        self._prepare_model(model)
        return self.parse_model(model, defaults)

    def parse_init_file(self, source: Path, defaults: TestDefaults) -> TestSuite:
        model = get_init_model(
            self._get_source(source),
            data_only=True,
            curdir=self._get_curdir(source),
            lang=self.lang,
        )
        model.source = source
        self._prepare_model(model)
        suite = TestSuite(
            name=TestSuite.name_from_source(source.parent),
            source=source.parent,
            rpa=None,
        )
        SuiteBuilder(suite, InitFileSettings(defaults)).build(model)
        return suite

    def parse_resource_file(self, source: Path) -> ResourceFile:
        model = get_resource_model(
            self._get_source(source),
            data_only=True,
            curdir=self._get_curdir(source),
            lang=self.lang,
        )
        model.source = source
        self._prepare_model(model)
        return self.parse_resource_model(model)

    def _prepare_model(self, model: File):
        """Hook for parsers that read the data out of another format."""

    ...


class MarkdownParser(RobotParser):
    extensions = (".robot.md", ".md", ".markdown")

    def _get_source(self, source: Path) -> str:
        with FileReader(source) as reader:
            return self._read_markdown_data(reader)

    def _read_markdown_data(self, mdfile: FileReader) -> str:
        fence_open = re.compile(r"\s*(`{3,}|~{3,})\s*(\S+)")
        fence_close = None
        blocks = []
        block = None
        self._indents = {}  # line number in the file -> columns removed by `dedent`
        lineno = 0
        for line in mdfile.readlines():
            lineno += 1
            if block is None:
                blocks.append([])  # keep one line per line of the file
                match = fence_open.match(line)
                if match:
                    fence, lang = match.groups()
                    if lang.lower() in ("robotframework", "robot"):
                        fence_close = re.compile(rf"\s*{fence[0]}{{{len(fence)},}}\s*")
                        block = []
                        first_line = lineno + 1
            elif fence_close.fullmatch(line):
                blocks.append(self._dedent(block, first_line))
                block = None
            else:
                block.append(line)
        if block:
            blocks.append(self._dedent(block, first_line))
        return "\n".join("".join(blk) for blk in blocks)

    def _dedent(self, block: "list[str]", first_line: int) -> "list[str]":
        margin = min((len(l) - len(l.lstrip()) for l in block if l.strip()), default=0)
        if margin:
            for offset in range(len(block)):
                self._indents[first_line + offset] = margin
        return dedent("".join(block)).splitlines(keepends=True)

    def _prepare_model(self, model: File):
        if self._indents:
            RestoreColumns(self._indents).visit(model)


class RestoreColumns(ModelVisitor):
    """Moves the tokens of dedented lines back to their columns in the file."""

    def __init__(self, indents: "dict[int, int]"):
        self.indents = indents

    def visit_Statement(self, node):
        for token in node.tokens:
            token.col_offset += self.indents.get(token.lineno, 0)

Verification. Both variants were tried out on Robot Framework 7.5 by replacing MarkdownParser with the code above, with the example from this issue, a suite file with a *** Settings *** block, a code block indented by two columns inside a list item and a plain block, and a Markdown resource file with an indented *** Keywords *** block. One parser instance was reused for all files, as Robot Framework does.

Pull request

I would be happy to provide a pull request for this, and would prefer variant 2: it gives tools exact lines and columns in every case, and the change to RobotParser is limited to one hook method that does nothing for the existing parsers. Variant 2 matters for editor support: with variant 1, the tokens of a code block inside a list item are shifted by the indentation of the block, so code completion, diagnostics, hover and highlighting would land at the wrong position there, even though the line is right. If the change to RobotParser is not wanted, variant 1 alone already fixes the line numbers. The pull request would also update utest/running/test_markdownparser.py: 20 of its 21 tests compare the exact text that _read_markdown_data returns, and that text now contains one line per line of the file, so their expected values change. New tests would cover the line numbers and, for variant 2, the columns.

Which variant would you prefer?

AI disclosure: I used AI assistance to analyse the problem, prototype the variants and write this issue. I reviewed everything and tried out and tested the example and both variants myself.