SCC's representation and involvement in LineJudge

Hi,

I am the creator of mezura, another line counter, and for the past month I have been building linejudge, a conformance suite for line counters, that checks whether each counter counts the way it says it counts. scc is one of the counters measured in it. I'm opening this issue to tell you the suite exists, explain how scc is represented in it, ask you to check that representation and invite you in the initiative.

What linejudge is

LineJudge holds a corpus of many small source files, each built around one thing that trips counters up: an escaped quote before a comment opener, a nested block comment, a string carrying its own delimiter, a line spliced onto the next with a backslash, a second language inside the file, etc. Each test file comes with a record of where every string and comment begins and ends, according to the language's spec or lexer, and always hand verified.
LineJudge also contains the tools around it: it downloads each counter's binary, runs it over every case, keeps a snapshot of what it answered so one release can be compared with the next, explains any single case line by line, and renders the results to a public webpage.

That record is the objective half: where a string or a comment begins and ends is settled by the language, and it is the same no matter which counter is asking. How the lines are then counted is the half with no objectively correct answer: counters disagree because they count by different rules, and most of those rules are defensible. So a counter is never judged against another counter, or against a predetermined idea of the right count. Each counter has a dialect file declaring the rules it counts by. From those rules and the record, the suite works out what that counter should answer on each file, then runs the counter and compares. A failure means one thing only: the counter did not do what its own rules say it should.

How scc is represented

The dialect file for scc is dialects/scc/default.toml. I wrote it from your documentation, from limited reading of the source, and from my observations about what the program does, so it is my best understanding of how scc counts, and it may be wrong in places. That file is what I'm asking you to check.
mezura is measured the same way and fails cases of its own.

How to read it. Each [[rule]] sorts lines into one of scc's three 'buckets', code, comments or blanks.
when is the condition, a list of yes/no questions about a line that must all be true, with ! meaning not.
in-string asks whether a string covers part of the line,
has-residue whether anything beyond whitespace sits outside every string and comment, and so on. The seven questions are defined in dialects/README.md and implemented once inside linejudge, so the file holds no code.
name is the rule's claim written as a sentence, and it is what the results page quotes when it says which rule decided a line.
The rules are not tried in order. Every line must land in exactly one bucket.

In words, the rules of this dialect file say that scc counts anything outside strings and comments as code, a plain string as code, a triple-quoted docstring with nothing beside it as comments, a comment with nothing beside it as comments, and a blank line outside everything as blank.
The [counts-as-its-own-language] block covers two stretches of a file that can fairly be counted either way, and says that scc counts Rust doc comments as comments of the Rust file and a Vue <template> as part of the Vue file.

Here is an example of what a disagreement looks like. A batch file, where the comment keyword is a whole word:

REM a comment
REMOVE /Q file.txt
echo done

$ linejudge explain 1230 --counter scc

scc.default on 1230-a_comment_word_at_the_head_of_a_longer_word
  by its rules    3 lines, 0 blanks, 2 code, 1 comments
  scc answers     3 lines, 0 blanks, 1 code, 2 comments   ✗ differs

scc 4.0.0 reads REMOVE as a comment: a line whose first word begins with REM is taken as one, whatever letters follow. Measured beside it, XREMOVE /Q file.txt counts as code and so does echo REM inside, so it is the head of the line that decides.

Every case where scc's answer differs from its own rules is published here, with the input, the strings and comments marked byte by byte, the rule that decided each line, and the command that reproduces it:

https://loc-conformance.github.io/linejudge/

What I'm asking from you

The immediate thing is the dialect file: if it says something different than what scc intends to do, it needs to be corrected. Actually, this file should probably belong to you.
I would like to set up a CODEOWNERS file for the maintainers of the counters that have native support in linejudge. Say the word and I will add you (@boyter), and anyone else who maintains scc and you trust with it.

Beside it are the notes. Every failure carries a line or two saying what scc did with the file. They live in recorded/scc.toml, the automatically written snapshot of what scc answered on every case, and the results page shows them from there. A few of them go further and say how scc read a file, which is a claim about the inside of a program I have only measured from the outside. Those are the ones that can be wrong. If any note says something untrue about scc, tell me and I will correct it.

The bigger thing is the project. linejudge lives under loc-conformance, an organization I created because I believe that this should not remain a one-person project. The goal is a collection of tools, tests and repos around line counting that the loc-counter authors run together. Today that means linejudge, with each counter's representation maintained by its own people and a corpus that grows with the cases each of us has been bitten by; tomorrow it is whatever we decide it should be (I have a few ideas about what should be built next). I would genuinely like you in it, you and whoever else you would bring from scc, whether that is taking part in the development or being active in the discussions and the decisions that will shape where the organization goes. The current design, where it goes next and how the organization is meant to be run are already up for review in the repository's discussions, and that is where the decisions get made.

Congratulations on 4.0.0, by the way: measured against the corpus, it fixes four cases that 3.7.0 failed (C++ raw strings holding their own closing bracket, Lua long brackets, and Rust raw strings ending in a backslash, in both the plain and the bare form).

Thanks for scc, it pushes the rest of us to do better!