syntax_tree

SyntaxTree

Syntax Tree is a suite of tools built on top of the internal CRuby parser. It provides the ability to generate a syntax tree from source, as well as the tools necessary to inspect and manipulate that syntax tree. It can be used to build formatters, linters, language servers, and more.

It is built with only standard library dependencies. It additionally ships with a plugin system so that you can build your own syntax trees from other languages and incorporate these tools.

• Installation
• CLI
  • ast
  • check
  • ctags
  • expr
  • format
  • json
  • match
  • search
  • write
  • Configuration
  • Globbing
• Library
  • SyntaxTree.read(filepath)
  • SyntaxTree.parse(source)
  • SyntaxTree.format(source)
  • SyntaxTree.mutation(&block)
  • SyntaxTree.search(source, query, &block)
  • SyntaxTree.index(source)
• Nodes
  • child_nodes
  • copy(**attrs)
  • Pattern matching
  • pretty_print(q)
  • to_json(*opts)
  • format(q)
  • ===(other)
  • construct_keys
• Visitor
  • visit_method
  • visit_methods
  • BasicVisitor
  • MutationVisitor
  • WithScope
• Language server
  • textDocument/formatting
  • textDocument/inlayHint
  • syntaxTree/visualizing
• Customization
  • Ignoring code
  • Plugins
  • Languages
• Integration
  • Rake
  • RuboCop
  • Editors
• Contributing
• License

Installation

Syntax Tree is both a command-line interface and a library. If you're only looking to use the command-line interface, then we recommend installing the gem globally, as in:

gem install syntax_tree

To run the CLI with the gem installed globally, you would run:

stree version

If you're planning on using Syntax Tree as a library within your own project, we recommend installing it as part of your gem bundle. First, add this line to your application's Gemfile:

gem "syntax_tree"

And then execute:

bundle install

To run the CLI with the gem installed in your gem bundle, you would run:

bundle exec stree version

CLI

Syntax Tree ships with the stree CLI, which can be used to inspect and manipulate Ruby code. Below are listed all of the commands built into the CLI that you can use.

For many commands, file paths are accepted after the configuration options. For all of these commands, you can alternatively pass in content through STDIN or through the -e option to specify an inline script.

ast

This command will print out a textual representation of the syntax tree associated with each of the files it finds. To execute, run:

stree ast path/to/file.rb

For a file that contains 1 + 1, you will receive:

(program (statements (binary (int "1") + (int "1"))))

check

This command is meant to be used in the context of a continuous integration or git hook. It checks each file given to make sure that it matches the expected format. It can be used to ensure unformatted content never makes it into a codebase.

stree check path/to/file.rb

For a file that matches the expected format, you will receive:

All files matched expected format.

If there are files with unformatted code, you will receive:

[warn] path/to/file.rb
The listed files did not match the expected format.

To change the print width that you are checking against, specify the --print-width option, as in:

stree check --print-width=100 path/to/file.rb

ctags

This command will output to stdout a set of tags suitable for usage with ctags.

stree ctags path/to/file.rb

For a file containing the following Ruby code:

class Foo
end

class Bar  (binary (int "1") :+ (int "1"))

copy(**attrs)

This method returns a copy of the node, with the given attributes replaced.

program = SyntaxTree.parse("1 + 1")

binary = program.statements.body.first
# => (binary (int "1") + (int "1"))

binary.copy(operator: :-)
# => (binary (int "1") - (int "1"))

Pattern matching

Pattern matching is another way to descend the tree which is more specific than using child_nodes. Using Ruby's built-in pattern matching, you can extract the same information but be as specific about your constraints as you like. For example, with minimal constraints:

program = SyntaxTree.parse("1 + 1")
program => { statements: { body: [binary] } }
binary
# => (binary (int "1") :+ (int "1"))

Or, with more constraints on the types to ensure we're getting exactly what we expect:

program = SyntaxTree.parse("1 + 1")
program => SyntaxTree::Program[statements: SyntaxTree::Statements[body: [SyntaxTree::Binary => binary]]]
binary
# => (binary (int "1") :+ (int "1"))

pretty_print(q)

Every node responds to the pretty_print Ruby interface, which makes it usable by the pp library. You can use this API manually, but it's mostly there for compatibility and not meant to be directly invoked. For example:

pp SyntaxTree.parse("1 + 1")
# (program (statements (binary (int "1") + (int "1"))))

to_json(*opts)

Every node responds to the to_json Ruby interface, which makes it usable by the json library. Much like pretty_print, you could use this API manually, but it's mostly used by JSON to dump the nodes to a serialized format. For example:

program = SyntaxTree.parse("1 + 1")
program => { statements: { body: [{ left: }] } }
puts JSON.dump(left)
# {"type":"int","value":"1","loc":[1,0,1,1],"cmts":[]}

format(q)

Every node responds to format, which formats the content nicely. The API mirrors that used by the pretty_print gem in that it accepts a formatter object and calls methods on it to generate its own internal representation of the text that will be outputted. Because of this, it's easier to not use this API directly and instead to call SyntaxTree.format. You can however use this directly if you create the formatter yourself, as in:

source = "1+1"
program = SyntaxTree.parse(source)
program => { statements: { body: [binary] } }

formatter = SyntaxTree::Formatter.new(source, [])
binary.format(formatter)

formatter.flush
formatter.output.join
# => "1 + 1"

===(other)

Every node responds to ===, which is used to check if the given other node matches all of the attributes of the current node except for location and comments. For example:

program1 = SyntaxTree.parse("1 + 1")
program2 = SyntaxTree.parse("1 + 1")

program1 === program2
# => true

construct_keys

Every node responds to construct_keys, which will return a string that contains a Ruby pattern-matching expression that could be used to match against the current node. It's meant to be used in tooling and through the CLI mostly.

program = SyntaxTree.parse("1 + 1")
puts program.construct_keys

# SyntaxTree::Program[
#   statements: SyntaxTree::Statements[
#     body: [
#       SyntaxTree::Binary[
#         left: SyntaxTree::Int[value: "1"],
#         operator: :+,
#         right: SyntaxTree::Int[value: "1"]
#       ]
#     ]
#   ]
# ]

Visitor

If you want to operate over a set of nodes in the tree but don't want to walk the tree manually, the Visitor class makes it easy. SyntaxTree::Visitor is an implementation of the double dispatch visitor pattern. It works by the user defining visit methods that process nodes in the tree, which then call back to other visit methods to continue the descent. This is easier shown in code.

Let's say, for instance, that you wanted to find every place in source where you have an arithmetic problem between two integers (this is pretty contrived, but it's just for illustration). You could define a visitor that only explicitly visits the SyntaxTree::Binary node, as in:

class ArithmeticVisitor '
	from (irb):1:in `'
	from bin/console:8:in `'

visit_methods

Similar to visit_method, visit_methods also checks that methods defined are valid visit methods. This variation however accepts a block and checks that all methods defined within that block are valid visit methods. It's meant to be used like:

class ArithmeticVisitor  "if a = 1\nend\n"

SyntaxTree::Formatter.format(source, program.accept(visitor))
# => "if (a = 1)\nend\n"

WithScope

The WithScope module can be included in visitors to automatically keep track of local variables and arguments defined inside each scope. A current_scope accessor is made available to the request, allowing it to find all usages and definitions of a local.

class MyVisitor < Visitor
  prepend WithScope

  def visit_ident(node)
    # find_local will return a Local for any local variables or arguments
    # present in the current environment or nil if the identifier is not a local
    local = current_scope.find_local(node)

    puts local.type # the type of the local (:variable or :argument)
    puts local.definitions # the array of locations where this local is defined
    puts local.usages # the array of locations where this local occurs
  end
end

Language server

Syntax Tree additionally ships with a language server conforming to the language server protocol. It can be invoked through the CLI by running:

stree lsp

By default, the language server is relatively minimal, mostly meant to provide a registered formatter for the Ruby language. However there are a couple of additional niceties baked in. There are related projects that configure and use this language server within IDEs. For example, to use this code with VSCode, see ruby-syntax-tree/vscode-syntax-tree.

textDocument/formatting

As mentioned above, the language server responds to formatting requests with the formatted document. It typically responds on the order of tens of milliseconds, so it should be fast enough for any IDE.

textDocument/inlayHint

The language server also responds to the relatively new inlay hints request. This request allows the language server to define additional information that should exist in the source code as helpful hints to the developer. In our case we use it to display things like implicit parentheses. For example, if you had the following code:

1 + 2 * 3

Implicitly, the 2 * 3 is going to be executed first because the * operator has higher precedence than the + operator. To ease mental overhead, our language server includes small parentheses to make this explicit, as in:

1 + ₍2 * 3₎

syntaxTree/visualizing

The language server additionally includes this custom request to return a textual representation of the syntax tree underlying the source code of a file. Language server clients can use this to (for example) open an additional tab with this information displayed.

Customization

There are multiple ways to customize Syntax Tree's behavior when parsing and formatting code. You can ignore certain sections of the source code, you can register plugins to provide custom formatting behavior, and you can register additional languages to be parsed and formatted.

Ignoring code

To ignore a section of source code, y