protoc plugin library for efficient proto-based code generation
protoc plugin library for efficient proto-based code generation
!!! THIS PROJECT IS A WORK-IN-PROGRESS | THE API SHOULD BE CONSIDERED UNSTABLE !!!
PG* is a protoc plugin library for efficient proto-based code generation
package main
import "github.com/lyft/protoc-gen-star/v2"
func main() {
pgs.Init(pgs.DebugEnv("DEBUG")).
RegisterModule(&myPGSModule{}).
RegisterPostProcessor(&myPostProcessor{}).
Render()
}
While this README seeks to describe many of the nuances of protoc plugin development and using PG*, the true documentation source is the code itself. The Go language is self-documenting and provides tools for easily reading through it and viewing examples. The docs can be viewed on GoDoc or locally by running make docs, which will start a godoc server and open them in the default browser.
Visitor][visitor] pattern and helpers for efficiently walking the dependency graphBuildContext][context] to facilitate complex generationParameters][params] accessModuleBase for quickly creating Modules and facilitating code generation[protoc-gen-example][pge], can be found in the testdata directory. It includes two Module implementations using a variety of the features available. It's protoc execution is included in the testdata/generated [Makefile][make] target. Examples are also accessible via the documentation by running make docs.
protoc FlowBecause the process is somewhat confusing, this section will cover the entire flow of how proto files are converted to generated code, using a hypothetical PG* plugin: protoc-gen-myplugin. A typical execution looks like this:
protoc \
-I . \
--myplugin_out="foo=bar:../generated" \
./pkg/*.proto
protoc, the PB compiler, is configured using a set of flags (documented under protoc -h) and handed a set of files as arguments. In this case, the I flag can be specified multiple times and is the lookup path it uses for imported dependencies in a proto file. By default, the official descriptor protos are already included.
myplugin_out tells protoc to use the protoc-gen-myplugin protoc-plugin. These plugins are automatically resolved from the system's PATH environment variable, or can be explicitly specified with another flag. The official protoc-plugins (eg, protoc-gen-python) are already registered with protoc. The flag's value is specific to the particular plugin, with the exception of the :../generated suffix. This suffix indicates the root directory in which protoc will place the generated files from that package (relative to the current working directory). This generated output directory is not propagated to protoc-gen-myplugin, however, so it needs to be duplicated in the left-hand side of the flag. PG* supports this via an output_path parameter.
protoc parses the passed in proto files, ensures they are syntactically correct, and loads any imported dependencies. It converts these files and the dependencies into descriptors (which are themselves PB messages) and creates a CodeGeneratorRequest (yet another PB). protoc serializes this request and then executes each configured protoc-plugin, sending the payload via stdin.
protoc-gen-myplugin starts up, receiving the request payload, which it unmarshals. There are two phases to a PG*-based protoc-plugin. First, PG* unmarshals the CodeGeneratorRequest received from protoc, and creates a fully connected abstract syntax tree (AST) of each file and all its contained entities. Any parameters specified for this plugin are also parsed for later consumption.
When this step is complete, PG* then executes any registered Modules, handing it the constructed AST. Modules can be written to generate artifacts (eg, files) or just performing some form of validation over the provided graph without any other side effects. Modules provide the great flexibility in terms of operating against the PBs.
Once all Modules are run, PG* writes any custom artifacts to the file system or serializes generator-specific ones into a CodeGeneratorResponse and sends the data to its stdout. protoc receives this payload, unmarshals it, and persists any requested files to disk after all its plugins have returned. This whole flow looks something like this:
foo.proto → protoc → CodeGeneratorRequest → protoc-gen-myplugin → CodeGeneratorResponse → protoc → foo.pb.go
The PG* library hides away nearly all of this complexity required to implement a protoc-plugin!
PG* Modules are handed a complete AST for those files that are targeted for generation as well as all dependencies. A Module can then add files to the protoc CodeGeneratorResponse or write files directly to disk as Artifacts.
PG* provides a ModuleBase struct to simplify developing modules. Out of the box, it satisfies the interface for a Module, only requiring the creation of Name and Execute methods. ModuleBase is best used as an anonyomous embedded field of a wrapping Module implementation. A minimal module would look like the following:
…
ModuleBase exposes a PG* [BuildContext][context] instance, already prefixed with the module's name. Calling Push and Pop allows adding further information to error and debugging messages. Above, each file from the target package is pushed onto the context before logging the "reporting" debug message.
The base also provides helper methods for adding or overwriting both protoc-generated and custom files. The above execute method creates a custom file at /tmp/report.txt specifying that it should overwrite an existing file with that name. If it instead called AddCustomFile and the file existed, no file would have been generated (though a debug message would be logged out). Similar methods exist for adding generator files, appends, and injections. Likewise, methods such as AddCustomTemplateFile allows for Templates to be rendered instead.
After all modules have been executed, the returned Artifacts are either placed into the CodeGenerationResponse payload for protoc or written out to the file system. For testing purposes, the file system has been abstracted such that a custom one (such as an in-memory FS) can be provided to the PG* generator with the FileSystem InitOption.
Artifacts generated by Modules sometimes require some mutations prior to writing to disk or sending in the response to protoc. This could range from running gofmt against Go source or adding copyright headers to all generated source files. To simplify this task in PG*, a PostProcessor can be utilized. A minimal looking PostProcessor implementation might look like this:
…
The copyrightPostProcessor struct satisfies the PostProcessor interface by implementing the Match and Process methods. After PG* recieves all Artifacts, each is handed in turn to each registered processor's Match method. In the above case, we return true if the file is a part of the targeted Artifact types. If true is returned, Process is immediately called with the rendered contents of the file. This method mutates the input, returning the modified value to out or an error if something goes wrong. Above, the notice is prepended to the input.
PostProcessors are registered with PG* similar to Modules:
g := pgs.Init(pgs.IncludeGo())
g.RegisterModule(some.NewModule())
g.RegisterPostProcessor(copyright.New("PG* Authors"))
While protoc ensures that all the dependencies required to generate a proto file are loaded in as descriptors, it's up to the protoc-plugins to recognize the relationships between them. To get around this, PG* uses constructs an abstract syntax tree (AST) of all the Entities loaded into the plugin. This AST is provided to every Module to facilitate code generation.
The hierarchy generated by the PG* gatherer is fully linked, starting at a top-level Package down to each individual Field of a Message. The AST can be represented with the following digraph:
A Package describes a set of Files loaded within the same namespace. As would be expected, a File represents a single proto file, which contains any number of Message, Enum or Service entities. An Enum describes an integer-based enumeration type, containing each individual EnumValue. A Service describes a set of RPC Methods, which in turn refer to their input and output Messages.
A Message can contain other nested Messages and Enums as well as each of its Fields. For non-scalar types, a Field may also reference its Message or Enum type. As a mechanism for achieving union types, a Message can also contain OneOf entities that refer to some of its Fields.
The structure of the AST can be fairly complex and unpredictable. Likewise, Module's are typically concerned with only a subset of the entities in the graph. To separate the Module's algorithm from understanding and traversing the structure of the AST, PG* implements the Visitor pattern to decouple the two. Implementing this interface is straightforward and can greatly simplify code generation.
Two base Visitor structs are provided by PG* to simplify developing implementations. First, the NilVisitor returns an instance that short-circuits execution for all Entity types. This is useful when certain branches of the AST are not interesting to code generation. For instance, if the Module is only concerned with Services, it can use a NilVisitor as an anonymous field and only implement the desired interface methods:
…
If access to deeply nested Nodes is desired, a PassthroughVisitor can be used instead. Unlike NilVisitor and as the name suggests, this implementation passes through all nodes instead of short-circuiting on the first unimplemented interface method. Setup of this type as an anonymous field is a bit more complex but avoids implementing each method of the interface explicitly:
type fieldVisitor struct {
pgs.Visitor
pgs.DebuggerCommon
}
func New(d pgs.DebuggerCommon) pgs.Visitor {
v := &fieldVisitor{DebuggerCommon: d}
v.Visitor = pgs.PassThroughVisitor(v)
return v
}
func (v *fieldVisitor) VisitField(f pgs.Field) (pgs.Visitor, error) {
v.Logf("%v.%v", f.Message().Name(), f.Name())
return nil, nil
}
Walking the AST with any Visitor is straightforward:
v := visitor.New(d)
err := pgs.Walk(v, pkg)
All Entity types and Package can be passed into Walk, allowing for starting a Visitor lower than the top-level Package if desired.
Modules registered with the PG* Generator are initialized with an instance of BuildContext that encapsulates contextual paths, debugging, and parameter information.
The BuildContext's OutputPath method returns the output directory that the PG* plugin is targeting. This path is also initially . but refers to the directory in which protoc is executed. This
No open issues yet, or sync has not completed.