mdsh

$ mdsh - a markdown shell pre-processor

The mdsh project describes a Markdown language extension that can be used to automate some common tasks in README.md files. Quite often I find myself needing to embed a snippet of code or markdown from a different file. Or I want to show the output of a command. In both cases this can be done manually, but what all you had to do was run mdsh and have the file updated automatically?

So the goal of this tool is first to extend the syntax of Markdown in a natural way. Something that you might type. And if the mdsh tool is run, the related blocks get updated in place. Most other tools would produce a new file but we really want a sort of idempotent operation here.

In the end this gives a tool that is a bit akin to literate programming or jupyer notebooks but for shell commands. It adds a bit of verbosity to the file and in exchange it allows to automate the refresh of those outputs.

See the source code of ./spec.clear.md and ./spec.processed.md for everything that mdsh can.

Usage

Run mdsh --help

…

mdsh command

The mdsh "Command" consists of these parts:

[langname]   [data_line]
[data]

in_cmd defines how and where to source data, it can be one of three:

• lang — produce code block with lang (similarly to current as lang statements).
• > — produce raw markdown output fenced by comment-tags
• ! — expand data to shell variables

with these 3 * 3 commands you get 9 combinations, for example:

• > py yml $ ./script.py foo $bar — execute script.py foo $bar in shell and produce yml code block
• >$ ./gen-md.py — execute gen-md.py and produce raw markdown
• ! foo=$bar — use foo=$bar as "raw data" and expand env variables that can be used in the next shell executions
• ! hello — would produce an empty code block with hello language.

Containers

Commands can be put into containers, here's all of them:

Inline code blocks

Must start from new line and end with newline. langname is skipped, parsing starts right from out_cmd, data is absent.

`>$ echo hi`

Code blocks

```[langname]   [data_line]
[data]
```

Source environment variables:

```env !
foo=$bar
```

Execute script and produce yaml block (you can even put shebang at the top and use other than bash scripting languages.

```sh > yaml $
echo 'foo: true'
```

Run data_line as oneline command and pass code block to it via stdin, producing raw markdown.

```> $ sed 's/.*/Hi, \0/'
Bobby
```

Oneline comments

Similar to inline code blocks but hidden:

`` — includes LICENSE.md

Multiline comment blocks

Behaves similarly to code blocks, but langname is not needed

Links

These slightly deviate from the rest of containers:

[  whatever here is ignored]()

Installation

The best way to install mdsh is with the rust tool cargo.

cargo install mdsh

If you are lucky enough to be a nix user:

nix-env -f https://github.com/NixOS/nixpkgs/archive/master.tar.gz -iA mdsh

If you are a nix + flakes user:

nix profile install github:zimbatm/mdsh

Running without installation

If you are a nix + flakes user:

nix run github:zimbatm/mdsh -- --help

Pre-commit hook

This project can also be installed as a pre-commit hook.

Add to your project's .pre-commit-config.yaml:

-   repo: https://github.com/zimbatm/mdsh.git
    rev: main
    hooks:
    -   id: mdsh

Make sure to have rust available in your environment.

Then run pre-commit install-hooks

Known issues

The tool currently lacks in precision as it doesn't parse the Markdown file, it just looks for the desired blocks by regexp. It means that in some cases it might misintepret some of the commands. Most existing Markdown parsers are used to generate HTML in the end and are thus not position-preserving. Eg: pulldown-cmark

The block removal algorithm doesn't support output that contains triple backtick or ``.

Related projects

• is the closest to this project. It has some interesting Pandoc filters that capture code blocks into outputs. The transformation is not in-place like mdsh.
• Enola.dev's ExecMD is another similar tool.
• Literate Programming is the practice of interspesing executable code into documents. There are many language-specific implementations out there. mdsh is a bit like a bash literate programming language.
• Jupyter Notebooks is a whole other universe of documentation and code. It's great but stores the notebooks as JSON files. A special viewer program is required to render them to HTML or text.

User Feedback

Issues

If you have any problems with or questions about this project, please contact us through a GitHub issue.

Contributing

You are invited to contribute new features, fixes or updates, large or small; we are always thrilled to receive pull requests, and do our best to process them as fast as we can.

License

>< LICENSE

MIT License

Copyright (c) 2019 zimbatm and contributors

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.