shyaml

================================= SHYAML: YAML for the command line

.. image:: https://img.shields.io/pypi/v/shyaml.svg :target: https://pypi.python.org/pypi/shyaml

.. image:: https://img.shields.io/travis/0k/shyaml/master.svg?style=flat :target: https://travis-ci.com/github/0k/shyaml/ :alt: Travis CI build status

.. image:: https://img.shields.io/appveyor/ci/vaab/shyaml.svg :target: https://ci.appveyor.com/project/vaab/shyaml/branch/master :alt: Appveyor CI build status

.. image:: http://img.shields.io/codecov/c/github/0k/shyaml.svg?style=flat :target: https://codecov.io/gh/0k/shyaml/ :alt: Test coverage

.. note::

Consider switching to shyaml-rs _, a Rust rewrite of shyaml.

This Python version remains available and is not archived, but it is in a lethargic state: no new features are planned, and maintenance is limited to critical fixes. The Rust version is where active development happens.

shyaml-rs is designed as a drop-in replacement — your existing shell scripts should work without modification. Reasons to switch:

• Performance: 14–42× faster than the Python version (2 ms vs 92 ms on a 7 KB file), with no startup overhead from a Python interpreter.
• Modern YAML 1.2 support: Uses libfyaml _ instead of the aging libyaml (last released in 2020, YAML 1.1 only). No artificial limits such as libyaml's 1024-character key cap.
• Write capabilities: set-value, del, and apply (YAML merging) are new actions not available in this Python version.
• Single static binary: No Python runtime needed. Download and run.

Install via cargo install shyaml-rs or grab a static Linux binary from the releases page _.

If you encounter any compatibility difference between the two versions, please open an issue _ — feedback is welcome and helps ensure a smooth transition.

Description

Simple script that allow read access to YAML files through command line.

This can be handy, if you want to get access to YAML data in your shell scripts.

This script supports only read access and it might not support all the subtleties of YAML specification. But it should support some handy basic query of YAML file.

Requirements

shyaml works in Linux, MacOSX, and Windows with python 2.7 and 3+.

Installation

You don't need to download the GIT version of the code as shyaml is available on the PyPI. So you should be able to run::

pip install shyaml

If you have downloaded the GIT sources, then you could add install the current version via::

pip install .

And if you don't have the GIT sources but would like to get the latest master or branch from github, you could also::

pip install git+https://github.com/0k/shyaml

Or even select a specific revision (branch/tag/commit)::

pip install git+https://github.com/0k/shyaml@master

On macOS, you can also install the latest release version via Homebrew _::

brew install shyaml

Or to install the master branch::

brew install shyaml --HEAD

Documentation

The following documented examples are actually tested automatically at each release for conformance on all platform and python versions.

Please note that there is some subtle benign differences in some output whether shyaml is using the libyaml C implementation or the full python implementation. The documentation can be run with both implementation but some examples will fail depending on the implementation. To make things clear, I'll use some annotation and you can yourself check which version you are using with::

$ shyaml -V | grep "^libyaml used:"  ## docshtest: if-success-set LIBYAML
libyaml used: True

Usage

shyaml takes its YAML input file from standard input ONLY. So let's define here a common YAML input for the next examples::

$ cat  test.yaml
name: "MyName !! héhé"  ## using encoding, and support comments !
subvalue:
    how-much: 1.1
    how-many: 2
    things:
        - first
        - second
        - third
    maintainer: "Valentin Lab"
    description: |
        Multiline description:
        Line 1
        Line 2
subvalue.how-much: 1.2
subvalue.how-much\more: 1.3
subvalue.how-much\.more: 1.4
EOF

General browsing struct and displaying simple values

Simple query of simple attribute::

$ cat test.yaml | shyaml get-value name
MyName !! héhé

Query nested attributes by using '.' between key labels::

$ cat test.yaml | shyaml get-value subvalue.how-much
1.1

Get type of attributes::

$ cat test.yaml | shyaml get-type name
str
$ cat test.yaml | shyaml get-type subvalue.how-much
float

Get length of structures or sequences::

$ cat test.yaml | shyaml get-length subvalue
5
$ cat test.yaml | shyaml get-length subvalue.things
3

But this won't work on other types::

$ cat test.yaml | shyaml get-length name
Error: get-length does not support 'str' type. Please provide or select a sequence or struct.

Parse structure

Get sub YAML from a structure attribute::

$ cat test.yaml | shyaml get-type subvalue
struct
$ cat test.yaml | shyaml get-value subvalue  ## docshtest: ignore-if LIBYAML
how-much: 1.1
how-many: 2
things:
- first
- second
- third
maintainer: Valentin Lab
description: 'Multiline description:

  Line 1

  Line 2

  '

Iteration through keys only::

$ cat test.yaml | shyaml keys
name
subvalue
subvalue.how-much
subvalue.how-much\more
subvalue.how-much\.more

Iteration through keys only (\0 terminated strings)::

$ cat test.yaml | shyaml keys-0 subvalue | xargs -0 -n 1 echo "VALUE:"
VALUE: how-much
VALUE: how-many
VALUE: things
VALUE: maintainer
VALUE: description

Iteration through values only (\0 terminated string highly recommended)::

$ cat test.yaml | shyaml values-0 subvalue |
  while IFS='' read -r -d $'\0' value; do
      echo "RECEIVED: '$value'"
  done
RECEIVED: '1.1'
RECEIVED: '2'
RECEIVED: '- first
- second
- third
'
RECEIVED: 'Valentin Lab'
RECEIVED: 'Multiline description:
Line 1
Line 2
'

Iteration through keys and values (\0 terminated string highly recommended)::

$ read-0() {
    while [ "$1" ]; do
        IFS=$'\0' read -r -d '' "$1" || return 1
        shift
    done
  } &&
  cat test.yaml | shyaml key-values-0 subvalue |
  while read-0 key value; do
      echo "KEY: '$key'"
      echo "VALUE: '$value'"
      echo
  done
KEY: 'how-much'
VALUE: '1.1'

KEY: 'how-many'
VALUE: '2'

KEY: 'things'
VALUE: '- first
- second
- third
'

KEY: 'maintainer'
VALUE: 'Valentin Lab'

KEY: 'description'
VALUE: 'Multiline description:
Line 1
Line 2
'

Notice, that you'll get the same result using get-values. get-values will support sequences and struct, and key-values support only struct. (for a complete table of which function support what you can look at the usage line)

And, if you ask for keys, values, key-values on non struct like, you'll get an error::

$ cat test.yaml | shyaml keys name
Error: keys does not support 'str' type. Please provide or select a struct.
$ cat test.yaml | shyaml values subvalue.how-many
Error: values does not support 'int' type. Please provide or select a struct.
$ cat test.yaml | shyaml key-values subvalue.how-much
Error: key-values does not support 'float' type. Please provide or select a struct.

Parse sequence

Query a sequence with get-value::

$ cat test.yaml | shyaml get-value subvalue.things
- first
- second
- third

And access individual elements with python-like indexing::

$ cat test.yaml | shyaml get-value subvalue.things.0
first
$ cat test.yaml | shyaml get-value subvalue.things.-1
third
$ cat test.yaml | shyaml get-value subvalue.things.5
Error: invalid path 'subvalue.things.5', index 5 is out of range (3 elements in sequence).

Note that this will work only with integer (preceded or not by a minus sign)::

$ cat test.yaml | shyaml get-value subvalue.things.foo
Error: invalid path 'subvalue.things.foo', non-integer index 'foo' provided on a sequence.

More usefull, parse a list in one go with get-values::

$ cat test.yaml | shyaml get-values subvalue.things
first
second
third

Note that the action is called get-values, and that output is separated by newline char(s) (which is os dependent), this can bring havoc if you are parsing values containing newlines itself. Hopefully, shyaml has a get-values-0 to terminate strings by \0 char, which allows complete support of any type of values, including YAML. get-values outputs key and values for struct types and only values for sequence types::

$ cat test.yaml | shyaml get-values-0 subvalue |
  while IFS='' read -r -d '' key &&
        IFS='' read -r -d '' value; do
      echo "'$key' -> '$value'"
  done
'how-much' -> '1.1'
'how-many' -> '2'
'things' -> '- first
- second
- third
'
'maintainer' -> 'Valentin Lab'
'description' -> 'Multiline description:
Line 1
Line 2
'

Please note that, if get-values{,-0} actually works on struct, it's maybe more explicit to use the equivalent key-values{,0}. It should be noted that key-values{,0} is not completly equivalent as it is meant to be used with struct only and will complain if not.

You should also notice that values that are displayed are YAML compatible. So if they are complex, you can re-use shyaml on them to parse their content.

Of course, get-values should only be called on sequence elements::

$ cat test.yaml | shyaml get-values name
Error: get-values does not support 'str' type. Please provide or select a sequence or struct.

Parse YAML document streams

YAML input can be a stream of documents, the action will then be applied to each document::

$ i=0; while true; do
      ((i++))
      echo "ingests:"
      echo " - data: xxx"
      echo "   id: tag-$i"
      if ((i >= 3)); then
          break
      fi
      echo "---"
done | shyaml get-value ingests.0.id | tr '\0' '&'
tag-1&tag-2&tag-3

Notice that NUL char is used by default for separating output iterations if not used in -y mode. You can use that to separate each output. -y mode will use conventional YAML way to separate documents (which is ---).

So::

$ i=0; while true; do
      ((i++))
      echo "ingests:"
      echo " - data: xxx"
      echo "   id: tag-$i"
      if ((i >= 3)); then
          break
      fi
      echo "---"
done | shyaml get-value -y ingests.0.id  ## docshtest: ignore-if LIBYAML
tag-1
...
---
tag-2
...
---
tag-3
...

Notice that it is not supported to use any query that can output more than one value (like all the query that can be suffixed with *-0) with a multi-document YAML::

$ i=0; while true; do
      ((i++))
      echo "ingests:"
      echo " - data: xxx"
      echo "   id: tag-$i"
      if ((i >= 3)); then
          break
      fi
      echo "---"
done | shyaml keys ingests.0 >/dev/null
Error: Source YAML is multi-document, which doesn't support any other action than get-type, get-length, get-value

You'll probably notice also, that output seems buffered. The previous content is displayed as a whole only at the end.