ngs

Next Generation Shell is a modern programming language for DevOps.

• NGS Website
• NGS Discord chat
• The Problem
• Suggested Solution - NGS
• NGS Use Cases
• Project Status
• Installing
  • Using Script
  • Using MacPorts
  • Using Homebrew
  • Using Snap
  • Using Docker
  • Using Github Action
  • Using iPython or Jupyter Notebook
  • Using AWS Lambda
• Documentation
  • Code Examples
    • Arrays
    • Hashes
    • Functions (multimethods) and multi-dispatch
    • Basic Cloud
    • Sample Scripts
  • Index
  • Tutorial -- brief introduction through code examples
  • Language Reference -- language design principles, syntax, and features
  • Motivation
  • Library reference -- all available functions
• Manually Compiling and Running
  • Clone from Git
  • Install with Dependencies - Debian-based Linux and MacOS
  • Install without Dependencies
  • Run Tests
  • Compile
  • Run
  • Uninstall
  • Debug - Mac
  • Debug - Homebrew builds
  • Build and run docker
  • Generate Documentation
• Contributing
• Planned Features
  • Cross-system
  • Development
  • Later / Unformed / Unfinished Thoughts
• How to Run the POC
• Have you heard of project X? How it compares to NGS?
• Discussion / requests / comments

The Problem

bash or Python? That's a square pegs for a round hole situation. Both are inadequate for Ops tasks. NGS aims to reduce your frustration and let you be more productive. There is no reason that you should be in this situation. (Python here represents also other general purpose programming languages).

bash does not meet any modern expectations for syntax, error handling nor has ability to work with structured data (beyond arrays and associative arrays which can not be nested). Let it go. You are not usually coding in assembly, FORTRAN, C, or C++, do you? They just don’t match the typical Ops tasks. Don’t make your life harder than it should be. Let it go. (Let’s not make it a blanket statement. Use your own judgement when to make an exception).

Python along with many other languages are general purpose programming languages which were not intended to solve specifically Ops problems. The consequence is longer and less readable scripts when dealing with files or running external programs, which are both pretty common for Ops.

See bash or Python? The Square Pegs and a Round Hole Situation blog post.

Shell user interface, the atrocity. Current shells as well as proposed alternatives treat UI as if nothing happened since the 70-s: mostly typing commands and getting some text back. Watch Unix shell - We can do better now 15 minutes video about what's wrong with the user interface and how NGS is fixing (WIP) that.

Suggested Solution - NGS

I like helping people. Reducing frustration and making people more productive just feels good. A scalable way to help is thorugh tools. That is why I created NGS. It is a programming language with typical Ops tasks in mind. See the use cases below. The next big planned part is the interactive shell.

NGS Use Cases

You as the Ops person, probably wondering whether NGS would be a good fit for your tasks. NGS aims to be best fit for the use cases below.

• Data manipulation: Got some JSON from an API call / external program / stdin and want to do something with it? Extract, modify, shove it into next API call / external program / stdout.
• Typical Ops Scripting
  • Run external programs
    • NGS has syntax for pipes and i/o redirection
    • Use built-in Argv facility for convenient building of command line arguments
    • Let NGS built-in mechanism to throw exceptions when exit codes indicate error
    • Let NGS built-in mechanism to parse commands' output
    • Easily pipe data in and out of external program
  • Easily output JSON or a human-readable nicely formatted table
  • Use built-in debug() function which prints debug information, conditionally, depending on environment variable per-section setting
  • Use built-in status() reporting function (previous status is overridden with new status on screen)
  • Use built-in log() function which prints timestamped output
  • Use built-in retry() and stop re-implementing it hundreds of times
• Testing: test CLI programs and API endpoints using built-in testing mini-framework
• WIP/PoC: AWS library
  • Easy integration with existing infra
  • Create new or modify existing infrastructure
  • Easier scripting as opposed to declarative tools (CloudFormation, Terraform) allows modeling of processes and not just final state
• Future: interactive shell with UI which is not from the 70s.

See Use Cases wiki page for more information about the use cases and examples.

Project Status

The language is very useful. See the the bin folder for examples. NGS is used in Beame.io for miscellaneous scripting such as testing CLI tools, performance tests orchestration, cloud manipulation, etc.

The work on the shell has just started. It is not usable yet. The shell is being implemented in NGS. See the design document.

From George Nachman, creator of iTerm2:

Neat! This is totally the project I would do if I had unlimited free time :) I've wished for something like this for a long time. I think there's a lot of room for improvement over a basic TTY, and you've nailed some of the important things

Installing

Using Script

On Linux: make sure curl and sudo are installed

On MacOS: make sure you have brew installed.

curl https://ngs-lang.org/install.sh | bash
ngs -pi 'sum(0..10)'

Using MacPorts

sudo port install ngs

Using Homebrew

brew install ngs

Using Snap

sudo snap install ngs

Using Docker

Docker repository is at https://hub.docker.com/r/ngslang/ngs/

docker run -it --rm ngslang/ngs

Using GitHub Action

Add to your Github Action the following (make sure to release the version as required)

steps:
	- uses: ngs-lang/[email protected]

After that, ngs can be used by simply calling ngs in any run step

steps:
	- uses: ngs-lang/[email protected]
	- run: ngs -pi 'sum(10..100)'

Running NGS as AWS Lambda Custom Runtime

To run NGS as AWS Lambda Function see https://github.com/ngs-lang/ngs-aws-lambda

Using iPython or Jupyter Notebook

Please refer to extension located in https://github.com/ngs-lang/ngs-ipython-extension

Using AWS Lambda

Please refer to extension located in https://github.com/ngs-lang/ngs-aws-lambda

Manually Compiling and Running

Clone from Git

git clone https://github.com/ngs-lang/ngs.git

# or if using git over ssh
git clone [email protected]:ngs-lang/ngs.git

cd ngs

Install with Dependencies - Debian-based Linux and MacOS

./install.sh

Install without Dependencies

sudo make install

Run Tests

make tests

Compile

make build

Build is currently tested with the GitHub action in .github/workflows/build.yml and we currently test the following 64-bit architectures:

Notes:

• 32-bit architectures are currently not supported.
• Debian 11 with gcc 10 is also tested with the following architectures: aarch64, s390x, ppc64le

Run

# If NGS is not installed (from the root of ngs project):
NGS_PATH=lib ./build/ngs SCRIPT_NAME.ngs

# If NGS is installed:
ngs SCRIPT_NAME.ngs

Tested as follows (some time ago):

• Pandoc 2.2
• Debian Stretch: gcc 4.8.5 + 4.9.3 + 5, clang 3.6
• Debian Jessie: gcc 4.8.4 + 4.9.2, clang 3.5
• Ubunty Trusty: gcc 4.8.4, clang 3.4

If you have troubles compiling, please try to compile the commit tagged tested.

Uninstall

cd build
for i in $(<install_manifest.txt);do rm "$i";done

Debug - Mac

# Debug when stuck (note to self mostly)
killall -SIGSEGV ngs
lldb --core /cores/core.XXXXX

Debug - Homebrew builds

• Checkout the relevant Homebrew repository (probably a fork, example: https://github.com/alebcay/homebrew-core.git).
• Copy the original formula from the original repository (at Formula/ngs.rb) somewhere else.
• In the ngs.rb copy, modify:
  • url "https://github.com/ngs-lang/ngs/archive/v0.2.8.tar.gz" -> url "file:///PATH/TO/ngs.tgz"
  • Add version "0.0.1-dev" (or alike)
• Development cycle:
  • In a folder just above ngs, run tar czf ngs.tgz --exclude=build ngs
  • brew install -d -s ./ngs.rb (wherever the ngs.rb is)

Build and run docker

# Build the docker
docker build -t ngs .
# Run the container
docker run -it --rm ngs
# Use NGS inside the container
ngs -pi 'sum(0..10)'

Generate Documentation

On Linux

# --- Linux ---
cd doc
rm -r out
mkdir out
./make.ngs out

On Mac, follow the instructions to create case sensitive volume.

# --- Mac --- One time setup ---
cd doc/
ln -s /Volumes/YOUR-VOLUME-NAME out

# --- Mac --- Each time ---
cd doc/
rm -r out/*;
./make.ngs out

Code Examples

Arrays

a = [1, 2, 3]
arr = a.map(X*2) # arr is now [2, 4, 6]
for i in arr {
	echo(i)
}

Hashes

h = {"a": 1, "b1": 2, "b2": 3}
echo(h.filterk(/^b/).mapv(X+10))  # {b1=12, b2=13}

Functions (multimethods) and multi-dispatch

F my_func(x:Int) x*10 # Single expression does not require { ... } syntax

doc This method is documented!
F my_func(s:Str) {
	t = s * 2
	"[" + t + "]" # Last value returned as the result
}

echo(my_func(1))      # 10
echo(my_func("xyz"))  # [xyzxyz]
echo(my_func)         # <MultiMethod with 2 method(s)>

More information about the language and syntax in particular is in ngslang.1

Basic Cloud

NGS has AWS library based on concept of Declarative Primitives

TLDR:

• top-to-bottom execution
• resource-level idempotence
• just a more convenient scripting
  • no dependency graph
  • no state file

This is how an instance can be created using NGS (real working code).

{
	NGS_