Open NSynth Super is an experimental physical interface for the NSynth algorithm
Open NSynth Super is an experimental physical interface for the NSynth algorithm
|
This project is no longer actively maintained by the Google Creative Lab but remains here in a read-only Archive mode so that it can continue to assist developers that may find the examples helpful. We aren’t able to address all pull requests or bug reports but outstanding issues will remain in read-only mode for reference purposes. Also, please note that some of the dependencies may not be up to date and there hasn’t been any QA done in a while so your mileage may vary.
For more details on how Archiving affects Github repositories see this documentation . We welcome users to fork this repository should there be more useful, community-driven efforts that can help continue what this project began. |
Open NSynth Super is an experimental physical interface for NSynth, a machine learning algorithm developed by Google Brain’s Magenta team to generate new, unique sounds that exist between different sounds. Open NSynth Super allows you to create and explore new sounds that it generates through a simple to use hardware interface that integrates easily into any musician’s production rig. To find out more, visit the NSynth Super website.
This repository contains all the instructions and files required to build an Open NSynth Super instrument from scratch, and you can use them to make your own version. Anyone can fork the code and schematics to remix the instrument in any way they wish.
This document has three chapters – an overview of the instrument itself, a 'make your own' guide, and an overview of the audio creation process.
The 'how it works' overview summarises the technical and user interface features of the Open NSynth Super instrument.
The 'make your own' guide takes you through the steps required to build an Open NSynth Super unit and load it with an example Linux image containing pre-computed sample audio, helping you to get started with the instrument straight away.
The audio creation overview gives a high-level summary of the audio creation pipeline, which enables you to process audio files (on a separate computer) and load the instrument with your own input sounds.
This repository also includes individual chapters on every aspect of Open NSynth Super, so you can build a unit from scratch, or hack and customise any part of it:
Open NSynth Super is super simple to integrate into any production music rig. Like many other synthesizer modules, it receives MIDI input via a 5-pin DIN connector, and outputs audio through a 3.5mm jack cable. This simple interface allows it to work with almost any MIDI source, like keyboards, DAWs, or hardware sequencers.
The physical interface of Open NSynth Super is constructed around a square touch interface. Using dials in the corners of the touch surface, musicians can select four source sounds and use the touch interface to explore the sounds that the NSynth algorithm has generated between them. In addition, the instrument also has controls for tuning the new sounds via the fine controls.
(A) Instrument selectors & patch storage - These rotary dials are used to select the instruments that are assigned to the corners of the interface. In version 1.2.0, these selectors can be pushed down to store or clicked to recall settings patches.
(B) OLED display - A high-contrast display shows you the state of the instrument and additional information about the controls that you are interacting with.
(C) Fine controls - These six dials are used to further customize the audio output by the device:
(D) Touch interface - This is a capacitive sensor, like the mouse pad on a laptop, which is used to explore the world of new sounds that NSynth has generated between your chosen source audio.
There are several distinct components to each Open NSynth Super unit: a custom PCB with a dedicated microcontroller and firmware for handling user inputs; a Raspberry Pi 3 computer running Raspbian Linux and an openFrameworks-based audio synthesiser application; and a series of scripts for preparing audio on a GPU-equipped Linux server using the NSynth algorithm. In addition to the software, firmware, and hardware, there are files for creating a casing and dials for the unit.
The electronics are built around a Raspberry Pi 3 running Raspbian Linux, and a custom PCB used to read the inputs and control the outputs. A microcontroller on the PCB manages the physical inputs: there are four rotary encoders on the four corners for instrument selection; six potentiometers below the interface to control the position, envelope, and volume settings; and a capacitive grid on the surface of the PCB (exposed through the top layer of the case) used to select the mixing point of the four instruments.
More information on the electronics, hardware, and firmware can be found here, and here. A complete bill of materials for the case and electronics is available in this Excel spreadsheet.
Open NSynth Super runs a multisampler audio application built on openFrameworks to process incoming MIDI note data and generate sound output. More information about this application can be found in the software readme file.
We designed a case for the electronics that can be easily manufactured with a laser cutter, and held together with standard screws and fittings. This design is easily customisable using different materials, colors, dials, and shapes. Read more about this in the case readme file.
Audio for the Open NSynth Super instrument is produced by the NSynth algorithm. Because generating audio requires a great deal of processing power, this repository includes a set of scripts that you can run on a server which will take any audio recordings of your choice and convert them into a format compatible with the instrument. For more information, see the audio readme file or the audio creation overview in this document.
This guide gives you an overview of how to create casing and electronics for Open NSynth Super, and load the device with a premade Linux image which is already set up with application software, hardware support, and example sounds.
The example image file includes the following software features:
You will need the following off-the-shelf items to complete this guide:
You will also need the following Open NSynth Super-specific items, which are detailed in the steps below:
This repository contains a reference design for a laser-cut shell for Open NSynth Super. As well as the case design, there are two sizes of 3D-printed dials which can optionally be added to the build. Parts for the case can be cut manually, or obtained easily from a laser-cutting service by supplying these files. More detailed information and specifications for the case can be found in the case readme file.
You should make or order a case following the above guide before moving forward with the build. Don't assemble the case yet, as you will need access to the ports on the Raspberry Pi.
Open NSynth Super is built around a custom PCB with inputs for the hardware controls, audio and data I/O ports, and an OLED display for the UI. The PCB can be ordered fully assembled (this is most cost effective when ordering several boards), or be assembled by hand in a few hours. Further detail on the manufacture and assembly of the board can be found in the PCB readme file.
You will need to obtain or build your PCB before proceeding further with this guide.
In order to test the board, you will need to have a working Open NSynth Super software installation, which you will prepare at the next step. Before proceeding, connect the Raspberry Pi GPIO pins to the Open NSynth Super mainboard.
There are two preconfigured OS images available, both loaded with sample audio and fully configured for use with NSynth Super. Depending on the size of your SD card, you can choose either the 64GB image or 16GB image.
The easiest way to create the SD card is to use GUI software like Etcher, Win32DiskImager, or the SD Card Association's formatter to burn the example image to a card.
The supplied images are compressed in bz2 format. These can be decompressed on the fly when writing to an SD card using the following command:
$ bzcat <IMAGE NAME> | sudo dd of=<DISK NAME>Connect a keyboard and screen to the Raspberry Pi, insert the SD card, and plug a USB power cable into the socket on the Open NSynth Super mainboard. You should see the Pi booting up on the connected display; when you receive a prompt, you can login with the default username and password: pi / raspberry. Note that the device might run a file system check when you first boot up from the new card – this will take about 5 minutes depending on the size of the disk.
The firmware will need to be loaded to the microcontroller on your Open NSynth Super mainboard. This allows the software application running on the Raspberry Pi to interact with the controls.
Because the file system is locked for robustness, you will need to remount the disk as read/write by running the following command on your Raspberry Pi:
$ sudo mount -o remount,rw /Next, navigate to the firmware directory and run the install command:
$ cd /home/pi/open-nsynth-super-master/firmware/src
$ make installThis command will build and install the firmware to the microcontroller. You will receive a message on screen confirming the operation has been successful:
** Programming Started **
auto erase enabled
Info : device id = 0x10006444
Info : flash size = 32kbytes
wrote 12288 bytes from file ../bin/main.elf in 0.607889s (19.740 KiB/s)
** Programming Finished **
** Verify Started **
verified 11788No open issues yet, or sync has not completed.