elixir

= The Elixir Cross Referencer :doctype: book :pp: {plus}{plus} :toc: :toc-placement!:

Elixir is a source code cross-referencer inspired by https://en.wikipedia.org/wiki/LXR_Cross_Referencer[LXR]. It's written in Python and its main purpose is to index every release of a C or C{pp} project (like the Linux kernel) while keeping a minimal footprint.

It uses Git as a source-code file store and Berkeley DB for cross-reference data. Internally, it indexes Git blobs rather than trees of files to avoid duplicating work and data. It has a straightforward data structure (reminiscent of older LXR releases) to keep queries simple and fast.

You can see it in action on https://elixir.bootlin.com/

link:CHANGELOG.adoc[Changelog]

toc::[]

= Requirements

• Python >= 3.8
• Git >= 1.9
• The Jinja2 and Pygments (>= 2.7) Python libraries
• Berkeley DB (and its Python binding)
• Universal Ctags
• Perl (for non-greedy regexes and automated testing)
• Falcon and mod_wsgi (for the REST API)

= Architecture

The shell script (script.sh) is the lower layer and provides commands to interact with Git and other Unix utilities. The Python commands use the shell script's services to provide access to the annotated source code and identifier lists (query.py) or to create and update the databases (update.py). Finally, the web interface (web.py) and uses the query interface to generate HTML pages and to answer REST queries, respectively.

When installing the system, you should test each layer manually and make sure it works correctly before moving on to the next one.

= Manual Installation

== Install Dependencies

For Debian

sudo apt install python3-pip python3-venv libdb-dev python3-dev build-essential universal-ctags perl git apache2 libapache2-mod-wsgi-py3 libjansson4

== Download Elixir Project

git clone https://github.com/bootlin/elixir.git /usr/local/elixir/

== Create a virtualenv for Elixir

python -m venv /usr/local/elixir/venv . /usr/local/elixir/venv/bin/activate pip install -r /usr/local/elixir/requirements.txt

== Create directories for project data

mkdir -p /path/elixir-data/linux/repo mkdir -p /path/elixir-data/linux/data

== Set environment variables

Two environment variables are used to tell Elixir where to find the project's local git repository and its databases:

• LXR_REPO_DIR (the git repository directory for your project)
• LXR_DATA_DIR (the database directory for your project)

Now open /etc/profile and append the following content.

export LXR_REPO_DIR=/path/elixir-data/linux/repo export LXR_DATA_DIR=/path/elixir-data/linux/data

And then run source /etc/profile.

== Clone Kernel source code

First clone the master tree released by Linus Torvalds:

cd /path/elixir-data/linux git clone --bare https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git repo

Then, you should also declare a stable remote branch corresponding to the stable tree, to get all release updates:

cd repo git remote add stable git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git git fetch stable

Then, you can also declare an history remote branch corresponding to the old Linux versions not present in the other repos, to get all the old version still available:

cd repo git remote add history https://github.com/bootlin/linux-history.git git fetch history --tags

Feel free to add more remote branches in this way, as Elixir will consider tags from all remote branches.

== First Test

cd /usr/local/elixir/ ./script.sh list-tags

== Create Database

. ./venv/bin/activate ./update.py

Generating the full database can take a long time: it takes about 15 hours on a Xeon E3-1245 v5 to index 1800 tags in the Linux kernel. For that reason, you may want to tweak the script (for example, by limiting the number of tags with a "head") in order to test the update and query commands. You can even create a new Git repository and just create one tag instead of using the official kernel repository which is very large.

== Second Test

Verify that the queries work:

$ python3 -m utils.query ident v4.10 raw_spin_unlock_irq C $ python3 -m utils.query file v4.10 /kernel/sched/clock.c

NOTE: v4.10 can be replaced with any other tag. NOTE: Don't forget to activate the virtual environment!

== Configure httpd

The CGI interface (web.py) is meant to be called from your web server. Since it includes support for indexing multiple projects, it expects a different variable (LXR_PROJ_DIR) which points to a directory with a specific structure:

• ** *** data *** repo ** *** `data` *** `repo` ** *** data *** repo

It will then generate the other two variables upon calling the query command.

Now replace /etc/apache2/sites-enabled/000-default.conf with docker/000-default.conf. Note: If using httpd (RedHat/Centos) instead of apache2 (Ubuntu/Debian), the default config file to edit is /etc/httpd/conf.d/elixir.conf.

Finally, start the httpd server.

systemctl restart apache2

== Configure SELinux policy

When running systemd with SELinux enabled, httpd server can only visit limited directories. If your /path/elixir-data/ is not one of these allowed directories, you will be responded with 500 status code.

To allow httpd server to visit /path/elixir-data/, run following codes:

chcon -R -t httpd_sys_rw_content_t /path/elixir-data/

To check if it takes effect, run the following codes:

ls -Z /path/elixir-data/

In case you want to check SELinux log related with httpd, run the following codes:

audit2why -a | grep httpd | less

== Configure systemd log directory

By default, the error log of elixir will be put in /tmp/elixir-errors. However, systemd enables PrivateTmp by default. And, the final error directory will be like /tmp/systemd-private-xxxxx-httpd.service-xxxx/tmp/elixir-errors. If you want to disable it, configure httpd.service with the following attribute:

PrivateTmp=false

== Configuration for other servers

Other HTTP servers (like nginx or lighthttpd) may not support WSGI and may require a separate WSGI server, like uWSGI.

Information about how to configure uWSGI with Lighthttpd can be found here: https://redmine.lighttpd.net/projects/lighttpd/wiki/HowToPythonWSGI#Python-WSGI-apps-via-uwsgi-SCGI-FastCGI-or-HTTP-using-the-uWSGI-server

Pull requests with example uWSGI configuration for Elixir are welcome.

= REST API usage

After configuring httpd, you can test the API usage:

== ident query

Send a get request to /api/ident//?version=&family=. For example:

curl http://127.0.0.1/api/ident/barebox/cdev?version=latest&family=C

The response body is of the following structure:

{ "definitions": [{"path": "commands/loadb.c", "line": 71, "type": "variable"}, ...], "references": [{"path": "arch/arm/boards/cm-fx6/board.c", "line": "64,64,71,72,75", "type": null}, ...]

}

= Maintenance and enhancements

== Using a cache to improve performance

At Bootlin, we're using the https://varnish-cache.org/[Varnish http cache] as a front-end to reduce the load on the server running the Elixir code.

.-------------. .---------------. .-----------------------. | Http client | --------> | Varnish cache | --------> | Apache running Elixir | '-------------' '---------------' '-----------------------'

== Keeping Elixir databases up to date

To keep your Elixir databases up to date and index new versions that are released, we're proposing to use a script like index /srv/elixir-data --all which is called through a daily cron job.

You can set $ELIXIR_THREADS if you want to change the number of threads used by update.py for indexing (by default the number of CPUs on your system).

= Building Docker images

Dockerfiles are provided in the docker/ directory. To build the image, run the following commands:

git clone https://github.com/bootlin/elixir.git ./elixir

docker build -t elixir --build-arg ELIXIR_VERSION=git rev-parse --short HEAD -f ./elixir/docker/Dockerfile ./elixir

ELIXIR_VER build argument is optional. Since .git directory is not copied into Docker image by default, the option is used to pass a version string to Elixir.

You can then run the image using docker run. Here we mount a host directory as Elixir data:

mkdir ./elixir-data

docker run -v ./elixir-data/:/srv/elixir-data -d --name elixir-container elixir

The Docker image does not contain any repositories. To index a repository, you can use the index-repository script. For example, to add the https://musl.libc.org/[musl] repository, run:

docker exec -it elixir-container index /srv/elixir-data musl

Or, to run indexing in a separate container:

docker run -v ./elixir-data/:/srv/elixir-data \

--entrypoint index elixir /srv/elixir-data musl

You can also use index /srv/elixir-data --all to start indexing all officially supported repositories.

After indexing is done, Elixir should be available under the following URL on your host: http://172.17.0.2/musl/latest/source

If 172.17.0.2 does not answer, you can check the IP address of the container by running:

docker inspect elixir-container | grep IPAddress

== Automatic repository updates

The Docker image does not automatically update repositories by itself. You can, for example, start index /srv/elixir-data --all in the container (or in a separate container, with Elixir data volume/directory mounted) from cron on the host to periodically update repositories.

== Using Docker image as a development server

You can easily use the Docker image as a development server by following the steps above, but mounting Elixir source directory from the host into /usr/local/elixir/ in the container when running docker run elixir.

Changes in the code made on the host should be automatically reflected in the container. You can use apache2ctl to restart Apache. Error logs are available in /var/log/apache2/error.log within the container.

= Hardware requirements

Performance requirements depend mostly on the amount of traffic that you get on your Elixir service. However, a fast server also helps for the initial indexing of the projects.

SSD storage is strongly recommended because of the frequent access to git repositories.

At Bootlin, here are a few details about the server we're using:

• As of July 2019, our Elixir service consumes 17 GB of data (supporting all projects), or for the Linux kernel alone (version 5.2 being the latest), 12 GB for indexing data, and 2 GB for the git repository.
• We're using an LXD instance with 8 GB of RAM on a cloud server with 8 CPU cores running at 3.1 GHz.

= Contributing to Elixir

== Supporting a new project

Elixir has a very simple modular architecture that allows to support new source code projects by just adding a new file to the Elixir sources.

Elixir's assumptions:

• Project sources have to be available in a git repository
• All project releases are associated to a given git tag. Elixir only considers such tags.

First make an installation of Elixir by following the above instructions. See the projects subdirectory for projects that are already supported.

Once Elixir works for at least one project, it's time to clone the git repository for the project you want to support:

cd /srv/git git clone --bare https://github.com/zephyrproject-rtos/zephyr

After doing this, you may also reference and fetch remote branches for this project, for example corresponding to the stable tree for the Linux kernel (see the instructions for Linux earlier in this document).

Now, in your LXR_PROJ_DIR directory, create a new directory for the new project:

cd $LXR_PROJ_DIR mkdir -p zephyr/data ln -s /srv/git/zephyr.git repo export LXR_DATA_DIR=$LXR_PROJ_DIR/data export LXR_REPO_DIR=$LXR_PROJ_DIR/repo

Now, go back to the El