Mozilla code search website. (Please file bugs in bugzilla at https://mzl.la/2YtXmoN)
Mozilla code search website. (Please file bugs in bugzilla at https://mzl.la/2YtXmoN)
Mozsearch is the backend for the Searchfox code indexing tool. Searchfox runs inside AWS, but you can develop on Searchfox locally using Vagrant.
We've moved from using Vagrant to docker for development because it's comparably painless and because, especially on linux, it's great for the searchfox container to be able to use all your CPU cores and only use memory when it needs it (instead of all the time).
That said, if you want to use something else, that's fine. If you look at our docker scripts, they're just a bunch of shell scripts we run under Ubuntu and they're basically the same as they were under Vagrant. And we also just run those scripts on top of a basic Ubuntu AMI for provisioning our instances. If you can run Ubuntu under a VM, you can run Searchfox in there.
Could you run searchfox outside of a container/VM? Probably? But, dependencies are a hassle.
Can you already run "docker" on your command line and have it work? Then you can skip this section!
[!WARNING] If you're running docker on macOS, make sure the Rosetta option is disabled in the Docker Desktop settings, otherwise you may run into segfaults on installing rust components. See https://github.com/rust-lang/rustup/issues/3902 and friends.
Docker has changed their licensing of their "Docker Desktop" packages. We do not recommend using Docker Desktop; you don't need it as long as you already are able to run Ubuntu directly or under (para)virtualization like WSL2.
If you do want to use it and you work at Mozilla, you should acquire a license through Service Desk before doing anything else. If you already have a license, that's fine too.
Podman can be used almost as a drop-in replacement to Docker. Just make sure that:
Note for Nix users: the devShell in flake.nix provides both of those requirements.
I don't think anyone has tried this yet, but it seems like there are a variety of options. Here are some I've just briefly researched:
If you use a thing and it works, please let us know and we can update these docs. Or better yet, you can submit a pull request for this doc!
First, you need Ubuntu installed under WSL2. If you already have a sufficiently up-to-date version (I would suggest Ubuntu 22.04 or later), you can skip this next step.
From the windows store, install "Ubuntu 22.04.1 LTS" or whatever updated version there is. Then I launched the "Terminal" app (which you may need to also install from the Windows store). I used the drop-down arrow to the right of the add a tab "+" button and picked "Ubuntu 22.04.1 LTS". This launched an installer that ran for ~30 seconds and then prompted me to be okay to switch back to the terminal. Specific steps:
Now let's keep going through the next sections.
Run the following commands in your normal Linux terminal on Linux, and under WSL2 on Windows, use the Terminal app with your "Ubuntu 22.04.1 LTS" tab open.
sudo apt update to update the package registry, you may need to enter your
password because you’re using sudo.sudo apt install -y docker.iosudo adduser $USER docker to add yourself to the docker group.groups.su - $USER and this will dump you in a freshly logged in shell where
you should see “docker” when you run groups.sudo update-alternatives --config iptables and pick the
“iptables-legacy” option, probably by hitting 1 and hitting enter. This
switches iptables to legacy mode. This was necessary to make docker.io happy
under WSL2 for me.VS Code is a very nice editor and it hooks up quite nicely to rust-analyzer via the language server protocol (LSP). Rust-analyzer needs to run somewhere and VS code's remote support is quite excellent. I personally do the following on these platforms:
Installation steps:
sudo apt install -y clang pkg-config libssl-dev cmakegit clone https://github.com/mozsearch/mozsearch.git to check out searchfox
if you haven’t alreadygit submodule update --init --recursive to check out the relevant submodules./build-docker.sh to provision and setup the “VM” / container. This
will run our shell scripts to create multiple layers of filesystem. Note that
the "livegrep" build phase may seem to hang for several minutes at one point
when it's looking for boost dependencies and this is unfortunately expected.
Don't ctrl-c out (although it's harmless if you do, you can re-run the command
later)../run-docker.sh to start up the VM/container. It will shutdown when you
exit from the shell so leave the shell running to keep the web-server
accessible../run-docker.sh from
other terminal tabs and they will connect to that same running container.
Note that if you log out of the first invocation that actually started the
container, it will close all the other terminals.Those scripts have default container, image, and volume names ("searchfox", "searchfox", "searchfox-vol") that can be overridden. You would usually only do this if you want multiple orthogonal searchfox checkouts on the same machine. (But do note that currently the script does not make it possible to override the decision to use port 16995, so you can't run both containers at the same time.)
Once you are able to successfully ./run-docker.sh, once you are inside at that
shell, you can do the following:
cd /vagrant
make build-test-repoThe above process will:
After that, you can connect to http://localhost:16995/ and see Searchfox at work!
Once you've done that, you might want to read the Manual Indexing doc for more details on what's happening under the hood when you run the above make rule.
You can also test the UI part of searchfox by the following inside the docker:
make webtestIf you want to run specific tests, you can run the following, where the
FILTER is a substring-match against the test file's path, which
./scripts/webtest.sh FILTERFor more details, please refer the webtest document.
The tests configuration defined at tests/config.json is very helpful, but it
isn't configured to use a blame repository or generate blame UI output. If
you're making changes that affect the blame UI or might interact with it, it
helps to test with it!
The searchfox configuration defined at tests/searchfox-config.json exists
for this purpose. It indexes the entirety of the repository. It can be built
via the Makefile by invoking the following to build the index at
~/searchfox-index (whereas tests is built at ~/index).
make build-searchfox-repoNote that you will need to do a couple things for this to work right:
output-file.rs depends on the blame repository having
lines that match up exactly with the state of the source files checked out
from git or it can panic because of accessing beyond the end of vectors. (The
blame data will also be wrong.)indexer-run.sh
manually without first running indexer-setup.sh, you may experience
problems.Also note that this will terminate any previously running tests web servers
even though the indexes live at different directories (~/index versus
~/searchfox-index). If you find that you want both the tests and searchfox
configurations to be served at the same time, you can use the webtest
configuration.
make build-webtest-repoIf you are making more extensive changes to searchfox, it's usually advisable to test them against firefox-main before landing them. While it's possible to do this locally, the normal way to do this is:
trigger_indexer.py.Details below.
For testing changes to the clang-plugin, run these steps, followed by the steps in the next section.
./mach try fuzzy --full -q "'searchfox" -q "'bugzilla-component"No open issues yet, or sync has not completed.