Copilot Arena | Discord | X | Copilot Arena is an open source AI coding assistant that provides paired autocomplete completions from differ…
Copilot Arena | Discord | X | Copilot Arena is an open source AI coding assistant that provides paired autocomplete completions from differ…
Copilot Arena is an open source AI coding assistant that provides paired autocomplete completions from different LLMs, which include state-of-the-art models like GPT-4o, Codestral, Llama-3.1 and more.
Step 1: Download the Copilot Arena extension from the Visual Studio Code Store.
Step 2: Copilot Arena currently supports two main feature: read autocomplete and in-line editing below to understand how to use each one. Since we show paired responses, the way you use them are slightly different than your standard AI coding tools!
Step 3: This step is optional. If applicable, you can change what data is saved by Copilot Arena by following the instructions in "Privacy Settings".
Step 4: Create a username by clicking the Copilot Arena icon on the sidebar; detailed instructions are also in "Create an account". Your username will be used for a future leaderboard to compare individual preferences.
You are now ready to get coding with Copilot Arena!
New! We have recently open-sourced our backend code. Check out this README if you are interested in contributing prompts and new models. We welcome folks to get involved to improve Copilot Arena!
In-line edits allow you to modify larger portions of your code with a simple prompt.
Here's how to generate in-line edits:
⌘+i (Windows: Ctrl+i). Enter in your prompt (e.g., what would you like changed about the highlighted code). When edits are being generated, the check mark changes to a spinning circle.⌘+1, Windows Ctrl+1⌘+2, Windows Ctrl+2⌘+3, Windows Ctrl+3. For old users, this was originally ⌘+n.Any changes will be automatically applied and the files showing diffs will be closed. You can always undo the change after if you would like. We welcome feedback on this feature! Please get in touch via Discord or raise an issue on the repo.
Understanding completions: When a completion is being generated, the check mark changes to a spinning circle. Once a pair of completions appear, you will notice that Copilot Arena adopts a slightly different user interface compared to a typical code completion.
Accepting completions: Press Tab to accept the top completion and Shift-Tab to accept the bottom completion. You can also choose to accept neither completion and continue typing.
We've observed that individuals have different model preferences! We currently support personal leaderboards for autocomplete. In-line editing leaderboard is coming soon!
Copilot Arena icon on the sidebar.Soon we will be releasing public leaderboards to compare individual preferences. To enable you to find your position on these leaderboards and help us collect helpful background information, creating an account is as simple as:
Copilot Arena icon on the sidebar and click Register.Your privacy is important to us. Please read carefully to determine which settings are most appropriate for you.
To generate completions, the code in your current file is sent to our servers and sent to various API providers. This cannot be changed.
By default, we collect your code for research purposes. You can opt-out of this. If you are working on code containing sensitive information, we recommend that you opt out of data collection.
arena.codePrivacySettings to Debug. We will only log your code for debugging.arena.codePrivacySettings to Private. Opting-out means any bugs you encounter will be non-reproducable on our end.You can find these settings by searching for "arena" in your vscode settings or clicking the gear button of the Copilot Arena extension -> Extension Settings.
If you would like to have the option in the future for us to delete any of your data, you must create an account on Copilot Arena following instructions described in "Create an account". To remove your data, you can email any of the Copilot Arena maintainers with your username.
Prior to releasing any collected code snippets to enable future research efforts, we will run a PII detector and remove any identified entities to further ensure no personal information is released.
If you have feedback or suggestions, please submit an issue or join the conversation on Discord! Check out our FAQs.
If you find this repository useful, pleaes cite copilot arena:
@misc{chi2025copilotarenaplatformcode,
title={Copilot Arena: A Platform for Code LLM Evaluation in the Wild},
author={Wayne Chi and Valerie Chen and Anastasios Nikolas Angelopoulos and Wei-Lin Chiang and Aditya Mittal and Naman Jain and Tianjun Zhang and Ion Stoica and Chris Donahue and Ameet Talwalkar},
year={2025},
eprint={2502.09328},
archivePrefix={arXiv},
primaryClass={cs.SE},
url={https://arxiv.org/abs/2502.09328},
}No open issues yet, or sync has not completed.