refinery

Does one of these scenarios sounds familiar to you? - You are working on your own side-project in NLP, but you don't have enough labeled data to train a good model. - You are working in a team and already have some labeled data, but your training data is just stored in a spreadsheet or some TXT-file, and you have no idea how _good_ it actually is. - You are working in a team about to start a new project with limited resources (annotators, budget, time), and now want to understand how you can best make use of them If so, you are one of the people we've built refinery for. refinery helps you to build better NLP models in a data-centric approach. Semi-automate your labeling, find low-quality subsets in your training data, and monitor your data in one place. _refinery_ doesn't get rid of manual labeling, but it makes sure that your valuable time is spent well. Also, the makers of _refinery_ currently work on integrations to other labeling tools, such that you can easily switch between different choices. > **_DEMO:_** You can interact with the application in a (mostly read-only) online playground. Check it out [here](https://demo.kern.ai) _refinery_ is a multi-repository project, you can find all integrated services in the architecture below. The app builds on top of [ Hugging Face](https://www.huggingface.co) and [spaCy](https://spacy.io/) to leverage pre-built language models for your NLP tasks, as well as [qdrant](https://github.com/qdrant/qdrant) for neural search. ## Table of contents - [‍ Why _refinery_?](#-why-refinery) - [Enabling ideas of one-person-armies](#enabling-ideas-of-one-person-armies) - [Extending your existing labeling approach](#extending-your-existing-labeling-approach) - [Put structure into unstructured data](#put-structure-into-unstructured-data) - [Pushing collaboration](#pushing-collaboration) - [Open-source, and treating training data as a software artifact](#open-source-and-treating-training-data-as-a-software-artifact) - [Integrations](#integrations) - [Your benefits](#your-benefits) - [How does Kern AI make money, if refinery is open-source?](#how-does-kern-ai-make-money-if-refinery-is-open-source) - [ Features](#-features) - [(Semi-)automated labeling workflow for NLP tasks](#semi-automated-labeling-workflow-for-nlp-tasks) - [Extensive data management and monitoring](#extensive-data-management-and-monitoring) - [Team workspaces in the managed version](#team-workspaces-in-the-managed-version) - [☕ Installation](#-installation) - [From pip](#from-pip) - [From repository](#from-repository) - [Persisting data](#persisting-data) - [ Documentation and tutorials](#-documentation-and-tutorials) - [‍ Need help?](#-need-help) - [ Community and contact](#-community-and-contact) - [ Contributing](#-contributing) - [❓ FAQ](#-faq) - [Concept questions](#concept-questions) - [Technical questions](#technical-questions) - [Service and hosting questions](#service-and-hosting-questions) - [ Python SDK](#-python-sdk) - [ Architecture](#-architecture) - [ Glossary](#-glossary) - [‍‍ Team and contributors](#-team-and-contributors) - [ Star History](#-star-history) - [ License](#-license) ## ‍ Why _refinery_? There are already many other tools available to build training data. Why did we decide to build _yet another one_? ### Enabling ideas of one-person-armies We believe that developers can have crazy ideas, and we want to lower the barrier for them to go for that idea. _refinery_ is designed to build labeled training data much faster, so that it takes you very little time to prototype an idea. We've received much love for exactly that, so make sure to give it a try for your next project. ### Extending your existing labeling approach _refinery_ is more than a labeling tool. It has a built-in labeling editor, but its main advantages come with automation and data management. You can integrate any kind of heuristic to label what is possible automatically, and then focus on headache-causing subsets afterwards. Whether you do the labeling in _refinery_ or any other tool (even crowd labeled) doesn't matter! ### Put structure into unstructured data _refinery_ is the tool that brings new perspectives into your data. You're working on multilingual, human-written texts? Via our integration to [bricks](https://github.com/code-kern-ai/bricks), you can easily enrich your texts with metadata such as the detected language, sentence complexity and many more. You can use this both to analyze your data, but also to orchestrate your labeling workflow. ### Pushing collaboration While doing so, we aim to improve the collaboration between engineers and subject matter experts (SMEs). In the past, we've seen how our application was being used in meetings to discuss label patterns in form of labeling functions and distant supervisors. We believe that data-centric AI is the best way to leverage collaboration. ### Open-source, and treating training data as a software artifact We hate the idea that there are still use cases in which the training data is just a plain CSV-file. That is okay if you _really_ just quickly want to prototype something at hand with a few records, but any serious software should be maintainable. We believe an open-source solution for training data management is what's needed here. _refinery_ is the tool helping you to document your data. That's how you treat training data as a software artifact. ### Integrations Lastly, _refinery_ supports [SDK actions](https://github.com/code-kern-ai/refinery-python) like pulling and pushing data. Data-centric AI redefines labeling to be more than a one-time job by giving it an iterative workflow, so we aim to give you more power every day by providing end-to-end capabilities, growing the large-scale availability of high-quality training data. Use our SDK to program integrations with your existing landscapes. ## Your benefits You can automate tons of repetitive tasks, gain better insights into the data labeling workflow, receive an implicit documentation for your training data, and can ultimately build better models in shorter time. Our goal is to make training data building feel more like a programmatic and enjoyable task, instead of something tedious and repetitive. _refinery_ is our contribution to this goal. And we're constantly aiming to improve this contribution. If you like what we're working on, please leave a ⭐! ## How does Kern AI make money, if refinery is open-source? You won't believe how often we get that question - and it is a fair one Put short, the open-source version of _refinery_ is currently a single-user version, and you can get access to a multi-user environment with our commercial options. Additionally, we have commercial products on top of _refinery_, e.g. to use the _refinery_ automations as an actual realtime prediction API. Generally, we are passionate about open-source and want to contribute as much as possible. ## Features For a detailed overview of features, please look into our [docs](https://docs.kern.ai). ### (Semi-)automated labeling workflow for NLP tasks - Both manual and programmatic for classifications and span-labeling - Integration with state-of-the-art libraries and frameworks - Creation and management of lookup lists/knowledge bases to support during labeling - Neural search-based retrieval of similar records and outliers - Sliceable labeling sessions to drill-down on specific subsets - Multiple labeling tasks possible per project - Rich library of ready-made automations in our open-source [bricks](https://github.com/code-kern-ai/bricks) library ### Extensive data management and monitoring - Best-in-class data management capabilities via our databrowser. Filter, sort and search your data e.g. by confidence, heuristic overlap, user, note, etc. - Integration with [ Hugging Face](https://www.huggingface.co) to automatically create document- and token-level embeddings - JSON-based data model for up- and downloads - Overview of project metrics like confidence and label distributions and confusion matrix - Data accessible and extendable via our [Python SDK](https://github.com/code-kern-ai/refinery-python) - Attribute modifications to extend your attributes (e.g. with sentence complexity metrics) in-place - Again, you can use [bricks](https://github.com/code-kern-ai/bricks) to enrich your data with metadata ### Team workspaces in the [managed version](https://www.kern.ai/pricing) - Allow multiple users to label your data with role-based access and minimized labeling views - Integrate crowd labeling workflows - Automated calculation of inter-annotator agreements ## ☕ Installation ### From pip ``` pip install kern-refinery ``` Once the library is installed, go to the directory where you want to store the data and run `refinery start`. This will automatically `git clone` this repository first if you haven't done so yet. To stop the server, run `refinery stop`. ### From repository **TL;DR:** ``` $ git clone https://github.com/code-kern-ai/refinery.git $ cd refinery ``` If you're on Mac/Linux: ``` $ ./start ``` If you're on Windows: ``` $ start.bat ``` To stop, type `./stop` (Mac/Linux) or `stop.bat`. _refinery_ consists of multiple services that need to be run together. To do so, we've set up a setup file, which will automatically pull and connect the respective services for you. The file is part of this repository, so you can just clone it and run `./start` (Mac/Linux) or `start.bat` (Windows) in the repository. After some minutes (now is a good time to grab a coffee ☕), the setup is done and you can access `http://localhost:4455` in your browser. To stop the server, run `./stop` (Mac/Linux) or `./stop.bat` (Windows). **You're ready to start! ** If you run into any issues during installation, please don't hesitate to reach out to us (see community section below). ### Persisting data By default, we store the data to the directory `refinery/postgres-data`. If you want to change that path, you need to modify the variable `LOCAL_VOLUME` of the `start` script of your operating system. To remove data, simply delete the volume folder. **Make sure to delete only if you don't need the data any longer - this is irreversible!** ## Documentation and tutorials The best way to start with _refinery_ is our [**quick start**](https://docs.kern.ai/refinery/quickstart). You can find extensive guides in our [docs](https://docs.kern.ai) and [tutorials](https://www.youtube.com/@kern_ai/videos) on our YouTube channel. We've also prepared a [repository with sample projects](https://github.com/code-kern-ai/sample-projects) which you can clone. If you need help writing your first labeling functions, look into our open-source content library [bricks](https://github.com/code-kern-ai/bricks). You can find our changelog [here](https://changelog.kern.ai). ## ‍ Need help? No worries, we've got you. If you have questions, reach out to us on [Discord](https://discord.gg/qf4rGCEphW), or [open a ticket](https://github.com/code-kern-ai/refinery/discussions/categories/q-a) in the "q&a" category of our forum. ## Community and contact Feel free to join our [Discord](https://discord.gg/qf4rGCEphW), where we'll happily help you building your training data: We send out a (mostly) weekly