A low-latency prediction-serving system
Clipper is a prediction serving system that sits between user-facing applications and a wide range of commonly used machine learning models and frameworks. Learn more about Clipper and view documentation at our website http://clipper.ai.
Clipper simplifies integration of machine learning techniques into user facing applications by providing a simple standard REST interface for prediction and feedback across a wide range of commonly used machine learning frameworks. Clipper makes product teams happy.
Clipper simplifies model deployment and helps reduce common bugs by using the same tools and libraries used in model development to render live predictions. Clipper makes data scientists happy.
Clipper improves throughput and ensures reliable millisecond latencies by introducing adaptive batching, caching, and straggler mitigation techniques. Clipper makes the infra-team less unhappy.
Clipper improves prediction accuracy by introducing state-of-the-art bandit and ensemble methods to intelligently select and combine predictions and achieve real-time personalization across machine learning frameworks. Clipper makes users happy.
Note: This quickstart works for the latest version of code. For a quickstart that works with the released version of Clipper available on PyPi, go to our website
This quickstart requires Docker and supports Python 2.7, 3.5, 3.6 and 3.7.
Basic query: https://github.com/ucbrise/clipper/tree/develop/examples/basic_query
Image query: https://github.com/ucbrise/clipper/tree/develop/examples/image_query
Examples including metrics: https://github.com/ucbrise/clipper/tree/develop/examples
Install Clipper
You can either install Clipper directly from GitHub:
pip install git+https://github.com/ucbrise/clipper.git@develop#subdirectory=clipper_admin
or by cloning Clipper and installing directly from the file system:
pip install -e </path/to/clipper_repo>/clipper_admin
Start a local Clipper cluster
First start a Python interpreter session.
$ python
# Or start one with iPython
$ conda install ipython
$ ipython
Create a ClipperConnection object and start Clipper. Running this command for the first time will
download several Docker containers, so it may take some time.
from clipper_admin import ClipperConnection, DockerContainerManager
clipper_conn = ClipperConnection(DockerContainerManager())
clipper_conn.start_clipper()
17-08-30:15:48:41 INFO [docker_container_manager.py:95] Starting managed Redis instance in Docker
17-08-30:15:48:43 INFO [clipper_admin.py:105] Clipper still initializing.
17-08-30:15:48:44 INFO [clipper_admin.py:107] Clipper is running
Register an application called "hello-world". This will create a prediction REST endpoint at http://localhost:1337/hello-world/predict
clipper_conn.register_application(name="hello-world", input_type="doubles", default_output="-1.0", slo_micros=100000)
17-08-30:15:51:42 INFO [clipper_admin.py:182] Application hello-world was successfully registered
Inspect Clipper to see the registered apps
clipper_conn.get_all_apps()
[u'hello-world']
Define a simple model that just returns the sum of each feature vector. Note that the prediction function takes a list of feature vectors as input and returns a list of strings.
def feature_sum(xs):
return [str(sum(x)) for x in xs]
Import the python deployer package
from clipper_admin.deployers import python as python_deployer
Deploy the "feature_sum" function as a model. Notice that the application and model
must have the same input type.
python_deployer.deploy_python_closure(clipper_conn, name="sum-model", version=1, input_type="doubles", func=feature_sum)
…
Possible Error
If start_clipper() is stuck at this logs, try pip install -U cloudpickle==0.5.3
…
It is because of cloudpickle dependency version issue. You may see this error logs from model container docker log.
…
Tell Clipper to route requests for the "hello-world" application to the "sum-model"
clipper_conn.link_model_to_app(app_name="hello-world", model_name="sum-model")
17-08-30:16:08:50 INFO [clipper_admin.py:224] Model sum-model is now linked to application hello-world
Your application is now ready to serve predictions
Now that you've deployed your first model, you can start requesting predictions at the REST endpoint that clipper created for your application: http://localhost:1337/hello-world/predict
With cURL:
$ curl -X POST --header "Content-Type:application/json" -d '{"input": [1.1, 2.2, 3.3]}' 127.0.0.1:1337/hello-world/predict
With Python:
import requests, json, numpy as np
headers = {"Content-type": "application/json"}
requests.post("http://localhost:1337/hello-world/predict", headers=headers, data=json.dumps({"input": list(np.random.random(10))})).json()
If you closed the Python REPL you were using to start Clipper, you will need to start a new Python REPL and create another connection to the Clipper cluster. If you still have the Python REPL session active from earlier, you can re-use your existing ClipperConnection object.
Create a new connection. If you have still have the Python REPL from earlier, you can skip this step.
from clipper_admin import ClipperConnection, DockerContainerManager
clipper_conn = ClipperConnection(DockerContainerManager())
clipper_conn.connect()
Stop all Clipper docker containers
clipper_conn.stop_all()
17-08-30:16:15:38 INFO [clipper_admin.py:1141] Stopped all Clipper cluster and all model containers
To file a bug or request a feature, please file a GitHub issue. Pull requests are welcome. Additional help and instructions for contributors can be found on our website at http://clipper.ai/contributing.
You can contact us at [email protected]
This research is supported in part by DHS Award HSHQDC-16-3-00083, DOE Award SN10040 DE-SC0012463, NSF CISE Expeditions Award CCF-1139158, and gifts from Ant Financial, Amazon Web Services, CapitalOne, Ericsson, GE, Google, Huawei, Intel, IBM, Microsoft and VMware.
No open issues yet, or sync has not completed.