sci-pype

> 数据库

开源

一个带有本地机器学习缓存和使用 S3 进行导出和导入的 Redis API。使用用于构建、训练、测试、分析整个数据集的 API。

100 stars0 点赞0 次浏览

访问官网 GitHub

工具介绍

一个带有本地机器学习缓存和使用 S3 进行导出和导入的 Redis API。使用用于构建、训练、测试、分析整个数据集的 API。

======================================================================= Sci-Pype - A Machine Learning Framework for Sharing Models and Analysis

This is now deployed under the https://redten.io cloud service for analyzing datasets.

Sci-Pype is a framework for analyzing datasets using Python 2.7 and extended from the Jupyter Scipy-Notebook_ with a supported command line version (no docker or Jupyter required). It was built to make data analysis easier by providing an API to build, train, test, predict, validate, analyze, extract, archive, and import Models and Analysis datasets with: S3 and redis (Kafka coming soon). After building and training the requested Models with a dataset, they are cached in redis along with their respective Analysis. After they are cached, they can be extracted and shared using S3. From S3, the Models can be imported back into redis for making new predictions using the same API.

.. figure:: ./examples/images/v2/Scipype-Seaborn-Visuals.png :alt: Sci-Pype - A Machine Learning Framework for Sharing Models and Analysis :align: center

Analyzing the IRIS dataset with Sci-Pype

Common use cases for this framework are sharing Analysis notebooks and then automating new predictions with email delivery using AWS SES. With this native caching + deployment layer, you can build, train and use the supported Machine Learning Algorithms and Models across multiple environments (including multi-tenant ones). Once trained, you can extract the Models as a compressed, serialized Model file (like a build artifact) that is uploaded to S3. Importing a Model file decompresses the file and stores the Pickle-serialized Models + Analysis objects in redis. In production, it might be useful to house larger Models in something like a load-balanced redis cluster_ for sharing and making new predictions across a team or by automation.

Please note this is a large docker container_ so it may take some time to download and it extracts to ~8.1 GB on disk.

.. _Jupyter Scipy-Notebook: https://github.com/jupyter/docker-stacks/tree/master/scipy-notebook .. _load-balanced redis cluster: https://github.com/jay-johnson/docker-redis-haproxy-cluster .. _large docker container: https://hub.docker.com/r/jayjohnson/jupyter/tags/

Notebook Examples

Please refer to the examples directory_ for the latest notebooks. Most of the notebooks and command line tools require running with a redis server listening on port 6000 (/dev-start.sh will start one).

.. _examples directory: https://github.com/jay-johnson/sci-pype/tree/master/examples

#. ML-IRIS-Analysis-Workflow-Classification.ipynb_

Build a unique Machine Learning Classifier (parameterized XGB by default) for each column in the IRIS dataset. After training and testing the Models, perform a general analysis on each column and save + display images generated during each step. After running, the Models + Analysis are Pickled into a set of objects stored in a set of unique redis cache keys. These leaf nodes are organized into a set of redis keys contained in the ``manifest`` node for retrieval as needed in the future (like a tree of Machine Learning Algorithm Models with their associated pre-computed Analysis in memory).

#. ML-IRIS-Analysis-Workflow-Regression.ipynb_

Build a unique Machine Learning Regressor (parameterized XGB by default) for each column in the IRIS dataset. After training and testing the Models, perform a general analysis on each column and save + display images generated during each step. After running, the Models + Analysis are Pickled into a set of objects stored in a set of unique redis cache keys. These leaf nodes are organized into a set of redis keys contained in the ``manifest`` node for retrieval as needed in the future (like a tree of Machine Learning Algorithm Models with their associated pre-computed Analysis in memory).

#. ML-IRIS-Extract-Models-From-Cache.ipynb_

Extract all Models and Analysis records from redis and compile a large Pickle-serialized dictionary. Create a ``manifest`` for decoupling Model + Analysis nodes and compress the dictionary object (using zlib) and write it to disk as a ``Model file`` (``*.cache.pickle.zlib``). After creating the file on disk, upload it to the configured S3 Bucket and Key.

Once uploaded to the S3 Bucket you should be able to view, download and share the ``Model files``:

.. figure:: ./examples/images/scipype_s3_bucket_with_xgb_classifier_and_regressor_models_as_pickled_object_files.png

    S3 Bucket containing the IRIS ``Model Files``

#. ML-IRIS-Import-and-Cache-Models-From-S3.ipynb_

Download the S3 IRIS ``Model file`` from the configured S3 Bucket + Key and decompress the previously-built Analysis and Models using Pickle to store them all in the redis cache according to the ``manifest``. This includes examples from the IRIS sample dataset and requires you to have a valid S3 Bucket storing the Models and are comfortable paying for the download costs to retrieve the ``Model file`` from S3 (https://aws.amazon.com/s3/pricing/).

#. ML-IRIS-Predict-From-Cache-for-New-Predictions-and-Analysis-Classifier.ipynb_

This notebook shows how to make new predictions with cached IRIS Classifier Models + Analysis housed in redis.

#. ML-IRIS-Predict-From-Cache-for-New-Predictions-and-Analysis-Regressor.ipynb_

This notebook shows how to make new predictions with cached IRIS Regressor Models + Analysis housed in redis.

Command Line Examples

Most of the notebooks and command line tools require running with a redis server listening on port 6000 (/dev-start.sh will start one). The command line versions that do not require docker or Jupyter can be found:

├── bins
│   ├── demo-running-locally.py - Simple validate env is working test
│   ├── ml
│   │   ├── builders - Build and Train Models then Analyze Predictions without display any plotted images (automation examples)
│   │   │   ├── build-classifier-iris.py
│   │   │   ├── build-regressor-iris.py
│   │   │   ├── rl-build-regressor-iris.py
│   │   │   └── secure-rl-build-regressor-iris.py
│   │   ├── demo-ml-classifier-iris.py - Command line version of: ML-IRIS-Analysis-Workflow-Classification.ipynb
│   │   ├── demo-ml-regressor-iris.py - Command line version of: ML-IRIS-Analysis-Workflow-Regression.ipynb
│   │   ├── demo-rl-regressor-iris.py - Command line version of: ML-IRIS-Redis-Labs-Cache-XGB-Regressors.ipynb
│   │   ├── demo-secure-ml-regressor-iris.py - Demo with a Password-Required Redis Server running locally
│   │   ├── demo-secure-rl-regressor-iris.py - Demo with a Password-Required Redis Labs Cloud endpoint
│   │   ├── downloaders
│   │   │   ├── download_boston_house_prices.py
│   │   │   └── download_iris.py - Command line tool for downloading + preparing the IRIS dataset
│   │   ├── extractors
│   │   │   ├── extract_and_upload_iris_classifier.py - Command line version of: ML-IRIS-Extract-Models-From-Cache.ipynb (Classifier)
│   │   │   ├── extract_and_upload_iris_regressor.py - Command line version of: ML-IRIS-Extract-Models-From-Cache.ipynb (Regressor)
│   │   │   ├── rl_extract_and_upload_iris_regressor.py - Command line version of:  ML-IRIS-Redis-Labs-Extract-From-Cache.ipynb
│   │   │   └── secure_rl_extract_and_upload_iris_regressor.py - Command line version with a password for: ML-IRIS-Redis-Labs-Extract-From-Cache.ipynb 
│   │   ├── importers
│   │   │   ├── import_iris_classifier.py - ML-IRIS-Import-and-Cache-Models-From-S3.ipynb (Classifier)
│   │   │   ├── import_iris_regressor.py - ML-IRIS-Import-and-Cache-Models-From-S3.ipynb (Regressor)
│   │   │   ├── rl_import_iris_regressor.py - Command line version of: ML-IRIS-Redis-Labs-Import-From-S3.ipynb
│   │   │   └── secure_rl_import_iris_regressor.py - Command line version with a password for: ML-IRIS-Redis-Labs-Import-From-S3.ipynb
│   │   └── predictors
│   │       ├── predict-from-cache-iris-classifier.py - ML-IRIS-Predict-From-Cache-for-New-Predictions-and-Analysis-Classifier.ipynb (Classifier)
│   │       ├── predict-from-cache-iris-regressor.py - ML-IRIS-Predict-From-Cache-for-New-Predictions-and-Analysis-Regressor.ipynb (Regressor)
│   │       ├── rl-predict-from-cache-iris-regressor.py - Command line version of: ML-IRIS-Redis-Labs-Predict-From-Cached-XGB.ipynb
│   │       └── secure-rl-predict-from-cache-iris-regressor.py - Command line version with a password for: ML-IRIS-Redis-Labs-Predict-From-Cached-XGB.ipynb

Now you can share, test, and deploy Models and their respective Analysis from a file in S3 for other Sci-Pype users running on different environments.

.. _ML-IRIS-Analysis-Workflow-Classification.ipynb: ./examples/ML-IRIS-Analysis-Workflow-Classification.ipynb .. _ML-IRIS-Analysis-Workflow-Regression.ipynb: ./examples/ML-IRIS-Analysis-Workflow-Regression.ipynb .. _ML-IRIS-Extract-Models-From-Cache.ipynb: ./examples/ML-IRIS-Extract-Models-From-Cache.ipynb .. _ML-IRIS-Import-and-Cache-Models-From-S3.ipynb: ./examples/ML-IRIS-Import-and-Cache-Models-From-S3.ipynb .. _ML-IRIS-Predict-From-Cache-for-New-Predictions-and-Analysis-Classifier.ipynb: ./examples/ML-IRIS-Predict-From-Cache-for-New-Predictions-and-Analysis-Classifier.ipynb .. _ML-IRIS-Predict-From-Cache-for-New-Predictions-and-Analysis-Regressor.ipynb: ./examples/ML-IRIS-Predict-From-Cache-for-New-Predictions-and-Analysis-Regressor.ipynb

Overview

The docker container runs a Jupyter web application. The web application runs Jupyter Notebooks_ as kernels. For now the examples and core included in this repository will only work with Python 2.

.. _Jupyter Notebooks: http://jupyter-notebook.readthedocs.io/en/latest/

This container can run in four modes:

#. Default development

This mode will mount your changes from the repository into the container at runtime for local testing.

To start the local development version run: dev-start.sh_

::

    ./dev-start.sh

You can login to the container with: ``./ssh.sh``

.. _dev-start.sh: https://github.com/jay-johnson/sci-pype/blob/s3_and_machine_learning_api/dev-start.sh

#. Docker Run Single Container

To start the local development version run: start.sh_

::

    ./start.sh

You can login to the container with: ``./ssh.sh``

.. _start.sh: https://github.com/jay-johnson/sci-pype/blob/master/start.sh

#. Full Stack

To start the full stack mode run: compose-start-full.sh_

::

    ./compose-start-full.sh

The full-stack-compose.yml_ will deploy three docker containers using docker compose:

- **MySQL Database container with phpMyAdmin** for Stock Data (from the `schemaprototyping repo`_) 
- **Jupyter**
- **Redis server** (`jayjohnson/redis-single-node`_) on port 6000

.. _compose-start-full.sh: https://github.com/jay-johnson/sci-pype/blob/master/compose-start-full.sh
.. _full-stack-compose.yml: https://github.com/jay-johnson/sci-pype/blob/master/full-stack-compose.yml
.. _schemaprototyping repo: https://github.com/jay-johnson/docker-schema-prototyping-with-mysql
.. _jayjohnson/redis-single-node: https://hub.docker.com/r/jayjohnson/redis-single-node

#. Standalone Testing

To start the full stack mode run: compose-start-jupyter.sh_

::

    ./compose-start-jupyter.sh

The jupyter-docker-compose.yml_ is used to deploy a single Jupyter container.

.. _compose-start-jupyter.sh: https://github.com/jay-johnson/sci-pype/blob/master/compose-start-jupyter.sh
.. _jupyter-docker-compose.yml: https://github.com/jay-johnson/sci-pype/blob/master/jupyter-docker-compose.yml