Trident-ML: 实时在线 机器学习 库
Trident-ML is a realtime online machine learning library. It allows you to build real time predictive features using scalable online algorithms. This library is built on top of Storm, a distributed stream processing framework which runs on a cluster of machines and supports horizontal scaling. The packaged algorithms are designed to fit into limited memory and processing time but they don't work in a distributed way.
Trident-ML currently supports :
Trident-ML is based on Trident, a high-level abstraction for doing realtime computing. If you're familiar with high level batch processing tools like Pig or Cascading, the concepts of Trident will be very familiar.
It's recommended to read the Storm and Trident documentation.
Trident-ML process unbounded streams of data implemented by an infinite collection of Instance or TextInstance. Creating instances is the first step to build a prediction tools. Trident-ML offers Trident functions to convert Trident tuples to instances :
TridentTopology toppology = new TridentTopology();
toppology
// Emit tuples with 2 random features (named x0 and x1) and an associated boolean label (named label)
.newStream("randomFeatures", new RandomFeaturesSpout())
// Transform trident tuple to instance
.each(new Fields("label", "x0", "x1"), new InstanceCreator(), new Fields("instance"));TridentTopology toppology = new TridentTopology();
toppology
// Emit tuples containing text and associated label (topic)
.newStream("reuters", new ReutersBatchSpout())
// Convert trident tuple to text instance
.each(new Fields("label", "text"), new TextInstanceCreator(), new Fields("instance"));Trident-ML includes differents algorithms to do supervised classification :
Theses classifiers learn from a datastream of labeled Instance using a ClassifierUpdater. Another datastream of unlabeled instance can be classified with a ClassifyQuery.
The following example learn a NAND function and classify instances comming from a DRPC stream.
…Trident-ML provides the KLDClassifier which implements a text classifier using the Kullback-Leibler Distance.
Here's the code to build a news classifier using Reuters dataset :
…KMeans is an implementation of the well known k-means algorithm which partitions instances into clusters.
Use a ClusterUpdater or a ClusterQuery to respectively udpate clusters or query the clusterer :
…Stream statistics such as mean, standard deviation and count can be easily computed using Trident-ML. Theses statistics are stored in a StreamStatistics object. Statistics update and query are performed respectively using a StreamStatisticsUpdater and a StreamStatisticsQuery :
…Note that Trident-ML can suppport concept drift in a sliding window manner. Use StreamStatistics#adaptive(maxSize) instead of StreamStatistics#fixed() to construct StreamStatistics implementation with a maxSize length window.
Data preprocessing is an important step in the data mining process. Trident-ML provides Trident functions to transform raw features into a representation that is more suitable for machine learning algorithms.
TridentTopology toppology = new TridentTopology();
toppology
// Emit tuples with 2 random features (named x0 and x1) and an associated boolean label (named label)
.newStream("randomFeatures", new RandomFeaturesSpout())
// Convert trident tuple to instance
.each(new Fields("label", "x0", "x1"), new InstanceCreator(), new Fields("instance"))
// Scales features to unit norm
.each(new Fields("instance"), new Normalizer(), new Fields("scaledInstance"));…Trident-ML includes a pre-trained twitter sentiment classifier. It was built on a subset of the Twitter Sentiment Corpus by Niek Sanders with a multi class PA classifier and classifies raw tweets as positive (true) or negative (false). This classifier is implemented as a trident function and can be easily used in a trident topology :
TridentTopology toppology = new TridentTopology();
// Classification stream
toppology.newDRPCStream("classify", localDRPC)
// Query classifier with text instance
.each(new Fields("args"), new TwitterSentimentClassifier(), new Fields("sentiment"));Trident-Ml is hosted on Clojars (a Maven repository). To include Trident-ML in your project , add the following to your pom.xml: :
clojars.org
http://clojars.org/repo
com.github.pmerienne
trident-ml
0.0.4
Storm allows trident-ml to process batches of tuples in a distributed way (batches will be computed among several nodes). This means that trident-ml can scale horizontally with workload.
However Storm prevents state updates to append simultaneously and the model learning is done in a state update. That's why, the learning step can't be distributed. Thankfully this lack of parallelization isn't a real bottle neck because the incremental algorithms are very fast (and simple!).
Distributed algorithms will not be implemented in trident-ml, the whole design prevents this.
So you can't achieve distributed learning however but you can still partition the streams to pre-process/enrich your data in a distributed manner.
Copyright 2013-2015 Pierre Merienne
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
暂无开放 Issues,或尚未同步最近议题。