百科.dev
全部条目在线工具AI 编程趋势榜开源项目技术资讯提交条目
登录
< 返回工具列表
R

redstorm

> 编程语言
开源

JRuby在风暴上

295 stars0 点赞3 次浏览
访问官网GitHub

工具介绍

JRuby在风暴上

RedStorm - JRuby on Storm

RedStorm provides a Ruby DSL using JRuby integration for the Storm distributed realtime computation system.

Like RedStorm? visit us on IRC at #redstorm on freenode

Check also these related projects:

  • redstorm-starter
  • redstorm-benchmark

Documentation

Chances are new versions of RedStorm will introduce changes that will break compatibility or change the developement workflow. To prevent out-of-sync documentation, per version specific documentation are kept in the wiki when necessary.

Dependencies

Stable 0.6.6

  • Tested on OSX 10.8.3, Ubuntu Linux 12.10 using Storm 0.9.0-wip16, JRuby 1.7.4, OpenJDK 7

Current 0.7.0.beta1

  • Tested on OSX 10.9.1, Ubuntu Linux 12.10 using Storm 0.9.1-incubating, JRuby 1.7.11, OpenJDK 7

Installation

  • RubyGems

    bash
    $ gem install redstorm
  • Bundler

    ruby
    source "https://rubygems.org"
    gem "redstorm", "~> 0.6.6"

Usage

Overview

  • create a project directory

  • install the RedStorm gem

  • create a subdirectory for your topology code

  • perform the initial setup as to build and install dependencies

    bash
    $ redstorm install
  • run your topology in local mode

    bash
    $ redstorm local 

Initial setup

bash
$ redstorm install

This will install default Java jar dependencies in target/dependency, generate & compile the Java bindings in target/classes.

By default, the Java compilation will use the current JVM version as source/target compilation compatibility. If you want to force a specific source/target version, use the --[JVM VERSION] option. For example, to force compiling for 1.6 use:

bash
$ redstorm install --1.6

Create a topology

Create a subdirectory for your topology code and create your topology class using this naming convention: underscore topology_class_file_name.rb MUST correspond to its CamelCase class name.

Here's an example hello_world_topology.rb

ruby
require 'red_storm'

class HelloWorldSpout  false do |tuple|
    log.info(tuple[0])
  end
end

class HelloWorldTopology 
  
    
    
      
      
    
  
  • ivy/topology_dependencies.xml

    xml
    
    
      
      
        
    
        
        
        
          
          
        
    
      
    

The jars repositories can be configured by adding the ivy/settings.xml file in the root of your project. For information on the Ivy settings format, see the Ivy Settings Documentation. Below is the current default:

  • ivy/settings.xml

    xml
    
    
      
      
        
          
          
          
          
        
      
    

Run in local mode

bash
$ redstorm local 

note that the topology can also be launched with the following command:

bash
$ java -cp "target/classes:target/dependency/storm/default/*:target/dependency/topology/default/*:" redstorm.TopologyLauncher local 

See examples below to run examples in local mode or on a production cluster.

Run on production cluster

Locally installing the Storm distribution is not required. Note that RedStorm does not provide the Storm command line utilities and you will need to install the Storm distribution to use the Storm command line utilities.

  1. create the Storm config file ~/.storm/storm.yaml and add the following
yaml
nimbus.host: "host_name_or_ip"

you can also use an alternate path and use the --config

  1. generate target/cluster-topology.jar. This jar file will include your sources directory plus the required dependencies
bash
$ redstorm jar   ...
  1. submit the cluster topology jar file to the cluster
bash
$ redstorm cluster 

or if you have an alternate Storm config path:

bash
$ redstorm cluster --config  

note that the cluster topology jar can also be submitted using the storm command with:

bash
$ storm jar target/cluster-topology.jar -Djruby.compat.version=RUBY1_9 redstorm.TopologyLauncher cluster 

The Storm wiki has instructions on setting up a production cluster. You can also manually submit your topology.

Examples

RedStorm includes several example topologies to help get you started. You can find documentation for the examples here.

Ruby DSL

Ruby DSL Documentation

Multilang ShellSpout & ShellBolt support

Please refer to Using non JVM languages with Storm for the complete information on Multilang & shelling in Storm.

In RedStorm ShellSpout and ShellBolt are supported using the following construct in the topology definition:

ruby
bolt JRubyShellBolt, ["python", "splitsentence.py"] do
  output_fields "word"
  source SimpleSpout, :shuffle
end
  • JRubyShellBolt must be used for a ShellBolt and the array argument ["python", "splitsentence.py"] are the arguments to the class constructor and are the commands to the ShellBolt.

  • The directory containing the topology class must contain a resources/ directory with all the shell files.

See the shell topology example

Transactional and LinearDRPC topologies

Despite the fact that both transactional and linear DRPC topologies are now deprecated as of Storm 0.8.1 work on these has been merged in RedStorm 0.6.5. Lots of the work done on this is required toward Storm Trident topologies. Documentation and examples for transactional and linear DRPC topologies will be added shorty.

Known issues

  • SnakeYAML conflict between Storm and JRuby

    See issue. This is a classic Java world jar conflict. Storm 0.9.0 uses snakeyaml 1.9 and JRuby 1.7.x uses snakeyaml 1.11. If you try to use YAML serialization in your topology it will crash with an exception. This problem is easy to solve when running topologies in local mode, simply override in the storm dependencies with the correct jar version. You can do this be creating a custom storm dependencies:

    • ivy/storm_dependencies.xml

      xml
      
      
        
        
          
          
          
        
      

    In remote cluster mode you will have to update snakeyaml manually or with your favorite deployment/provisioning tool.

RedStorm Development

It is possible to fork the RedStorm project and run local and remote/cluster topologies directly from the project sources without installing the gem. This is a useful setup when contributing to the project.

Requirements

  • JRuby 1.7.4

Workflow

  • fork project and create branch

  • install RedStorm required gems

    bash
    $ bundle install
  • install dependencies in target/dependencies

    bash
    $ bundle exec redstorm deps
  • generate and build Java source into target/classes

    bash
    $ bundle exec redstorm build

    By default, the Java compilation will use the current JVM version as source/target compilation compatibility. If you want to force a specific source/target version, use the --[JVM VERSION] option. For example, to force compiling for 1.6 use:

    bash
    $ bundle exec redstorm build --1.6

    if you modify any of the RedStorm Ruby code or Java binding code, you need to run this to refresh code and rebuild the bindings

  • follow the normal usage patterns to run the topology in local or remote cluster.

    bash
    $ bundle exec redstorm bundle ...
    $ bundle exec redstorm local ...
    $ bundle exec redstorm jar ...
    $ bundle exec redstorm cluster ...

Remote cluster testing

Vagrant & Chef configuration to create a single node test Storm cluster is available in https://github.com/colinsurprenant/redstorm/tree/master/vagrant/

Notes about 1.8/1.9 JRuby compatibility

Ruby 1.9 is the default runtime mode in JRuby 1.7.x

If you require Ruby 1.8 support, there are two ways to have JRuby run in 1.8 runtime mode:

  • by setting the JRUBY_OPTS env variable

    bash
    $ export JRUBY_OPTS=--1.8
  • by using the --1.8 option

    bash
    $ jruby --1.8 -S redstorm ...

By defaut, a topology will be executed in the same mode as the interpreter running the $ redstorm command. You can force RedStorm to choose a specific JRuby compatibility mode using the [--1.8|--1.9] parameter for the topology execution in local or remote cluster.

bash
$ redstorm local|cluster [--1.8|--1.9] ...

If you are not using the DSL and only using the proxy classes (like in examples/native) you will need to manually set the JRuby version in the Storm Backtype::Config object like this:

ruby
class SomeTopology
  RedStorm::Configuration.topology_class = self

  def start(base_class_path, env)
    builder = TopologyBuilder.new
    builder.setSpout ...
    builder.setBolt ...

    conf = Backtype::Config.new
    conf.put("topology.worker.childopts", "-Djruby.compat.version=RUBY1_8")

    StormSubmitter.submitTopology("some_topology", conf, builder.createTopology);
  end
end

How to contribute

Fork the project, create a branch and submit a pull request.

Some ways you can contribute:

  • by reporting bugs using the issue tracker
  • by suggesting new features using the issue tracker
  • by writing or editing documentation
  • by writing specs
  • by writing code
  • by refactoring code
  • ...

Projects using RedStorm

If you want to list your RedStorm project here, contact me.

  • Tweigeist - realtime computation of the top trending hashtags on Twitter. See Live Demo.

Author

Colin Surprenant, http://github.com/colinsurprenant/, @colinsurprenant, [email protected], http://colinsurprenant.com/

Contributors

  • Theo Hultberg, https://github.com/iconara
  • Paul Bergeron, https://github.com/dinedal
  • Phil Pirozhkov, https://github.com/pirj
  • Evan Broderm, https://github.com/ebroder
  • Shay Elkin, https://github.com/shayel
  • adsummos, https://github.com/adsummos

License

Apache License, Version 2.0. See the LICENSE.md file.

Issues· 35 开放

查看全部 Issues在 GitHub 打开

暂无开放 Issues,或尚未同步最近议题。

> 标签

Ruby

暂无评论,来聊聊你的看法吧

> 工具信息

发布日期2026年8月1日
最后更新2026年9月18日
分类编程语言
定价开源

> 相关工具

T
TypeScript
JavaScript 的超集,为前端与全栈提供静态类型
P
Python
通用编程语言,广泛用于 Web、数据与 AI
G
Go
Google 推出的简洁高效系统语言