redis-objects

Redis::Objects - Map Redis types directly to Ruby objects

Important 2.0 changes

Several longstanding bugs have been addressed in this release. However, this may require some code changes as part of the upgrade.

Renaming of lock Method

The lock method that collided with ActiveRecord::Base has been renamed redis_lock. This means your classes need to be updated to call redis_lock instead:

class YouClassNameHere < ActiveRecord::Base
  include Redis::Objects
  redis_lock :mylock  # formerly just "lock"
end

For more details on the issue and fix refer to #196.

New key naming method

A new method to determine the internal key naming scheme has been added to fix a longstanding bug. (Refer to #213) By default, backwards compatibility is maintained, but this means that in some cases, Nested classes (for example Dog::Behavior and Cat::Behavior) would have key names that collide (they would both start with Behavior as anything before :: is stripped).

To ensure names are unique, you can set Redis::Objects.prefix_style = :modern after you load the module. By default, it is set to Redis::Objects.prefix_style = :legacy.

Migrating old keys

If your Redis::Object subclasses are nested such as Dog::Behavior and Cat::Behavior, then you should upgrade them using the below proceedure. (only needed once)

Create a script like this:

class Dog::Behavior < ActiveRecord::Base
  include Redis::Objects
  # ... your relevant redis_object definitions here (counters/sets) ...
end

Dog::Behavior.migrate_redis_legacy_keys

You need to find a time when you can temporarily pause writes to your redis server so that you can run that script. It uses redis.scan internally so it should be able to handle a high number of keys. For large data sets, it could take a while.

After migrating all of your redis keys, update Redis::Objects.prefix_style = :modern to start using the new keys.

Note: Your existing data in Redis will not be accessible after running migrate_redis_legacy_keys until the prefix_style has been changed.

---

Overview

This is not an ORM (Object-Relational Mapping). People that are wrapping ORM’s around Redis are missing the point.

The killer feature of Redis is that it allows you to perform atomic operations on individual data structures, like counters, lists, and sets. The atomic part is HUGE. Using an ORM wrapper that retrieves a "record", updates values, then sends those values back, removes the atomicity, and thus the major advantage of Redis. Just use MySQL, k?

This gem provides a Rubyish interface to Redis, by mapping Redis data types to Ruby objects, via a thin layer over the redis gem. It offers several advantages over the lower-level redis-rb API:

1. Easy to integrate directly with existing ORMs - ActiveRecord, DataMapper, etc. Add counters to your model!
2. Complex data structures are automatically Marshaled (if you set :marshal => true)
3. Integers are returned as integers, rather than '17'
4. Higher-level types are provided, such as Locks, that wrap multiple calls

This gem originally arose out of a need for high-concurrency atomic operations; for a fun rant on the topic, see An Atomic Rant, or scroll down to Atomic Counters and Locks in this README.

There are two ways to use Redis::Objects, either as an include within a model class (to tightly integrate with ORMs or other classes), or standalone by using classes such as Redis::List and Redis::SortedSet.

Installation and Setup

Add it to your Gemfile as:

gem 'redis-objects'

Redis::Objects needs a handle created by Redis.new or a ConnectionPool.

If you're using Rails, config/initializers/redis.rb is a good place for this. However, there are no dependencies on Rails. Redis::Objects can be used in any Ruby code; Sinatra, Resque, or Standalone - no problem.

The recommended approach is to use a ConnectionPool since this guarantees that most timeouts in the redis client do not pollute your existing connection.

require 'connection_pool'
Redis::Objects.redis = ConnectionPool.new(size: 5, timeout: 5) { Redis.new(:host => '127.0.0.1', :port => 6379) }

However, you need to make sure that both :timeout and :size are set appropriately in a multithreaded environment.

Alternatively, you can set the redis handle directly:

Redis::Objects.redis = Redis.new(...)

Redis::Objects will also default to Redis.current if Redis::Objects.redis is not set.

Redis.current = Redis.new(:host => '127.0.0.1', :port => 6379)

Finally, you can even set different handles for different classes:

class User
  include Redis::Objects
end
class Post
  include Redis::Objects
end

# you can also use a ConnectionPool here as well
User.redis = Redis.new(:host => '1.2.3.4')
Post.redis = Redis.new(:host => '5.6.7.8')

As of 0.7.0, redis-objects now autoloads the appropriate Redis::Whatever classes on demand. Previous strategies of individually requiring redis/list or redis/set are no longer required.

Option 1: Model Class Include

Including Redis::Objects in a model class makes it trivial to integrate Redis types with an existing ActiveRecord, DataMapper, Mongoid, or similar class. Redis::Objects will work with any class that provides an id method that returns a unique value. Redis::Objects automatically creates keys that are unique to each object, in the format:

model_name:id:field_name

For illustration purposes, consider this stub class:

class User
  include Redis::Objects
  counter :my_posts
  def id
    1
  end
end

user = User.new
user.id  # 1
user.my_posts.increment
user.my_posts.increment
user.my_posts.increment
puts user.my_posts.value # 3
user.my_posts.reset
puts user.my_posts.value # 0
user.my_posts.reset 5
puts user.my_posts.value # 5

Here's an example that integrates several data types with an ActiveRecord model:

…

Familiar Ruby array operations Just Work™:

@team = Team.find_by_name('New York Yankees')
@team.on_base << 'player1'
@team.on_base << 'player2'
@team.on_base << 'player3'
@team.on_base    # ['player1', 'player2', 'player3']
@team.on_base.pop
@team.on_base.shift
@team.on_base.length  # 1
@team.on_base.delete('player2')
@team.on_base = ['player1', 'player2']  # ['player1', 'player2']

Sets work too:

@team.outfielders << 'outfielder1'
@team.outfielders << 'outfielder2'
@team.outfielders << 'outfielder1'   # dup ignored
@team.outfielders  # ['outfielder1', 'outfielder2']
@team.outfielders.each do |player|
  puts player
end
player = @team.outfielders.detect{|of| of == 'outfielder2'}
@team.outfielders = ['outfielder1', 'outfielder3']  # ['outfielder1', 'outfielder3']

Hashes work too:

@team.pitchers_faced['player1'] = 'pitcher2'
@team.pitchers_faced['player2'] = 'pitcher1'
@team.pitchers_faced = { 'player1' => 'pitcher2', 'player2' => 'pitcher1' }

And you can do unions and intersections between objects (kinda cool):

@team1.outfielders | @team2.outfielders   # outfielders on both teams
@team1.outfielders & @team2.outfielders   # in baseball, should be empty :-)

Counters can be atomically incremented/decremented (but not assigned):

@team.hits.increment  # or incr
@team.hits.decrement  # or decr
@team.hits.incr(3)    # add 3
@team.runs = 4        # exception

Defining a different method as the id field is easy

class User
  include Redis::Objects
  redis_id_field :uid
  counter :my_posts
end

user.uid                # 195137a1bdea4473
user.my_posts.increment # 1

You can also define globals redis attributes that are accessed through the class itself. No id needed/used for these.

class Team < ActiveRecord::Base
  include Redis::Objects

  sorted_set :rank, :global => true
end

Team.rank['Yankees']   = 12
Team.rank['Red Socks'] = 5
Team.rank['Mariners']  = 7
Team.rank.members(:with_scores => true) # => [["Red Socks", 5], ["Mariners", 7], ["Yankees", 12]]

Finally, for free, you get a redis method that points directly to a Redis connection:

Team.redis.get('somekey')
@team = Team.new
@team.redis.get('somekey')
@team.redis.smembers('someset')

You can use the redis handle to directly call any Redis API command.

Option 2: Standalone Usage

There is a Ruby class that maps to each Redis type, with methods for each Redis API command. Note that calling new does not imply it's actually a "new" value - it just creates a mapping between that Ruby object and the corresponding Redis data structure, which may already exist on the redis-server.

Counters

The counter_name is the key stored in Redis.

@counter = Redis::Counter.new('counter_name')
@counter.increment  # or incr
@counter.decrement  # or decr
@counter.increment(3)
puts @counter.value

This gem provides a clean way to do atomic blocks as well:

@counter.increment do |val|
  raise "Full" if val > MAX_VAL  # rewind counter
end

See the section on Atomic Counters and Locks for cool uses of atomic counter blocks.

Locks

A convenience class that wraps the pattern of using setnx to perform locking.

@lock = Redis::Lock.new('serialize_stuff', :expiration => 15, :timeout => 0.1)
@lock.lock do
  # do work
end

This can be especially useful if you're running batch jobs spread across multiple hosts.

Values

Simple values are easy as well:

@value = Redis::Value.new('value_name')
@value.value = 'a'
@value.delete

Complex data is no problem with :marshal => true:

@account = Account.create!(params[:account])
@newest  = Redis::Value.new('newest_account', :marshal => true)
@newest.value = @account.attributes
puts @newest.value['username']

Compress data to save memory usage on Redis with :compress => true:

@account = Account.create!(params[:account])
@marshaled_value = Redis::Value.new('marshaled', :marshal => true, :compress => true)
@marshaled_value.value = @account.attributes
@unmarshaled_value = Redis::Value.new('unmarshaled', :compress => true)
@unmarshaled_value = 'Really Long String'
puts @marshaled_value.value['username']
puts @unmarshaled_value.value

Lists

Lists work just like Ruby arrays:

@list = Redis::List.new('list_name')
@list << 'a'
@list << 'b'
@list.include? 'c'   # false
@list.values  # ['a','b']
@list << 'c'
@list.delete('c')
@list[0]
@list[0,1]
@list[0..1]
@list.shift
@list.pop
@list.clear
# etc

You can bound the size of the list to only hold N elements like so:

# Only holds 10 elements, throws out old ones when you reach :maxlength.
@list = Redis::List.new('list_name', :maxlength => 10)

Complex data types are serialized with :marshal => true:

@list = Redis::List.new('list_name', :marshal => true)
@list << {:name => "Nate", :city => "San Diego"}
@list << {:name => "Peter", :city => "Oceanside"}
@list.each do |el|
  puts "#{el[:name]} lives in #{el[:city]}"
end

Note: If you run into issues, with Marshal errors, refer to the fix in Issue #176.

Hashes

Hashes work like a Ruby Hash, with a few Redis-specific additions. (The class name is "HashKey" not just "Hash", due to conflicts with the Ruby core Hash class in other gems.)

@hash = Redis::HashKey.new('hash_name')
@hash['a'] = 1
@hash['b'] = 2
@hash.each do |k,v|
  puts "#{k} = #{v}"
e