refile

Refile

Refile is a modern file upload library for Ruby applications. It is simple, yet powerful.

Links:

• API documentation
• Source Code
• Contributing
• Code of Conduct
• Example Application

Features:

• Configurable backends, file system, S3, etc...
• Convenient integration with ORMs
• On the fly manipulation of images and other files
• Streaming IO for fast and memory friendly uploads
• Works across form redisplays, i.e. when validations fail, even on S3
• Effortless direct uploads, even to S3
• Support for multiple file uploads
• Support for single file upload

Sponsored by:

Quick start, Rails

Add the gem:

gem "refile", require: "refile/rails"
gem "refile-mini_magick"

We're requiring both Refile's Rails integration and image processing via the MiniMagick gem, which requires ImageMagick (or GraphicsMagick) to be installed. To install it simply run:

brew install imagemagick # OS X
sudo apt-get install imagemagick # Ubuntu

Use the attachment method to use Refile in a model:

class User < ActiveRecord::Base
  attachment :profile_image
end

Generate a migration:

rails generate migration add_profile_image_to_users profile_image_id:string &&
profile_image_filename:string && profile_image_size:string &&
profile_image_content_type:string

rake db:migrate

Add an attachment field to your form:

<%= form_for @user do |form| %>
  <%= form.attachment_field :profile_image %>
<% end %>

Set up strong parameters:

def user_params
  params.require(:user).permit(:profile_image)
end

And start uploading! Finally show the file in your view:

<%= image_tag attachment_url(@user, :profile_image, :fill, 300, 300, format: "jpg") %>

How it works

Refile consists of several parts:

1. Backends: cache and persist files
2. Model attachments: map files to model columns
3. A Rack application: streams files and accepts uploads
4. Rails helpers: conveniently generate markup in your views
5. A JavaScript library: facilitates direct uploads

Let's look at each of these in more detail!

1. Backend

Files are uploaded to a backend. The backend assigns an ID to this file, which will be unique for this file within the backend.

Let's look at a simple example of using the backend:

backend = Refile::Backend::FileSystem.new("tmp")

file = backend.upload(StringIO.new("hello"))
file.id # => "b205bc..."
file.read # => "hello"

backend.get(file.id).read # => "hello"

As you may notice, backends are "flat". Files do not have directories, nor do they have names or permissions, they are only identified by their ID.

Refile has a global registry of backends, accessed through Refile.backends.

There are two "special" backends, which are only really special in that they are the default backends for attachments. They are cache and store.

By default files will be uploaded to ./tmp/uploads/store. If you would like to persist them between deploys of your application, you can override the upload folder by adding an initializer like this:

# config/initializers/refile.rb

Refile.backends['store'] = Refile::Backend::FileSystem.new('/etc/projectname-uploads/')

The cache is intended to be transient. Files are added here before they are meant to be permanently stored. Usually files are then moved to the store for permanent storage, but this isn't always the case.

Suppose for example that a user uploads a file in a form and receives a validation error. In that case the file has been temporarily stored in the cache. The user might decide to fix the error and resubmit, at which point the file will be promoted to the store. On the other hand, the user might simply give up and leave, now the file is left in the cache for later cleanup.

Refile has convenient accessors for setting the cache and store, so for example if you add the refile-s3 gem to your Gemfile:

gem "refile-s3"

Now you can upload files to S3 easily by using these accessors:

# config/initializers/refile.rb
require "refile/s3"

aws = {
  access_key_id: "xyz",
  secret_access_key: "abc",
  region: "sa-east-1",
  bucket: "my-bucket",
}
Refile.cache = Refile::S3.new(prefix: "cache", **aws)
Refile.store = Refile::S3.new(prefix: "store", **aws)

Try this in the quick start example above and your files are now uploaded to S3.

Backends also provide the option of restricting the size of files they accept. For example:

Refile.cache = Refile::S3.new(max_size: 10.megabytes, ...)

The Refile gem only ships with a FileSystem backend. Additional backends are provided by other gems.

• Amazon S3
• Fog provides support for a ton of different cloud storage providers, including Google Storage and Rackspace CloudFiles.
• Postgresql
• Gridfs
• In Memory

Uploadable

The upload method on backends can be called with a variety of objects. It requires that the object passed to it behaves similarly to Ruby IO objects, in particular it must implement the methods size, read(length = nil, buffer = nil), eof?, rewind, and close. All of File, Tempfile, ActionDispatch::UploadedFile and StringIO implement this interface, however String does not. If you want to upload a file from a String you must wrap it in a StringIO first.

2. Attachments

You've already seen the attachment method:

class User < ActiveRecord::Base
  attachment :profile_image
end

Calling attachment generates a getter and setter with the given name. When you assign a file to the setter, it is uploaded to the cache:

User.new

# with a ActionDispatch::UploadedFile
user.profile_image = params[:file]

# with a regular File object
File.open("/some/path", "rb") do |file|
  user.profile_image = file
end

# or a StringIO
user.profile_image = StringIO.new("hello world")

user.profile_image.id # => "fec421..."
user.profile_image.read # => "hello world"

When you call save on the record, the uploaded file is transferred from the cache to the store. Where possible, Refile does this move efficiently. For example if both cache and store are on the same S3 account, instead of downloading the file and uploading it again, Refile will simply issue a copy command to S3.

Other ORMs

Refile comes with ActiveRecord integration built-in, but is built to integrate with any ORM, so building your own should not be too difficult. Some integrations are already available via gems:

• refile-sequel
• refile-mongoid

Pure Ruby classes

You can also use attachments in pure Ruby classes like this:

class User
  extend Refile::Attachment

  attr_accessor :profile_image_id

  attachment :profile_image
end

Keeping uploaded files

By default Refile will delete a stored file when its model is destroyed. You can change this behaviour by passing in the destroy option.

class User < ActiveRecord::Base
  attachment :profile_image, destroy: false
end

Now Refile will not delete the profile_image file from the store if the user is destroyed.

3. Rack Application

Refile includes a Rack application (an endpoint, not a middleware), written in Sinatra. This application streams files from backends and can even accept file uploads and upload them to backends.

Important: Unlike other file upload solutions, Refile always streams your files through your application. It cannot generate URLs to your files. This means that you should always put a CDN or other HTTP cache in front of your application. Serving files through your app takes a lot of resources and you want it to happen rarely.

Setting this up is actually quite simple, you can use the same CDN you would use for your application's static assets. This blog post explains how to set this up (bonus: faster static assets!). Once you've set this up, simply configure Refile to use your CDN:

Refile.cdn_host = "https://your-dist-url.cloudfront.net"

Using the HTTPS protocol for Refile.cdn_host is recommended. There aren't any performance concerns, and it is always safe to request HTTPS assets.

Mounting

If you are using Rails and have required refile/rails.rb, then the Rack application is mounted for you at /attachments. You should be able to see this when you run rake routes.

You can configure Refile to use a different mount_point than /attachments:

Refile.mount_point = "/your-preferred-mount-point"

You could also run the application on its own, it doesn't need to be mounted to work.

If you are using a catch-all route (such as required by Comfy CMS), you will need to turn off Automounting and add the refile route before your catch all route.

(in initializers/refile.rb)

Refile.automount = false

in routes.rb

  mount Refile.app, at: Refile.mount_point, as: :refile_app

  # Make sure this routeset is defined last
  comfy_route :cms, :path => '/', :sitemap => true

Retrieving files

Files can be retrieved from the application by calling:

GET /attachments/:token/:backend_name/:id/:filename

The :filename serves no other purpose than generating a nice name when the user downloads the file, it does not in any way affect the downloaded file. For caching purposes you should always use the same filename for the same file. The Rails helpers default this to the name of the column.

The :token is a generated digest of the request path when the Refile.secret_key is configured; otherwise, the application will raise an error. The digest feature provides a security measure against unverified requests.

NOTICE: If you don't set the Refile.secret_key we will use rails secret_key_base to generate the token. We suggest you not to change the secret_key_base after you generated and hardcoded some attachment URLs in your application (e.g. blog post images), because the token will change and you'll not be able to retrieve in this case, the images.

Processing

Refile provides on the fly processing of files. You can trigger it by calling a URL like this:

GET /attachments/:token/:backend_name/:processor_name/*args/:id/:filename

Suppose we have uploaded a file:

Refile.cache.upload(StringIO.new("hello")).id # => "a4e8ce"

And we've defined a processor like this:

Refile.processor :reverse do |file|
  StringIO.new(file.read.reverse)
end

Then you could do the following.

curl http://127.0.0.1:3000/attachments/token/cache/reverse/a4e8ce/some_file.txt
elloh

Refile calls call on the processor and passes in the retrieved file, as well as all additional arguments sent through the URL.

4. Rails helpers

Refile provides the attachment_field form helper which generates a file field as well as a hidden field. This field keeps track of the file in case it is not yet permanently stored, for example if validations fail. It is also used for direct and presigned uploads. For this reason it is highly recommended to use attachment_field instead of `file_f