Process your uploads in the background by uploading directly to S3
Process your uploads in the background by uploading directly to S3
CarrierWave is a great way to upload files from Ruby applications, but since processing and saving is done in-process, it doesn't scale well. A better way is to upload your files directly then handle the processing and saving in a background process.
CarrierWaveDirect works on top of CarrierWave and provides a simple way to achieve this.
For a concrete example on how to use CarrierWaveDirect in a Rails application check out the Example Application.
Right now, CarrierWaveDirect works with Amazon S3. Adding support for Google Storage for Developers should be fairly straight forward since the direct upload form is essentially the same. Please see the contributing section if you would like support for Google Storage for Developers or any other service that provides direct upload capabilities.
Please be aware that this gem (and S3 in general) only support single file uploads. If you want to upload multiple files simultaneously you'll have to use a javascript or flash uploader.
Install the latest release:
gem install carrierwave_directIn Rails, add it to your Gemfile:
gem 'carrierwave_direct'Note that CarrierWaveDirect is not compatible with Rails 2.
Please read the CarrierWave readme first
CarrierWaveDirect works with fog so make sure you have CarrierWave set up and initialized with your fog credentials, for example:
CarrierWave.configure do |config|
config.fog_credentials = {
:provider => 'AWS', # required
:aws_access_key_id => 'xxx', # required
:aws_secret_access_key => 'yyy', # required
:region => 'eu-west-1' # optional, defaults to 'us-east-1'
}
config.fog_directory = 'name_of_your_aws_bucket' # required
# see https://github.com/jnicklas/carrierwave#using-amazon-s3
# for more optional configuration
endIf you haven't already done so generate an uploader
rails generate uploader Avatarthis should give you a file in:
app/uploaders/avatar_uploader.rbCheck out this file for some hints on how you can customize your uploader. It should look something like this:
class AvatarUploader /foo.png
If you're *not* using Rails you can generate a direct upload form to S3 similar to [this example](http://doc.s3.amazonaws.com/proposals/post.html#A_Sample_Form) by making use of the CarrierWaveDirect helper methods.
### Sinatra
Here is an example using Sinatra and Haml
```ruby
# uploader_test.rb
CarrierWave.configure do |config|
config.fog_credentials = {
:provider => 'AWS',
:aws_access_key_id => ENV['AWS_ACCESS_KEY_ID'],
:aws_secret_access_key => ENV['AWS_SECRET_ACCESS_KEY']
}
config.fog_directory = ENV['AWS_FOG_DIRECTORY'] # bucket name
end
class ImageUploader @uploader.direct_fog_url, :method => "post", :enctype => "multipart/form-data"}
- @uploader.direct_fog_hash.each do |key, value|
- if key != :uri
%input{:type => "hidden", :name => key, :value => value}
%input{:type => "hidden", :name => "success_action_redirect", :value => @uploader.success_action_redirect}
%input{:name => "file", :type => "file"}
%input{:type => "submit", :value => "Upload to S3"}If you are using Rails and you've mounted your uploader like this:
class User
After uploading to S3, You'll need to update the uploader object with the returned key in the controller action that corresponds to new_user_url:
@uploader.update_attribute :avatar_key, params[:key]You can also pass html options like this:
{ :target => "_blank_iframe" } do |f| %>
Note if User is not an ActiveRecord object e.g.
class User
extend CarrierWave::Mount
extend CarrierWaveDirect::Mount
mount_uploader :avatar, AvatarUploader
endyou can still use the form helper by including the ActiveModel modules your uploader:
class AvatarUploader
# Content-Type will automatically be set to the default
You could use a manual select as well.
# Must tell the file_field helper not to include content type
Or you can use the helper which shows all possible content types as a select, with the default content type selected.
# Must tell the file_field helper not to include its own content type
…
ruby
class UsersController
then in your controller you can do something like this:
def create
@user = User.new(params[:user])
if @user.save_and_process_avatar
flash[:notice] = "User being created"
redirect_to :action => :index
else
render :new
end
endNow that the basic building blocks are in place you can process and save your avatar in the background. This example uses Resque but the same logic could be applied to DelayedJob or any other background processor.
class User true)
end
endThe method self.remote_avatar_url= from CarrierWave downloads the avatar from S3 and processes it. save then re-uploads the processed avatar to S3
Your users may find it convenient to upload a file from a location on the Internet via a URL. CarrierWaveDirect gives you another accessor to achieve this.
class User trueValidates that your mounted model has an avatar uploaded from file or specified by remote url. It does not check that an your mounted model actually has a valid avatar after the download has taken place. Turned off by default
validates :avatar, :is_attached => trueValidates that your mounted model has an avatar attached. This checks whether there is an actual avatar attached to the mounted model after downloading. Turned off by default
validates :avatar, :filename_uniqueness => trueValidates that the filename in the database is unique. Turned on by default
validates :avatar, :filename_format => trueValidates that the uploaded filename is valid. As well as validating the extension against the extension_allowlist it also validates that the upload_dir is correct. Turned on by default
validates :avatar, :remote_net_url_format => trueValidates that the remote net url is valid. As well as validating the extension against the extension_allowlist it also validates that url is valid and has only the schemes specified in the url_scheme_whitelist. Turned on by default
As well as the built in validations CarrierWaveDirect provides, some validations, such as max file size and upload expiration can be performed on the S3 side.
CarrierWave.configure do |config|
config.validate_is_attached = true # defaults to false
config.validate_is_uploaded = true # defaults to false
config.validate_unique_filename = false # defaults to true
config.validate_filename_format = false # defaults to true
config.validate_remote_net_url_format = false # defaults to true
config.min_file_size = 5.kilobytes # defaults to 1.byte
config.max_file_size = 10.megabytes # defaults to 5.megabytes
config.upload_expiration = 1.hour # defaults to 10.hours
config.will_include_content_type = 'video/mp4' # defaults to false; if true, content-type will be set
# on s3, but you must include an input field named
# Content-Type on every direct upload form
config.use_action_status = true # defaults to false; if true, you must set
# success_action_status in your uploader:
# uploader.success_action_status = "201"
# and add use_action_status to the file field:
# f.file_field :avatar, use_action_status: true'
endIf you are using a javascript uploader (e.g. Dropzone, jQuery Upload, Uploadify, etc.) you will need to add CORS configuration to your bucket, otherwise default bucket configuration will deny CORS requests. To do that open your bucket in Amazon console, click on its properties and add a CORS configuration similar to this
*
GET
POST
3000
Authorization
Content-Type
Origin
…
ruby
Fog.mock!
def fog_directory
ENV['AWS_FOG_DIRECTORY']
end
connection = ::Fog::Storage.new(
:aws_access_key_id => ENV['AWS_ACCESS_KEY_ID'],
:aws_secret_access_key => ENV['AWS_SECRET_ACCESS_KEY'],
:provider => 'AWS'
)
connection.directories.create(:key => fog_directory)If you're using Capybara with Cucumber or RSpec, CarrierWaveDirect gives you a few useful helpers. To get the Capybara helpers, include the module into your test file or helper
describe AvatarUploadSpec
include CarrierWaveDirect::Test::CapybaraHelpers
endTo attach a file to the direct upload form you can use
attach_file_for_direct_upload('path/to/file.ext')To simulate a successful upload and redirect to S3 you can use
upload_directly(AvatarUploader.new, "Upload to S3")This will click the Upload to S3 button on the form and redirect you to the success_action_redirect url (in the form) with a sample response from S3
To simulate an unsuccessful upload you can pass :success => false and you'll remain on the upload page e.g.
upload_directly(AvatarUploader.new, "Upload to S3", :success => false)You can also use find_key and find_upload_path to get the key and upload path from the form
If your mounted model validates a file is uploaded you might want to make use of the sample_key method
include CarrierWaveDirect::Test::Helpers
Factory.define :user |f|
f.email "[email protected]"
f.key { sample_key(AvatarUploader.new) }
endThis will return a valid key based off your upload_dir and your extension_allowlist
If you wanted to fake a download in the background you could do something like this
uploader = AvatarUploader.new
upload_path = find_upload_path
redirect_key = sample_key(:base => find_key, :filename => File.basename(upload_path))
uploader.key = redirect_key
download_url = uploader.url
# Register the download url and return the uploaded file in the body
FakeWeb.register_uri(:get, download_url, :body => File.open(upload_path))The Active Record validations use the Rails i18n framework. Add these keys to your translations file:
en:
errors:
messages:
carrierwave_direct_filename_taken: filename was already taken
carrierwave_direct_upload_missing: upload is missing
carrierwave_direct_attachment_missing: attachment is missing
carrierwave_direct_filename_invalid: "is invalid. "
carrierwave_direct_allowed_extensions: Allowed file types are %{extensions}
carrierwave_direct_allowed_schemes: Allowed schemes are %{schemes}If you're Ra
No open issues yet, or sync has not completed.