How can large files be sent efficiently in Ruby on Rails with authentication and caching?

Sending large files in web applications can be challenging. When you add the need to verify permissions to that, things become even more complicated. Fortunately, Rails offers elegant solutions to this problem. In this article, I will demonstrate how to effectively handle large files while maintaining control over access and utilising cache mechanisms.

The problem with sending big files

Imagine you have a Ruby on Rails application that needs to handle files of several hundred megabytes in size, such as architectural projects, training videos or data packages. The standard approach, where the Rails application loads the file into memory and then sends it to the user, has serious drawbacks:

It uses a lot of server memory
It puts a strain on the application server, which is not optimised for this task.
It slows down the processing of other parallel requests.

Fortunately, there is a better solution: delegating the sending of files to an HTTP server while maintaining the Rails authentication logic.

X-Sendfile to the rescue

X-Sendfile (or variants such as X-Accel-Redirect for Nginx) is an HTTP header that changes the way files are served. It works like this:

The user requests access to a file
The Rails application checks permissions
Instead of sending the file itself, Rails adds an X-Sendfile header with the file's location.
The HTTP server (e.g. Nginx or Apache) takes over the task and effectively sends the file.

Let's take a look at what this looks like in code:

class DocumentsController < ApplicationController
  before_action :authenticate_user!

  def download
    @document = Document.find(params[:id])

    # check permissions
    unless current_user.can_access?(@document)
      return head :forbidden
    end

    # prepare response headers
    response.headers["Content-Type"] = @document.content_type
    response.headers["Content-Disposition"] = "attachment; filename=\"#{@document.filename}\""

    # crucial part - delete the file sending to the HTTP server
    response.headers["X-Sendfile"] = @document.file_path

    head :ok
  end
end

Configuring NGINX server

In order for X-Sendfile to work with Nginx, we need to configure the server accordingly.

server {
  listen 80;
  server_name mojasuperapka.pl;

  root /var/www/mojapp/public;

  location / {
    proxy_pass http://upstream_rails_app;
    ... other stuff
  }

  # crucial part for X-Accel-Redirect (X-Sendfile variant Nginx)
  location /protected-files/ {
    internal; # folder not accessible from outside
    alias /var/www/mojapp/storage/;
  }
}

Implementation with cache considerations

What about the cache? We can combine X-Sendfile with caching mechanisms.

class DocumentsController < ApplicationController
  before_action :authenticate_user!

  def download
    @document = Document.find(params[:id])

    unless current_user.can_access?(@document)
      return head :forbidden
    end

    # check headers: If-Modified-Since, If-None-Match
    if stale?(etag: @document.file_hash, last_modified: @document.updated_at)
      response.headers["Content-Type"] = @document.content_type
      response.headers["Content-Disposition"] = "attachment; filename=\"#{@document.filename}\""

      # set up cache
      response.headers["Cache-Control"] = "private, max-age=3600"

      # set X-Sendfile
      response.headers["X-Sendfile"] = @document.file_path

      head :ok
    end
    # if not stale, then it will return 304 Not Modified
  end
end

Testing in the development environment

In the Rails development environment, Nginx or Apache do not work by default. Fortunately, the gem send_file emulates the X-Sendfile function.

# config/environments/development.rb
config.middleware.insert(0, Rack::SendFile)
config.action_dispatch.x_sendfile_header = nil # turn off real x-sendfile

And in the controller:

def download
  @document = Document.find(params[:id])

  if Rails.env.development?
    # for development, use Rack::SendFile
    send_file @document.file_path, 
              type: @document.content_type,
              disposition: "attachment",
              filename: @document.filename
  else
    # on production, use X-Sendfile
    response.headers["Content-Type"] = @document.content_type
    response.headers["Content-Disposition"] = "attachment; filename=\"#{@document.filename}\""
    response.headers["X-Sendfile"] = @document.file_path
    head :ok
  end
end

Other HTTP servers

For different HTTP servers, headers may differ:

Apache uses X-Sendfile.
Nginx uses X-Accel-Redirect. Thruster (used by Kamal) also supports the X-Accel-Redirect header.

In Rails, this can be configured:

# config/environments/production.rb
config.action_dispatch.x_sendfile_header = 'X-Accel-Redirect' # for nginx
# config.action_dispatch.x_sendfile_header = 'X-Sendfile' # for apache

When is it worth using X-Sendfile?

X-Sendfile is particularly useful when:

You are sending large files (videos, ZIP, RAR, PDFs).
You require authentication /authorization before sending a file.
You encounter performance or memory issues when handling files.

However, this may not be necessary in the case of:

Tiny files
Dynamically generated content for each request
Developer environment (although we can emulate it).

Example of a comprehensive solution:

Below is a more detailed example of file handling that takes into account various scenarios:

class SecureFilesController < ApplicationController
  before_action :authenticate_user!

  def download
    @file = SecureFile.find(params[:id])
    # authorize the user
    authorize! :download, @file
    # complicated logic
    @file.register_download(current_user)
    # support range requests for large files
    response.headers["Accept-Ranges"] = "bytes"

    # check cache
    if @file.public?
      # public files can be cached via proxy
      response.headers["Cache-Control"] = "public, max-age=86400"
    else
      # private files, only cached by the browser
      response.headers["Cache-Control"] = "private, max-age=3600"
    end

    # prepare header for response
    response.headers["Content-Type"] = @file.content_type
    response.headers["Content-Disposition"] = @file.force_download? ? 
      "attachment; filename=\"#{@file.filename}\"" : 
      "inline; filename=\"#{@file.filename}\""

    # set proper x-sendfile header
    x_sendfile_header = Rails.application.config.action_dispatch.x_sendfile_header

    case x_sendfile_header
    when "X-Sendfile" # apache
      response.headers["X-Sendfile"] = @file.file_path
    when "X-Accel-Redirect" # nginx
      # nginx expects a relative path
      response.headers["X-Accel-Redirect"] = "/protected-files/#{@file.relative_path}"
    else
      # no x-sendfile? go back with normal send_file (for dev or test for example)
      return send_file @file.file_path, 
                      type: @file.content_type,
                      disposition: @file.force_download? ? "attachment" : "inline",
                      filename: @file.filename
    end

    head :ok
  end
end

Summary

X-Sendfile is a powerful tool that allows you to efficiently serve large files in Ruby on Rails applications while maintaining full control over access. Although implementing X-Sendfile requires proper HTTP server configuration, the performance benefits are worthwhile, especially when your application needs to handle large files for multiple users simultaneously.

Happy downloading!