Send files faster with X-Sendfile | John Guenin

Send files faster with X-Sendfile

Rails' send_file has a beautiful, simple interface. Unfortunately, it ties up your Rails process with the rather mundane task of reading a file from disk. If you use Mongrel, you'll quickly notice that the :stream option doesn't seem to make any difference in memory usage and your mongrel process stubbornly doesn't want to let go of that memory once the file has been sent. This is a very, very bad thing.

Please don’t let this post scare you away from Mongrel. I have found it much more stable and easier to set up than FCGI. Mongrel + Capistrano saved me from the brink of insanity when deployment came around. The memory issue with send_file is indicative of a bigger issue with Ruby’s garbage collection—not a problem with Mongrel itself.

There’s this slick HTTP header called “X-Sendfile” that Apache and Lighttpd support. It was made to address this kind of problem (isn’t that convenient?). Now, when you send an HTTP response, all you have to do is tack on an X-Sendfile header with the path of the file you need to send—don’t worry about actually reading the file or sending any of those bytes yourself. The web server will load the file you specified and send it downstream.

This is loads easier on your Rails process (it just sets the headers and gets on with life), and you get all the cool file transfer functionality already baked into your web server (like proper caching, resuming, etc). Of course, you can send any file that the web server can read, not just files in your normal public directory. This makes it perfect for sending private or protected content to specific users of your app.

Setting those magical headers

You shouldn’t be too surprised that Rails makes it easy to set this header correctly. You could do something like this:

This code sample has been excerpted from the Ruby on Rails Wiki.

filename = "/var/www/myfile.xyz" 
response.headers['Content-Type'] = "application/force-download"
response.headers['Content-Disposition'] = "attachment; filename=\"#{File.basename(filename)}\""
response.headers["X-Sendfile"] = filename
response.headers['Content-length'] = File.size(filename)
render :nothing => true

Apache users will need mod_xsendfile. Lighttpd users should look into mod_fastcgi and mod_proxy_core.

Now, as long as your server is correctly configured, you file should be zipping along at record speeds. If you need this functionality in more than one action or more than one app it’s not terribly DRY… and you should probably write some functional tests for this… ugh. Someone really ought to write a plugin.

The XSendfile plugin

The advanced or impatient may want to jump straight to the rdoc or source.

I wrapped up all this goodness in a plugin. I’ve had this running on a production server (Apache2 + Mongrel) for a couple of weeks with no issues. For the good of humanity, it comes complete with rdoc and tests.

Installation

You can install the plugin using Rails’ built-in plugin command.

ruby script/plugin install http://john.guen.in/svn/plugins/x_send_file/

Usage

The interface is as close as possible to Rails’ send_file command. If your server requires it, you can override the HTTP header used for X-Sendfile with the new :header option (the default is X-Sendfile).

x_send_file('/path/to/file')
x_send_file('/path/to/image.jpg', :type => 'image/jpeg', :disposition => 'inline')
x_send_file('/path/to/file/', :header => 'X-LIGHTTPD-SEND-FILE')

If you want x_send_file jump in and take over any time send_file is used, add this to your environment.rb:

That ! stands for “danger!” Use this method responsibly.

XSendFile::Plugin.replace_send_file!

More information

If you’re ready to dig further, check out these resources:

Comments

There are 16 comments on this post.  Post yours →

hi! first, it’s a great feature! my problem is, that i store all the blob data in the database. so there is no direct access to the files for apache. is there a possibility to tell mongrel to fetch the blobs and then first write them to the disk before telling apache where to get them? thanx!

Kostya August 07, 2007 at 04:24 AM

I was having trouble on a Windows install. Apache didn’t want to access the file and was giving me a 200 response page but a 404 response. Weird. However, it turns out that apache didn’t like the rails way of linking to a file. all that …

./script/../config/../[insert your directory file files here]

… stuff was making Apache on windows unhappy. In order to alleviate this issue I had to resort to using File.expand_path… like so.

xsendfile File.expandpath(file.publicfilename)

FYI

PS - Is it considered bad form (read, potential security risk) to output to the production log a line saying that this plugin is sending the x-sendfile header with this server file location, etc.? I don’t know if I like the file to contain the actual server location of the file I’m sending. Is there a config line that we can use to turn off plugin logging?

otherwise, sweet plugin! thanx.

(e)

matte October 02, 2007 at 05:11 PM

and actually I changed it to determine the platform since it’s working beautifully on my freebsd production box…

filename = RUBYPLATFORM =~ /mswin32/ ? File.expandpath(@asset.publicfilename) : @asset.publicfilename xsendfile filename

matte October 02, 2007 at 05:52 PM

I can’t manage to have X-Sendfile work with Rails. I’m sure my Apache setup is OK since X-Sendfile works fine with PHP. With Rails 1.1.6 on Mac OS X, I always get a zero Ko file as a result (response.headers or xsendfile plugin both give the same result) after a few seconds wait. Any idea ?

Cyril October 12, 2007 at 06:32 AM

Update to my concern about outputting a full server path to a file showing up in the log. Here’s a sooper simple patch to only show the filename, for security reasons…

http://pastie.caboo.se/116075

(e)

matte November 09, 2007 at 02:47 PM

221195076097_serv378431 http://milfxxxpass.com#0 - milf older milf sex [URL=http://milfxxxpass.com#2] hot milf[/URL] [http://milfxxxpass.com#3 milf next doors] [link]http://milfxxxpass.com#4[/link]

iSHMAel@ku.com95 November 14, 2007 at 08:50 PM

Great post, helped me understand X-Sendfile really easily.

You should be pleased to know that edge Rails now allows you to do sendfile :xsendfile => true and it sets the proper response headers and returns a HEAD response.

See http://dev.rubyonrails.org/changeset/8628 for the changeset.

Chu Yeow January 13, 2008 at 06:07 AM

Thanx for this plugin! I have one problem, please help me!

I put my private files in to ‘tmp/files’ folder. xsendfile plugin is installed.

My controller has this code: xsendfile(‘tmp/files/for_download.pdf’)

Folowing this action i download this pdf file with the correct name “for_download.pdf”, but the file is broken and its size is only 1kb

Vyacheslav March 18, 2008 at 07:53 AM

@Vyacheslav: 1 byte, you mean, right? that happens to me too. Seems like if I am using the x_sendfile parameter, I never get the actual file size, only 1 byte. Always.

Attila September 10, 2008 at 11:58 AM

@Vyacheslav, Attila: you must give absolute path to a file, because Apache is not aware of Rails current directory. Try this one: xsendfile(“#{Rails.root}/tmp/files/for_dowload.pdf”)

gertas October 01, 2008 at 02:49 AM

Hi, are you likely to clone this up to github? I’m happy to if not, but thought it sensible to ask first in case you wanted to maintain it on github. Mostly because you can pull in using a git submodule quite easily, saving maintaining the code within our own applications.

Tom Simnett December 08, 2008 at 10:55 AM

any github repo? Thanks

Daniel Lopes December 22, 2008 at 03:21 PM

One of the best file centers is Megaupload! For a proper search and downloading use http://megauploadfiles.com/

klever January 07, 2009 at 07:36 AM

Hi,

I’m also having problems with the download of a PDF file. I’m using the following code:

def sendthefile(filename) xsendfile (“#{Rails.root}/doc/” + filename, :type => ‘application/pdf’) end

and I’ve checked that the file has the necessary permissions but I still only get 1byte downloaded. Any ideas? This is running on localhost by the way, although I get the same on my prod site.

TIA,

Alan

Alan February 14, 2009 at 06:34 AM

I have been trying to install mod_xsendfile for apache and to get it working with no succees…

I have test.php: $file = “video.mp4”; header(“X-Sendfile: “. $file); header(“content-type: video/mp4”);

and .htaccess file in same directory that contains:

XSendFile on

I’m using flowplayer (www.flowplayer.org) to read the mp4 file trough test.php and flowplayer gives an error “streamnotfound”. If i use readfile($file) in php-script. it works…

apache2ctl -t -D DUMPMODULES shows that xsendfilemodule is installed.

Do you have any ideas what would be wrong?

I really really appreciate your help!

Hese March 05, 2009 at 02:12 PM

I’ve replaced underscores with dashes so they’re visible in this comment.

There is a documentation bug in x-send-file.rb: instead of saying # The normal send-file method can still be accessed using send-file-without-xsendfile. it should say # the normal send-file method can still be accessed using send-file-without-x-send-file.

Responses

Please Login to respond

Get Gleanr!

What is Gleanr?

Gleanr is the networking engine for digital-age professionals. Get impact (& income!) in the information streams you care about.

How does it work?

Your custom Gleanr channels automate information flow relevant to you. All you do is "click" - we do the rest (instant capture, indexing, and networking).

What is the value?

Gleanr is the only web service where professionals can manage and monetize their expertise.

Is this more web 2.0?

Yes, but for work. Now you can capitalize on your unique ability to filter and enrich professional information streams.

Show me!

Explore the public parts of professional information streams here, or take the Gleanr Tour.

Sign me up!