Secure, fast, low-resource file delivery

Here’s the problem. I have a handful of files between 500MB-800MB that I need to host up on my website. I need to get these files to people that belong to a certain vBulletin community, and only people of that vBulletin community. Furthermore, the people of that community aren’t the sharpest knives in the drawer, so it has to be easy. I have limited resources, which include a shared LAMP webserver, and a 150/150 VPS.

The requirements? Just a few:

• Must be able to track users to sessions to monitor and log abuse
• Links generated should be anti-leech and secured with one-time use tokens
• Links generated can only be accessible from this vBulletin community
• Links must be easy to use

The limitations? It’s like being handcuffed to a telephone pole:

• Hosting is done on shared hosting (no persistant readfile() sessions in PHP or other ugly CPU intensive scipting)
• Hosting can also be done on 150/150 VPS. I’ve used lighttpd, but it chokes on sending files that are bigger than my RAM capacity

Things I’ve tried, and failed to work:

• Doing the classic PHP readfile() and writing to the buffer
  • Far too slow
  • Persistant PHP session isn’t nice to shared hosting
  • readfile() in lighttpd and Apache cause the process to consume massive amounts of memory for large files, doesn’t work in VPS
  • More than 5 users or so and the load spikes to high heaven. Server /suicides
• Using lighttpd and mod_secdownload
  • Far faster than PHP readfile()
  • Will only work on VPS as shared hosting is Apache only
  • If you don’t have enough RAM (which I don’t on the VPS) – the process with explode and hang. I can’t kill it, and the server refuses connections until the watchdog resets the process or until I reboot the VPS. Server /suicides again.
• Using mod_rewrite in conjunction with rewritemap and prg in Apache
  • Sounds perfect – you can create tokens, and then revoke them with the rewritemap program, then use mod_rewrite to hide path source!
  • …Until you realize that you need to modify your apache.conf to specify the rewritemap program (requires root or sudo, not happening on either VPS or shared)
  • Your rewrite program is executed on server start, and is persistent – which is not allowed on shared hosting anyways. I /suicide
• Apache with mod_xsendfile
  • Can’t install custom modules on Apache in shared or VPS hosting
  • Apache is still a fat hog of a server, and using Apache to send binary files is about as efficient as using Humvees for mass transit.
• Dynamic mod_rewrite definitions in .htaccess files
  • Use a manual lockfile with a SQL database to dynamically generate .htaccess mod_rewrite definitions
  • Each rewrite entry for a unique URL a time limit
  • Each successful request for token updates both the database and the .htaccess file
  • This seems like the best solution – it will work with the most hosting environments too
  • This behavior is basically emulating mod_secdownload for lighttpd in Apache with mod_rewrite

The last entry (emulating mod_secdownload in Apache with mod_rewrite) looks good, and is probably going to be what I’m going to use – I’ll be posting the results here as soon as I finish coding it. Weeeee!