So I updated the site code to do some more caching. Things are going quite a bit faster between the Varnish ESI caching and the redis caching for fragments and such.
Still need to figure out how to get the gallery pages' images caching properly, but as long as the main page and reply trees are cached, the rails app itself will bear less of a burden.