New page caching implemented on the blogs.law.harvard.edu server

We’ve implemented a new page caching system that should help improve response time and ensure better uptime for the blogs.law.harvard.edu server. A “page cache” will intercept requests and serve pages WITHOUT invoking all of the wordpress code – allowing our site to serve many more requests over all. The page cache only comes in to play for non-logged-in users, and for those that haven’t posted a comment recently to a blog we host.

The page cache has one significant side effect: WP-slimstats will be even less accurate, as most pages served to non-logged in users come directly from a page cache. This means that WP-slimstats doesn’t know about all traffic. Wp-Slimstats over-counts illegitimate traffic and is inefficient, unsupported and inaccurate – we’re looking into other options to provide some kind of analytics for our sites that will work with the new page cache.

The backstory:

We get a significant amount of legitimate – and not so legitimate – traffic. We served over 3 million legitimate page views and around 300,000 unique visitors this August – on top of all the ‘bot traffic. Badly behaved ‘bots request too many pages simultaneously and can cause a significant service interruption in combination with all the legitimate traffic we’re already handling.

Our traffic – bad and good – has been increasing over time and we have been seeing more service interruptions due to high load: we have reached the point where we simply can’t continue to provide a high quality service without a page cache in place. We’ve already implemented a behaviour based ‘bot catcher and numerous other tricks to optimize how we serve content: the page cache is the newest weapon in our arsenal.

Thanks for your patience! The site should feel snappier. If you want to browse a fully cached version of your site, log out and clear any “blogs.law.harvard.edu” cookies. Logging out by itself isn’t enough, you have to clear your blogs.law cookies, too.