Right. Those of you who stayed up late on Thursday to watch the launch of the new site (and our second attempt on Friday morning) are probably wondering what the hell happened, so in a spirit of openness I thought I'd talk you through the issues we faced, what we're doing about them, and when you can expect us to have another go! It really boiled down to three issues - importing the remaining site content which had built up since the last time we did it, the unexpectedly heavy load that was placed on the site the moment we went live, and the supporting infrastructures response to that load. Importing new content We had the site up to date apart from forum posts, which hadn't been updated since about April 20th - about 15,000 new posts altogether. In order to import all of those, we had to make sure that any new users who'd posted since that date were imported, then import the threads, then import all the replies - after that we had to create the URLs for them (the new forum has pretty browser links for each forum thread - like whathifi.com/forum/what-are-you-listening-to instead of community.whathifi.com/forums/t/57127.aspx), make sure they all matched up and were in the right forum, etc etc. We'd anticipated that this was going to take about an hour, but in the end it took about eight - the new user import in particular didn't want to play nice, and for a reason that we didn't get to the bottom of in the time alloted, we lost the turntables and headphones forums altogether. In the end, however, we got there (except for turntables and headphones, which we decided we would go live without and fix the next day), and we readied for launch... Unprecedented Load Just to give you some background, the new site behaves differently from a technical point of view if you're logged in or you're not. If you're not logged in, the site serves up cached HTML pages (provided they've been visited before), and that cache is refreshed at short intervals. Therefore, if nobody's logged in at all, the site doesn't hit the database to get fresh content until it's told to. Conversely, if you are logged in, you get fresh content all the time - practically none of the content is cached and therefore every page you see is a database hit. With that in mind, the most logged in users we normally see is about 20 - the most posts we've ever had in the space of a minute is 21, which whilst not exactly a measure of how many people were logged in at that point, it gives us an idea of our normal authenticated user base. When we went live at around 11pm on Thursday, we saw database activity that suggested we had hundreds of people logging in - ten times our usual activity. As you might expect, the majority of this activity was centred around forum landing pages - the latest posts page and the main forums landing page - and since everybody was logged in (or trying), the pages were all calling the database. Now we did notice that some of those database queries were suboptimal, and needed improving straight away, but we hoped we'd be able to weather the storm, albeit at fairly slow speed, until... Self-Inflicted Hardware Meltdown The new site runs on a public cloud. Which means that, rather than have physical servers that are limited by memory and processing power, we have access to highly scalable hardware which can cope with peaks and troughs in traffic seamlessly. However, this (theoretically) infinitely scalable hardware does have its limits, and when we put our site live, the enormous database load that we suddenly created threatened to bring down lots of other sites on the same cloud, and our provider, as is their right, shut our site down. The reason that we tried twice was that a red herring of a broken server appeared late on Thursday, which made us think that we'd been unlucky so we tried again on Friday. Turned out that the reason the server was broken was probably us! So just after lunch Friday we decided that enough was enough and reverted to the old site, while we licked our wounds (and had a pint)... Now, we've assessed over the weekend and today and think we can address the first two issues above pretty quickly, which kind of make the third one go aay. So we're going to make those changes today and tomorrow, and then hit the site with some heavy load to make sure that it can cope with the activity we saw Thursday and Friday, with a view to going live again towards the end of next week. Any questions, I know you won't be afraid to ask...
Facebook
Twitter
Instagram