Site slowness - we need your help!

[Andy Clough]

UPDATE: WE NEED ANYONE WHO IS FINDING THE SITE VERY SLOW TO RUN SOME TESTS TO HELP US DIAGNOSE THE PROBLEM – SEE MY MOST RECENT POST FOR INSTRUCTIONS ON HOW TO DO THIS

Hi everyone

We're acutely aware that the site for logged-in users has been running painfully slowly over the last few weeks. After investigation by our tech team, it turns out we've been hit by a monumental amount of spam: our database currently holds 93 million records, and last month alone, there were approx 54,000 spam posts held by Mollom (the anti-spam system).

Yesterday we upgraded to a newer version of Mollom, and over the next few days we'll be purging all existing spam from the system, which hopefully will improve the site's speed. While this happens, you may be asked to fill in a 'Captcha' box when posting on the Forums if the system thinks your post might be spam. We think this mild inconvenience is worth it to help us improve the site's performance and will be monitoring the situation closely. Apologies in advance if the occasional post accidentally gets deleted. Thanks for your patience and understanding.

 

[jjbomber]

54,000 spam posts? The AVI boys have been busy :rofl: :rofl: :rofl: :rofl:

 

[MajorFubar]

Thanks for the update Andy.

 

[Andrew Everard]

54,000 spam posts? The AVI boys have been busy :rofl: :rofl: :rofl: :rofl:

 

[relocated]

54,000 spam posts? The AVI boys have been busy :rofl: :rofl: :rofl: :rofl:

Oh, you boys are such a hoot. Shame that to get into this thread, about speeding things up, took 1minute and 53 seconds. :doh:

Edit: Then 1minute 14seconds to place the above post.

 

[Andrew Everard]

Oh, you boys are such a hoot. Shame that to get into this thread, about speeding things up, took 1minute and 53 seconds. :doh:

Edit: Then 1minute 14seconds to place the above post.

As you can imagine, purging that amount of spam data from the servers is going to take some time: I gather we're talking some days, maybe even longer, to get rid of it all.

So it's not an instant fix, but once it's all cleared we are assured things will speed up.

 

[BigH]

Going even slower now, takes about 1 min to open each post. 15.47 BST.

 

[Andy Clough]

Yes, it will take several days to clear the backlog of spam posts, so we don't expect to see any improvement until next week. I never said it would be an instantaneous fix

 

[BigH]

Why has it built up so much? I clear mine several times a day.

 

[Andy Clough]

We assumed Mollom was deleting the spam automatically. It wasn't.

 

[BigH]

I see.

 

[simonlewis]

54,000 spam posts, that is a lot, i'm speechless, how awfull.

 

[drummerman]

There was an AVAST alert a couple of days ago. Apparently one of the major Anti Spam sites has been attacked in retaliation.

If I remember right, the attack was partly dealt with by spreading it to different servers around the globe causing slow down and problems.

Something to do with that perhaps?

regards

 

[Andrew Everard]

Something to do with that perhaps?

No, nothing to do with that, AFAICS: it seems the spam has just been pouring in over an extended period of time, with the number of posts increasing as the site has become more visible, and overall traffic has grown.

Several years back, a spam post was a novelty, an occasional annoyance we could deal with manually; but now that has turned into a deluge.

We have several thousand sign-ups a week from would-be spammers, sometimes with a block of similar nonsense usernames (just a jumble of letters) being signed up within a few minutes, suggesting this is being done on an industrial scale, and (for example) getting on for 10,000 posts in our 'spam corral' whose subject is a particular brand of expensive handbag.

I guess it's just a case of the more visible a site becomes to the internet as a whole, and moreover to its search engines, the more attractive it is for spammers and others who want to try any means to get their product and their message in front of as many users as possible.

 

[Andy Clough]

Hi everyone

We're conducting further tests on why the site is running so slowly, and our internet ops team have identified a problem with the cacheing on our CDN server, which particularly seems to affect users on Virgin Media (users on our sister site PistonHeads are experiencing the same problem). They've asked if users who are having difficulty getting the site to load could run a traceroute by following the instructions below, then we can analyse the results:

In order to facilitate the debugging of the slowness we need to run tracert on some of the users computers. This is going to help Limelight to pinpoint the issues to certain geographical areas and subsequently identify the points of presence which have the issue.

Tracert capabilities are explained here: http://support.microsoft.com/kb/162326

The best guide on how to use it can be found here: http://pcsupport.about.com/od/commandlinereference/p/tracert-command.htm

The users can either provide us with screenshots of the command prompt windows or copy/paste the content of the window.

To start the command prompt:

Start > All programs > Accessories > Command Prompt

In the command prompt window write :

tracert cdn.code.haymarket.com

press Enter and wait for the command to complete

Either screenshot the window and post it to the forum thread or right click on the window > Mark > select all the content of the window > right click again will act as copy > paste the content into the forum reply box.

For Mac:

1. Go to the Applications folder in the Finder.

2. Go into the Utilities folder and double click on the Network Utility to open it.

3. Click on the Traceroute tab and then type cdn.code.haymarket.com into the box and click the Trace button.

4. Once completed, copy and paste the results from the results box into a new reply on the forum.

Your help is greatly appreciated.

 

[simonlewis]

I'm doing it right now, how do you screen shot. ?

 

[John Duncan]

Tracing route to haymarket.vo.llnwd.net [178.79.195.246]
over a maximum of 30 hops:

1 36 ms 1 ms 1 ms 192.168.79.1
2 32 ms 31 ms 32 ms gateway.uk.psie.net [154.32.253.1]
3 32 ms 33 ms 32 ms reserved.uk.murphx.net [109.170.249.82]
4 33 ms 33 ms 33 ms 154.32.0.1
5 46 ms 32 ms 41 ms tge1-4.fr3.lon.llnw.net [195.66.224.133]
6 32 ms 34 ms 36 ms ve8.fr3.lcy1.llnw.net [68.142.125.90]
7 * 68 ms 57 ms cdn-178-79-195-246.lcy.llnw.net [178.79.195.246]

Trace complete.

Note that this isn't giving me a particular problem at the moment as am at work. I did notice that cdn.code appeared to be taking forever last night (though didn't run a tracert, and am on Virgin there)

 

[simonlewis]

Microsoft Windows [Version 6.0.6002]
Copyright (c) 2006 Microsoft Corporation. All rights reserved.

C:Userssimon lewis>tracert cdn.code.heymarket.com

Tracing route to cdn.code.heymarket.com [107.20.206.69]
over a maximum of 30 hops:

1 1 ms <1 ms <1 ms Broadcom.Home [192.168.1.1]
2 13 ms 13 ms 13 ms 86.54.183.225
3 13 ms 13 ms 13 ms 91.85.10.113
4 13 ms 13 ms 13 ms 86.54.135.181
5 13 ms 13 ms 13 ms supanet.v18.lon-th3cr.spn.kcom.com [86.54.183.12
1]
6 13 ms 13 ms 13 ms ldn-s10-rou-1041.UK.eurorings.net [134.222.146.5
2]
7 14 ms 14 ms 13 ms ldn-s2-rou-1041.UK.eurorings.net [134.222.231.29
]
8 96 ms 94 ms 95 ms ldn-s2-rou-1101.UK.eurorings.net [134.222.228.93
]
9 94 ms 97 ms 94 ms nntr-s1-rou-1101.FR.eurorings.net [134.222.232.1
82]
10 88 ms 89 ms 89 ms nyk-s2-rou-1021.US.eurorings.net [134.222.226.16
5]
11 96 ms 95 ms 95 ms ahbn-s1-rou-1041.US.eurorings.net [134.222.227.1
34]
12 94 ms 95 ms 95 ms ahbn-s1-rou-1001.US.eurorings.net [134.222.226.5
3]
13 95 ms 95 ms 95 ms 72.21.221.132
14 95 ms 95 ms 95 ms 72.21.220.53
15 95 ms 131 ms 95 ms 72.21.222.149
16 * * * Request timed out.
17 * * * Request timed out.
18 * * * Request timed out.
19 106 ms 98 ms 99 ms 216.182.232.51
20 * * * Request timed out.
21 * * * Request timed out.
22 * * * Request timed out.
23 * * * Request timed out.
24 * * * Request timed out.
25 * * * Request timed out.
26 * * * Request timed out.
27 * * * Request timed out.
28 * * * Request timed out.
29 * * * Request timed out.
30 * * * Request timed out.

Trace complete.

C:Userssimon lewis>

 

[Andy Clough]

I'm doing it right now, how do you screen shot. ?

It's shift/apple/4 on a Mac, Alt-print/screen on a PC which copies it to the clipboard, then save it as a jpeg file.

 

[John Duncan]

I'm doing it right now, how do you screen shot. ?

On windows - Alt-PrintScreen will copy the active window into the clipboard. You can then run MSPaint and paste the screenshot into there (Ctrl-V), and save it as a jpg file.

Then host it on flickr or whatever and paste it here. If you can't be -rsed with the hosting/flickr thing, mail the file to me at the address below and I'll sort it out for you.

 

[chebby]

Microsoft Windows [Version 6.0.6002]
Copyright (c) 2006 Microsoft Corporation. All rights reserved.

C:Userssimon lewis>tracert cdn.code.heymarket.com

Tracing route to cdn.code.heymarket.com [107.20.206.69]
over a maximum of 30 hops:

1 1 ms <1 ms <1 ms Broadcom.Home [192.168.1.1]
2 13 ms 13 ms 13 ms 86.54.183.225
3 13 ms 13 ms 13 ms 91.85.10.113
4 13 ms 13 ms 13 ms 86.54.135.181
5 13 ms 13 ms 13 ms supanet.v18.lon-th3cr.spn.kcom.com [86.54.183.12
1]
6 13 ms 13 ms 13 ms ldn-s10-rou-1041.UK.eurorings.net [134.222.146.5
2]
7 14 ms 14 ms 13 ms ldn-s2-rou-1041.UK.eurorings.net [134.222.231.29
]
8 96 ms 94 ms 95 ms ldn-s2-rou-1101.UK.eurorings.net [134.222.228.93
]
9 94 ms 97 ms 94 ms nntr-s1-rou-1101.FR.eurorings.net [134.222.232.1
82]
10 88 ms 89 ms 89 ms nyk-s2-rou-1021.US.eurorings.net [134.222.226.16
5]
11 96 ms 95 ms 95 ms ahbn-s1-rou-1041.US.eurorings.net [134.222.227.1
34]
12 94 ms 95 ms 95 ms ahbn-s1-rou-1001.US.eurorings.net [134.222.226.5
3]
13 95 ms 95 ms 95 ms 72.21.221.132
14 95 ms 95 ms 95 ms 72.21.220.53
15 95 ms 131 ms 95 ms 72.21.222.149
16 * * * Request timed out.
17 * * * Request timed out.
18 * * * Request timed out.
19 106 ms 98 ms 99 ms 216.182.232.51
20 * * * Request timed out.
21 * * * Request timed out.
22 * * * Request timed out.
23 * * * Request timed out.
24 * * * Request timed out.
25 * * * Request timed out.
26 * * * Request timed out.
27 * * * Request timed out.
28 * * * Request timed out.
29 * * * Request timed out.
30 * * * Request timed out.

Trace complete.

C:Userssimon lewis>

You spelt Haymarket as 'Heymarket'.

 

[simonlewis]

Thanks both, already copy & pasted.

 

[simonlewis]

@ chebby, -does that mean i have to do it again ?

 

[chebby]

Tracing route to haymarket.vo.llnwd.net [87.248.210.254]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms 10.0.1.1
2 8 ms 11 ms 29 ms cpc4-pmth11-2-0-gw.6-1.cable.virginmedia.com [213.104.148.1]
3 10 ms 7 ms 7 ms cosh-core-1b-ae1-1114.network.virginmedia.net [80.3.163.245]
4 11 ms 11 ms 71 ms glfd-bb-1b-ae4-0.network.virginmedia.net [212.43.163.213]
5 10 ms 11 ms 11 ms glfd-bb-1a-ae0-0.network.virginmedia.net [213.105.172.5]
6 15 ms 14 ms 15 ms popl-bb-1b-ae7-0.network.virginmedia.net [213.105.159.62]
7 14 ms 22 ms 22 ms popl-bb-1c-ae1-0.network.virginmedia.net [213.105.159.190]
8 * * * Request timed out.
9 17 ms 23 ms 24 ms tge7-6.fr3.lon.llnw.net [87.248.208.141]
10 28 ms 23 ms 29 ms ve5.fr4.lon.llnw.net [69.28.171.138]
11 68 ms 18 ms 24 ms cdn-87-248-210-254.lon.llnw.net [87.248.210.254]

Trace complete.

Another one....

Tracing route to haymarket.vo.llnwd.net [87.248.210.254]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms 10.0.1.1
2 129 ms 43 ms 37 ms cpc4-pmth11-2-0-gw.6-1.cable.virginmedia.com [213.104.148.1]
3 12 ms 8 ms 10 ms cosh-core-1b-ae1-1114.network.virginmedia.net [80.3.163.245]
4 11 ms 11 ms 11 ms glfd-bb-1b-ae4-0.network.virginmedia.net [212.43.163.213]
5 11 ms 11 ms 11 ms glfd-bb-1a-ae0-0.network.virginmedia.net [213.105.172.5]
6 12 ms 15 ms 22 ms popl-bb-1b-ae7-0.network.virginmedia.net [213.105.159.62]
7 16 ms 13 ms 31 ms popl-bb-1c-ae1-0.network.virginmedia.net [213.105.159.190]
8 13 ms 15 ms 15 ms tele-ic-5-ae0-0.network.virginmedia.net [213.105.159.117]
9 21 ms 28 ms 19 ms tge7-6.fr3.lon.llnw.net [87.248.208.141]
10 18 ms 22 ms 15 ms ve5.fr4.lon.llnw.net [69.28.171.138]
11 19 ms 19 ms * cdn-87-248-210-254.lon.llnw.net [87.248.210.254]
12 25 ms 19 ms 19 ms cdn-87-248-210-254.lon.llnw.net [87.248.210.254]

Trace complete.

 

[simonlewis]

Microsoft Windows [Version 6.0.6002]
Copyright (c) 2006 Microsoft Corporation. All rights reserved.

C:Userssimon lewis>tracert cdn.code.haymarket.com

Tracing route to haymarket.vo.llnwd.net [87.248.210.254]
over a maximum of 30 hops:

1 1 ms 1 ms <1 ms Broadcom.Home [192.168.1.1]
2 13 ms 13 ms 13 ms 86.54.183.225
3 14 ms 13 ms 12 ms 91.85.10.117
4 13 ms 13 ms 13 ms 62.164.130.61
5 13 ms 13 ms 15 ms supanet.v19.lon-gs2cr.spn.kcom.com [86.54.183.19
3]
6 15 ms 23 ms 27 ms tge1-4.fr3.lon.llnw.net [195.66.224.133]
7 14 ms 24 ms 29 ms ve5.fr4.lon.llnw.net [69.28.171.138]
8 14 ms 13 ms 42 ms cdn.code.haymarket.com [87.248.210.254]

Trace complete.

C:Userssimon lewis>