Comments for "Why Sometimes it takes a long time for a Sift page to load" Blog at VideoSift.com

By dag

dag (http://dag.videosift.com) — Thu, 17 Apr 2008 15:05:50 -0700

If anyone has any good ideas on what might make webservers spike on a regular interval like this - we'd love to hear your crackpot theories.

By Thylan

Thylan (http://www.videosift.com/member/Thylan) — Thu, 17 Apr 2008 15:15:24 -0700

Automated google web crawling because they recognize our premier ness on their search categories for all vids of import.

No genuine answers though. Do the spikes link to some freak net traffic spikes or is it just their own load that has this pattern?

By maatc

maatc (http://maatc.videosift.com) — Thu, 17 Apr 2008 15:49:39 -0700

I know nothing about this, but it does seem automated.
A quick search for "server spike automated" over at google brought up this:

http://www.xav.com/scripts/search/help/1177.html

Could it be that?

By jwray

jwray (http://jwray.videosift.com) — Thu, 17 Apr 2008 17:08:20 -0700

It could be that people go to videosift first thing after the end of whatever TV show they were watching that ended on the hour. What does the net traffic graph look like?

Putting a 15 second cooldown between searches from the same IP is a good idea regardless. Someone could easily DDoS by spamming search queries. And make sure you disallow the search page in your robots.txt.

By dag

dag (http://dag.videosift.com) — Thu, 17 Apr 2008 19:44:32 -0700

It could be Googlebot- but they usually play nice- and our new search tool is pretty lightweight/not load producing.

By maatc

maatc (http://maatc.videosift.com) — Fri, 18 Apr 2008 07:37:10 -0700

>> ^jwray:
It could be that people go to videosift first thing after the end of whatever TV show they were watching that ended on the hour.

Like a server version of the superbowl water pressure myth?

By blankfist

blankfist (http://blankfist.videosift.com) — Fri, 18 Apr 2008 19:53:50 -0700

It would sound a lot like spambots or searchbots, though I can't imagine they'd be pounding your servers at such a regular interval. Just in case, you could think about globally changing href links to no follow to keep bot indexing down.

I think you're using php, so you could look into a code accelerator like eAccelerator, though don't hold me accountable if your Xeon machines start spewing sparks. And if you're using MySQL, here's a nifty article on query caching which you may or may not find useful. Dunno. If it is rogue bot traffic, I've heard of people using bot traps, as well.

I hope you find some of this helpful at least.

By jonny

jonny (http://www.videosift.com/member/jonny) — Fri, 18 Apr 2008 20:40:13 -0700

Is there any correlation between cpu load and network load? If not, then looks inwards. Could it be siftbot doing maintenance, checking the age of queued vids?

By dag

dag (http://dag.videosift.com) — Sun, 20 Apr 2008 01:34:31 -0700

We thought it might have been our scheduled processes - it could be, though when we're watching them - they never seem to be the problem. It might be our search.

By jonny

jonny (http://www.videosift.com/member/jonny) — Mon, 21 Apr 2008 11:14:59 -0700

You don't need to sit there and watch them. Just use a profiler. Hopefully it wouldn't add too much load, but I don't know for certain as I've never run one on a live website before.