Crazy Python Crawler
Someone emailed me doubting my crawler could operate at the speeds I posted last week so here is a video I took this morning. I should have waited a few minutes after launching it before starting the video as it really starts cranking once all the threads get rocking and you can see that near the end of the video. Also notice my streaming internet radio going in and out thanks to no available bandwidth left on my 5Mbps line.
You can also hear a ticking sound. That is my new 1TB drive. It makes these weird ticking noises even when its not in use. REally sounds like the arm hitting something its not supposed to hit. Hope its not defective.
Wow. Although, that ticking sound is never good. I would replace that drive immediately.
Hi please send that script to me.
Hello, are you willing to post your source code so that we can all learn from your technique. I would really like to take a look at it. Are you recognizing robots.txt or parsing meta data at all? Thanks!
Hi Michael
Unfortunately this crawler was developed for a client and as a result I have to obey the terms of my contract with them which does not allow me to release the code.
Tony