__/ [Luigi Donatello Asero] on Tuesday 03 January 2006 02:49 \__
>
> "Marco Schmidt" <marcoschmidt@xxxxxxxxxxxxx> skrev i meddelandet
> news:41u6opF1g7kd9U1@xxxxxxxxxxxxxxxxx
>> The Alexa Internet Archive crawler has visited each of my pages listed
>> at DMOZ 50 times each day for at least the last couple of days.
>> They're all static content and change s rarely. Is anyone else
>> experiencing this? I don't want to exclude their crawler right away,
>> but why on earth would they want to have 50 copies per page per day
>> for archiving purposes? Requests are all coming from one host
>> (crawl23-public.alexa.com).
>>
>> Regards,
>> Marco
>
> I am not experiencing this with Alexa.
> But I have experienced long visits from Google (several robots)
> for a long time although these visits do not seem to show significant
> changes in the ranking of my pages. So I am wondering what is Google
> actually doing?
On my main site, Alexa snatched 250 MB of pages in the past 2 days. They
never consumed that much traffic, not even in a whole month. I haven't
looked for any patterns yet as the server can cope, so it's not worrisome.
What's the benefit of permitting Alexa to crawl though? Having the site
archived for someone to look back at deleted content in the future? Have
we not learned the lesson yet?
http://web.archive.org/web/20021030152640/www-db.stanford.edu/~sergey/photos/drag96.jpg
Roys
|
|
