FC Forums •And now we have a crawler

Page 1 of 1

And now we have a crawler

Posted: Thu May 08, 2014 8:13 am

by Archduke Daynel, PhD

In addition to Google, Bing and Majestic 12 bots, and a Baidu spider, we now also have a Heritrix crawler.
No screenshot because I'm at my phone atm.

Re: And now we have a crawler

Posted: Thu May 08, 2014 8:52 am

by Ivan2006

don't worry, it's just
http://en.wikipedia.org/wiki/Internet_Archive
They're cool guys.

Re: And now we have a crawler

Posted: Thu May 08, 2014 9:18 am

by Luna

But, spider is here right now... watching.

EDIT: Here

Re: And now we have a crawler

Posted: Thu May 08, 2014 9:32 am

by Prototype

What? A Heretic crawler?

HERESY!

Re: And now we have a crawler

Posted: Thu May 08, 2014 4:46 pm

by Vinyl

prototype wrote:What? A Heretic crawler?

HERESY!