Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

ArchiveTeam definitely do not intend to kill websites with too fast crawling, but definitely have done that unintentionally and always will stop/slow the crawling when it happens.

Even the distributed crawling system has monitoring and controls to ensure it doesn't kill sites.



Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: