Well, I've worked for organizations that had active defenses against crawlers...... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		PaulHoule on Dec 17, 2010 \| parent \| context \| favorite \| on: Delicious's Data Policy is Like Setting a Museum o... Well, I've worked for organizations that had active defenses against crawlers... Make too many HTTP request and hour and ~poof~ a deny directive goes in the .htaccess file, or if they really like to play a rough game they'll firewall you. I know delicious had active defenses because I ran afoul of them.

kevindication on Dec 17, 2010 | [–]

And really, slowing your connections down to rates that are measured in baud would be the most fun.

jamiequint on Dec 17, 2010 | | [–]

True, there are plenty of relatively cheap proxy services made for exactly this reason though that will roll your IP on each request.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact