Websites are closing their doors to AI agents
I think the pushback was inevitable. Sites pay for content and traffic, and an agent that visits once and scrapes everything doesn't pay the bills, especially when publishers depend on ad views or subscriptions to stay afloat. The interesting part is that nobody has settled on a business model for machine readers yet, and the default posture is becoming "block first, ask later."
I'd want to know how this gets resolved, because a web that locks agents out also locks out the convenience those companies keep promising. OpenAI and Google have the leverage to negotiate deals with big sites, but the long tail of the internet, the blogs and forums and local businesses, probably won't get the same treatment. That split, where the top of the web stays open and the middle gets walled off, would quietly change how agents actually work in practice.
It's early, and the back-and-forth between crawlers and anti-bot measures is as old as the web itself. But this time the stakes feel higher because the agents are meant to act, not just index. If they can't reach the sites, they're just demos.