Agent Traffic Needs House Rules

The old web had a useful fiction at its center: a visitor was probably a person. Not always, of course. There were crawlers, scrapers, uptime checks, spam rigs, and the occasional mysterious IP address with a private obsession with your archive page. But the everyday language still pointed toward a human being somewhere nearby. A visit. A reader. A session. A conversion. A bounce, which sounded like a small human disappointment.

AI agents are making that fiction harder to maintain.

Cloudflare's July 1 updates are a good marker of the shift. Its new default controls for AI crawlers let sites block AI crawlers unless the publisher opts in, while Pay Per Crawl experiments with charging selected crawlers for access. The mechanics are technical: bot categories, HTTP status codes, crawler verification, billing experiments. The real issue is older and stranger. A website is becoming a place where the owner has to distinguish among kinds of nonhuman attention.

One bot may be indexing a page so a search engine can send readers back. Another may be feeding a training corpus that never returns. Another may be fetching a page at a user's request because a personal agent is comparing products, summarizing a policy, or checking whether a medical billing form has changed. Another may be doing the same thing with no clear user behind it, treating the site's work like unguarded office snacks.

These are not morally identical visits. They should not be flattened into one permission switch labeled "AI."

The current argument often collapses into a familiar fight about blocking bots or getting paid. That fight matters. Publishers, documentation maintainers, independent writers, and small businesses have been asked to donate legibility to powerful systems, then be grateful for whatever attention washes back.

But the harder design problem is not only compensation. It is legibility. Site owners need to know whether an agent is acting as a librarian, a courier, a customer, a student, a training pipeline, a competitor, or a very expensive vacuum cleaner in a blazer. Users need to know whether the agent they delegated is welcome in the room, and whether the source can see the difference between "Kevin asked me to fetch this" and "a platform is harvesting another million pages before lunch."

That distinction is where interface design turns into civic plumbing. Robots.txt was a little sign on the garden gate. API keys were velvet ropes. Login walls were doors. Those metaphors worked while most automated access was either search infrastructure or obvious scraping. They wobble when an agent is genuinely standing in for a person.

Imagine a benefits applicant using an assistant to gather county forms and draft questions for an office that is only open while the applicant is working. Blocking all AI agents may protect the site from extraction, but it may also make the web less usable for the person who needed help crossing the paperwork swamp. Now imagine a recipe site copied into a chatbot answer without a visit, subscription, attribution, or even the popup the publisher depended on to pay rent. Allowing all AI access may help the user in the moment while quietly destroying the places users were asking agents to consult.

The agentic web has to hold both cases at once. That is not a vibe. It is a permissions architecture.

This is why efforts like Content Signals are interesting even if they are early and imperfect. They try to separate search, AI training, and AI input use into clearer declarations. A page can be available for search, unavailable for training, and conditionally available as input to a user-requested AI system. That will not solve enforcement by itself. The web has always contained people willing to ignore polite signs. Still, a better sign matters. A society that cannot name its boundaries usually ends up enforcing them badly.

The same pressure is visible from the agent side. Papers such as A2A: Architecting Advanced Agent-to-Agent Interactions show a world where agents increasingly need to identify roles, negotiate tasks, exchange state, and preserve accountability across systems. That sounds abstract until you put it next to a normal afternoon on the web. A person asks an agent to find a warranty policy. The site wants to know if this is search indexing, training, shopping, support, fraud, or a customer's assistant doing exactly what the customer would have done, only with less squinting at a PDF.

The browser used to carry a lot of this social meaning for us. A person clicked. A page loaded. The site could show terms, ask for consent, require login, or make the experience terrible in one of the web's many beloved traditions. Agents break that choreography. They can read without seeing, act without lingering, summarize without sending, and transact without the familiar ceremony of arrival. A website built only for human eyes becomes an institution with a loading dock it never designed.

House rules are not the same as hostility. A decent shop can welcome customers and still refuse to let strangers empty the shelves into a truck. A public office can let residents use assistive tools without letting vendors quietly map the entire service system for resale. The question is whether we can make automated purpose visible enough for trust to survive.

That will require product work that is more specific than the slogans. Agents may need ways to present delegated authority: who asked, for what purpose, within what scope, and with what retention rules. Sites may need controls that distinguish training from user-requested retrieval, commercial aggregation from accessibility, monitoring from action, and checkout from research. Analytics may need to stop pretending every successful answer is simply a failed visit.

There is a human cost to getting this wrong. If we make the agentic web too closed, we turn helpful delegation into another privilege for people with time, skill, and patience. If we make it too open, we hollow out the places agents depend on until the web becomes a set of summaries arguing over rooms nobody maintains. Both failures can look efficient from a dashboard. The damage shows up later, in thinner sources, worse recourse, more brittle services, and a sense that nobody quite knows who is in the room anymore.

The web has always been a negotiation between access and care. AI agents do not end that negotiation. They make it harder to fake. The next useful interface may not be another magical answer box. It may be a better door: one that can tell the difference between a reader, a representative, a miner, and a machine with its hat pulled low, while remembering that a person may be standing just behind it.