What is the agent-first internet?

It describes a web where a growing share of reading, comparing and buying is performed by AI agents acting for people rather than by people browsing directly. Your website stops being purely a destination and becomes a data source and a transaction endpoint. The practical consequence is that machine legibility starts to matter as much as human persuasion.

Will AI agents replace website traffic entirely?

No. Human visits remain the majority of most sites' revenue today. What is changing is composition: fewer low-intent browsing sessions, because agents absorb that filtering, and a higher-converting remainder. Plan for declining session volume with steady or rising revenue per session, not for traffic disappearing.

How do I know if AI agents can read my website?

Load your key pages with JavaScript disabled and check whether prices, specifications, availability and policies still appear. Then confirm your robots.txt, WAF and CDN are not blocking the agents you want. Then validate your structured data. Most failures are found in those three checks.

Should I publish an llms.txt file?

It costs little and does no harm, so publish one if you want. Be realistic about the return: Google has said it does not support llms.txt, and no major provider has committed to it as a production signal. It genuinely helps for developer documentation consumed by coding agents, and it is not a substitute for structured, server-rendered content.

What is the Agentic Commerce Protocol?

ACP is an open standard developed by OpenAI and Stripe that lets merchants sell through AI agents with a single integration, powering Instant Checkout in ChatGPT. The merchant keeps control of pricing, fulfilment and the customer relationship, and remains the merchant of record. It is one of several competing and complementary standards.

What is the difference between ACP and AP2?

ACP focuses on the commerce transaction: product feeds, carts and checkout between a merchant and an agent. AP2, from Google and now under the FIDO Alliance, focuses on payment authorisation, using signed intent, cart and payment mandates to prove a human approved the purchase. They address different layers and can be used together.

Should I block AI crawlers on my website?

Not indiscriminately. Distinguish training crawlers, which take content and rarely send visitors, from search and shopping agents, which can produce citations, referrals and transactions. Blocking the first while allowing the second is defensible, and cryptographic agent identity through Web Bot Auth makes that distinction enforceable rather than aspirational.

How do I measure AI agent traffic in analytics?

Create a dedicated channel group for known AI referrers so they stop being counted as direct traffic, and review origin server logs for agent user agents, which catches activity that never reaches your JavaScript analytics. Report revenue per session alongside session counts, because volume metrics alone will misrepresent the shift.

Is agent-readiness a marketing job or an engineering job?

Both, plus security. Marketing owns which facts are published and how clearly, engineering owns rendering, structured data and transactable endpoints, and security owns the agent allow list. Projects sitting entirely inside marketing stall at the first CDN change they cannot make themselves.

The Agent-First Internet: Your Next Customer Won't Visit

An AI agent reads a dozen supplier sites in four seconds, shortlists three, and nobody loads a page. Nine suppliers never learn they were considered. This covers what the agent-first internet actually is, why the traffic data says two opposite things at once, and a 90-day plan to make your site readable to the thing doing the reading.

✦Key Takeaways

Agent traffic breaks into three behaviours: answering in place, researching on a human's behalf, and transacting directly. Most teams lump them together, which is why the response usually fails.
The largest AI crawlers request thousands of pages for every visitor they send back, so raw traffic will keep falling even as your influence on purchase decisions holds steady.
Adobe Analytics found AI-referred retail visits converting around 60% better than non-AI traffic and generating 53% more revenue per visit. Fewer sessions, materially better ones.
Adobe also found roughly half of retail site content is machine-readable, from about 63% in cosmetics down to 47% in furniture and home. The gap is not visibility, it is legibility.
Failing an agent's qualification filter produces no signal at all. You do not appear in analytics as a lost deal because you never appeared at all.
The commerce protocol layer is already multi-standard: ACP from OpenAI and Stripe, AP2 from Google, plus card network schemes. Betting on a single winner is the wrong shape of bet.
llms.txt is the most over-recommended and least evidenced item on most agent-readiness checklists. Google has publicly declined to support it.
The highest-return work is unglamorous: get facts out of prose and PDFs, render server-side, publish parseable policies, stop blocking agents you want, and expose a transactable endpoint.

A buyer needs a supplier. Instead of typing keywords into Google, they describe the problem to an assistant: the budget, the deadline, the two integrations that are non-negotiable. The assistant reads a dozen supplier sites in about four seconds, checks each one against the stated constraints, and comes back with three names. The buyer picks one. Total human page views across all twelve suppliers: zero.

Nine of those suppliers will never know they were considered. There is no impression in their analytics, no bounce, no session, no abandoned form. They were read, assessed against a requirement, and dropped, and the entire event is invisible on their side of the glass. That is what makes the agent-first internet hard to plan for: the failure mode is silence.

This is not a forecast. Payment protocols, browser APIs and agent identity standards all landed in production within the last eighteen months, and the traffic data has started to move with them. What follows is what the agent-first internet actually is, why most websites are unreadable to the thing doing the reading, and a 90-day plan to fix it.

What "Agent-First" Actually Means

Agent-first does not mean humans stop buying. It means the reading, comparing and qualifying that used to happen in a browser increasingly happens inside a model, and your website is consumed as a data source rather than experienced as a destination.

It helps to separate three behaviours flattened into the single phrase "AI traffic", because they have different economics and different fixes.

Answer-in-place. Someone asks a question, the model answers from what it knows or from a quick retrieval pass, and nobody leaves the chat. Your only leverage is whether your facts were in reach and stated clearly enough to be repeated accurately. This is the behaviour behind AI Overviews eating organic traffic, and the tier most businesses have already felt.

Agent-assisted research. A person delegates a task: compare these five options, find a supplier who meets these conditions, check whether this is in stock in my size. The agent visits real pages, often many of them, and returns a shortlist. Sometimes a human clicks through, which shows up as a referral. Often they do not, and the summary is the only thing the buyer sees.

Agent-executed transaction. The agent completes the action: places the order, books the appointment, submits the enquiry. No human ever loads your interface. This is what the commerce protocols were built for, and it is the tier most businesses are least prepared for, because being findable and being transactable are separate engineering problems.

The strategic error is treating all three as a traffic problem. Only the middle tier is about traffic. The first is about factual availability, the third is about machine-executable transactions, and neither improves because you published more blog posts.

The Traffic Data Says Two Opposite Things at Once

Look at crawler data and the agent-first internet looks like a disaster. Look at conversion data and it looks like the best channel you have. Both readings are correct, and holding them together is the whole trick.

On the crawl side, Cloudflare has published crawl-to-refer ratios on Radar since 2025, comparing how many pages an AI platform requests against how many visitors it sends back. The ratios are brutal and have stayed brutal: the largest crawlers pull thousands of pages for every referral, and for some the figure has run into the tens of thousands. The old bargain, where a search engine indexed your content and repaid you in clicks, has been renegotiated without your signature.

On the conversion side, the picture inverts. Adobe Analytics, tracking transactions across more than 200 of the top 2,000 US retailers, reported in July 2026 that AI-referral traffic was up 62% year over year, converting roughly 60% better than non-AI traffic and generating 53% more revenue per visit. Those visitors spent 59% more time on site, bounced 33% less, and added to cart 28% more often. It was the eleventh consecutive month of AI traffic outperforming everything else.

The reconciliation is straightforward once you stop treating sessions as the unit of value. An agent does the filtering a human used to do by opening fifteen tabs. By the time a person arrives, the mismatches have already been discarded. You lose the browsing volume and keep, disproportionately, the qualified end of it.

Your analytics is under-reporting the shift

The measurement problem is real and it flatters the status quo. Assistant referrals frequently land in your reports as direct traffic because the referrer is stripped. Agentic browsers such as Comet and Atlas produce sessions that look human because, technically, a human is driving. Answer-in-place interactions produce nothing at all.

The fix is not exotic. Segment known AI referrers into their own channel group rather than letting them dissolve into direct, log the user agents hitting your origin, and report revenue per session alongside session volume. If your board dashboard tracks only sessions, it will show a decline your revenue does not share, and you will make the wrong decision from a true number.

Three horizontal lanes showing answer-in-place with zero sessions, agent-assisted research producing an occasional referral, and agent-executed transaction placing an order with no human visit — Three behaviours, one label. Only the middle one is a traffic problem.

Why an Agent Cannot Use Most of Your Website

Twenty-five years of web design optimised for a specific reader: a distracted human who scans, responds to imagery, and needs persuading. Almost every convention that serves that reader works against a machine reader extracting facts under a token budget.

Facts dissolved into persuasion. "Industry-leading turnaround" is not a delivery estimate. "Flexible pricing to suit your needs" is not a price. An agent matching a buyer's constraint of delivery within five working days cannot do anything with a phrase that avoids committing to a number. The information is not hidden by intent, it is hidden by house style.

Pricing behind a form. Gating price behind a contact form assumed a motivated human would complete it. An agent will not. It notes that price is unavailable, and depending on how the buyer phrased the constraint, that is either a soft mark against you or an immediate disqualification.

Policies in PDFs and images. Returns windows, delivery cutoffs, SLAs, warranty terms and compliance certifications routinely live in PDFs or support portal screenshots. These are exactly the fields an agent needs, stored in the least parseable formats on your site.

JavaScript-only rendering. Plenty of agents execute JavaScript, but not all do, and many work under time and token constraints that make a slow client-rendered page a poor bet next to a competitor whose content is in the initial HTML response. If your specifications only exist after hydration, treat them as optional content.

Blocked at the door. Many sites sit behind bot mitigation tuned for scrapers, and it does not distinguish a scraper from an agent carrying a real buyer's request. Cloudflare has moved toward blocking AI crawlers by default in several configurations, a defensible content policy and a commercial decision most marketing teams were never consulted on. Somebody in your organisation may already have opted you out through a security setting.

None of these are design failures in the traditional sense. They are the correct answers to the previous question.

The Qualification Layer: How Agents Actually Shortlist

Think of the agent as an unusually literal procurement analyst working from a constraint list. It is not judging your brand. It is checking whether stated facts satisfy stated conditions, and it discards anything it cannot verify quickly.

Consider a realistic B2B prompt: find a UK payroll provider that is ISO 27001 certified, integrates with Xero, supports weekly payroll, and can onboard within three weeks. Four constraints. A provider that meets all four but publishes the certification as a badge image, the integration list in a JavaScript carousel, and the onboarding timeline nowhere at all fails three of the four checks despite being the best fit in the market.

That failure is completely silent. No impression, no bounce, no abandoned form, no signal of any kind. This is why agent-readiness resists the usual optimisation loop: the feedback that would normally tell you something is broken does not exist.

The scale of the gap is measurable. Adobe's analysis of retail sites found roughly half of site content is readable by AI systems, with wide variation by category: around 63% in cosmetics at the top, down to about 47% in furniture and home. These are large, well-resourced retailers. The median mid-market site is not doing better.

Adobe Analytics, May 2026. Roughly half of a well-resourced retail site is invisible to the reader deciding whether you qualify.

The work is less about writing new content than surfacing facts you already have: structured product and offer markup, specifications in tables, policies as HTML with explicit numbers, named entities rather than pronouns. Much of it overlaps with structuring content for featured snippets and AI citations, which is convenient, because the same discipline serves both.

From Findable to Transactable

Being cited in an answer is the first half. The second is whether an agent can complete a transaction with you without a human taking over, and that is now a protocol question rather than a UX question.

Four layers matter, and they stack rather than compete.

Layer	What it does	Where it stands
Discovery and facts	Structured data, feeds, server-rendered content	Mature. Nothing new to learn, just work to do
Site tools	WebMCP lets a page declare callable tools to an agent	Incubating at the W3C, shipping behind Chrome flags and early trials
Commerce	ACP, from OpenAI and Stripe, standardises product feeds and checkout for agents	Live in ChatGPT Instant Checkout with real merchants
Payment authorisation	AP2 signs intent, cart and payment as verifiable mandates; Visa and Mastercard run card-rail schemes	AP2 donated to the FIDO Alliance; network schemes running in parallel

WebMCP is the most interesting of these for non-retail businesses. It inverts the current arrangement: instead of an agent screenshotting your page and guessing which button to click, your site registers named tools with typed parameters and the agent calls them. A booking system exposes a checkAvailability tool and a createBooking tool rather than hoping a model can drive a date picker. That is a far more reliable contract than visual automation, and considerably safer.

On the commerce side, OpenAI's Instant Checkout and the Agentic Commerce Protocol let a purchase complete inside the conversation while the merchant keeps the customer relationship and remains the merchant of record. Google's Agent Payments Protocol tackles the adjacent problem of proving a human actually authorised what the agent is doing, using signed mandates rather than trust. Nobody credible expects a single winner soon, which argues for keeping product and service data in a clean, protocol-neutral form so you can serve whichever standard your customers arrive on. The same reasoning applies to the agentic shopping shift in ecommerce more broadly.

The llms.txt Distraction

Almost every agent-readiness checklist opens with "publish an llms.txt file". It is the most over-recommended item in the category and the least supported by evidence.

llms.txt is a sensible proposal: a Markdown file at your root giving a model a curated map of your important content. Adoption is real, helped considerably by documentation platforms enabling it by default across thousands of sites at once. The problem is the demand side. Google has said on the record that it does not support llms.txt and has no plans to, and no major model provider has committed to using it as a signal in production answer surfaces.

Where it demonstrably works is developer documentation consumed by coding agents. Cursor, Claude Code and similar tools do fetch llms.txt when pointed at a docs site, and if you sell a developer product that is a genuine reason to ship one.

For everyone else: it costs an hour, it does no harm, publish it if you like. But it is not the unlock, and treating it as the centrepiece of an agent strategy substitutes a file for the actual work. The actual work is getting your facts into structured, server-rendered HTML that every crawler already knows how to read. Less satisfying, and where the returns are. The same discipline underpins generative engine optimisation.

Identity: The Bot You Blocked Is Now Your Customer

Here is the awkward inversion. You have spent a decade building defences against automated traffic, and now a meaningful share of your qualified demand arrives automated.

The resolution is identity rather than blanket policy. Cloudflare's Web Bot Auth and signed agents work lets an agent cryptographically sign its requests, so your origin can verify who is asking rather than guessing from a user agent string anyone can forge. AWS WAF, Vercel, Akamai and Shopify have added support on the same foundation. It is the first credible answer to a question that has been unanswerable since the web began.

That changes the decision from "allow bots or not" into specific commercial choices. Which agents may read your content. Which may transact. What a training crawler gets versus what a shopping agent acting for a named buyer gets.

It also introduces a governance question most organisations have not scoped. An agent transacting on a customer's behalf holds some version of that customer's data and intent, and the handoffs are where things go wrong, as we covered in securing agent-to-agent conversations. Agent-readiness is not purely a marketing project: somebody in security needs to own the allow list.

What This Changes by Business Type

Ecommerce is furthest along and has the clearest checklist: complete structured product data, accurate real-time availability, parseable delivery and returns terms, and a decision on the commerce protocols. The conversion upside in the Adobe data is real, and the merchants capturing it are the ones whose catalogues are legible.

B2B and professional services face the harder version. Pricing is often deliberately opaque and the sale runs on relationships. That model still works, but the shortlist forms earlier and increasingly without you. Publishing indicative pricing bands, explicit engagement models, named certifications and clear capacity constraints is uncomfortable, and it is what gets you into the final three.

SaaS should already treat documentation as a product surface. Docs are the most agent-consumed content most software companies own, and llms.txt genuinely earns its place here.

Local and service businesses need the boring fundamentals most: correct structured data for hours, service areas and pricing, consistent details across directories, and content that answers situational questions in plain language.

A 90-Day Agent-Readiness Plan

Three phases, each with something you can actually check.

Days 1 to 30: see what is happening. Audit robots.txt, your WAF and your CDN settings to establish which agents you are currently blocking, deliberately or otherwise. Segment known AI referrers into a distinct analytics channel so they stop hiding inside direct. Pull server logs and inventory which agent user agents already hit you and which pages they request.
Verify: you can state, from data, which agents can reach your site and what they read most.

Days 31 to 60: make the facts legible. Pick the twenty pages carrying commercial facts and fix those first. Add structured data for products, offers, services and FAQs. Move prices, delivery windows, returns terms and certifications out of PDFs and images into HTML with explicit numbers.
Verify: fetch each page with JavaScript disabled and confirm every fact a buyer would use as a constraint is present in the raw HTML.

Days 61 to 90: become transactable. Decide whether agent-completed transactions are in scope this year. If they are, prepare a clean product or service feed, review your checkout or enquiry endpoint for programmatic access, evaluate ACP or AP2 against where your customers actually are, and pilot WebMCP tools for your two highest-value actions. If they are not, document the decision with a review date rather than letting it drift.
Verify: an agent can complete one real end-to-end task on your site without human intervention, or you have a dated, written decision not to allow it yet.

What Not to Do

Do not rebuild your website for agents. The human experience still drives most revenue, and a site optimised purely for machine consumption will convert humans worse. This is additive work, not a replacement.

Do not block everything reflexively. Blocking training crawlers while permitting shopping and search agents is a legitimate, well-supported position. Blocking all automated traffic in 2026 is closer to unplugging the phone.

Do not buy an agent-readiness product before doing the audit. Most of the first ninety days is work you already know how to do, and the tooling market is far ahead of the evidence.

Conclusion

The agent-first internet is not the end of websites. It is the end of the website as the only place a customer forms an opinion about you. Your site is becoming two things at once: an experience for the humans who still arrive, and a structured, verifiable supply of facts for the machines that increasingly arrive first.

The businesses adapting fastest are not the ones adopting the most standards. They are the ones that made their prices, terms, availability and capabilities unambiguous, kept the right doors open, and made it possible to transact without a human clicking anything. That work is legible to agents, and it happens to be clearer for people too.

Frequently Asked Questions

What is the agent-first internet?: It describes a web where a growing share of reading, comparing and buying is performed by AI agents acting for people rather than by people browsing directly. Your website stops being purely a destination and becomes a data source and a transaction endpoint. The practical consequence is that machine legibility starts to matter as much as human persuasion.
Will AI agents replace website traffic entirely?: No. Human visits remain the majority of most sites' revenue today. What is changing is composition: fewer low-intent browsing sessions, because agents absorb that filtering, and a higher-converting remainder. Plan for declining session volume with steady or rising revenue per session, not for traffic disappearing.
How do I know if AI agents can read my website?: Load your key pages with JavaScript disabled and check whether prices, specifications, availability and policies still appear. Then confirm your robots.txt, WAF and CDN are not blocking the agents you want. Then validate your structured data. Most failures are found in those three checks.
Should I publish an llms.txt file?: It costs little and does no harm, so publish one if you want. Be realistic about the return: Google has said it does not support llms.txt, and no major provider has committed to it as a production signal. It genuinely helps for developer documentation consumed by coding agents, and it is not a substitute for structured, server-rendered content.
What is the Agentic Commerce Protocol?: ACP is an open standard developed by OpenAI and Stripe that lets merchants sell through AI agents with a single integration, powering Instant Checkout in ChatGPT. The merchant keeps control of pricing, fulfilment and the customer relationship, and remains the merchant of record. It is one of several competing and complementary standards.
What is the difference between ACP and AP2?: ACP focuses on the commerce transaction: product feeds, carts and checkout between a merchant and an agent. AP2, from Google and now under the FIDO Alliance, focuses on payment authorisation, using signed intent, cart and payment mandates to prove a human approved the purchase. They address different layers and can be used together.
Should I block AI crawlers on my website?: Not indiscriminately. Distinguish training crawlers, which take content and rarely send visitors, from search and shopping agents, which can produce citations, referrals and transactions. Blocking the first while allowing the second is defensible, and cryptographic agent identity through Web Bot Auth makes that distinction enforceable rather than aspirational.
How do I measure AI agent traffic in analytics?: Create a dedicated channel group for known AI referrers so they stop being counted as direct traffic, and review origin server logs for agent user agents, which catches activity that never reaches your JavaScript analytics. Report revenue per session alongside session counts, because volume metrics alone will misrepresent the shift.
Is agent-readiness a marketing job or an engineering job?: Both, plus security. Marketing owns which facts are published and how clearly, engineering owns rendering, structured data and transactable endpoints, and security owns the agent allow list. Projects sitting entirely inside marketing stall at the first CDN change they cannot make themselves.

Your Next Customer Won't Visit Your Website: The Rise of the Agent-First Internet