Why are platforms blocking agents?
On The Chopping Block a few weeks ago, there was a discussion about how Amazon is blocking Meta’s Muse agent.
Robert: “Amazon is incentivized for Muse to be on shopping sprees”
Haseeb: “No, no, no. Absolutely not. Absolutely not. Because Amazon makes most of their money from advertising.”
Haseeb says Amazon has low margins on their products, but makes money by charging sellers for placement (“Adwords for products”), whereas DoorDash charges for fulfillment, so it makes sense that Amazon bans agents (like Muse) whereas DoorDash welcomes them.
This is an interesting economic point. We typically think of user bots as spendthrifts who’d increase platform sales. But if bots ignore ads (or are less swayed by them), then platforms that make their money from ad placement would turn bots away, whereas platforms that make their money by taking a cut of sales would encourage them.
Amazon
First, is Amazon actually an advertising business? We can look at Amazon’s FY2025 10-K. In fiscal year 2025, ads brought in $68.6B of Amazon’s $716.9B in net sales (9.6%). But, as Haseeb pointed out, net sales may not be the right comparison since most of Amazon’s sales of physical goods are very low margin. Unfortunately, Amazon’s filings only report advertising revenue, not advertising profit. If we knew the margin on ads, then we could get the profit. The figure below shows Amazon’s profits from ads at different (hypothetical) margins. If Amazon’s ad margins are above 25%, then the majority of Amazon’s retail profit comes from ads. If they’re above 50%, then Amazon’s profit from ads exceeds its profit in the entire retail category (of which advertising is a part). That would mean that if ad margins are above 50%, then the rest of Amazon’s retail business is losing money. This is exactly the point Haseeb was making: (if ad margins are reasonably high) sales of goods are a loss-leader for Amazon’s real retail business: advertising. If Amazon’s ad margins are even higher (above 58%), then even when you consider Amazon’s non-retail business (e.g. AWS), Amazon would be making the majority of its profits from ads. Although we can’t be sure of Amazon’s margins on its ad business, it seems like Haseeb was right, Amazon is an advertising business.

Other platforms
We can run the same calculation for other prominent platforms. Like Amazon, most platforms don’t report their ad margins. The plot below shows the ad margin at which ads would earn half of each platform’s operating income (blue) and all of its operating income (red) for the 2025 fiscal year.

Where the figure says “never,” that means the total ad revenue is below operating income, so even if the margin on ads were 100%, ads couldn’t earn that share of profit.
Stance on bots
But the big question we’d like to address is whether a firm’s advertising revenue drives its stance on bots. We’ve got an idea about which companies are “ad companies,” so now we just need to get a sense of each company’s stance on bots.
For this, we’ll use each company’s robots.txt file. If you’re not familiar with robots.txt, it’s a file that servers can publish that gives guidance about how “robots” should behave on its site. The robots.txt standard was created long before AI agents so that sites could give guidance on how they wanted to be crawled by search engines. Today, robots.txt files also give guidance to popular AI agents as well as search bots and crawlers associated with AI training. The robots.txt files do not have any enforcement mechanism, they are just requests to bots (and bots frequently ignore them). Nevertheless, robots.txt files give a sense of whether the site owner would like bots on their site.
The table below shows what each site’s robots.txt says to the major AI agents that act for a user. Each site was tested on three of its own pages: the homepage, a search-results page and an item page (a product, listing, store or hotel). block means all three were disallowed, allow means all three were allowed, and search blocked means only the search page was disallowed. Googlebot is included as a baseline, since many sites keep every crawler, including Google’s, off their internal search-results pages. For comparison, the last column summarizes how the site treats AI training crawlers (e.g. GPTBot, ClaudeBot, Google-Extended, CCBot, meta-externalagent and Amazonbot).
| Site | Googlebot | ChatGPT-User | Claude-User | Perplexity-User | AI training crawlers |
|---|---|---|---|---|---|
| amazon.com | allow | block | block | block | all blocked except Amazonbot |
| yelp.com | allow | block | block | block | all blocked |
| instacart.com | allow | allow | block | block | all blocked |
| ebay.com | search blocked | allow | allow | search blocked | all blocked except Google-Extended |
| expedia.com | search blocked | allow | allow | allow | search blocked |
| airbnb.com | search blocked | search blocked | search blocked | search blocked | GPTBot, ClaudeBot blocked |
| zillow.com | allow | allow | allow | allow | meta-externalagent, Amazonbot blocked |
| chewy.com | search blocked | search blocked | search blocked | search blocked | Amazonbot blocked |
| temu.com, wayfair.com, booking.com, costco.com, newegg.com, craigslist.org, doordash.com | allow | allow | allow | allow | all allowed |
| walmart.com, target.com, etsy.com, homedepot.com, bhphotovideo.com, ubereats.com | search blocked | search blocked | search blocked | search blocked | search blocked |
Only three sites (Amazon, Yelp and Instacart) treat user-directed AI agents more strictly than Google’s search crawler, Googlebot. Two go the other way. eBay keeps its search results away from Googlebot but explicitly allows them for OpenAI’s and Anthropic’s agents, and Expedia does the same for OpenAI’s, Anthropic’s and Perplexity’s.
robots.txt Caveats
There are a couple of caveats associated with the bot-blocking measurements. First, the robots.txt rules are not global, and some sites allow bots in certain areas, but not others. The robots.txt standard allows path-specific rules (e.g. separate rules for example.com/product and example.com/docs). To account for this, we tried to identify the most important paths, and then used Protego to test the rules for these paths. Second, robots.txt has no enforcement mechanism, so some sites use other methods to block bots (e.g. blocking bots at the network level, or disallowing them in their terms of service). Third, robots.txt files change frequently. This is where they were in mid-October 2026.
Findings
On the whole, Haseeb’s observation seems to hold up well. Companies whose business is primarily based on ads do seem more willing to restrict AI agents, whereas companies whose primary profits come from elsewhere are more welcoming to AI agents.
In the figure below, each row is a company.
Since we don’t have data on the profit from ads, the x-axis shows the ad revenue divided by the company’s operating income (profit).

The figure includes fewer companies than the robots.txt table above because it needs both ad revenue and operating income, and many companies (including Etsy, Home Depot, Wayfair and Chewy) don’t report their ad revenue separately.
DoorDash, which Haseeb held up as the counterpoint to Amazon, is the interesting case. It doesn’t report its ad revenue. It has only said its ad business passed a $1 billion annualized run-rate in 2024. That’s roughly the size of its 2025 operating income ($723M), which would put it among the more ad-dependent platforms, but the estimate is too rough to rank it exactly. Even so, DoorDash welcomes agents. So by revenue, DoorDash is the fulfillment business Haseeb described (ads are only about 7% of its revenue), but by the measure we used for Amazon, it may look more like an ad business than he suggested.
Measuring profit differently doesn’t change which platforms look most ad-dependent. If we use EBITDA (operating income plus depreciation and amortization) instead of operating income, Yelp, Instacart and Amazon’s retail business are still the three most ad-dependent platforms. It does matter for DoorDash, whose depreciation and amortization ($747M) is larger than its operating income, so switching to EBITDA roughly halves its ratio. About half of that is amortized software development, though, which is a real ongoing cost, so EBITDA probably flatters DoorDash’s profit. It also makes the case for Amazon weaker: Amazon’s warehouses and data centers carry a lot of depreciation, so on EBITDA, ads would earn all of Amazon’s retail profit only at margins above about 80% (rather than 50%), and half of its total profit only above about 89% (rather than 58%).
Other explanations
Although Haseeb’s claim is appealing, it’s not the only possible explanation.
- Security. Officially, Amazon says it blocked Muse because Meta never asked permission, Muse doesn’t identify itself while browsing, and Muse appears to capture and store customer credentials (GeekWire).
- Control. A lot of people claim that it’s about controlling the relationship with the customer. If Muse becomes the dominant shopper on Amazon, Amazon will lose its relationship with its customers. As Nicolas Bustamante, a founder who now works on AI at Microsoft, put it, platforms “want to own the customer relationship, not become interchangeable suppliers behind someone else’s interface.” This is actually pretty similar to the ad-revenue claim, because one of the main reasons you want a direct relationship with the customer is so you can advertise to them. (Bustamante’s own sentence ends with ”+ protect their ad business.”)
- Bargaining. Amazon says it expects agents to “offer a mutual exchange of value” (Business Insider), Andy Jassy has said Amazon is talking with companies that want to run shopping agents, and Mark Zuckerberg has said Muse would “take a very small cut of whatever the transaction is” (GeekWire). So Amazon will likely open up to agents where it can get an acceptable cut of the profits.
Our robots.txt data can help separate these explanations. If security were the main reason, we’d expect any site that handles customer accounts and payments to block agents. But Walmart, Target, Booking and Expedia all handle accounts and payments, and none of them treat agents more strictly than Googlebot. Walmart just launched an Agent Gateway to make it easier for outside agents to shop. If control were the main reason, we’d expect every aggregator to block, but Booking and Expedia are aggregators too, and they welcome agents. Bargaining is hard to test with robots.txt, since a block works as a bargaining chip whatever the underlying reason. The ad story fits the data best: the three sites that block agents are the three most dependent on ads, with DoorDash as the main exception.
For a platform that takes a cut of each sale, an agent is a customer who shows up ready to buy. For a platform that sells placement, an agent is a customer who, as Perplexity put it, doesn’t “have eyeballs to see the pervasive advertising.” So yes, Amazon, Yelp and Instacart are blocking customers, just the ones they can’t advertise to.
Data
Most of these companies report a full-year ad figure, but a few don’t, and their figures are approximate (marked with ”~” in the Findings figure). Uber’s data is based on the “well over $2 billion in annualized revenue” from its Q4 2025 remarks. Walmart said its global ad business was “nearly $6.4 billion.” DoorDash doesn’t report its ad revenue at all. It has only said that its ad business passed a $1 billion annualized run-rate in 2024. Its ads have likely grown since then, so we use $1 billion as a rough floor for 2025. Instacart’s figure is its “advertising & other” revenue, so its ad revenue alone is somewhat lower than shown. Booking’s figure is its “advertising and other” revenue, most of which comes from KAYAK and OpenTable. Expedia’s figure is its “advertising and media” revenue, which includes $417M from trivago. Airbnb has no ad business yet, though in September 2026 its CEO called sponsored listings a potential $1B opportunity.
These data were pulled from the following sources:
- Yelp: Q4 2025 earnings release and FY2025 10-K
- Instacart: Q4 2025 shareholder letter
- Amazon: FY2025 10-K
- DoorDash: FY2025 10-K; ad run-rate from its June 2025 ads announcement
- eBay: Q4 2025 earnings release (ad revenue is the sum of the four 2025 quarterly releases)
- Expedia: Q4 2025 earnings release
- Uber: Q4 2025 earnings release; ad run-rate from its prepared remarks
- Walmart: Q4 FY2026 earnings release (fiscal year ended January 31, 2026)
- Target: Q4 2025 earnings release (fiscal year ended January 31, 2026)
- Booking: Q4 2025 earnings release and FY2025 10-K
- Airbnb: Q4 2025 shareholder letter