CarragentBot: what it reads and how to stop it

Carragent is software that runs AI agents for small businesses. One of them, Prospecting, finds businesses that may want what its owner sells. Once a week, and when its owner asks for a search, it looks businesses up on the web and CarragentBot reads their public pages.

It names itself with this User-Agent:

CarragentBot/1.0 (+https://carragent.com/bot)

Add these lines to your robots.txt. CarragentBot reads that file before it opens any page of your site, so from the agent’s next run it does not read your site:

User-agent: CarragentBot
Disallow: /

To keep it out of one part of your site only, use this instead, with that part’s path:

User-agent: CarragentBot
Disallow: /private/

Or write to admin@carragent.com with your website’s address and ask us to add it to our block list. Every search checks that list before it opens a page, and businesses already listed from your site are removed from our customers’ lists.

Public web pages only, at http and https addresses, and each site’s robots.txt. It skips social networks, marketplaces and directories whose terms keep automated programs out, and it never opens an address on a private network.

On each visit it reads at most 3 pages of a site: the page the search found and, from the links on that page, up to two more of the same site that look like a contact page or an about page. With the robots.txt, that is at most 4 requests, one after another. A search looks at no more than 36 sites, and one run of the agent at no more than 9 of them, so a longer search carries on in a later run.

It waits at most 20 seconds for a page (10 for a robots.txt), and it reads the first 512 KiB of a robots.txt and ignores the rest. If a visit cannot be finished, because the site does not answer or because of a problem on our side, it tries again later, and it gives up on a site that does not answer after 3 attempts.

A customer’s agent searches at most once a week, and the customer can start one more search by hand, at most once a day. Once a site has been looked at for a customer, it is not looked at again for that customer for 35 days. The limit is per customer, so a site that suits several of our customers can be visited for each of them.

Before it opens a page, CarragentBot fetches the site’s robots.txt and follows it (RFC 9309). It obeys the group for CarragentBot or, if there is none, the group for *. The longest matching rule wins, and Allow wins a tie.

With no robots.txt (a 404, a 410 or any other 4xx except 401, 403 and 429) nothing is forbidden. If the robots.txt answers 401 or 403, nothing is read from that site in that search. If it answers 429 or a server error (500 and above), or does not answer in time, nothing is read for the moment: CarragentBot tries again later and gives up on the site after 3 attempts.

A site whose robots.txt shuts the crawlers of AI companies out of its root (GPTBot, ClaudeBot, CCBot or Google-Extended, for example) is not read either, whatever it says to everyone else.

The file is fetched again for each run of the agent, so a change applies from the next run.

It never logs in, and it never fills in or submits a form: it only loads public pages.

It never contacts anyone. The agent sends no email or message and posts no comment. It makes a list that our customer reviews, and the customer decides whether to write to a business.

At the moment CarragentBot does not sign its requests, so its User-Agent is the only thing that names it, and anyone can copy a User-Agent. For any question about a request, write to admin@carragent.com.