About our crawler
AdContextBot reads a few small public files on each site in our index, about once a week. This page says exactly which ones, and how to keep it out.
How it identifies itself
AdContextBot/1.0 (+https://adcontext.io/bot)
What it requests
- /ads.txt
- /robots.txt
- /.well-known/adagents.json
- The home page, to read its title and description only, and only when robots.txt allows it
That is four requests per site per week at most. It does not log in, follow links, run scripts or collect anything about your readers.
Why your site is in the index
An advertising exchange lists your company in its public seller list. That file is how we learned your domain.
Keep it off your home page
Add this to your robots.txt. The crawler will stop reading the page, and your site will no longer be filed under a topic.
User-agent: AdContextBot
Disallow: /
Disallow: /
Leave the index entirely
Write to us with your domain and we remove it and stop visiting.