What it does
UsefullyBot fetches a page when one of our API customers asks to read it (for example to answer a question with up-to-date sources). It fetches only the pages requested; it doesn't crawl your site on its own.
For site owners
User agent: UsefullyBot/1.0 (+https://usefully.info/bot)
UsefullyBot fetches a page when one of our API customers asks to read it (for example to answer a question with up-to-date sources). It fetches only the pages requested; it doesn't crawl your site on its own.
Before fetching, it checks your robots.txt for rules for the UsefullyBot user agent (or *). A disallowed page is never fetched, and isn't fetched another way.
Pages are fetched once per request, with redirects limited and private or internal addresses refused.
Add a rule to your robots.txt. Changes take effect within an hour.
# robots.txt User-agent: UsefullyBot Disallow: /
Questions or problems? Email support@usefully.info.