Technical information about how Moderator crawls and indexes content from the open web.
Our crawler identifies itself with the following User-Agent string:
FriendlyModeratorBot/1.0 (+https://moderator.so/policies/crawling; contact@moderator.so)
The User-Agent includes a link to this policy page and contact information.
Moderator fully respects robots.txt directives. We check robots.txt before crawling and honor all path-level blocking rules.
We enforce conservative rate limits to avoid impacting your servers.
To explicitly allow our crawler, add the following to your robots.txt:
User-agent: FriendlyModeratorBot Allow: / Crawl-delay: 10
If no robots.txt is present or returns 404, we assume access is allowed with a default 10-second delay.
To block our crawler entirely, add the following to your robots.txt:
User-agent: FriendlyModeratorBot Disallow: /
You can also contact us directly at contact@moderator.so to request removal.
The most reliable way for publishers to make content available to Moderator is through RSS feeds. RSS is an open standard for web content syndication that allows you to control exactly what metadata is shared—titles, descriptions, publication dates, and links—without exposing full article text.
To help Moderator (and other aggregators) find your feed automatically, add an RSS autodiscovery link to your site's HTML <head>:
<link rel="alternate" type="application/rss+xml" title="Your Site Name" href="https://example.com/feed.xml" />
This meta tag tells feed readers where to find your RSS feed. Most CMS platforms like WordPress, Ghost, and Hugo include this by default.
For the full RSS 2.0 specification, see the RSS Advisory Board specification.