Crawler
WebmasterHelperBot
If you found this page from a user agent in your access log, here is exactly what made that request.
User agent
Mozilla/5.0 (compatible; WebmasterHelperBot/1.0; +https://webmasterhelper.com/bot)What it is
It is not a crawler in the usual sense. It makes requests only when a person types a domain into the scanner on this site and presses Scan. There is no queue, no schedule and no discovery — it never follows links to find new URLs.
What a single scan requests
- The URL given, following redirects, by
GET. /robots.txt,/sitemap.xmlor/sitemap_index.xml.- One randomly named URL, to check the 404 response. It will appear in your logs as a 404 with a
wmh-probe-prefix, and that is expected. http://and the www or apex variant of the hostname, one request each.- Up to twelve
GETrequests for assets referenced in the page, aborted as soon as the response headers arrive. Expect these in your log as complete requests with few or no bytes sent.
Around 20 to 25 requests in total, over a few seconds, once per scan.
Blocking it
The scanner checks robots.txt but, like a browser fetching a page a person asked for, it fetches the URL you entered regardless. To refuse it at the server instead:
# nginx
if ($http_user_agent ~* "WebmasterHelperBot") { return 403; }Blocking it means nobody can scan your site with this tool, including you.
Rate limits
Scans are rate limited per client address so the tool cannot be used to generate load against a third party. A burst of identical scans will be refused with a 429 before it reaches your server.