Fetch
Fetch is the bot that collects. It is run by WPTopDog and is the only part of WPTopDog that makes requests to WordPress.org.
Status: not started. Fetch has not yet made any request to WordPress.org. This page says how it will behave, so that anyone can hold it to that.
How to recognize Fetch
| User-Agent | WPTopDogFetch/0.1 (+https://fetch.wptopdog.com/) |
|---|---|
| Address | 45.76.166.24 (IPv4 only). Every request comes from this one address. |
| Contact | fetch@wptopdog.com |
| Pace | At least ten seconds between requests to one host, and longer if that host states a longer crawl-delay. |
What it will fetch
New and updated plugins and themes, release information and security announcements, from the WordPress.org sources it is allowed to read. It runs once a day, at a jittered off-peak time, and works incrementally: it asks only for what changed since it last looked.
How it behaves
- Robots first. It reads each host's robots rules before its first request of the day, and makes no request to anything a host disallows. A refusal is reported, never worked around.
- Stated pace. It waits at least as long as the host states between requests, and never less than ten seconds.
- Small and incremental. No bulk crawls. It does not ask for what it already holds.
- Capped. Every response has a size limit; an oversized response is dropped.
- No retries or redirects. A failed request waits for the next run. A redirect is recorded, not followed.
- Stops on refusal. A 403 or 429 halts the run, and it stays halted until a person has looked at it.
- One identity. The same address and the same User-Agent, every time.
What Fetch does not do
- It does not test, install, activate or run any plugin or theme.
- It does not log in to anything or request anything that needs credentials.
- It does not publish. It collects.
If Fetch is a problem for you
Write to fetch@wptopdog.com. Fetch will pause on request.