collect_content.RdEvery requested domain comes back, in order, whether or not it was reachable. Failures carry a code from [fetch_error_codes()] rather than a bare `NA`.
collect_content(
domains = NULL,
delay = 1,
timeout = 10,
max_bytes = 10 * 1024^2,
obey_robots = TRUE,
max_crawl_delay = 30,
max_redirects = 5,
cache_dir = NULL,
cache_ttl = 7 * 86400,
cache_max_size = 1024^3,
allow_hosts = character(),
archive_fallback = TRUE,
archive_date = NULL,
user_agent = rdomains_user_agent()
)Character vector of domains or URLs.
Minimum seconds between requests to the same host. `Crawl-delay` overrides this upward.
Per-request timeout, seconds.
Cap on the response body actually read. The default matches piedomains' 10 MB. A smaller cap looks prudent and is not: cnn.com's homepage alone is roughly 6 MB, so a 2 MB limit rejects major news sites as `content_too_large` while they return HTTP 200.
Whether to fetch and honour robots.txt. Turning this off is discouraged and is your responsibility, not the package's.
Skip a host that asks for a longer delay than this rather than sleeping on it.
Maximum redirect hops to follow. Every hop is re-validated, so a redirect cannot be used to reach an address the first check refused.
Where to cache fetched pages. Defaults to a directory under [tempdir()], so nothing persists past the session; pass [rdomains_cache_dir()] to opt into a cache that does.
Seconds a cached page stays fresh.
Prune the cache above this many bytes, oldest first.
Hosts exempt from the private-address checks, by exact name. Only the hosts named are exempt, so a redirect elsewhere is still refused. Intended for testing against a local server.
When a host serves an anti-bot interstitial, try the Internet Archive's most recent capture. Rows recovered this way carry `source = "archive"` and a `snapshot_timestamp`, so the vintage is never hidden. Dead domains are deliberately *not* recovered this way.
Fetch each domain as it was on this date (`"YYYYMMDD"`) from the Internet Archive instead of fetching it live. This is how you ask what a domain *used to be* – and, set against a live run, how you measure whether a label has gone stale. Rows carry `source = "archive"` and the realised `snapshot_timestamp`, which is the capture actually found and not necessarily the date you asked for.
Override the identifying user-agent.
A tibble with one row per input: `domain_name`, `status`, `stage`, `error_code`, `retryable`, `http_status`, `final_url`, `fetched_at`, `content_bytes`, `title`, `description`, `lang`, `text`, `n_tokens`, `page_state`, `block_vendor`, `robots_allowed`, `source_last_published`.
The crawler identifies itself as `rdomains/<version>`, obeys `robots.txt` including `Crawl-delay`, spaces requests to the same host, caps the response body, and refuses to fetch hosts that resolve to private or link-local addresses.
[fetch_report()] to summarise the run, [page_signals()] for what the page states are, [source_vintage()] for how a live fetch compares with the static lists.
if (FALSE) { # \dontrun{
res <- collect_content(c("example.com", "wikipedia.org"))
fetch_report(res)
# retry only what is worth retrying
again <- collect_content(res$domain_name[res$retryable])
} # }