Every requested domain comes back, in order, whether or not it was reachable. Failures
carry a code from fetch_error_codes() rather than a bare NA.
collect_content(
domains = NULL,
delay = 1,
timeout = 10,
max_bytes = 10 * 1024^2,
obey_robots = TRUE,
max_crawl_delay = 30,
max_redirects = 5,
cache_dir = NULL,
cache_ttl = 7 * 86400,
cache_max_size = 1024^3,
allow_hosts = character(),
archive_fallback = FALSE,
archive_date = NULL,
user_agent = rdomains_user_agent()
)Character vector of domains or URLs.
Minimum seconds between requests to the same host. Crawl-delay overrides
this upward.
Per-request timeout, seconds.
Cap on the response body actually read. The default matches
piedomains' 10 MB. A smaller cap looks prudent and is not: cnn.com's homepage
alone is roughly 6 MB, so a 2 MB limit rejects major news sites as
content_too_large while they return HTTP 200.
Whether to fetch and honour robots.txt. Turning this off is discouraged and is your responsibility, not the package's.
Skip a host that asks for a longer delay than this rather than sleeping on it.
Maximum redirect hops to follow. Every hop is re-validated, so a redirect cannot be used to reach an address the first check refused.
Where to cache fetched pages. Defaults to a directory under
tempdir(), so nothing persists past the session; pass rdomains_cache_dir() to opt
into a cache that does.
Seconds a cached page stays fresh.
Prune the cache above this many bytes, oldest first.
Hosts exempt from the private-address checks, by exact name. Only the hosts named are exempt, so a redirect elsewhere is still refused. Intended for testing against a local server.
Opt in to archived content (default FALSE). When a host serves an
anti-bot interstitial, try the
Internet Archive's most recent capture. Rows recovered this way carry
source = "archive" and a snapshot_timestamp, so the vintage is never
hidden. Dead domains and explicitly exempted hosts are not recovered this way.
Fetch each domain as it was on this date ("YYYYMMDD") from the
Internet Archive instead of fetching it live. This is how you ask what a domain
used to be – and, set against a live run, how you measure whether a label has gone
stale. Rows carry source = "archive" and the realised snapshot_timestamp, which
is the latest capture on or before the requested day.
Override the identifying user-agent.
A tibble with one row per input: input_id, input, domain_name,
archive_error_code, status, stage, error_code,
retryable, http_status, final_url, fetched_at, content_bytes, title,
description, lang, text, html, n_tokens, page_state, block_vendor,
robots_allowed, from_cache, source, snapshot_timestamp, source_last_published.
Live pages have no inferred publication date; fetched_at records retrieval.
The crawler identifies itself as rdomains/<version>, obeys robots.txt including
Crawl-delay, spaces requests to the same host, caps the response body, and refuses to
fetch hosts that resolve to private or link-local addresses.
fetch_report() to summarise the run, page_signals() for what the page
states are, source_vintage() for how a live fetch compares with the static lists.
if (FALSE) { # \dontrun{
res <- collect_content(c("example.com", "wikipedia.org"))
fetch_report(res)
# retry only what is worth retrying
again <- collect_content(res$input[res$retryable])
} # }