What kind of page is this?

Source: R/page_signals.R

Inspects already-fetched HTML for the three things that are not a classifiable site: an anti-bot interstitial, a domain-parking placeholder, and a server's "nothing here" page. No network access.

page_signals(html, text = NULL, domain = "", status = NULL)

Arguments

html

Raw response body.

text

Extracted page text. If NULL, derived from html.

domain

The domain requested, used to spot a page whose only content is its own name.

status

HTTP status code, if known.

Value

A one-row tibble: page_state (one of "content", "blocked", "parked", "unavailable", "thin"), blocked, block_vendor, block_reason, parked, unavailable, thin, n_tokens.

Details

parked and unavailable are answers, not failures – they are facts about the domain, plainly stated in the page, that a caller can act on. They are also free: no classification service needs to be consulted.

See also

fetch_error_codes() for how these map onto run outcomes.

Examples

# A Cloudflare challenge, not a website
page_signals("<html><title>Just a moment...</title><body>cf_chl_opt</body></html>")
#> # A tibble: 1 × 8
#>   page_state blocked block_vendor block_reason parked unavailable thin  n_tokens
#>   <chr>      <lgl>   <chr>        <chr>        <lgl>  <lgl>       <lgl>    <int>
#> 1 blocked    TRUE    cloudflare   strong anti… FALSE  FALSE       TRUE         1

# A parking page
page_signals("<html><body>This domain is for sale. Inquire now.</body></html>")
#> # A tibble: 1 × 8
#>   page_state blocked block_vendor block_reason parked unavailable thin  n_tokens
#>   <chr>      <lgl>   <chr>        <chr>        <lgl>  <lgl>       <lgl>    <int>
#> 1 parked     FALSE   NA           NA           TRUE   FALSE       TRUE         7

# A real page that merely embeds reCAPTCHA is not blocked
page_signals(paste0(
  "<html><title>Reddit</title><body>",
  paste(rep("real discussion content", 200), collapse = " "),
  "<script src='recaptcha/api.js'></script></body></html>"
))
#> # A tibble: 1 × 8
#>   page_state blocked block_vendor block_reason parked unavailable thin  n_tokens
#>   <chr>      <lgl>   <chr>        <chr>        <lgl>  <lgl>       <lgl>    <int>
#> 1 content    FALSE   NA           NA           FALSE  FALSE       FALSE      600