What kind of page is this?

Inspects already-fetched HTML for the three things that are not a classifiable site: an anti-bot interstitial, a domain-parking placeholder, and a server's "nothing here" page. No network access.

page_signals(html, text = NULL, domain = "", status = NULL)

Arguments

html: Raw response body.
text: Extracted page text. If `NULL`, derived from `html`.
domain: The domain requested, used to spot a page whose only content is its own name.
status: HTTP status code, if known.

Value

A one-row tibble: `page_state` (one of `"content"`, `"blocked"`, `"parked"`, `"unavailable"`, `"thin"`), `blocked`, `block_vendor`, `block_reason`, `parked`, `unavailable`, `thin`, `n_tokens`.

Details

`parked` and `unavailable` are **answers, not failures** – they are facts about the domain, plainly stated in the page, that a caller can act on. They are also free: no classification service needs to be consulted.

Examples

# A Cloudflare challenge, not a website
page_signals("<html><title>Just a moment...</title><body>cf_chl_opt</body></html>")
#> # A tibble: 1 × 8
#>   page_state blocked block_vendor block_reason parked unavailable thin  n_tokens
#>   <chr>      <lgl>   <chr>        <chr>        <lgl>  <lgl>       <lgl>    <int>
#> 1 blocked    TRUE    cloudflare   strong anti… FALSE  FALSE       TRUE         1

# A parking page
page_signals("<html><body>This domain is for sale. Inquire now.</body></html>")
#> # A tibble: 1 × 8
#>   page_state blocked block_vendor block_reason parked unavailable thin  n_tokens
#>   <chr>      <lgl>   <chr>        <chr>        <lgl>  <lgl>       <lgl>    <int>
#> 1 parked     FALSE   NA           NA           TRUE   FALSE       TRUE         7

# A real page that merely embeds reCAPTCHA is not blocked
page_signals(paste0(
  "<html><title>Reddit</title><body>",
  paste(rep("real discussion content", 200), collapse = " "),
  "<script src='recaptcha/api.js'></script></body></html>"
))
#> # A tibble: 1 × 8
#>   page_state blocked block_vendor block_reason parked unavailable thin  n_tokens
#>   <chr>      <lgl>   <chr>        <chr>        <lgl>  <lgl>       <lgl>    <int>
#> 1 content    FALSE   NA           NA           FALSE  FALSE       FALSE      600

Arguments

Value

Details

See also

Examples