pie_cat.RdFetches each domain's homepage and classifies its **current** content, rather than looking the domain up in a list written years ago. This is the answer to the problem [source_vintage()] documents: `shalla_cat()` reports what a domain was in January 2022, `pie_cat()` reports what it is today.
pie_cat(domains = NULL, pages = NULL, threshold = 0.1, ...)Character vector of domains. Ignored when `pages` is supplied.
A tibble from [collect_content()], to classify pages you have already fetched. Rows that failed to fetch are returned untouched, so a partial run can be completed without refetching what already worked.
Probability floor for the `pie_categories` list. The argmax is always included even when it falls below this. Applied to whatever piedomains returns; the model's own multi-label threshold is a piedomains setting.
Passed to [collect_content()] when `pages` is not supplied.
The [collect_content()] tibble plus `pie_category`, `pie_confidence`, `pie_categories` (a list-column of category/probability tibbles), `label_source` (`"model"` or `"heuristic"`) and `model_repo`.
Through the Python package `piedomains`, via `reticulate`. That is deliberate rather than convenient. The model input has four pieces that are easy to get subtly wrong and impossible to notice when you do:
* the text is prefixed with the domain minus its TLD – training fed `domain + " " + text`, and a serving path once omitted it; * probabilities are `softmax(logits / temperature)` with a temperature read from the checkpoint, not 1.0; * merged classes have their probabilities **summed**, not overwritten; * the text cleaner collapses whitespace and lowercases, and does *not* deduplicate or sort, which it used to.
Reimplementing those in R would make this package a second place they can drift. Calling the Python package means parity is structural: the weights come from the same Hugging Face checkpoint and the input contract upgrades with a `pip` bump.
“`r install.packages("reticulate") reticulate::py_install("piedomains", pip = TRUE) “`
The model weights (~1.2 GB) download from Hugging Face on first use and are cached by `transformers` afterwards. Set `RETICULATE_PYTHON` to point at an existing environment if you already have `piedomains` installed somewhere.
[collect_content()] for the fetch, [source_vintage()] for how a live answer compares with the static lists.
if (FALSE) { # \dontrun{
pie_cat(c("cnn.com", "wikipedia.org"))
# fetch once, classify later, without refetching
pages <- collect_content(c("cnn.com", "wikipedia.org"))
pie_cat(pages = pages)
} # }