NEWS.md
openai_cat() and claude_cat() now run on the ellmer package and a new llm_cat() accepts any ellmer chat object, so Gemini, Ollama, Bedrock and other providers work too. Replies are constrained to the category vocabulary through the providers’ structured-output APIs, transient failures (429, 503) are retried, and repeated hosts are sent once.mode argument. The default, "batch", uses the provider batch API at about half the price with a state_file for interruption and resume; when state_file is not given it is derived from the provider, model and prompts under rdomains_cache_dir(), so the same call collects the same job. A batch that is not yet complete returns rows with the new retryable error_code "batch_pending". Providers without a batch API are refused with a pointer to "parallel" (max_active, rpm); "sequential" sends one request at a time.delay argument; ellmer’s rate limiting replaces the fixed sleep. provider for Claude results is now "anthropic". http_status is NA for successful requests because ellmer does not expose the response. details gains per-request input_tokens, output_tokens and ellmer’s cost estimate.cs.stanford.edu matches a list entry for stanford.edu; details$matched_key records which key applied. Steven Black stays host-level.get_ut1_data() no longer follows the seven symlink aliases in the UT1 archive (porn -> adult, ads -> publicite, drugs -> drogue, aggressive and violence -> agressif, mail -> forums, proxy -> redirector). Every domain on those lists was written twice under two names, so ut1_cat() reported no single category for 4.6 million adult domains. Re-run get_ut1_data(overwrite = TRUE).
html_text_content() hands the body to xml2 as bytes. A tagless string was opened as a file path or URL.
The private-address guard checks every resolved address, not curl’s first.
VirusTotal input rejections are invalid_domain and not retryable.
Page-cache entries are bound to the robots, allow_hosts and max_bytes settings they were fetched under, and are stored byte for byte. The cache namespace moved to v3, so old entries are ignored. cache_clear() validates its argument.
Documentation: fetch_report()$missing lists failed rows; the result contract describes the top-scoring category that pie_cat() returns.
Explicit model selections and R environment changes reach an already initialized Python interpreter. Model loading restores both runtimes’ prior environment values, including when a checkpoint fails to load.
Classifiers now return a common result with input_id, original input, provider, evidence type, structured outcomes, typed labels, and source-specific details. Provider-specific category columns become category; pie_confidence becomes confidence. Vendor labels and multiple list memberships are retained in labels. University records, UT1 usage, and piedomains fetch/model metadata move into details.
Empty domain vectors return typed empty results without I/O. no_match distinguishes absent memberships from execution failures. All download helpers return a path invisibly.
rate_limit is renamed delay, measured in seconds. web_categories() exposes the shared default LLM vocabulary. API failures retain status, reason and model identity.
not_news() is replaced by news_url_signals() with independent soft_news and hard_news flags. adult_ml1_cat() is replaced by adult_name_probability(); its fitted model and training-era feature encoding are unchanged. No compatibility aliases are retained.
Steven Black results describe membership only: blocked or no_match. Hostname fragments no longer imply advertising or malware. UT1 retains every category membership.
uni_cat() accepts a saved JSON file and caches its automatic download for the session.
Current-content collection defaults to archive_fallback = FALSE. Historical and fallback requests preserve the full URL. Archive outages and robots transport failures have distinct retryable outcomes; not-found pages and unresolved redirects do not report success.
Corrupt cache records are misses. Substantive articles mentioning parking marketplaces are not classified as parked by that mention alone. Acquired pages share one processing path.
VirusTotal preserves typed authentication, quota and HTTP failures; invalid vendor labels fail the affected row. Archive resource guards retain their permanent failure reasons.
Inference-only retries preserve caller-restored batch IDs. Mocked model tests restore the session cache, so mocked and real-model checks can run together.
pie_cat(model = ...) selects a checkpoint explicitly. The pinned default is reused within a session; custom checkpoints are loaded afresh. Saved results support inference-only retry.
R 4.5 is the tested minimum. A dedicated R/Python job exercises the bridge and verifies pinned-weight parity both online and offline. The unused XML dependency is removed.
This is an API and correctness release, not a newly trained classifier. The pinned piedomains checkpoint’s drugs/retail limitations and the historical adult model’s lack of current calibration remain documented. No new model-quality claim is made.
pages inputs must be fetched again.gpt-6-luna and claude-haiku-5-5 with their supported request parameters. Reject invalid/truncated categories and preserve explicit VirusTotal key precedence. Live paid-API tests now require explicit opt-in.domains and both Steven Black and VirusTotal output to domain_name. VirusTotal credentials now use api_key.no_match; it is not a safety assessment.collect_content() now caches fetched pages. A cache hit replays the stored provenance – the time the page was really fetched, the URL it really resolved to – rather than synthesising a fresh timestamp, so a cached row still says when it is from. Entries expire (cache_ttl), the cache is pruned to a size budget (cache_max_size), and a corrupt entry is a miss rather than an error. Nothing persists past the session unless you pass rdomains_cache_dir().rdomains_cache_dir() and cache_clear().collect_content() and pie_cat() take an archive_date, fetching each domain as it was on or before that date. Rows carry the realised snapshot_timestamp, which is the capture found and rarely the date asked for.collect_content() recovers pages from the Internet Archive when a host serves an anti-bot interstitial – detection plus a fallback rather than an evasion arms race. Recovered rows carry source = "archive" and a snapshot_timestamp, so the vintage is never hidden. A domain that no longer resolves is deliberately not recovered this way: answering “what was this” while looking like “what is this” is the staleness this package exists to surface.pie_cat(): fetches a domain’s homepage and classifies its current content with the piedomains model, rather than looking the domain up in a list written years ago. It reaches the model through the Python package via reticulate, so the four fragile pieces of the model input – the domain prefix, temperature scaling, label projection and the text cleaner – stay on the Python side rather than becoming a second place they can drift. parked and unavailable pages are labelled from the page itself and cost no model call.collect_content()’s max_bytes default is now 10 MB, matching piedomains. At 2 MB it rejected cnn.com – roughly 6 MB of homepage – as content_too_large while the server returned 200.ut1_cat() and get_ut1_data(): the UT-Capitole blacklists, the maintained successor to Shallalist. Where shalla_cat() answers from a list that stopped in January 2022, this one is updated continuously. Categories are UT1’s own and are reported verbatim, with ut1_usage saying whether UT1 maintains the list for blocking or for allowing – both describe content.collect_content() honours an explicit http:// or https:// in the input, keeping its port and path. Forcing https:// meant http-only hosts were unreachable.content_too_large rather than connection_error. curl aborts the transfer, so the limit surfaced as a request failure – and connection_error is retryable, so callers would have retried a too-large page forever.webfakes), covering redirects to private addresses, robots.txt refusal, oversized bodies, bot walls, and whether the cache actually prevents a second request.Two of the four lookup sources are no longer published: DMOZ closed in March 2017 and Shallalist stopped in 2022. Their labels were correct when assigned, but domains expire and change hands, so a lookup today can return the previous registrant’s category. Until now that answer was presented identically to one from a list updated last week.
source_vintage() reports every category source, when it was last published, whether it is still maintained, and its successor where one exists.shalla_cat(), dmoz_cat() and stevenblack_cat() now return a source_last_published column. For the two dead lists this is a constant; for Steven Black’s actively-maintained hosts file it is the fetched file’s own date.The sibling project piedomains measured what this confusion costs: its worst class disagreed with its own page content 71% of the time, and the cause was not bad annotation but roughly a decade between the label and the page. Only 60% of the domains it trained on still resolve.
New collect_content() fetches homepage HTML and text, so a domain can be classified on what it says today rather than on what a list said years ago. It returns one row per requested domain, never dropped, each carrying status, stage, error_code and retryable – so a transient failure is distinguishable from a permanent one and only the right rows get retried. fetch_error_codes() documents the closed set of reasons; fetch_report() summarises a run.
Supporting functions, all usable on their own if you already hold HTML:
page_signals() reports whether a page is an anti-bot interstitial, a domain-parking placeholder, a server’s “nothing here” page, or too thin to classify. Vendor presence alone is not a block: reddit, walmart and quora all serve real pages while embedding reCAPTCHA.html_text_content() extracts text, title, description and language.The crawler identifies itself as rdomains/<version> with a contact URL, obeys robots.txt including Crawl-delay, spaces requests to the same host, caps the response body, follows redirects by hand so every hop is re-validated, and refuses hosts resolving to private or link-local addresses.
Static HTML only – no headless browser, so a JavaScript-rendered page comes back thin and says so.
get_dmoz_data() built its output path with paste0(), so the documented default outdir = "." produced a hidden .dmoz_domain_category.csv that dmoz_cat() would then fail to find. It now uses file.path(), matching its two siblings.stevenblack_cat(use_file = NULL) re-downloaded roughly 4 MB on every call. The file is now cached for the session.virustotal_cat() to use VirusTotal API v3 (previously v2.0)virustotal_cat() implementation to properly extract categories from v3 API response structureclean_domains() - standardized domain cleaningvalidate_domains() - comprehensive input validationvalidate_data_file() - consistent file validationget_api_key() - unified API key retrievalbuild_categorization_prompt() - LLM prompt constructionapply_rate_limit() - rate limiting logic:: notation for imported functions (cleaner code, consistent with @importFrom)get_alexa_data() has been removed (service discontinued)virustotal_cat() parameter renamed from domain to domains for consistencyopenai_cat() and claude_cat() functions*_cat() functions for seamless integration