---
title: "Classify, compare, and resume"
author: "Gaurav Sood"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Classify, compare, and resume}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r setup, include=FALSE}
library(rdomains)
```

A domain-list lookup answers a different question from inspecting a page. Choose
an evidence source first, then retain its identity with the result. The examples
below use local fixtures so the vignette can run without network access or keys.

## Preserve every membership

```{r lookup}
path <- tempfile(fileext = ".csv")
write.csv(data.frame(
  domain = c("example.com", "example.com", "university.example"),
  category = c("news", "shopping", "education"),
  usage = c("black", "black", "white")
), path, row.names = FALSE)
results <- ut1_cat(c("https://example.com/article", "missing.example"), use_file = path)
results[c("input_id", "input", "provider", "status", "category")]
results$labels[[1]]
fetch_report(results)
```

The first domain has two memberships, so no single category is selected. The
second has no match, which is different from a failed download. UT1 usage and raw
records remain in `details`. The fixture has no verified metadata, so its dates
are unknown.

## Preserve identifiers when resuming

```{r empty-retry}
pending <- which(results$retryable)
retried <- ut1_cat(results$input[pending], use_file = path)
retried$input_id <- results$input_id[pending]
results[pending, ] <- retried
stopifnot(nrow(results) == 2L, nrow(retried) == 0L)
unlink(path)
```

For a paid provider, save completed chunks with `saveRDS()` and retry only failed
inputs using the same provider, model, and vocabulary. Keep original identifiers
when combining chunks. A result's `no_match` state does not call for a retry.

## Distinguish page state from content classification

```{r page-state}
page_signals("<html><body>This domain is for sale. Inquire now.</body></html>")
news_url_signals(c("https://example.com/sport", "https://example.com/politics"))
```

Page-state detection is heuristic. URL keyword flags are neither mutually
exclusive nor a substitute for reading the page. The content classifier uses
Python's own HTML extraction and its pinned model weights. Its documented
quality limitations still apply.

```{r content, eval=FALSE}
reticulate::py_require("piedomains==0.15.0", python_version = ">=3.12")
pages <- collect_content("https://example.com/article")
classified <- pie_cat(pages = pages)
classified$details[[1]][c("fetched_at", "source", "snapshot_timestamp", "fetch_status")]
```

Historical collection uses the exact URL and the most recent capture on or before
`archive_date`. Current collection does not use archived fallback unless requested.