Some servers refuse any request that does not come from a web browser, with a browser check or a block page instead of the file, 'Cloudflare' being the common case. This package opens a real 'Chrome' with a throwaway profile, lets any check run, and fetches the file or page: through 'curl' carrying the clearance the browser was given where the server accepts that, and through the browser itself where it does not. Nothing is simulated; where a check asks a person to click, a window waits for them. Built on 'chromote'.

Fetch files and pages through a real browser when a plain request is refused.
Some servers refuse any request that does not come from a web browser.
Cloudflare, the common case, refuses in two ways: a browser check, a 403
page that runs a script, sets a cf_clearance cookie and reloads; or a block,
a 403 page that says "Sorry, you have been blocked", because the rule is
about what the client looks like. curl, download.file() and friends never
get past either. A browser does, and this package lets a browser do what a
browser is allowed to do:
safeconduct::fetch(
"https://www12.statcan.gc.ca/census-recensement/2021/geo/sip-pis/rnf-frr/files-fichiers/lrnf000r21a_e.zip",
"lrnf000r21a_e.zip")
If the server serves the file to a plain request, that is what happens and no
browser is involved. Otherwise a Chrome with a throwaway profile is opened on
the host, the check runs if there is one, and the file is downloaded: through
curl carrying the browser's clearance where the server accepts that, and in
the browser itself where it does not. A clearance curl can use is kept for
the rest of the R session, so later files from the host need no browser. A
file downloaded in the browser lands under tempdir() and is moved into
place when complete.
Nothing is simulated. When the check shows a "Verify you are human" box, the
window is brought to the front and waits up to two minutes for the person at
the screen to click it (it wants to stay in front until it is through), and
the browser is not disguised beyond dropping the word Headless from its
user-agent string, which some hosts refuse on that word alone. A download
that must run unattended cannot rely on this package where such a box is
shown, and a server that blocks the browser too is reported as such within
seconds.
Documentation, with the function reference and a getting-started vignette that walks through two refused downloads, is at https://mountainmath.github.io/safeconduct/.
read_page() is fetch() for a page or a feed: the response comes back in
memory, shaped like curl::curl_fetch_memory()'s, by the same route.
page_json(), page_text() and page_html() parse it, taking the charset
from its headers and raising an HTTP error status; each also takes a URL, so
page_html(url) stands in for rvest::read_html(url). For a feed read every
few minutes, keep one browser open on the host:
s <- safeconduct::browser_session("https://results.vancouver.ca/")
r <- safeconduct::read_page("https://results.vancouver.ca/data/arearesults.json",
session = s)
safeconduct::page_json(r)
close(s)
A check the server asks for again meanwhile is passed and the read repeated;
a browser that has gone away is reopened at the next call.
fetch(url, destfile, session = s) downloads a file in the same browser.
By default headless Chrome goes first, and a window is opened only when the
headless run ends on a check or a block, and only when there is a display to
show it on. headed = TRUE opens a window at once, FALSE never does, and
useragent = NULL keeps Chrome's own user-agent string; both arguments are
on every function that may open a browser. Some hosts run a second check at
the file's URL, which headless Chrome passes unclicked after about two
minutes where a window is through as soon as its box is clicked; the package
says so and waits, and headed = TRUE skips the wait. At the console, once a
host has been found to need a window, a tip says so, once per host and R
session.
chromote::find_chrome(); set CHROMOTE_CHROME if it is somewhere
unusual.options(safeconduct.browser = FALSE) never opens a browser.
Errors are classed under safeconduct_error: safeconduct_unavailable when
no browser can be opened here, safeconduct_not_passed when the check was
not passed in time or the server blocks the browser too, and
safeconduct_download_error when the transfer failed. A calling package can
catch these and turn them into instructions to download by hand.
clearance(url) returns the browser's user-agent string and cookies for a
host, and clearance_handle(clearance) puts them on a curl handle, for
requests this package does not make itself. probe(url) says what a server
serves at a URL, and whether it is a browser check (is_challenge()) or a
block page (is_block()).
Console tips are not shown for calls made from another package, and need no
configuration to silence; they are messages of class safeconduct_hint.
# install.packages("pak")
pak::pak("mountainMath/safeconduct")