Functions to Automate Downloading Geospatial Data Available from Several Federated Data Sources

Functions to automate downloading geospatial data available from several federated data sources (mainly sources maintained by the US Federal government). Currently, the package enables extraction from seven datasets: The National Elevation Dataset digital elevation models (1 and 1/3 arc-second; USGS); The National Hydrography Dataset (USGS); The Soil Survey Geographic (SSURGO) database from the National Cooperative Soil Survey (NCSS), which is led by the Natural Resources Conservation Service (NRCS) under the USDA; the Global Historical Climatology Network (GHCN), coordinated by National Climatic Data Center at NOAA; the Daymet gridded estimates of daily weather parameters for North America, version 3, available from the Oak Ridge National Laboratory's Distributed Active Archive Center (DAAC); the International Tree Ring Data Bank; and the National Land Cover Database (NLCD).


cranversion CRAN downloads permonth CRANdownloads BuildStatus ZenodoDOI ROpenSciStatus

FedData is an R package implementing functions to automate downloading geospatial data available from several federated data sources (mainly sources maintained by the US Federal government).

FedData version 2.5 will be the final minor CRAN release of FedData 2. FedData 3 will be released in the coming months, but some code built on FedData 2 will not be compatible with FedData 3.

Currently, the package enables extraction from seven datasets:

This package is designed with the large-scale geographic information system (GIS) use-case in mind: cases where the use of dynamic web-services is impractical due to the scale (spatial and/or temporal) of analysis. It functions primarily as a means of downloading tiled or otherwise spatially-defined datasets; additionally, it can preprocess those datasets by extracting data within an area of interest (AoI), defined spatially. It relies heavily on the sp, raster, and rgdal packages.

This package has been built and tested on a source (Homebrew) install of R on macOS 10.13 (High Sierra), and has been successfully run on Ubuntu 14.04.5 LTS (Trusty), Ubuntu 16.04.1 LTS (Xenial) and binary installs of R on Mac OS 10.13 and Windows 10.

Contributors

  • Dylan Beaudette - USDA-NRCS Soil Survey Office, Sonora, CA
  • Scott Chamberlain - ROpenSci and Museum of Paleontology at UC Berkeley

Install FedData

  • From CRAN:

    install.packages('FedData')
  • Development version from GitHub:

    install.packages("devtools")
    devtools::install_github("ropensci/FedData")
  • Linux (Ubuntu 14.04.5 or 16.04.1):

    First, in terminal: bash sudo add-apt-repository ppa:ubuntugis/ppa -y sudo apt-get update -q sudo apt-get install libssl-dev libcurl4-openssl-dev netcdf-bin libnetcdf-dev gdal-bin libgdal-dev Then, in R:

    update.packages("survival")
    install.packages("devtools")
    devtools::install_github("ropensci/FedData")

Demonstration

This demonstration script is available as an R Markdown document in the GitHub repository: https://github.com/ropensci/FedData.

Load FedData and define a study area

# FedData Tester
library(FedData)
library(magrittr)
 
# Extract data for the Village Ecodynamics Project "VEPIIN" study area:
http://veparchaeology.org
vepPolygon <- polygon_from_extent(raster::extent(672800, 740000, 4102000, 4170000),
                                  proj4string = "+proj=utm +datum=NAD83 +zone=12")

Get and plot the National Elevation Dataset for the study area

# Get the NED (USA ONLY)
# Returns a raster
NED <- get_ned(template = vepPolygon,
               label = "VEPIIN")
# Plot with raster::plot
raster::plot(NED)

Get and plot the Daymet dataset for the study area

# Get the DAYMET (North America only)
# Returns a raster
DAYMET <- get_daymet(template = vepPolygon,
               label = "VEPIIN",
               elements = c("prcp","tmax"),
               years = 1980:1985)
# Plot with raster::plot
raster::plot(DAYMET$tmax$X1985.10.23)

Get and plot the daily GHCN precipitation data for the study area

# Get the daily GHCN data (GLOBAL)
# Returns a list: the first element is the spatial locations of stations,
# and the second is a list of the stations and their daily data
GHCN.prcp <- get_ghcn_daily(template = vepPolygon, 
                            label = "VEPIIN", 
                            elements = c('prcp'))
# Plot the NED again
raster::plot(NED)
# Plot the spatial locations
sp::plot(GHCN.prcp$spatial,
         pch = 1,
         add = TRUE)
legend('bottomleft',
       pch = 1,
       legend="GHCN Precipitation Records")

Get and plot the daily GHCN temperature data for the study area

# Elements for which you require the same data
# (i.e., minimum and maximum temperature for the same days)
# can be standardized using standardize==T
GHCN.temp <- get_ghcn_daily(template = vepPolygon, 
                            label = "VEPIIN", 
                            elements = c('tmin','tmax'), 
                            years = 1980:1985,
                            standardize = TRUE)
#> Warning: Unknown or uninitialised column: 'ELEMENT'.
#> Warning in get_ghcn_daily_station(ID = station, elements = elements, years
#> = years, : Elements not available: TMAX, TMIN
#> Warning: Unknown or uninitialised column: 'ELEMENT'.
#> Warning in get_ghcn_daily_station(ID = station, elements = elements, years
#> = years, : Elements not available: TMAX, TMIN
#> Warning: Unknown or uninitialised column: 'ELEMENT'.
#> Warning in get_ghcn_daily_station(ID = station, elements = elements, years
#> = years, : Elements not available: TMAX, TMIN
# Plot the NED again
raster::plot(NED)
# Plot the spatial locations
sp::plot(GHCN.temp$spatial,
         add = TRUE,
         pch = 1)
legend('bottomleft',
       pch = 1,
       legend = "GHCN Temperature Records")

Get and plot the National Hydrography Dataset for the study area

# Get the NHD (USA ONLY)
NHD <- get_nhd(template = vepPolygon, 
               label = "VEPIIN")
# Plot the NED again
raster::plot(NED)
# Plot the NHD data
NHD %>%
  lapply(sp::plot,
         col = 'black',
         add = TRUE)

Get and plot the NRCS SSURGO data for the study area

# Get the NRCS SSURGO data (USA ONLY)
SSURGO.VEPIIN <- get_ssurgo(template = vepPolygon, 
                     label = "VEPIIN")
# Plot the NED again
raster::plot(NED)
# Plot the SSURGO mapunit polygons
plot(SSURGO.VEPIIN$spatial,
     lwd = 0.1,
     add = TRUE)

Get and plot the NRCS SSURGO data for particular soil survey areas

# Or, download by Soil Survey Area names
SSURGO.areas <- get_ssurgo(template = c("CO670","CO075"), 
                           label = "CO_TEST")
 
# Let's just look at spatial data for CO675
SSURGO.areas.CO675 <- SSURGO.areas$spatial[SSURGO.areas$spatial$AREASYMBOL=="CO075",]
 
# And get the NED data under them for pretty plotting
NED.CO675 <- get_ned(template = SSURGO.areas.CO675,
                            label = "SSURGO_CO675")
               
# Plot the SSURGO mapunit polygons, but only for CO675
plot(NED.CO675)
plot(SSURGO.areas.CO675,
     lwd = 0.1,
     add = TRUE)

Get and plot the ITRDB chronology locations in the study area

# Get the ITRDB records
ITRDB <- get_itrdb(template = vepPolygon,
                        label = "VEPIIN",
                        makeSpatial = TRUE)
#> Warning in eval(jsub, SDenv, parent.frame()): NAs introduced by coercion
# Plot the NED again
raster::plot(NED)
# Map the locations of the tree ring chronologies
plot(ITRDB$metadata,
     pch = 1,
     add = TRUE)
legend('bottomleft',
       pch = 1,
       legend = "ITRDB chronologies")

Get and plot the National Land Cover Dataset for the study area

# Get the NLCD (USA ONLY)
# Returns a raster
NLCD <- get_nlcd(template = vepPolygon,
                 year = 2011,
                 dataset = "landcover",
                 label = "VEPIIN")
# Plot with raster::plot
raster::plot(NLCD)

 
# You can also download the Canopy or impervious datasets:
NLCD_canopy <- get_nlcd(template = vepPolygon,
                 year = 2011,
                 dataset = "canopy",
                 label = "VEPIIN")
# Plot with raster::plot
raster::plot(NLCD_canopy)

 
NLCD_impervious <- get_nlcd(template = vepPolygon,
                 year = 2011,
                 dataset = "impervious",
                 label = "VEPIIN")
# Plot with raster::plot
raster::plot(NLCD_impervious)


Acknowledgements

This package is a product of SKOPE (Synthesizing Knowledge of Past Environments) and the Village Ecodynamics Project. This software is licensed under the MIT license.

FedData was reviewed for rOpenSci by @jooolia, and was greatly improved as a result. rOpenSci onboarding was coordinated by @sckott.

News

FedData 2.5.1

  • Switch to laze-loading data.
  • Updated NHD paths to new National Map directory structure.

FedData 2.5.0

  • Added functions for the National Land Cover Database.

FedData 2.4.7

  • SSURGO fixed test where supplying an unavailable survey area now returns NULL instead of an error.
  • SSURGO zip directory encoding changes as of late October 2017 forced changes in the FedData:::get_ssurgo_study_area function.
  • Fixed issue where NHD template wouldn't load because they added a jpeg preview to the directory.

FedData 2.4.6

  • DAYMET functions now do not operate in parallel. This was breaking the download functions.
  • Final update for version 2 of FedData.
  • Accepted to ROpenSci! Migrating to the ROpenSci organization on GitHub.

FedData 2.4.3

  • writeOGR for SSURGO and NHD were failing on Windows when the extraction.dir included a trailing slash. Paths are now normalized to remove the trailing slash.

FedData 2.4.2

  • Updated the get_ned function to provide more useful errors and warnings when downloads are unsuccessful.

FedData 2.4.1

  • Added pkgdown site.
  • SSURGO functions (e.g., get_ssurgo) now doesn't bomb on large (> 1 billion sq meter) requests. Now, the area of interest is broken into smaller chunks to build the download list.

FedData 2.4.0

  • Added a NEWS.md file to track changes to the package.
  • Updated DAYMET functions to fix a bug that downloaded only one tile at a time.
  • Linted all code.

Reference manual

It appears you don't have a PDF plugin for this browser. You can click here to download the reference manual.

install.packages("FedData")

2.5.3 by R. Kyle Bocinsky, a month ago


https://github.com/ropensci/FedData


Report a bug at https://github.com/ropensci/FedData/issues


Browse source code at https://github.com/cran/FedData


Authors: R. Kyle Bocinsky [aut, cre], Dylan Beaudette [ctb], Scott Chamberlain [ctb]


Documentation:   PDF Manual  


MIT + file LICENSE license


Imports data.table, devtools, soilDB, igraph, curl, methods, rgdal, raster, Hmisc, rgeos, readr, lubridate, dplyr, magrittr, foreach, ncdf4, stringr, sf, httr

Depends on sp

Suggests testthat, covr, roxygen2


See at CRAN