General Purpose Client for 'ERDDAP' Servers

General purpose R client for 'ERDDAP' servers. Includes functions to search for 'datasets', get summary information on 'datasets', and fetch 'datasets', in either 'csv' or 'netCDF' format. 'ERDDAP' information: < https://upwell.pfeg.noaa.gov/erddap/information.html>.


rerddap is a general purpose R client for working with ERDDAP servers.

From CRAN

install.packages("rerddap")

Or development version from GitHub

devtools::install_github("ropensci/rerddap")
library('rerddap')

ERDDAP is a server built on top of OPenDAP, which serves some NOAA data. You can get gridded data (griddap), which lets you query from gridded datasets, or table data (tabledap) which lets you query from tabular datasets. In terms of how we interface with them, there are similarties, but some differences too. We try to make a similar interface to both data types in rerddap.

rerddap supports NetCDF format, and is the default when using the griddap() function. NetCDF is a binary file format, and will have a much smaller footprint on your disk than csv. The binary file format means it's harder to inspect, but the ncdf4 package makes it easy to pull data out and write data back into a NetCDF file. Note the the file extension for NetCDF files is .nc. Whether you choose NetCDF or csv for small files won't make much of a difference, but will with large files.

Data files downloaded are cached in a single hidden directory ~/.rerddap on your machine. It's hidden so that you don't accidentally delete the data, but you can still easily delete the data if you like.

When you use griddap() or tabledap() functions, we construct a MD5 hash from the base URL, and any query parameters - this way each query is separately cached. Once we have the hash, we look in ~/.rerddap for a matching hash. If there's a match we use that file on disk - if no match, we make a http request for the data to the ERDDAP server you specify.

You can get a data.frame of ERDDAP servers using the function servers(). Most I think serve some kind of NOAA data, but there are a few that aren't NOAA data. If you know of more ERDDAP servers, send a pull request, or let us know.

First, you likely want to search for data, specify either griddadp or tabledap

ed_search(query = 'size', which = "table")
#> 11 results, showing first 20 
#>                                                                                         title
#> 1                                                                          CalCOFI Fish Sizes
#> 2                                                                        CalCOFI Larvae Sizes
#> 3                Channel Islands, Kelp Forest Monitoring, Size and Frequency, Natural Habitat
#> 4                                                         CalCOFI Larvae Counts Positive Tows
#> 5                                                                                CalCOFI Tows
#> 6                                                  OBIS - ARGOS Satellite Tracking of Animals
#> 7                                                     GLOBEC NEP MOCNESS Plankton (MOC1) Data
#> 8                                                 GLOBEC NEP Vertical Plankton Tow (VPT) Data
#> 9                             NWFSC Observer Fixed Gear Data, off West Coast of US, 2002-2006
#> 10                                 NWFSC Observer Trawl Data, off West Coast of US, 2002-2006
#> 11 AN EXPERIMENTAL DATASET: Underway Sea Surface Temperature and Salinity Aboard the Oleander
#>             dataset_id
#> 1     erdCalCOFIfshsiz
#> 2     erdCalCOFIlrvsiz
#> 3       erdCinpKfmSFNH
#> 4  erdCalCOFIlrvcntpos
#> 5       erdCalCOFItows
#> 6            aadcArgos
#> 7        erdGlobecMoc1
#> 8         erdGlobecVpt
#> 9   nwioosObsFixed2002
#> 10  nwioosObsTrawl2002
#> 11            nodcPJJU
ed_search(query = 'size', which = "grid")
#> 311 results, showing first 20 
#>                                                                                                 title
#> 1         ROMS3.0 CBLAST2007 Ripples with SWAN-40m res (his case7 ar0fd 0008) [time][eta_rho][xi_rho]
#> 2             ROMS3.0 CBLAST2007 Ripples with SWAN-40m res (his case7 ar0fd 0008) [time][eta_u][xi_u]
#> 3             ROMS3.0 CBLAST2007 Ripples with SWAN-40m res (his case7 ar0fd 0008) [time][eta_v][xi_v]
#> 4  ROMS3.0 CBLAST2007 Ripples with SWAN-40m res (his case7 ar0fd 0008) [time][s_rho][eta_rho][xi_rho]
#> 5   ROMS3.0 CBLAST2007 Ripples with SWAN-40m res (his case7 ar0fd 0008) [time][Nbed][eta_rho][xi_rho]
#> 6         ROMS3.0 CBLAST2007 Ripples with SWAN-40m res (his case7 ar0fd 0009) [time][eta_rho][xi_rho]
#> 7             ROMS3.0 CBLAST2007 Ripples with SWAN-40m res (his case7 ar0fd 0009) [time][eta_u][xi_u]
#> 8             ROMS3.0 CBLAST2007 Ripples with SWAN-40m res (his case7 ar0fd 0009) [time][eta_v][xi_v]
#> 9  ROMS3.0 CBLAST2007 Ripples with SWAN-40m res (his case7 ar0fd 0009) [time][s_rho][eta_rho][xi_rho]
#> 10  ROMS3.0 CBLAST2007 Ripples with SWAN-40m res (his case7 ar0fd 0009) [time][Nbed][eta_rho][xi_rho]
#> 11        ROMS3.0 CBLAST2007 Ripples with SWAN-40m res (his case7 ar0fd 0010) [time][eta_rho][xi_rho]
#> 12            ROMS3.0 CBLAST2007 Ripples with SWAN-40m res (his case7 ar0fd 0010) [time][eta_u][xi_u]
#> 13            ROMS3.0 CBLAST2007 Ripples with SWAN-40m res (his case7 ar0fd 0010) [time][eta_v][xi_v]
#> 14 ROMS3.0 CBLAST2007 Ripples with SWAN-40m res (his case7 ar0fd 0010) [time][s_rho][eta_rho][xi_rho]
#> 15  ROMS3.0 CBLAST2007 Ripples with SWAN-40m res (his case7 ar0fd 0010) [time][Nbed][eta_rho][xi_rho]
#> 16        ROMS3.0 CBLAST2007 Ripples with SWAN-40m res (his case7 ar0fd 0011) [time][eta_rho][xi_rho]
#> 17            ROMS3.0 CBLAST2007 Ripples with SWAN-40m res (his case7 ar0fd 0011) [time][eta_u][xi_u]
#> 18            ROMS3.0 CBLAST2007 Ripples with SWAN-40m res (his case7 ar0fd 0011) [time][eta_v][xi_v]
#> 19 ROMS3.0 CBLAST2007 Ripples with SWAN-40m res (his case7 ar0fd 0011) [time][s_rho][eta_rho][xi_rho]
#> 20  ROMS3.0 CBLAST2007 Ripples with SWAN-40m res (his case7 ar0fd 0011) [time][Nbed][eta_rho][xi_rho]
#>             dataset_id
#> 1  whoi_cbae_ef31_0877
#> 2  whoi_61f3_6dac_36c1
#> 3  whoi_4eff_1b8e_513a
#> 4  whoi_4d06_1f87_db0b
#> 5  whoi_4849_5c78_58dc
#> 6  whoi_28c3_4a74_191a
#> 7  whoi_e627_874c_2b1b
#> 8  whoi_9183_ea0f_9417
#> 9  whoi_322d_c428_66b4
#> 10 whoi_5f75_6229_f722
#> 11 whoi_02c9_858d_bc77
#> 12 whoi_689d_0109_9280
#> 13 whoi_4f6e_439a_8e4b
#> 14 whoi_5451_efbd_60ac
#> 15 whoi_1ace_a9ee_f343
#> 16 whoi_0524_c15f_28da
#> 17 whoi_e5c5_72b2_af51
#> 18 whoi_1fcf_de1b_c428
#> 19 whoi_e305_a468_eabc
#> 20 whoi_0657_2b81_14df

Then you can get information on a single dataset

info('whoi_62d0_9d64_c8ff')
#> <ERDDAP info> whoi_62d0_9d64_c8ff 
#>  Dimensions (range):  
#>      time: (2012-06-25T01:00:00Z, 2015-10-14T00:00:00Z) 
#>      eta_v: (0, 334) 
#>      xi_v: (0, 895) 
#>  Variables:  
#>      bedload_Vsand_01: 
#>          Units: kilogram meter-1 s-1 
#>      bedload_Vsand_02: 
#>          Units: kilogram meter-1 s-1 
#>      bedload_Vsand_03: 
#>          Units: kilogram meter-1 s-1 
#>      bedload_Vsand_04: 
#>          Units: kilogram meter-1 s-1 
#>      bedload_Vsand_05: 
#>          Units: kilogram meter-1 s-1 
#>      bedload_Vsand_06: 
#>          Units: kilogram meter-1 s-1 
#>      svstr: 
#>          Units: newton meter-2 
#>      vbar: 
#>          Units: meter second-1 
#>      wetdry_mask_v:
(out <- info('noaa_esrl_027d_0fb5_5d38'))
#> <ERDDAP info> noaa_esrl_027d_0fb5_5d38 
#>  Dimensions (range):  
#>      time: (1850-01-01T00:00:00Z, 2014-05-01T00:00:00Z) 
#>      latitude: (87.5, -87.5) 
#>      longitude: (-177.5, 177.5) 
#>  Variables:  
#>      air: 
#>          Range: -20.9, 19.5 
#>          Units: degC
(res <- griddap(out,
  time = c('2012-01-01', '2012-01-12'),
  latitude = c(21, 20),
  longitude = c(-80, -79)
))
#> <ERDDAP griddap> noaa_esrl_027d_0fb5_5d38
#>    Path: [~/.rerddap/0c0d352c6ec861f6efadce493e270fd0.nc]
#>    Last updated: [2016-01-12 12:21:40]
#>    File size:    [0 mb]
#>    Dimensions (dims/vars):   [3 X 1]
#>    Dim names: time, latitude, longitude
#>    Variable names: CRUTEM3: Surface Air Temperature Monthly Anomaly
#>    data.frame (rows/columns):   [1 X 4]
#>                   time  lat   lon air
#> 1 2012-01-01T00:00:00Z 22.5 -77.5  NA
(out <- info('erdCalCOFIfshsiz'))
#> <ERDDAP info> erdCalCOFIfshsiz 
#>  Variables:  
#>      calcofi_species_code: 
#>          Range: 19, 1550 
#>      common_name: 
#>      cruise: 
#>      fish_1000m3: 
#>          Units: Fish per 1,000 cubic meters of water sampled 
#>      fish_count: 
#>      fish_size: 
#>          Units: mm 
#>      itis_tsn: 
#>      latitude: 
#>          Range: 32.515, 38.502 
#>          Units: degrees_north 
#>      line: 
#>          Range: 46.6, 93.3 
#>      longitude: 
#>          Range: -128.5, -117.33 
#>          Units: degrees_east 
#>      net_location: 
#>      net_type: 
#>      order_occupied: 
#>      percent_sorted: 
#>          Units: %/100 
#>      sample_quality: 
#>      scientific_name: 
#>      ship: 
#>      ship_code: 
#>      standard_haul_factor: 
#>      station: 
#>          Range: 28.0, 114.9 
#>      time: 
#>          Range: 9.94464E8, 9.9510582E8 
#>          Units: seconds since 1970-01-01T00:00:00Z 
#>      tow_number: 
#>          Range: 2, 10 
#>      tow_type: 
#>      volume_sampled: 
#>          Units: cubic meters
tabledap(out, fields = c('longitude', 'latitude', 'fish_size', 'itis_tsn'),
         'time>=2001-07-07', 'time<=2001-07-10')
#> <ERDDAP tabledap> erdCalCOFIfshsiz
#>    Path: [~/.rerddap/52894d2daf4c71796c44775f06dc3f16.csv]
#>    Last updated: [2016-01-12 11:04:28]
#>    File size:    [0.02 mb]
#>    Dimensions:   [558 X 4]
#> 
#>     longitude  latitude fish_size itis_tsn
#> 2  -118.10667 32.738335      31.5   623625
#> 3  -118.10667 32.738335      48.3   623625
#> 4  -118.10667 32.738335      15.5   162221
#> 5  -118.10667 32.738335      16.3   162221
#> 6  -118.10667 32.738335      17.8   162221
#> 7  -118.10667 32.738335      18.2   162221
#> 8  -118.10667 32.738335      19.2   162221
#> 9  -118.10667 32.738335      20.0   162221
#> 10 -118.10667 32.738335      21.0   162221
#> 11 -118.10667 32.738335      21.5   162221
#> ..        ...       ...       ...      ...
  • Please report any issues or bugs.
  • License: MIT
  • Get citation information for rerddap in R doing citation(package = 'rerddap')
  • Please note that this project is released with a Contributor Code of Conduct. By participating in this project you agree to abide by its terms.

News

rerddap 0.3.4

MINOR IMPROVEMENTS

  • Removed use of ncdf package, which has been taken off CRAN. Using ncdf4 now for all NetCDF file manipulation. (#35)
  • Failing better now with custom error catching (#31)
  • Added many internal checks for parameter inputs, warning or stopping as necessary - ERDDAP servers silently drop with no informative messages (#32)

BUG FIXES

  • Using now file.info()$size instead of file.size() to be backwards compatible with R versions < 3.2

rerddap 0.3.0

NEW FEATURES

  • Cache functions accept the outputs of griddap() and tabledap() so that the user can easily see cache details or delete the file from the cache without having to manually get the file name. (#30)

MINOR IMPROVEMENTS

  • All package dependencies now use importFrom so we only import functions we need instead of their global namespaces.

BUG FIXES

  • Fixed bug in parsing data from netcdf files, affected the griddap() function (#28)

rerddap 0.2.0

NEW FEATURES

  • Added a suite of functions to manage local cached files (#17)

MINOR IMPROVEMENTS

  • Added new ERDDAP server to list of servers in the servers() function (#21)

BUG FIXES

  • Fixed a few cases across a number of functions in which an empty list passed to query parmaeter in httr::GET() caused an error (#23)
  • Fixed retrieval of path to file written to disk by httr::write_disk() (#24)
  • last is a value accepted by ERDDAP servers, but internal functions weren't checking correctly, fixed now. (#25)
  • as.info() wasn't passing on the url parameter to the info() function. fixed now. (#26)

rerddap 0.1.0

NEW FEATURES

  • released to CRAN

Reference manual

It appears you don't have a PDF plugin for this browser. You can click here to download the reference manual.

install.packages("rerddap")

0.4.2 by Scott Chamberlain, 4 months ago


https://github.com/ropensci/rerddap


Report a bug at https://github.com/ropensci/rerddap/issues


Browse source code at https://github.com/cran/rerddap


Authors: Scott Chamberlain [aut, cre], Ben Tupper [ctb], Roy Mendelssohn [ctb]


Documentation:   PDF Manual  


MIT + file LICENSE license


Imports utils, httr, dplyr, data.table, jsonlite, xml2, digest, ncdf4, tibble, hoardr

Suggests roxygen2, knitr, rmarkdown, testthat, akima, ggplot2, mapdata, plot3D

Enhances taxize


See at CRAN