Select and Download Climate Data from 'DWD' (German Weather Service)

Handle climate data from the 'DWD' ('Deutscher Wetterdienst', see < https://www.dwd.de/EN/climate_environment/cdc/cdc.html> for more information). Choose files with 'selectDWD()', download and process data sets with 'dataDWD()' and 'readDWD()'.


rdwd is an R package to select, download and read climate data from the German Weather Service (Deutscher Wetterdienst, DWD). They provide over 228 thousand datasets with weather observations online at ftp://ftp-cdc.dwd.de/pub/CDC/observations_germany/climate.

rdwd is available on CRAN: CRAN_Status_Badge downloads Rdoc

It has been presented at FOSDEM 2017 and UseR!2017 in Brussels, featured in Rstudios data package list and written about in OSOR.

Usage of the package usually looks something like the following:

# download and install the rdwd package (only needed once):
install.packages("rdwd")
berryFunctions::instGit("brry/rdwd") # latest development version (if wanted)
 
# load the package into library (needed in every R session):
library(rdwd)
 
# select a dataset (e.g. last year's daily climate data from Potsdam City):
link <- selectDWD("Potsdam", res="daily", var="kl", per="recent")
 
# Actually download that dataset, returning the local storage file name:
file <- dataDWD(link, read=FALSE)
 
# Read the file from the zip folder:
clim <- readDWD(file)
 
# Inspect the data.frame:
str(clim)

You can also select datasets with the interactive map.
Installation instructions and more examples are available in the package vignette.

vignette("mapDWD") # interactive map, likely faster than CRAN link above
vignette("rdwd")   # package instructions and examples

A real-life usage example of the package can be found at https://github.com/brry/prectemp/blob/master/Code_analysis.R

help

I'm looking for someone to help implement multiple downloads in dataDWD via e.g. curl or wget. The requirements are as follows:

  • works cross-platform
  • is called from R
  • has as few dependencies as possible
  • optimally enables a progress bar

News

rdwd ToDo-list / wishlist / issues

  • dataDWD: potentially use curl to download data without sleep need
  • readDWD: figure out reason for fread trouble (reported in emails by some users)

rdwd: DWD Climate Data access package by Berry Boessenkool, [email protected] Any feedback is welcome, per email or on github.com/brry/rdwd

rdwd Version history



Version 1.0.0 (2019-03-17) First major version increase! Tests are now systematic and comprehensive (albeit run locally and manually to reduce CRAN load and enable local file writing). selectDWD got a very noticable performance boost and an interactive res/var/per selection. Handling raster files is now supported in rdwd. Examples and vignettes have been refined. Detailed changes can be found on https://github.com/brry/rdwd/commits, here's the main overview:

  • dataDWD: dots now passed to readDWD, download errors checked and returned as informative warning
  • readDWD: reading functions added for binary, raster and multi_annual files, MESS_DATUM POSIXct column added in monthly data, list output now named
  • indexFTP: now removes duplicates from file list, sped up in default 1min/prec usage
  • rdwd now works fine with multi_annual and subdaily data
  • readVars: parameter abbreviations completed, output now visible
  • selectDWD: order of arguments changed, interactive res/var/per selection enabled, computing speed extremely increased
  • localtests.R heavily expanded, examples reduced and refined
  • indexes + vignettes updated, use cases expanded with nearbyStations example New functions and objects: readMeta, dwdbase (central package base url)


Version 0.11.0 (2018-11-26)

  • dataDWD: argument force can now be NA to download files older than 24 hours
  • hourly/solar timestamp now processed automatically
  • use cases vignette added
  • vignettes precompiled + CRAN tests removed to reduce automated load on DWD Server
  • indexFTP: got much smarter in discerning files from folders
  • indexes expanded for new DWD data New function: readVars, along with parameter_abreviations


Version 0.10.0 (2018-03-26)

  • created new index with sub-hourly data, reflected in data overview at selectDWD
  • dataDWD,readDWD,selectDWD: file ending checks more elegant
  • createIndex: updated to handle the new files
  • some minor improvements, see https://github.com/brry/rdwd/commits


Version 0.9.0 (2017-11-03)

  • readDWD: readDWD.data and readDWD.meta are now separate (non-exported) functions
  • readDWD: fread default is now FALSE
  • dataDWD: fread and overwrite arguments added
  • references dutch meteo package and useR!2017 rdwd presentation
  • importFrom entries completed
  • DESCRIPTION: BugReports entry added


Version 0.8.0 (2017-06-09) reflects DWD FTP update June 1

  • indexFTP: ftp blockage chance reduced, data loss avoided, progress bar added
  • errors/warnings/messages now often include a traceback
  • metaInfo tells about non-public files
  • createIndex + readDWD read meta files in German locale to handle Umlaute
  • Index documentation and creation unified
  • Vignettes and readme improved
  • various minor improvements and fixes, see https://github.com/brry/rdwd/commits renamed: indexDWD -> indexFTP added: lldist, maxlldist, nearbyStations no longer exported in NAMESPACE: rowDisplay, dirDWD removed: fileDWD, geoIndexAll


Version 0.7.0 (2017-02-03)

  • interactive map now colored by availability of recent file
  • stations with slightly varying coordinates (<900 m apart) aggregated in geoIndex
  • documentation corrections, improved messages in fileDWD and dirDWD
  • map vignette expanded, package vignette abbreviated slightly
  • metaInfo printout greatly improved
  • duplicate index entries removed New function: rowDisplay New object: geoIndexAll Removed object: mapDWD (to reduce package size)


Version 0.6.1 (2017-01-24) Initial release of the package, development can be seen at https://github.com/brry/rdwd/compare/[email protected]{2016-10-19}[email protected]{2017-01-24}

'selectDWD' uses index information to find files matching a path or station criteria. With the returned path/file names, 'dataDWD' and 'readDWD' download and read data. 'dirDWD' and 'fileDWD' control that no file will be overwritten and path messages are useful. 'indexDWD' lists all the files on the FTP server with DWD data. 'createIndex' uses that list to create 'fileIndex', 'metaIndex' and 'geoIndex'. 'mapDWD' is an interactive leaflet map also useful to explore datasets.

Note: 'dataDWD' and 'readDWD' were started in June 2016 within my misc package berryFunctions, from which they will be deleted after rdwd is on CRAN



Reference manual

It appears you don't have a PDF plugin for this browser. You can click here to download the reference manual.

install.packages("rdwd")

1.1.0 by Berry Boessenkool, 4 months ago


https://github.com/brry/rdwd


Report a bug at https://github.com/brry/rdwd/issues


Browse source code at https://github.com/cran/rdwd


Authors: Berry Boessenkool


Documentation:   PDF Manual  


Task views: Hydrological Data and Modeling


GPL (>= 2) license


Imports berryFunctions, pbapply

Suggests RCurl, leaflet, knitr, rmarkdown, testthat, data.table, OSMscale, R.rsp, raster, R.utils

System requirements: Pandoc (>= 1.12.3), pandoc-citeproc


Suggested by segmentr.


See at CRAN