The goal of this document is to get you up and running with rsdmx as quickly as possible.
rsdmx
provides a set of classes and methods to read data and metadata documents exchanged through the Statistical Data and Metadata Exchange (SDMX) framework.
The SDMX framework provides two sets of standard specifications to facilitate the exchange of statistical data:
SDMX allows to disseminate both data (a dataset) and metadata (the description of the dataset).
For this, the SDMX standard provides various types of documents, also known as messages. Hence there will be:
Generic
and Compact
ones. The latter aims to provide a more compact XML document. They are other data document types derivating from the ones previously mentioned.Data Structure Definition
(DSD). As its name indicates, it describes the structure and organization of a dataset, and will generally include all the master/reference data used to characterize a dataset. The 2 main types of metadata are (1) the concepts
, which correspond to the dimensions and/or attributes of the dataset, and (2) the codelists
which inventory the possible values to be used in the representation of dimensions and attributes.For more information about the SDMX standards, you can visit the SDMX website, or this introduction by EUROSTAT.
rsdmx offers a low-level set of tools to read data and metadata in the SDMX-ML format. Its strategy is to make it very easy for the user. For this, a unique function named readSDMX
has to be used, whatever it is a data
or metadata
document, or if it is local
or remote
datasource.
What rsdmx
does support:
a SDMX format abstraction library, with focus on the the main SDMX standard XML format (SDMX-ML), and the support of the three format standard versions (1.0
, 2.0
, 2.1
)
an interface to SDMX web-services for a list of well-known data providers, such as OECD, EUROSTAT, ECB, UN FAO, UN ILO, etc (a list that should grow in a near future!). See it in action!
Let's see then how to use rsdmx
!
rsdmx
can be installed from CRAN or from its development repository hosted in Github. For the latter, you will need the devtools
package and run:
devtools::install_github("opensdmx/rsdmx")
To load rsdmx in R, do the following:
library(rsdmx)
This section will introduce you on how to read SDMX dataset documents, either from remote datasources, or from local SDMX files.
The following code snipet shows you how to read a dataset from a remote data source, taking as example the OECD StatExtracts portal: http://stats.oecd.org/restsdmx/sdmx.ashx/GetData/MIG/TOT../OECD?startTime=2000&endTime=2011
myUrl <- "http://stats.oecd.org/restsdmx/sdmx.ashx/GetData/MIG/TOT../OECD?startTime=2000&endTime=2011"
dataset <- readSDMX(myUrl)
stats <- as.data.frame(dataset)
You can try it out with other datasources, such as from the EUROSTAT portal: http://ec.europa.eu/eurostat/SDMX/diss-web/rest/data/cdh_e_fos/..PC.FOS1.BE/?startperiod=2005&endPeriod=2011
The online rsdmx documentation also provides a list of data providers, either from international or national institutions, and more request examples.
Now, the service providers above mentioned are known by rsdmx
which let users using readSDMX
with the helper parameters. The list of service providers can be retrieved doing:
providers <- getSDMXServiceProviders();
as.data.frame(providers)
## agencyId name
## 1 ECB European Central Bank
## 2 ESTAT Eurostat (Statistical office of the European Union)
## 3 OECD Organisation for Economic Cooperation and Development
## 4 FAO Food and Agriculture Organization of the United Nations
## 5 ILO International Labour Organization of the United Nations
## 6 UIS UNESCO Institute of Statistics
## 7 ABS Australian Bureau of Statistics
## 8 NBB National Bank of Belgium
## 9 INSEE Institut national de la statistique et des études économiques
## scale country
## 1 international <NA>
## 2 international <NA>
## 3 international <NA>
## 4 international <NA>
## 5 international <NA>
## 6 international <NA>
## 7 national AUS
## 8 national BEL
## 9 national FRA
Note it is also possible to add an SDMX service provider at runtime. For registering a new SDMX service provider by default, please contact me!
Let's see how it would look like for querying an OECD
datasource:
sdmx <- readSDMX(agencyId = "OECD", resource = "data", flowRef = "MIG",
key = list("TOT", NULL, NULL), start = 2010, end = 2011)
df <- as.data.frame(sdmx)
head(df)
## CO2 VAR GEN COU attrs.df obsTime obsValue OBS_STATUS
## 1 TOT B11 WMN AUS P1Y 2010 107740 <NA>
## 2 TOT B11 WMN AUS P1Y 2011 108865 <NA>
## 3 TOT B11 TOT AUS P1Y 2010 206714 <NA>
## 4 TOT B11 TOT AUS P1Y 2011 210704 <NA>
## 5 TOT B12 TOT AUS P1Y 2010 29307 <NA>
## 6 TOT B12 TOT AUS P1Y 2011 31204 <NA>
This example shows you how to use rsdmx
with local SDMX files, previously downloaded from EUROSTAT.
#bulk download from Eurostat
tf <- tempfile(tmpdir = tdir <- tempdir()) #temp file and folder
download.file("http://ec.europa.eu/eurostat/estat-navtree-portlet-prod/BulkDownloadListing?sort=1&file=data%2Frd_e_gerdsc.sdmx.zip", tf)
sdmx_files <- unzip(tf, exdir = tdir)
#read local SDMX (set isURL = FALSE)
sdmx <- readSDMX(sdmx_files[2], isURL = FALSE)
stats <- as.data.frame(sdmx)
By default, readSDMX
considers the data source is remote. To read a local file, add isURL = FALSE
.
This section will introduce you on how to read SDMX metadata documents, including concepts
, codelists
and a complete data structure definition
(DSD)
Read concept schemes from FAO data portal
csUrl <- "http://data.fao.org/sdmx/registry/conceptscheme/FAO/ALL/LATEST/?detail=full&references=none&version=2.1"
csobj <- readSDMX(csUrl)
csdf <- as.data.frame(csobj)
Read codelists from FAO data portal
clUrl <- "http://data.fao.org/sdmx/registry/codelist/FAO/CL_FAO_MAJOR_AREA/0.1"
clobj <- readSDMX(clUrl)
cldf <- as.data.frame(clobj)
###Data Structure Definition (DSD)
This example illustrates how to read a complete DSD using a OECD StatExtracts portal data source.
dsdUrl <- "http://stats.oecd.org/restsdmx/sdmx.ashx/GetDataStructure/TABLE1"
dsd <- readSDMX(dsdUrl)
rsdmx
is implemented in object-oriented way with S4
classes and methods. The properties of S4
objects are named slots
and can be accessed with the slot
method. The following code snippet allows to extract the list of codelists
contained in the DSD document, and read one codelist as data.frame
.
#get codelists from DSD
cls <- slot(dsd, "codelists")
#get list of codelists
codelists <- sapply(slot(cls, "codelists"), function(x) slot(x, "id"))
#get a codelist
codelist <- as.data.frame(slot(dsd, "codelists"), codelistId = "CL_TABLE1_FLOWS")
In a similar way, the concepts
of the dataset can be extracted from the DSD and read as data.frame
.
#get concepts from DSD
concepts <- as.data.frame(slot(dsd, "concepts"))