Easier access to scientific data
Brought to you by IGE NOAA NMFS SWFSC ERD
|
IGE's public ERDDAP
Easier access to scientific data |
Brought to you by IGE NOAA NMFS SWFSC ERD |
Data Access Protocol (DAP) and its
selection constraints.
The URL specifies what you want: the dataset, a description of the graph or the subset of the data, and the file
type for the response.
Tabledap request URLs must be in the form
http://osug-smos-rea.osug.fr:8081/erddap/tabledap/datasetID.fileType{?query}
For example,
http://osug-smos-rea.osug.fr:8081/erddap/tabledap/pmelTaoDySst.htmlTable?longitude,latitude,time,station,wmo_platform_code,T_25&time>=2015-05-23T12:00:00Z&time<=2015-05-31T12:00:00Z
Thus, the query is often a comma-separated list of desired variable names, followed by a collection of
constraints (e.g., variable<value),
each preceded by '&' (which is interpreted as "AND").
Details:
| Data fileTypes | Description | Info | Example |
|---|---|---|---|
| .asc | View OPeNDAP-style ISO-8859-1 comma-separated text. | info |
example |
| .csv | Download a ISO-8859-1 comma-separated text table (line 1: names; line 2: units; ISO 8601 times). | info |
example |
| .csvp | Download a ISO-8859-1 .csv file with line 1: name (units). Times are ISO 8601 strings. | info |
example |
| .csv0 | Download a ISO-8859-1 .csv file without column names or units. Times are ISO 8601 strings. | info |
example |
| .das | View the dataset's metadata via an ISO-8859-1 OPeNDAP Dataset Attribute Structure (DAS). | info |
example |
| .dds | View the dataset's structure via an ISO-8859-1 OPeNDAP Dataset Descriptor Structure (DDS). | info |
example |
| .dods | OPeNDAP clients use this to download the data in the DODS binary format. | info |
example |
| .esriCsv | Download a ISO_8859_1 .csv file for ESRI's ArcGIS 9.x and below (separate date and time columns). | info |
example |
| .fgdc | View the dataset's UTF-8 FGDC .xml metadata. | info |
example |
| .geoJson | Download longitude,latitude,otherColumns data as a UTF-8 GeoJSON .json file. | info |
example |
| .graph | View a Make A Graph web page. | info |
example |
| .help | View a web page with a description of tabledap. | info |
example |
| .html | View an OPeNDAP-style HTML Data Access Form. | info |
example |
| .htmlTable | View a UTF-8 .html web page with the data in a table. Times are ISO 8601 strings. | info |
example |
| .iso19115 | View the dataset's ISO 19115-2/19139 UTF-8 .xml metadata. | info |
example |
| .itx | Download an ISO-8859-1 Igor Text File. Each response column becomes a wave. | info |
example |
| .json | View a table-like UTF-8 JSON file (missing value = 'null'; times are ISO 8601 strings). | info |
example |
| .jsonlCSV | View a UTF-8 JSON Lines CSV file (missing value = 'null'; times are ISO 8601 strings). | info |
example |
| .jsonlKVP | View a UTF-8 JSON Lines file with Key:Value pairs (missing value = 'null'; times are ISO 8601 strings). | info |
example |
| .mat | Download a MATLAB binary file. | info |
example |
| .nc | Download a flat, table-like, NetCDF-3 binary file with COARDS/CF/ACDD metadata. | info |
example |
| .ncHeader | View the UTF-8 header (the metadata) for the NetCDF-3 .nc file. | info |
example |
| .ncCF | Download a NetCDF-3 CF Discrete Sampling Geometries file (Contiguous Ragged Array). | info |
example |
| .ncCFHeader | View the UTF-8 header (the metadata) for the .ncCF file. | info |
example |
| .ncCFMA | Download a NetCDF-3 CF Discrete Sampling Geometries file (Multidimensional Array). | info |
example |
| .ncCFMAHeader | View the UTF-8 header (the metadata) for the .ncCFMA file. | info |
example |
| .nccsv | Download a NetCDF-3-like 7-bit ASCII NCCSV .csv file with COARDS/CF/ACDD metadata. | info |
example |
| .nccsvMetadata | View the dataset's metadata as the top half of a 7-bit ASCII NCCSV .csv file. | info |
example |
| .ncoJson | Download an NCO lvl=2 JSON file with COARDS/CF/ACDD metadata. | info |
example |
| .odvTxt | Download longitude,latitude,time,otherColumns as an ISO-8859-1 ODV Generic Spreadsheet File (.txt). | info |
example |
| .subset | View an HTML form which uses faceted search to simplify picking subsets of the data. | info |
example |
| .tsv | Download a UTF-8 tab-separated text table (line 1: names; line 2: units; ISO 8601 times). | info |
example |
| .tsvp | Download a UTF-8 .tsv file with line 1: name (units). Times are ISO 8601 strings. | info |
example |
| .tsv0 | Download a UTF-8 .tsv file without column names or units. Times are ISO 8601 strings. | info |
example |
| .xhtml | View a UTF-8 XHTML (XML) file with the data in a table. Times are ISO 8601 strings. | info |
example |
ArcGIS
.esriCsv
- ArcGIS is a family of Geographical Information Systems (GIS) products from ESRI:
ArcView, ArcEditor, and ArcInfo. To get data from ERDDAP into your ArcGIS program (version 9.x and below):
.)
Ferret
is a free program for visualizing and analyzing large and complex
gridded datasets. Because tabledap's tabular datasets are very different
from gridded datasets, it is necessary to use Ferret in a very specific way to
avoid serious problems and misunderstandings:
IDL -
IDL is a commercial scientific data visualization program. To get data from
ERDDAP into IDL, first use ERDDAP to select a subset of data and download a .nc file.
Then, use these
instructions
to import the data from the .nc file into IDL.
JSON .json
files are widely used to transfer data to JavaScript scripts running on web pages.
All .json responses from ERDDAP (metadata, gridded data, and tabular/in-situ data) use the
same basic format: a database-like table. Since data from EDDTable datasets is already a table
(with a column for each requested variable), ERDDAP can easily store the data in a .json file.
For example,
{
"table": {
"columnNames": ["longitude", "latitude", "time", "bottle_posn", "temperature1"],
"columnTypes": ["float", "float", "String", "byte", "float"],
"columnUnits": ["degrees_east", "degrees_north", "UTC", null, "degree_C"],
"rows": [
[-124.82, 42.95, "2002-08-17T00:49:00Z", 1, 8.086],
[-124.82, 42.95, "2002-08-17T00:49:00Z", 2, 8.585],
[-124.82, 42.95, "2002-08-17T00:49:00Z", 3, 8.776],
...
[-124.1, 44.65, "2002-08-19T20:18:00Z", 3, null]
]
}
}
All .json responses from ERDDAP have
JSONP
(from .json and
.geoJson) -
Jsonp is an easy way for a JavaScript script on a web
page to import and access data from ERDDAP. Requests for .json and .geoJson files may
include an optional jsonp request by adding &.jsonp=functionName to the end of the query.
Basically, this just tells ERDDAP to add functionName( to the beginning of the response
and ")" to the end of the response.
The functionName must be a series of 1 or more (period-separated) words.
For each word, the first character of functionName must be an ISO 8859 letter or "_".
Each optional subsequent character must be an ISO 8859 letter, "_", or a digit.
If originally there was no query, leave off the "&" in your query.
After the data download to the web page has finished, the data is accessible to the JavaScript
script via that JavaScript function.
Here is an example using
jsonp with ERDDAP and jQuery (thanks to Jenn Patterson of CenCOOS).
MATLAB
.mat
- Matlab users can use tabledap's .mat file type to download data from within
MATLAB. Here is a one line example:
load(urlwrite('http://osug-smos-rea.osug.fr:8081/erddap/tabledap/pmelTaoDySst.mat?time,T_25&station="0n0e"&time>=2015-05-23T12:00:00Z&time<=2015-05-31T12:00:00Z&.draw=lines', 'test.mat'));
The data will be in a MATLAB structure. The structure's name will be the datasetID
plot(pmelTaoDySst.time, pmelTaoDySst.T_25)
ERDDAP stores datetime values in .mat files as "seconds since 1970-01-01T00:00:00Z".
To display one of these values as a String in Matlab, you can use, e.g.,
datastr(cwwcNDBCMet.time(1)/86400 + 719529)
86400 converts ERDDAP's "seconds since" to Matlab's "days since". 719529 converts
ERDDAP's base time of "1970-01-01T00:00:00Z" to Matlab's "0000-01-00T00:00:00Z".
.mat files have a maximum length for identifier names of 32 characters.
If a variable name is longer than that, ERDDAP changes the name of the variable
when writing the .mat file: it generates a hash digest of the variable name and
appends that after the first 25 characters. Thus, long variable names that only
differ at the end will still have unique names in the .mat file.
ERDDAP administrators: you can avoid this problem by specifying shorter variable
destinationNames.
NetCDF
.nc
- Requests for .nc files will always return the data in a table-like, NetCDF-3,
32-bit, .nc file:
Don't use NetCDF-Java, NetCDF-C, NetCDF-Fortran, NetCDF-Perl, or Ferret to try to
access a remote ERDDAP .nc file. It won't work. Instead, use
this approach.
NetCDF
.nc4
- Requests for .nc4 files will always return the data in a table-like, NetCDF-4,
64-bit, .nc file:
Don't use NetCDF-Java, NetCDF-C, NetCDF-Fortran, NetCDF-Perl, or Ferret to try to
access a remote ERDDAP .nc file. It won't work. Instead, use
this approach.
.ncHeader and .nc4Header
- Requests for .ncHeader and .nc4Header files will return the header information (text)
that would be generated if you used
ncdump -h fileName
on the corresponding NetCDF-3 or NetCDF-4 .nc file.
.ncCF
- Requests for a .ncCF file will return a version 3, 32-bit,
NetCDF .nc
file with the
Contiguous Ragged Array Representation associated with the dataset's cdm_data_type,
as defined in the newly ratified
CF
Discrete Geometries conventions
(which were previously named "CF Point Observation Conventions").
.ncCFHeader
- Requests for .ncCFHeader files will return the header information (text) that
would be generated if you used
ncdump -h fileName
on the corresponding .ncCF file.
.ncCFMA
- Requests for a .ncCFMA file will return a verion 3, 32-bit,
NetCDF .nc file
with the Complete or Incomplete, depending on the data, Multidimensional Array Representation
associated with the dataset's cdm_data_type, as defined in the
CF
Discrete Sampling Geometries
conventions, which were previously named "CF Point Observation Conventions".
This is the file type used by the NODC Templates.
A request will succeed only if the dataset has a cdm_data_type other than "Other"
and if the request includes at least one data variable (not just the outer, descriptive variables).
The file will include longitude, latitude, time, and other required descriptive variables, even if
you don't request them.
.ncCFMAHeader
- Requests for .ncCFMAHeader files will return the header information (text) that
would be generated if you used
ncdump -h fileName
on the corresponding .ncCFMA file.
NetCDF-Java, NetCDF-C, NetCDF-Fortran, and NetCDF-Perl
-
Don't try to access an ERDDAP tabledap dataset URL directly with a NetCDF library or tool
(by treating the tabledap dataset as an OPeNDAP dataset or by creating a URL with the .nc file extension).
It won't work.
Ocean Data View .odvTxt
- ODV users can download data in a
ODV Generic Spreadsheet Format .txt file
by requesting tabledap's .odvTxt fileType.
The selected data MUST include longitude, latitude, and time variables.
Any longitude values (0 to 360, or -180 to 180) are fine.
After saving the resulting file (with the extension .txt) in your computer:
OPeNDAP Libraries - Although ERDDAP is an
OPeNDAP-compatible data server,
you can't use
most OPeNDAP client libraries, including
NetCDF-Java, NetCDF-C, NetCDF-Fortran, NetCDF-Perl,
or
Ferret,
to get data directly from an ERDDAP tabledap dataset because those libraries don't
support the OPeNDAP Selection constraints that tabledap datasets use for requesting
subsets of the dataset, nor do they support the sequence data structure in the response.
(But see this other approach that works with NetCDF libraries.)
But you can use the Pydap Client or
Java-DAP2,
because they both support Selection
constraints. With both the Pydap Client and Java-DAP2, when creating the initial
connection to an ERDDAP table dataset, use the tabledap dataset's base URL, e.g.,
http://osug-smos-rea.osug.fr:8081/erddap/tabledap/pmelTaoDySst
Pydap Client
users
can access tabledap datasets via ERDDAP's standard OPeNDAP services.
See the
Pydap Client instructions for accessing sequential data.
Note that the name of a dataset in tabledap will always be a single word, e.g., pmelTaoDySst
in the OPeNDAP dataset URL
http://osug-smos-rea.osug.fr:8081/erddap/tabledap/pmelTaoDySst
and won't ever have a file extension (unlike, for example, .cdp for the sample dataset in the
Pydap instructions). Also, the name of the sequence in tabledap datasets will always be "s"
(unlike "location" for the sample dataset in the Pydap instructions).
Python
is a widely-used computer language that is very popular among scientists.
In addition to the Pydap Client, you can use Python to download various files from ERDDAP
as you would download other files from the web:
import urllib
urllib.urlretrieve("http://baseurl/erddap/tabledap/datasetID.fileType?query", "outputFileName")
Or download the content to an object instead of a file:
import urllib2
response = urllib2.open("http://baseurl/erddap/tabledap/datasetID.fileType?query")
theContent = response.read()
There are other ways to do this in Python. Search the web for more information.
"Jupyter Notebook
is an open-source web application that
allows you to create and
share documents that contain live code, equations, visualizations and explanatory text"
using any of over 40 programming languages, including Python and R.
Here are two sample Jupyter Notebooks that access ERDDAP using Python:
ERDDAP Advanced Search Test and
ERDDAP Timing.
Thanks to Rich Signell.
R Statistical Package -
R is an open source statistical package for many operating systems.
In R, you can download a .csv file from ERDDAP
and then import data from that .csv file
into an R structure (e.g., test). For example:
download.file(url="http://osug-smos-rea.osug.fr:8081/erddap/tabledap/pmelTaoDySst.csv?longitude,latitude,time,station,wmo_platform_code,T_25&time>=2015-05-23T12:00:00Z&time<=2015-05-31T12:00:00Z", destfile="/home/bsimons/test.csv") test<-read.csv(file="/home/bsimons/test.csv")
Making an Image File with a Graph or Map of Tabular Data
If a tabledap request URL specifies a subset of data which is suitable for making a graph or
a map, and the fileType is an image fileType, tabledap will return an image with a graph or map.
tabledap request URLs can include optional graphics commands which let you
customize the graph or map.
As with other tabledap request URLs, you can create these URLs by hand or have a computer
program do it.
Or, you can use the Make A Graph web pages, which simplify creating these URLs (see the
"graph" links in the table of tabledap datasets).
The fileType options for downloading images of graphs and maps of table data are:
| Image fileTypes | Description | Info | Example |
|---|---|---|---|
| .kml | View a .kml file, suitable for Google Earth. | info |
example |
| .smallPdf | View a small .pdf image file with a graph or map. | info |
example |
| View a standard, medium-sized .pdf image file with a graph or map. | info |
example | |
| .largePdf | View a large .pdf image file with a graph or map. | info |
example |
| .smallPng | View a small .png image file with a graph or map. | info |
example |
| .png | View a standard, medium-sized .png image file with a graph or map. | info |
example |
| .largePng | View a large .png image file with a graph or map. | info |
example |
| .transparentPng | View a .png image file (just the data, without axes, landmask, or legend). | info |
example |
Image Size - ".small" and ".large" were ERDDAP's original system for making
different-sized images. Now, for .png and .transparentPng images (not other
image file types), you can also use the
&.size=width|height
parameter to request
an image of any size.
.transparentPng - The .transparentPng file type will make a graph or map without
the graph axes, landmask, or legend, and with a transparent (not opaque white)
background. This file type can be used for any type of graph or map.
For graphs and maps, the default size is 360x360 pixels.
Or, you can use the &.size=width|height
parameter to request an image of any size.
Incompatibilities
Some results file types have restrictions. For example, Google Earth .kml is only
appropriate for results with longitude and latitude values. If a given request is
incompatible with the requested file type, tabledap throws an error.
Command Line Downloads with curl
If you want to download a series of files from ERDDAP, you don't have to request each file's
ERDDAP URL in your browser, sitting and waiting for each file to download.
If you are comfortable writing computer programs (e.g., with C, Java, Python, Matlab, r)
you can write a program with a loop that imports all of the desired data files.
Or, if you are comfortable with command line programs (just running a program, or using bash or tcsh
scripts in Linux or Mac OS X, or batch files in Windows), you can use curl to save results files
from ERDDAP into files on your hard drive, without using a browser or writing a computer program.
ERDDAP+curl is amazingly powerful and allows you to use ERDDAP in many new ways.
On Linux or Mac OS X, curl is probably already installed as /usr/bin/curl.
On Windows, or if your computer doesn't have curl already, you need to
download curl
and install it. To get to a command line in Windows, click on "Start" and type
"cmd" into the search textfield.
("Win32 - Generic, Win32, binary (without SSL)" worked for me in Windows 7.)
Please be kind to other ERDDAP users: run just one script or curl command at a time.
Instructions for using curl are on the
curl man page and in this
curl tutorial.
But here is a quick tutorial related to using curl with ERDDAP:
curl -g "http://coastwatch.pfeg.noaa.gov/erddap/tabledap/cwwcNDBCMet.png?time,atmp&time%3E=2010-09-03T00:00:00Z&time%3C=2010-09-06T00:00:00Z&station=%22TAML1%22&.draw=linesAndMarkers&.marker=5|5&.color=0x000000&.colorBar=|||||" -o NDBCatmpTAML1.pngThe erddapUrl must be percent encoded
. To do this, you need to convert
and JavaScript's
encodeURIComponent()) and there are
.
them in the erddapURL as %5B, %5D, %7B, %7D, respectively.
curl "http://coastwatch.pfeg.noaa.gov/erddap/tabledap/cwwcNDBCMet.png?time,atmp&time%3E=2010-09-03T00:00:00Z&time%3C=2010-09-06T00:00:00Z&station=%22{TAML1,41009,46088}%22&.draw=linesAndMarkers&.marker=5|5&.color=0x000000&.colorBar=|||||" -o NDBCatmp#1.png
curl -v --data 'user=myUserName&password=myPassword' -c cookies.txt -b cookies.txt -k https://baseurl:8443/erddap/login.html
curl -v -c cookies.txt -b cookies.txt -k https://baseurl:8443/erddap/tabledap/datasetID.fileType?query -o outputFileName
DAP
selection constraint query in the form: {resultsVariables}{constraints} .
DAP
selection constraints,
:
and JavaScript's
encodeURIComponent()) and there are
.
.
-
Many numeric variables have an attribute which identifies a
on the right.
(tutorial) as is used by Java.
standard (for example, degrees_C).
standard (for example, Cel).
Or, if you change the fileType in the URL from .png to .graph,
you can see a Make A Graph web page with that request loaded:
http://osug-smos-rea.osug.fr:8081/erddap/tabledap/pmelTaoDySst.graph?time,T_25&station="0n0e"&time>=2015-05-23T12:00:00Z&time<=2015-05-31T12:00:00Z&.draw=lines
That makes it easy for humans to modify an image request to make a
similar graph or map.
Or, if you change the fileType in the URL from .png to a data fileType
(e.g., .htmlTable), you can download the data that was graphed:
http://osug-smos-rea.osug.fr:8081/erddap/tabledap/pmelTaoDySst.htmlTable?time,T_25&station="0n0e"&time>=2015-05-23T12:00:00Z&time<=2015-05-31T12:00:00Z&.draw=lines
A sample map URL is
http://osug-smos-rea.osug.fr:8081/erddap/tabledap/pmelTaoDySst.png?longitude,latitude,T_25&time=2015-05-31T12:00:00Z&.draw=markers&.marker=5%7C5
Or, if you change the fileType in the URL from .png to .graph,
you can see a Make A Graph web page with that request loaded:
http://osug-smos-rea.osug.fr:8081/erddap/tabledap/pmelTaoDySst.graph?longitude,latitude,T_25&time=2015-05-31T12:00:00Z&.draw=markers&.marker=5%7C5
Or, if you change the fileType in the URL from .png to a data fileType
(e.g., .htmlTable), you can download the data that was mapped:
http://osug-smos-rea.osug.fr:8081/erddap/tabledap/pmelTaoDySst.htmlTable?longitude,latitude,T_25&time=2015-05-31T12:00:00Z&.draw=markers&.marker=5%7C5
Some ERDDAP installations do have authentication enabled.
Currently, ERDDAP only supports authentication via Google-managed email accounts,
which includes email accounts at NOAA and many universities.
If an ERDDAP has authentication enabled, anyone with a Google-managed email account
can log in, but they will only have access to the private datasets
that the ERDDAP administrator has explicitly authorized them to access.
For instructions on logging into ERDDAP from a browser or via a script, see
Access to Private Datasets in ERDDAP.
in the HTTP GET request header sent
To generate a citation for a dataset:
If you think of the dataset as a scientific article, you can generate a
citation based on the author (see the "creator_name" or "institution" metadata),
the date that you downloaded the data, the title (see the "title" metadata),
and the publisher (use the Data Access Form URL for the dataset, e.g.,
https://coastwatch.pfeg.noaa.gov/erddap/griddap/jplMURSST41.html ). If the
dataset's metadata includes a
Digital Object Identifier (DOI), please
include that in the citation you create.
and clicking on "Apply for membership".
![http://osug-smos-rea.osug.fr:8081/erddap/tabledap/pmelTaoDySst.png?time,T_25&station="0n0e"&time>=2015-05-23T12:00:00Z&time<=2015-05-31T12:00:00Z&.draw=lines](http://osug-smos-rea.osug.fr:8081/erddap/tabledap/pmelTaoDySst.png?time,T_25&station="0n0e"&time%3E=2015-05-23T12:00:00Z&time%3C=2015-05-31T12:00:00Z&.draw=lines){kind=link}