Easier access to scientific data
Brought to you by IGE NOAA NMFS SWFSC ERD
|
IGE's public ERDDAP
Easier access to scientific data |
Brought to you by IGE NOAA NMFS SWFSC ERD |
Data Access Protocol (DAP) and its
projection constraints.
The URL specifies what you want: the dataset, a description of the graph or the subset of the data, and the file
type for the response.
griddap request URLs must be in the form
http://osug-smos-rea.osug.fr:8081/erddap/griddap/datasetID.fileType{?query}
For example,
http://osug-smos-rea.osug.fr:8081/erddap/griddap/jplMURSST41.htmlTable?analysed_sst[(2015-06-09T09:00:00Z)][(-89.99):1000:(89.99)][(-179.99):1000:(180.0)]
Thus, the query is often a variable name (e.g., analysed_sst),
followed by a set of [] for each of the variable's dimensions
(for example, [time][latitude][longitude]),
with your desired (start):stride:(stop) values within each [].
Details:
| Data fileTypes | Description | Info | Example |
|---|---|---|---|
| .asc | View OPeNDAP-style ISO-8859-1 comma-separated text. | info |
example |
| .csv | Download a ISO-8859-1 comma-separated text table (line 1: names; line 2: units; ISO 8601 times). | info |
example |
| .csvp | Download a ISO-8859-1 .csv file with line 1: name (units). Times are ISO 8601 strings. | info |
example |
| .csv0 | Download a ISO-8859-1 .csv file without column names or units. Times are ISO 8601 strings. | info |
example |
| .das | View the dataset's metadata via an ISO-8859-1 OPeNDAP Dataset Attribute Structure (DAS). | info |
example |
| .dds | View the dataset's structure via an ISO-8859-1 OPeNDAP Dataset Descriptor Structure (DDS). | info |
example |
| .dods | OPeNDAP clients use this to download the data in the DODS binary format. | info |
example |
| .esriAscii | Download an ISO-8859-1 ESRI ASCII file (lat lon data only; lon must be all below or all above 180). | info |
example |
| .fgdc | View the dataset's UTF-8 FGDC .xml metadata. | info |
example |
| .graph | View a Make A Graph web page. | info |
example |
| .help | View a web page with a description of griddap. | info |
example |
| .html | View an OPeNDAP-style HTML Data Access Form. | info |
example |
| .htmlTable | View a UTF-8 .html web page with the data in a table. Times are ISO 8601 strings. | info |
example |
| .iso19115 | View the dataset's ISO 19115-2/19139 UTF-8 .xml metadata. | info |
example |
| .itx | Download an ISO-8859-1 Igor Text File. Each axis variable and each data variable becomes a wave. | info |
example |
| .json | View a table-like UTF-8 JSON file (missing value = 'null'; times are ISO 8601 strings). | info |
example |
| .jsonlCSV | View a UTF-8 JSON Lines CSV file (missing value = 'null'; times are ISO 8601 strings). | info |
example |
| .jsonlKVP | View a UTF-8 JSON Lines file with Key:Value pairs (missing value = 'null'; times are ISO 8601 strings). | info |
example |
| .mat | Download a MATLAB binary file. | info |
example |
| .nc | Download a NetCDF-3 binary file with COARDS/CF/ACDD metadata. | info |
example |
| .ncHeader | View the UTF-8 header (the metadata) for the NetCDF-3 .nc file. | info |
example |
| .ncml | View the dataset's structure and metadata as a UTF-8 NCML .xml file. | info |
example |
| .nccsv | Download a NetCDF-3-like 7-bit ASCII NCCSV .csv file with COARDS/CF/ACDD metadata. | info |
example |
| .nccsvMetadata | View the dataset's metadata as the top half of a 7-bit ASCII NCCSV .csv file. | info |
example |
| .ncoJson | Download an NCO lvl=2 JSON file with COARDS/CF/ACDD metadata. | info |
example |
| .odvTxt | Download time,lat,lon,otherVariables as an ODV Generic Spreadsheet File (.txt). | info |
example |
| .tsv | Download a UTF-8 tab-separated text table (line 1: names; line 2: units; ISO 8601 times). | info |
example |
| .tsvp | Download a UTF-8 .tsv file with line 1: name (units). Times are ISO 8601 strings. | info |
example |
| .tsv0 | Download a UTF-8 .tsv file without column names or units. Times are ISO 8601 strings. | info |
example |
| .xhtml | View a UTF-8 XHTML (XML) file with the data in a table. Times are ISO 8601 strings. | info |
example |
ArcGIS
.esriAsc
.esriAsc is an old and inherently limited file format. If you have ArcGIS 10 or higher, we strongly recommend
that you download gridded data from ERDDAP in a NetCDF .nc file, which can be opened directly by ArcGIS 10+
using the
Make
NetCDF Raster Layer tool in the Multidimension Tools toolbox.
If you have ArcGIS 9.x or lower:
ArcGIS is a family of Geographical Information Systems (GIS) products from ESRI: ArcView, ArcEditor, and ArcInfo.
To download data for use with ArcGIS 9.x or lower:
in ERDDAP, choose the .esriAscii file type to save a latitude longitude subset of data for just one point in time.
The file's extension will be .asc. This file format was designed by ESRI to transfer coverage data between computers.
Then, import the data file into your ArcGIS program:
Shapefiles - Sorry, ERDDAP currently does not distribute grid data as shapefiles.
Ferret
is a free program for visualizing and analyzing large and complex gridded
datasets. Ferret should work well with all datasets in griddap since griddap is
fully compatible with OPeNDAP. See the
Ferret documentation.
Note that the griddap dataset's OPeNDAP base URL that you use with Ferret's
set data, for example, http://osug-smos-rea.osug.fr:8081/erddap/griddap/jplMURSST41 ,
won't ever have a file extension at the end.
IDL -
IDL is a commercial scientific data visualization program. To get data from ERDDAP
into IDL, first use ERDDAP to select a subset of data and download a .nc file.
Then, use these
instructions
to import the data from the .nc file into IDL.
JSON .json
files are widely used to transfer data to JavaScript scripts running on web pages.
All .json responses from ERDDAP (metadata, gridded data, and tabular/in-situ data) use the
same basic format: a database-like table. For data from grid datasets, ERDDAP flattens the data
into a table with a column for each dimension and a column for each data variable. For example,
{
"table": {
"columnNames": ["time", "latitude", "longitude", "analysed_sst"],
"columnTypes": ["String", "float", "float", "double"],
"columnUnits": ["UTC", "degrees_north", "degrees_east", "degree_C"],
"rows": [
["2014-02-03T09:00:00Z", 34.9969, -134.995, 16.037],
["2014-02-03T09:00:00Z", 34.9969, -134.984, 16.033],
["2014-02-03T09:00:00Z", 34.9969, -134.973, null],
...
["2014-02-03T09:00:00Z", 36.9965, -132.995, 15.285]
]
}
}
All .json responses from ERDDAP have
JSONP
(from .json) -
Jsonp is an easy way for a JavaScript script on a web page to
import and access data from ERDDAP. Requests for .json files may include an optional
jsonp request by adding &.jsonp=functionName to the end of the query.
Basically, this just tells ERDDAP to add functionName( to the beginning of the
response and ")" to the end of the response.
The first character of functionName must be an ISO 8859 letter or "_".
Each optional subsequent character must be an ISO 8859 letter, "_", a digit, or ".".
If originally there was no query, leave off the "&" in your query.
After the data download to the web page has finished, the data is accessible to the
JavaScript script via that JavaScript function.
Here is an example using
jsonp with ERDDAP and jQuery (thanks to Jenn Patterson of CenCOOS).
MATLAB
.mat
users can use griddap's .mat file type to download data from within MATLAB.
Here is a one line example:
load(urlwrite('http://osug-smos-rea.osug.fr:8081/erddap/griddap/jplMURSST41.mat?analysed_sst[(2015-06-09T09:00:00Z)][(-89.99):1000:(89.99)][(-179.99):1000:(180.0)]', 'test.mat'));
The data will be in a MATLAB structure. The structure's name will be the datasetID
imagesc(jplMURSST41.longitude, jplMURSST41.latitude, squeeze(jplMURSST41.analysed_sst), [0 32]) set(gca, 'YDir', 'normal')The numbers at the end of the first line specify the range for the color mapping.
There are also Matlab
Xtractomatic scripts for ERDDAP,
which are particularly useful for
getting environmental data related to points along an animal's track (e.g.,
GTOPP data).
ERDDAP stores datetime values in .mat files as "seconds since 1970-01-01T00:00:00Z".
To display one of these values as a String in Matlab, you can use, e.g.,
datastr(cwwcNDBCMet.time(1)/86400 + 719529)
86400 converts ERDDAP's "seconds since" to Matlab's "days since". 719529 converts
ERDDAP's base time of "1970-01-01T00:00:00Z" to Matlab's "0000-01-00T00:00:00Z".
.mat files have a maximum length for identifier names of 32 characters.
If a variable name is longer than that, ERDDAP changes the name of the variable
when writing the .mat file: it generates a hash digest of the variable name and
appends that after the first 25 characters. Thus, long variable names that only
differ at the end will still have unique names in the .mat file.
ERDDAP administrators: you can avoid this problem by specifying shorter variable
destinationNames.
NetCDF
.nc
- Requests for .nc files return the requested subset of the dataset in a
standard, NetCDF-3, 32-bit, .nc file.
If you are using
NetCDF-Java,
don't try to directly access an ERDDAP dataset or subset
as a .nc file. (It won't work, mostly because that .nc file isn't a static, persistent file. It is a
virtual file.) Instead, use one of these two options:
In both cases, you can then do what you want with the nc object, for example,
request metadata or request a subset of a variable's data.
.ncHeader
- Requests for .ncHeader files will return the header information (text) that
would be generated if you used
ncdump -h fileName
on the corresponding .nc file.
Ocean Data View .odvTxt
- ODV users can download data in a
ODV Generic Spreadsheet Format .txt file
by requesting griddap's .odvTxt fileType.
The dataset MUST include longitude, latitude, and time dimensions.
Any longitude values (0 to 360, or -180 to 180) are fine.
After saving the resulting file (with the extension .txt) in your computer:
OPeNDAP Libraries - Since ERDDAP is an
OPeNDAP-compatible data server,
you can use
any OPeNDAP client library, such as
NetCDF-Java, NetCDF-C, NetCDF-Fortran, NetCDF-Perl,
Java-DAP2,
Ferret, or the
Pydap Client,
to get data from an ERDDAP griddap dataset.
When creating the initial connection to an ERDDAP griddap dataset from any OPeNDAP library:
For example, with the NetCDF-Java library, you can use:
NetcdfFile nc = NetcdfDataset.openFile("http://osug-smos-rea.osug.fr:8081/erddap/griddap/jplMURSST41", null);
(don't use NetcdfFile.open; it is for local files only) or
NetcdfDataset nc = NetcdfDataset.openDataset("http://osug-smos-rea.osug.fr:8081/erddap/griddap/jplMURSST41");
(NetcdfFiles are a lower level approach than NetcdfDatasets. It is your choice.)
Once you have the nc object, you can request metadata or a subset of a
variable's data.
Pydap Client
users
can access griddap datasets via ERDDAP's standard OPeNDAP services.
See the
Pydap Client instructions for accessing gridded data.
Note that the name of a dataset in ERDDAP will always be a single word,
(e.g., jplMURSST41 in the OPeNDAP dataset URL
http://osug-smos-rea.osug.fr:8081/erddap/griddap/jplMURSST41 )
and won't ever have a file extension (unlike, for example, .nc for the
sample dataset in the Pydap instructions).
Python
is a widely-used computer language that is very popular among scientists.
In addition to the Pydap Client, you can use Python to download various files from ERDDAP
as you would download other files from the web:
import urllib
urllib.urlretrieve("http://baseurl/erddap/griddap/datasetID.fileType?query", "outputFileName")
Or download the content to an object instead of a file:
import urllib2
response = urllib2.open("http://baseurl/erddap/griddap/datasetID.fileType?query")
theContent = response.read()
There are other ways to do this in Python. Search the web for more information.
"Jupyter Notebook
is an open-source web application that
allows you to create and
share documents that contain live code, equations, visualizations and explanatory text"
using any of over 40 programming languages, including Python and R.
Here are two sample Jupyter Notebooks that access ERDDAP using Python:
ERDDAP Advanced Search Test and
ERDDAP Timing.
Thanks to Rich Signell.
R Statistical Package -
R is an open source statistical package for many operating systems.
In R, you can download a NetCDF version 3 .nc file from ERDDAP. For example:
download.file(url="http://osug-smos-rea.osug.fr:8081/erddap/griddap/jplMURSST41.nc?analysed_sst[(2015-06-09T09:00:00Z)][(-89.99):1000:(89.99)][(-179.99):1000:(180.0)]", destfile="/home/bsimons/test.nc")Then import data from that .nc file into R with the RNetCDF, ncdf, or ncdf4 packages available
.
Or, if you want the data in tabular form, download and import the data in a .csv file.
download.file(url="http://osug-smos-rea.osug.fr:8081/erddap/griddap/jplMURSST41.csv?analysed_sst[(2015-06-09T09:00:00Z)][(-89.99):1000:(89.99)][(-179.99):1000:(180.0)]", destfile="/home/bsimons/test.csv") test<-read.csv(file="/home/bsimons/test.csv")There are also R Xtractomatic scripts for ERDDAP, which are particularly useful for getting
data).
Making an Image File with a Graph or Map of Gridded Data
If a griddap request URL specifies a subset of data which is suitable for making
a graph or a map, and the fileType is an image fileType, griddap will return an image
with a graph or map.
griddap request URLs can include optional graphics commands which let you
customize the graph or map.
As with other griddap request URLs, you can create these URLs by hand or have a
computer program do it. Or, you can use the Make A Graph web pages, which simplify
creating these URLs (see the "graph" links in the table of
griddap datasets).
The fileType options for downloading images of graphs and maps of grid data are:
| Image fileTypes | Description | Info | Example |
|---|---|---|---|
| .geotif | View a grayscale GeoTIFF .tif file (for lat lon data; lon must be all below or all above 180). | info |
example |
| .kml | View a Google Earth .kml file (for lat, lon data only). | info |
example |
| .smallPdf | View a small .pdf image file with a graph or map. | info |
example |
| View a standard, medium-sized .pdf image file with a graph or map. | info |
example | |
| .largePdf | View a large .pdf image file with a graph or map. | info |
example |
| .smallPng | View a small .png image file with a graph or map. | info |
example |
| .png | View a standard, medium-sized .png image file with a graph or map. | info |
example |
| .largePng | View a large .png image file with a graph or map. | info |
example |
| .transparentPng | View a .png image file (just the data, without axes, landmask, or legend). | info |
example |
Image Size - ".small" and ".large" were ERDDAP's original system for making
different-sized images. Now, for .png and .transparentPng images (not other
image file types), you can also use the
&.size=width|height
parameter to request
an image of any size.
.transparentPng - The .transparentPng file type will make a graph or map without
the graph axes, landmask, or legend, and with a transparent (not opaque white)
background. This option can be used for any type of graph or map.
When &.draw= is set to anything other than surface, the default image size
is 360x360 pixels. Use the &.size=width|height parameter
to specify a different size.
When &.draw= is set to surface, ERDDAP makes an image where each
data point becomes one pixel, which may result in a huge image. If the request
takes too long or fails (perhaps for an odd apparent reason, like a Proxy Error)
either use a stride (see below) value greater than 1 (e.g., 5) for the x and y axis
variables, or use the
&.size=width|height parameter
to restrict the image size,
so that the image isn't huge.
Incompatibilities
Some results file types have restrictions. For example, Google Earth .kml is only
appropriate for results with longitude and latitude values. If a given request is
incompatible with the requested file type, griddap throws an error.
Command Line Downloads with curl
If you want to download a series of files from ERDDAP, you don't have to request each file's
ERDDAP URL in your browser, sitting and waiting for each file to download.
If you are comfortable writing computer programs (e.g., with C, Java, Python, Matlab, r)
you can write a program with a loop that imports all of the desired data files.
Or, if you are comfortable with command line programs (just running a program, or using bash or tcsh
scripts in Linux or Mac OS X, or batch files in Windows), you can use curl to save results files
from ERDDAP into files on your hard drive, without using a browser or writing a computer program.
ERDDAP+curl is amazingly powerful and allows you to use ERDDAP in many new ways.
On Linux or Mac OS X, curl is probably already installed as /usr/bin/curl.
On Windows, or if your computer doesn't have curl already, you need to
download curl
and install it. To get to a command line in Windows, click on "Start" and type
"cmd" into the search textfield.
("Win32 - Generic, Win32, binary (without SSL)" worked for me in Windows 7.)
Please be kind to other ERDDAP users: run just one script or curl command at a time.
Instructions for using curl are on the
curl man page and in this
curl tutorial.
But here is a quick tutorial related to using curl with ERDDAP:
curl -g "http://coastwatch.pfeg.noaa.gov/erddap/griddap/erdBAssta5day.png?sst[%282010-09-01T12:00:00Z%29][][][]&.draw=surface&.vars=longitude|latitude|sst&.colorBar=|||||" -o BAssta5day20100901.pngThe erddapUrl must be percent encoded
. To do this, you need to convert
and JavaScript's
encodeURIComponent()) and there are
.
them in the erddapURL as %5B, %5D, %7B, %7D, respectively.
curl "http://coastwatch.pfeg.noaa.gov/erddap/griddap/erdBAssta5day.png?sst%5B%282010-09-[01-05]T12:00:00Z%29%5D%5B%5D%5B%5D%5B%5D&.draw=surface&.vars=longitude|latitude|sst&.colorBar=|||||" -o BAssta5day201009#1.png
curl -v --data 'user=myUserName&password=myPassword' -c cookies.txt -b cookies.txt -k https://baseurl:8443/erddap/login.html
curl -v -c cookies.txt -b cookies.txt -k https://baseurl:8443/erddap/griddap/datasetID.fileType?query -o outputFileName
DAP
projection constraint query
which can request:
Using [start:stride:stop]
When requesting dimension (axis) variables or data variables, the query may
specify a subset of a given dimension by identifying the [start{{:stride}:stop}]
indices for that dimension.
in parentheses, which griddap then converts to the
Or, if you change the fileType in the URL from .png to .graph, you can see a Make A Graph
web page with that request loaded:
http://osug-smos-rea.osug.fr:8081/erddap/griddap/jplMURSST41.graph?analysed_sst[(2015-05-01T09:00:00Z):(2015-06-01T09:00:00Z)][(29)][(-135)]&.draw=linesAndMarkers&.vars=time%7Canalysed_sst%7C&.marker=1%7C3&.color=0xFF9900&.colorBar=%7CC%7CLinear%7C%7C%7C
That makes it easy for humans to modify an image request to make a similar graph or map.
Or, if you change the fileType in the URL from .png to a data fileType (e.g., .htmlTable), you can download the data that was graphed:
http://osug-smos-rea.osug.fr:8081/erddap/griddap/jplMURSST41.htmlTable?analysed_sst[(2015-05-01T09:00:00Z):(2015-06-01T09:00:00Z)][(29)][(-135)]&.draw=linesAndMarkers&.vars=time%7Canalysed_sst%7C&.marker=1%7C3&.color=0xFF9900&.colorBar=%7CC%7CLinear%7C%7C%7C
Or, if you change the fileType in the URL from .png to .graph, you can see a Make A Graph
web page with that request loaded:
http://osug-smos-rea.osug.fr:8081/erddap/griddap/jplMURSST41.graph?analysed_sst[(2015-06-09T09:00:00Z)][(-89.99):1000:(89.99)][(-179.99):1000:(180.0)]&.draw=surface&.vars=longitude%7Clatitude%7Canalysed_sst&.colorBar=Rainbow%7CC%7CLinear%7C0%7C32%7C
Or, if you change the fileType in the URL from .png to a data fileType (e.g., .htmlTable),
you can download the data that was mapped:
http://osug-smos-rea.osug.fr:8081/erddap/griddap/jplMURSST41.htmlTable?analysed_sst[(2015-06-09T09:00:00Z)][(-89.99):1000:(89.99)][(-179.99):1000:(180.0)]&.draw=surface&.vars=longitude%7Clatitude%7Canalysed_sst&.colorBar=Rainbow%7CC%7CLinear%7C0%7C32%7C
Some ERDDAP installations do have authentication enabled.
Currently, ERDDAP only supports authentication via Google-managed email accounts,
which includes email accounts at NOAA and many universities.
If an ERDDAP has authentication enabled, anyone with a Google-managed email account
can log in, but they will only have access to the private datasets
that the ERDDAP administrator has explicitly authorized them to access.
For instructions on logging into ERDDAP from a browser or via a script, see
Access to Private Datasets in ERDDAP.
in the HTTP GET request header sent
To generate a citation for a dataset:
If you think of the dataset as a scientific article, you can generate a
citation based on the author (see the "creator_name" or "institution" metadata),
the date that you downloaded the data, the title (see the "title" metadata),
and the publisher (use the Data Access Form URL for the dataset, e.g.,
https://coastwatch.pfeg.noaa.gov/erddap/griddap/jplMURSST41.html ). If the
dataset's metadata includes a
Digital Object Identifier (DOI), please
include that in the citation you create.
and clicking on "Apply for membership".
][(-89.99):1000:(89.99)][(-179.99):1000:(180.0)]&.draw=surface&.vars=longitude%7Clatitude%7Canalysed_sst&.colorBar=Rainbow%7CC%7CLinear%7C0%7C32%7C){kind=link}
![http://osug-smos-rea.osug.fr:8081/erddap/griddap/jplMURSST41.png?analysed_sst[(2015-05-01T09:00:00Z):(2015-06-01T09:00:00Z)][(29)][(-135)]&.draw=linesAndMarkers&.vars=time%7Canalysed_sst%7C&.marker=1%7C3&.color=0xFF9900&.colorBar=%7CC%7CLinear%7C%7C%7C](http://osug-smos-rea.osug.fr:8081/erddap/griddap/jplMURSST41.png?analysed_sst[(2015-05-01T09:00:00Z):(2015-06-01T09:00:00Z)][(29)][(-135)]&.draw=linesAndMarkers&.vars=time%7Canalysed_sst%7C&.marker=1%7C3&.color=0xFF9900&.colorBar=%7CC%7CLinear%7C%7C%7C){kind=link}