Version 2
Copyright © 2004-2012 Delaware Environmental Observing System
Published: February 24th 2004
| Revision History | ||
|---|---|---|
| Revision 1.1 | 2004/02/24 | GEQ |
| Initial version | ||
| Revision 1.2 | 2005/02/02 | GEQ |
| Added description of which streams can be processed. | ||
| Revision 2.1 | 2006/09/26 | GEQ |
| Incremented version number of this document to 2. | ||
| Revision 2.2 | 2007/04/02 | GEQ |
| Incremented version number of this document to 2. | ||
| Revision 2.3 | 2007/05/17 | GEQ |
| Added DNERR stream. | ||
| Revision 2.4 | 2007/06/20 | GEQ |
| Added DNERR buoy streams. | ||
| Revision 2.5 | 2007/07/18 | GEQ |
| Added retries command line option. | ||
| Revision 2.6 | 2007/07/27 | GEQ |
| Added USACE streams. | ||
| Revision 2.7 | 2007/08/07 | GEQ |
| Added description of creating url for retrieval. | ||
| Revision 2.1.6 | 2008/01/02 | GEQ |
| Added description of the %2 option. | ||
| Revision 2.1.8 | 2010/01/31 | GEQ |
| Added NOS options. | ||
| Revision 2.1.11 | 2010/10/15 | GEQ |
| Added %B and %E options for URL construction. | ||
Abstract
This technical note describes the DEOS retrieve program, and the options for its use.
Table of Contents
The DEOS retrieve function provides the functionality to remotely retrieve meteorological data and store it locally to provide input to the retrieve program.
The retrieve program utilizes streams for retrieval. The following table describes the streams supported.
Table 1. Retrieve Station Metadata
| Stream Name | Description |
|---|---|
| METAR-SFSS | This is the stream for single-station in a single file METAR data as provided on the NWS web site. |
| USGS-STREAM | USGS stream-flow stations. |
| USGS-TIDAL | USGS tidal stations. |
| RAWS | NWS fire weather stations. |
| NDBC-BUOY | NDBC Ocean buoy stations. |
| DGS-Wells | Delaware Geologic Survey water wells. |
| DNERR-WQ-Multi | Delaware National Estuarine Research Reserve multi-station water quality data. |
| DNERR-Buoy-Status | Delaware National Estuarine Research Reserve buoy status data. |
| DNERR-Buoy-Wave | Delaware National Estuarine Research Reserve wave buoy data. |
| USACE-Buoy-Current | US Army Corps of Engineers buoy data (current data.) |
| NOS-Water-Level-Prediction | NOS data (prediction data.) |
| NOS-Water-Level-Obs | NOS data (observed data.) |
The program is executed by the command retrieve from the command line.
retrieve { -u username | --user username } { -p password | --password password } { -d database | --database database } { -s stream-name | --stream stream-name } [ -rvalue | --retries value ] [ -vvalue | --verbose value ] [ -h | --help ]
The retrieve program uses database configuration parameters to determine which data streams require data retrieval, as well as the method needed to retrieve them, and any associated parameters.
The specification for the user option is:
{ -u username | --user username | --user=username }
This option will allow the program to connect to the database using the username provided. The user must already exist, and have sufficient permissions to access the database specified. This item is required.
The specification for the password option is:
{ -p password | --password password | --password=password }
This option in combination with the username allows the program to connect to the database using the username provided. This item is required.
The specification for the database option is:
{ -d database | --database database | --database=database }
This option will specify which database the program should use as a source of data, configuration items and a destination for any event logging items. This item is required.
The specification for the stream option is:
{ -s stream name | --stream stream name | --stream=stream name }
This option specifies the stream name for retrieval. This is a required argument.
The specification for the retries option is:
[ -r value | --retries value | --retries=value ]
This option specifies the number of connection retries that the program will attempt. The default is 1.
The specification for the verbose option is:
[ -v value | --verbose value ]
The value provided for the verbose setting (a digit between 0 and 5) indicates how much output is generated, 0 being minimal and 5 being extremely verbose. Positive values of setting send data to the ELF, negative value also print the message to the screen. If no value is provided, the setting is assumed to be 1.
The following database tables provide configuration options for the retrieve program.
This table contains data about the various protocols, such as FTP, and associated information required to specify data retrieval.
This table contains data such as the specified retrieval method, protocol, hostname, user, source path, and associated information.
The heart of the retrieve program is the construction of a dynamic url according to the URL specification document. The components are defined in the Retrieve_Methods table and utilize a number of dynamic codes to represent changing data items as outlined in the following table.