Version 2
Copyright © 2004-2012 Delaware Environmental Observing System
Published: March 5th 2004
| Revision History | ||
|---|---|---|
| Revision 1.1 | 2004/03/05 | GEQ |
| Initial version | ||
| Revision 1.2 | 2004/12/15 | GEQ |
| Data file format adds time zone difference from GMT. | ||
| Revision 1.3 | 2004/12/17 | GEQ |
| Added sdu program description. | ||
| Revision 1.4 | 2005/02/02 | GEQ |
| Added table describing station and grid loading options. | ||
| Revision 1.5 | 2005/10/20 | GEQ |
| Added local settings column to station metadata settings table. | ||
| Revision 1.6 | 2006/01/24 | GEQ |
| Added DelDOT network. Clarified network names and added a hyperlink target to the names section. Fixed error in HRAP topography filename. | ||
| Revision 1.7 | 2006/05/05 | GEQ |
| Added City and County descriptions. | ||
| Revision 1.8 | 2006/05/08 | GEQ |
| Cleaned up structure of file. | ||
| Revision 1.9 | 2006/05/11 | GEQ |
| Added usgs_counties.sql file to USGS streamflow setup. | ||
| Revision 1.10 | 2006/07/13 | GEQ |
| Added metar stations county data installation. Clarified in first table, which file contains County data. | ||
| Revision 2.1 | 2006/09/26 | GEQ |
| Incremented version number of this document to 2. | ||
| Revision 2.2 | 2006/12/05 | GEQ |
| Added DGS well network. | ||
| Revision 2.3 | 2007/01/16 | GEQ |
| Corrected name for DGS wells station metadata. | ||
| Revision 2.4 | 2007/06/18 | GEQ |
| Added DNERR networks. | ||
| Revision 2.5 | 2007/07/27 | GEQ |
| Added USACE networks. | ||
| Revision 2.1.8 | 2010/01/31 | GEQ |
| Added NOS network. Changed numbering to match code version. | ||
| Revision 2.1.11 | 2011/03/20 | GEQ |
| Added soil location metadata loading program. Added DNREC WQ network. | ||
| Revision 2.2.1 | 2012/04/29 | GEQ |
| Added convenience scripts for station, topography and miscellaneous loading. | ||
| Revision 2.2.2 | 2012/07/28 | GEQ |
| Added DBOFS network. | ||
Abstract
This technical note describes the programs to load station, topographical, City and County metadata.
Table of Contents
This technical note describes the programs available to the DEOS systems administrator to load system metadata. Described herein are the station metadata loading program sdi, enhanced station metadata loading sdu and the topographical information loading program tdi. In addition, a convenience script install-all.sh is described.
The DEOS system supports a number of different network data retrieval, ingest and display. The following table summarizes the information for these networks and options needed for loading the data.
Table 1. Network Station Metadata
| Network | Number | Location Type | Metadata Filename | Local Settings [a] | Description |
|---|---|---|---|---|---|
| NWS | 3 | 1 | metar_meta.txt | deos_data.sql | The NWS METAR network is the only one so far to have enhanced station data available for ingesting using the sdu program. This data is in file StnSummary.txt. The station County data is added utilizing the program described in the METAR station county metadata ingest section. |
| NIDS | 5 | 1 | radar_meta.txt | nexrad_data.sql | NEXRAD radar products, primarily the digital precipitation array (DPA). |
| DEOS | 7 | 1 | deos_metadata.txt | deos_data.sql | The local settings file contains DEOS station County data. |
| USGS-Streamflow | 8 | 5 | usgs-stream.txt | usgs_water_data.sql and usgs_counties.sql | The usgs_counties.sql contains County data. |
| USGS-Tidal | 9 | 6 | usgs-tidal.txt | usgs_water_data.sql and usgs_counties.sql | The usgs_counties.sql contains County data. |
| RAWS | 10 | 1 | raws_metadata.txt | raws_data.sql | NWS fire weather network |
| NDBC | 11 | 7 | ndbc_metadata.txt | ndbc_data.sql | Buoy network |
| DelDOT | 12 | 1 | deldot_metadata.txt | deldot_data.sql | Delaware Department of Transport network. The local settings file contains County data. |
| DGS-Wells | 13 | 1 | dgs_wells_metadata.txt | dgs_data.sql | Delaware Geological Survey well network. The local settings file contains County data. |
| DNERR-Water-Quality | 14 | 9 | dnerr_metadata.txt | dnerr_data.sql | Delaware National Estuarine Research Reserve water quality station network. The local settings file contains County data. |
| DNERR-Buoy | 15 | 7 | dnerr_buoy_metadata.txt | dnerr_buoy_data.sql | Delaware National Estuarine Research Reserve buoy network. |
| USACE-Buoy | 16 | 7 | usace_buoy_metadata.txt | usace_buoy.sql | US Army Corps of Engineers buoy network. |
| NOS | 17 | 7 | nos_metadata.txt | nos_data.sql | NOS network. |
| DNREC-Water-Quality | 18 | 9 | DNREC_WQ-Stations.txt | DNREC_WQ_data.sql and DNREC_WQ_county.sql | DNREC water quality station network. |
| DBOFS | 19 | 8 | DBOFS_metadata.txt | dbofs_data.sql | DBOFS station network. |
[a] These settings are probably only useful for University of Delaware installations. | |||||
The DEOS system supports a number of different grid projections. The following table summarizes the information for these projections and options needed for loading the data.
The following sections describe the metadata loading programs of the DEOS system.
The program is executed by the command install-all.sh from the command line.
The program is executed by the command sdi from the command line.
sdi { -u user-name | --user user-name } { -p password | --password password } { -d database | --database database } { -D directory | --directory directory } { -f filename | --filename filename } {--network-number value} {--location-type value} {--address-desc text} {--location-desc text}
The sdi program takes the command line options as described below.
The specification for the user option is:
{ -u username | --user username | --user=username }
This option will allow the program to connect to the database using the username provided. The user must already exist, and have sufficient permissions to access the database specified. This item is required.
The sdi program assumes that the input file conforms to the description given in this section.
The first line of the file is a comment line.
Subsequent lines are assumed to be tab-delimited fields of the following data.
Station ID
Latitude (in Radians)
Longitude (in Radians)
Elevation (in Meters)
Standard Time Difference from GMT (in Hours)
State
City
The sdi populates the following database tables.
This table contains address data for the station. Ingested data from the input file includes:
City
State
This table contains location data for the station. Ingested data from the input file includes:
Latitude
Longitude
Elevation
The program is executed by the command sdu from the command line.
sdu { -u user-name | --user user-name } { -p password | --password password } { -d database | --database database } { -D directory | --directory directory } { -f filename | --filename filename }
The sdu program takes the command line options as described below.
The specification for the user option is:
{ -u username | --user username | --user=username }
This option will allow the program to connect to the database using the username provided. The user must already exist, and have sufficient permissions to access the database specified. This item is required.
The sdu program assumes that the input file conforms to the description given in the stations listing file.
The sdu populates the following database tables.
The program is executed by the command install-all.sh from the command line.
The program is executed by the command city_ingest from the command line.
city_ingest { -u user-name | --user user-name } { -p password | --password password } { -d database | --database database } { -f filename | --filename filename } [ -k value | --skip=value ]
The city_ingest program takes the command line options as described below.
The specification for the user option is:
{ -u username | --user username | --user=username }
This option will allow the program to connect to the database using the username provided. The user must already exist, and have sufficient permissions to access the database specified. This item is required.
The city_ingest program assumes that the input file conforms to the description given in this section.
The first line of the file is a comment line.
Subsequent lines are assumed to be comma-delimited fields of the following data, fields beyond the first two are ignored.
City description in double quotes
City name
State name
The city_ingest populates the following database tables.
The program is executed by the command county_ingest from the command line.
county_ingest { -u user-name | --user user-name } { -p password | --password password } { -d database | --database database } { -f filename | --filename filename } [ -k value | --skip=value ]
The county_ingest program takes the command line options as described below.
The specification for the user option is:
{ -u username | --user username | --user=username }
This option will allow the program to connect to the database using the username provided. The user must already exist, and have sufficient permissions to access the database specified. This item is required.
The county_ingest program assumes that the input file conforms to the description given in this section.
The first line of the file is a comment line.
Subsequent lines are assumed to be comma-delimited fields of the following data, fields beyond the first two are ignored.
County name
State name
The county_ingest populates the following database tables.
The program is executed by the command metar_county from the command line.
metar_county { -u user-name | --user user-name } { -p password | --password password } { -d database | --database database } { -f filename | --filename filename } [ -k value | --skip=value ]
The metar_county program takes the command line options as described below.
The specification for the user option is:
{ -u username | --user username | --user=username }
This option will allow the program to connect to the database using the username provided. The user must already exist, and have sufficient permissions to access the database specified. This item is required.
The metar_county program assumes that the input file conforms to the description given in this section.
This file is called metadata/Export_XYmetar_meta_wCounty.csv.
The first line of the file is a comment line.
Subsequent lines are assumed to be comma-delimited fields of the following data, fields beyond those mentioned are ignored.
Not used
Station Identifier
Not used
Not used
Not used
Not used
Not used
Not used
State Abbreviation
Not used
Not used
County Name
The metar_county populates the following database tables.
The program is executed by the command install-all.sh from the command line.
The program is executed by the command tdi from the command line.
tdi { -u user-name | --user user-name } { -p password | --password password } { -d database | --database database } { -D directory | --directory directory } { -f filename | --filename filename } { --projection-number value | --projection-number=value }
The tdi program takes the command line options as described below.
The specification for the user option is:
{ -u username | --user username | --user=username }
This option will allow the program to connect to the database using the username provided. The user must already exist, and have sufficient permissions to access the database specified. This item is required.
The tdi program assumes that the input file conforms to the description given in this section.
The first line of the file is a comment line.
Subsequent lines are assumed to be tab-delimited fields of the following data.
Grid cell X value
Grid cell Y value
Elevation (in Meters)
Latitude (in Radians)
Longitude (in Radians)
The tdi populates the following database tables.
This table contains location data for the station. Ingested data from the input file includes:
Latitude
Longitude
Elevation