Manual Upload¶
Explanation¶
This page explains the different import and reimport formats. Since most formats expect a quality flag, it is helpful to have a look at the different quality flags first.
Technical framework¶
- Fields for values (e.g. "AirTemperature2maGAvg10min" or "value") have a maximum length of 200 characters.
- Fields for comments (e.g. "quality_comment,") have a maximum length of 1000 characters
- Fields without a value are either empty, contain spaces or the string "NIL"
Example: All rows contain a valid zero value in the third column:
timestamp, AirTemperature2maGAvg10min, SurfacewaterConcentration1mbSMax1min, BatteryVoltage
2013-09-12 12:23:33, 20.3,, 11.7
2013-09-12 12:23:33, 20.3, , 11.7
2013-09-12 12:23:33, 20.3, NIL, 11.7
- Individual files must not exceed a maximum length of 100,000 lines
- Individual files must not exceed a maximum size of 10 megabytes
Manual Level 0 Upload¶
Description¶
Upload or import of raw data for loggers that cannot send data to the DMP automatically or data taken from the logger manually.
Specification¶
- Usually CSV with date and time field
- Table header with field labels is recommended
Specification data fields¶
- Data structure corresponds to the one configured in the logger component
Example¶
timestamp, AirTemperature2maGAvg10min, SurfacewaterConcentration1mbSMax1min, BatteryVoltage
2013-09-12 12:23:33, 20.3, 8.21, 11.7
2013-09-12 12:43:33, -273.1, 43.1, 8.6
Manual Level 2a Upload¶
Description¶
Upload or reimport of quality flags without further processed data in ASCII-CSV format. The reimport must be preceded by a data export because the measured value ID is required as a key for subsequent assignment to the raw data.
Specification¶
- Table header with field labels is required
Specification data fields¶
- Field level1_value_id (position 1, mandatory).
- Contains measurement value ID exported from database
- Mandatory to get reference to measured values
- Field quality_flag (position 2, mandatory)
- Contains the quality flag ( OK, DOUBTFUL or BAD )
- Mandatory required
- Field quality_cause (position 3, conditionally mandatory)
- Predefined reason (e.g. BATTERY_LOW) or OTHER
- Mandatory if quality_flag not OK
- List of predefined reasons to be maintained by TERENO data managers
- Field quality_comment (position 4, conditionally mandatory)
- Free text field for comment or extended description of quality_cause
- Mandatory if quality_cause matches OTHER
Example¶
level1_value_id, quality_flag, quality_cause, quality_comment
23443, OK, ,
23444, DOUBTFUL, BATTERY_LOW, ,
23445, BAD, OTHER, Signal cable broken
Manual Level 2b Upload¶
Description¶
Upload or reimport of quality-assured data and their quality flags. Required if data is recorded or further processed externally. Before further processing, a data export is recommended in order to obtain, if possible, the measured value ID as a key and thus the reference to the raw data.
Specification¶
- A dot (".") is to be used as decimal separator
- For zero values the character string NIL is to be used
- One measured value and its quality flags are described per line
- Table header with field designations required
Specification data fields¶
- Field timestamp (position 1, mandatory).
- Contains date and time when the measured value was recorded
- Format YYYY-MM-DD hh:mm:ss
- Field sensor name (position 2, mandatory)
- Name of the sensor, results from level 2 sensor configuration in the logger component
- Field value (position 3, mandatory)
- Measured value or NIL in case of non-existent/incorrect values
- Field quality_flag (position 3, mandatory)
- Contains the quality flag ( OK, DOUBTFUL or BAD )
- Mandatory
- Field quality_cause (position 4, conditionally mandatory)
- Predefined reason (e.g. BATTERY_LOW) or OTHER
- Mandatory if quality_flag is not OK
- List of predefined reasons to be maintained by TERENO data managers
- Field quality_comment (position 5, conditionally mandatory)
- Free text field for comment or extended description of quality_cause
- Mandatory if quality_cause corresponds to OTHER
- Field level1_value_id (position 6, optional)
- Contains measurement value ID exported from database
- Recommended to get reference to measured values
Example¶
timestamp, sensorname, value, quality_flag, quality_cause, quality_comment, level1_value_id
2013-09-12 12:23:33, AirTemperature2maGAvg10min, 20.3, OK, NIL, NIL, 23144
2013-09-12 12:23:33, SurfacewaterConcentration1mbSMax1min, 8.21, OK, NIL, NIL, 23145
2013-09-12 12:43:33, AirTemperature2maGAvg10min, -273.1, BAD, Other, Signal cable broken, 23146
2013-09-12 12:43:33, SurfacewaterConcentration1mbSMax1min, 43.1, DOUPTFUL, BATTERY_LOW, NIL, 23147
Further information¶
- empty fields at the end can be omitted (quality_cause, quality_comment, level1_value_id)
- already recorded values will be overwritten
- identification by unique sensor name and timestamp
- if a timestamp occurs in two different l2raw records, the one parsed last wins