Specification raw data transfer and format to the DMP¶

Download PDF: Datei:Spezifikation Rohdaten Transfer.pdf (german)

Transfer¶

Various endpoints are available for automated file-based transfer of measurement data to the DMP. Transfers can be made via Secure File Transfer Protocol (SFTP), File Transfer Protocol (FTP), or Hypertext Transfer Protocol (HTTP). SFTP and FTP transfers can be initiated from the Internet, HTTP transfers can only be initiated from the UFZ network. For transfers via mobile radio, the use of special data cards (M-VPN-SIM) of the UFZ is recommended, which enable a direct (restricted) connection to the UFZ network without additional technical requirements such as VPN client or router. In general, the transfer must actively originate from the data logger. No methods for cyclic fetching of measurement data are provided.

Devices with direct network access should support configuration via DHCP. Mobile modems must support PPP and the configuration of an individual APN 6. The resolution of the IP addresses of the different endpoints must be done dynamically via DNS.

Secure File Transfer Protocol (SFTP)¶

The most reliable way to transfer raw file-based data is via SFTP. Transport encryption ensures authenticity, confidentiality and integrity of the data and thus protects against misuse and transmission errors. SFTP clients are available for various environments and all relevant operating systems. However, encryption results in increased demands on the client's computing power, which is why it is not possible to implement it in every environment.

Further information and references to an application for creating SFTP accounts can be found on the UFZ intranet.

File Transfer Protocol (FTP)¶

Unlike transfers via SFTP, transfers via FTP transmit both access and payload data unencrypted. FTP clients are available for various environments and all relevant operating systems. Because of the lower computing power requirements, implementation is also possible in very simple environments optimized for very low power consumption.

Further information and references to an application for creating FTP accounts can be found on the UFZ intranet.

Hypertext Transfer Protocol (HTTP)¶

Via HTTP, data can be transferred both encrypted and unencrypted. The data is transferred via HTTP POST request to a server in the UFZ network (logger-worker.intranet.ufz.de). In this process, the file to be transferred is embedded as a parameter with the name "file". The assignment to a specific logger is done by a device ID, which is assigned in the DMP.

Example implementations¶

Python¶

import requests

device_id = 'fffffffffffff'
file      = 'test.csv'

paramter  = 'file'
base_url  = 'https://logger-worker.intranet.ufz.de/gateway/upload/device/'
url       = base_url + device_id

response = requests.post(url, files={
        paramter: open(file, 'rb')
    })

ret     = response.json()
success = response.status_code == requests.codes.ok

if len(ret['content']):
    print(ret['content'])

if not success and len(ret['exception']):
    print(ret['exception'])
    exit(1)

Curl/Bash¶

#!/usr/bin/env bash

FILE="/tmp/test.csv"
DEVICE_ID="fffffffffffff"

BASE_URL="https://logger-worker.intranet.ufz.de/gateway/upload/device/"
URL="${BASE_URL}${DEVICE_ID}"

curl "$URL" -sf -F "file=@${FILE}" > /dev/null
exit $?

.NET/Powershell¶

$file     = "C:\tmp\test.csv"
$deviceid = "fffffffffffff"
$base_url = "https://logger-worker.intranet.ufz.de/gateway/upload/device/"

$url       = $base_url + $deviceid
$webClient = New-Object System.Net.WebClient;

$webClient.UploadFile($url, "POST", $file);

Format¶

The raw data should be transferred UTF-8 encoded in the form of CSV text files.

Technical framework¶

A file should contain at least one time stamp with at least one measured value
Individual files may not exceed a maximum length of 100,000 lines
Individual files may not exceed a maximum size of 10 megabytes
Fields have a maximum length of 200 characters
Fields without value
1. are either empty,
2. contain blanks,
3. or the string "NIL"

Example:

timestamp, AirTemperature, Surfacewater, Battery
2013-09-12 12:23:33, 20.3,, 11.7
2013-09-12 12:23:33, 20.3,    , 11.7
2013-09-12 12:23:33, 20.3, NIL, 11.7

All rows contain a valid zero value in the third column.

Field separator¶

comma (",")

Line seperator¶

Configurable (see https://en.wikipedia.org/wiki/Newline)

Unix: Line feed ("LF", "0A" oder "\n")
Windows: Carriage return line feed ("CRLF", "0D0A" oder "\r\n")

Decimal point¶

dot (".")

Timestamp¶

In the first column/field

Format "YYYY-MM-DD HH:mi:ss",
for example 2013-02-19 10:50:00

First row/heading¶

Column headers (or empty), will be ignored

Strings¶

Strings must not contain commas.