Skip to content

Archive data

WKDV offers you an easy-to-use service for the irreversible archiving of files without time limitations.

Overview of performance characteristics

Getting started

Metrics (Login with UFZ account required)

Frequently asked questions (FAQ)

Archiving

What do I need to start?

For archiving, you will need a data project. It establishes the topic of archivings and regulates access authorisations. Here you will be provided with further information on data projects. (e.g. what are data projects; how can they be set up; where do I get a project overview, ...)

Why should I archive my data?

Your data are safe on the central WKDV storage systems. We provide for a backup of data on two locally separate storage media. You will profit from it not only with regard to any long-term archiving of data but already when using your personal user directory or the group directories.

As opposed to the personal directory and the group directory, you can also use the file archives for very large data volumes. Moreover, the data in the archives are protected against changes or deletion.

In any event, you should give preference to the archiving system – as opposed to unsafe solutions, such as USB hard disk drives or CDs/DVDs.

Where and how are files stored?

Our file system is an intelligent combination of fast hard disk drives as "buffers" and slower but safe high-capacity magnetic tapes. This is why archiving processes and future access to contents will take some time. Copies on two locally separated archiving systems and the additional verification by checksums will protect against loss of data. In any event, you should give preference to the archiving system – as opposed to unsafe solutions, such as USB hard disk drives or CDs/DVDs.

What is the procedure for archiving files?

The answer to this question is somewhat more complex and to be found on the page Procedure for archiving files.

Can I secure archiving with its own checksum?

Yes, that's possible. The archiving system is using, as a standard, checksums in MD5 format.

The checksum file must have the same name as the file to be archived and have the ending ".md5". The checksum file must also be in the same directory as the file to be archived. Under Linux, you will generate for the sample file archive.tar.gz a valid checksum file as follows:

md5sum archive.tar.gz > archive.tar.gz.md5

If these two prerequisites are met and the checksum is in the MD5 format, the file browser will automatically offer you – for the selection of the file to be archived – the appropriate checksum file for selection.

Caution: The archiving process fails if, after conclusion of archiving, the checksums of the two tape copies do not conform with the checksum you had selected!

If you do not provide any own checksum file, the archiving process automatically generates a checksup when the file to be archived is copied and, after completion of the archiving, the process checks it against the two generated tape copies. You can look at this checksum (or the one selected on your own) any time in the web front end of the archives component of the data management portal and you will find it additionally in the directory through which the archived file can be downloaded again.

How long does it take to archive a file?

There is no generalised answer to this question. The duration depends on the size of the file to be archived and also on how many archiving orders are waiting in parallel for their processing.

You should reckon, however, that archiving even smaller files may take several hours.

Will I be automatically informed after completion of the archiving process?

Yes – not only in case of a failed but also with a successful archiving process, you will be subsequently immediately advised thereof by email.

Can I also delete archived data again?

No –that is not within the sense or intention of archives.

We want to ensure that data once archived will be retained for a very long period of time and that they will be available even in more than 10 years, irrespective of the project's duration.

Use of the web interface is not feasible for me - can archiving be automated?

Aside from the web interface, the system provides an interface by means of which archiving can be triggered and tracked.

On the page Interface for automated archiving, you will find further information.

Something went wrong

I cannot set up any new archiving order - what is the reason for it?

As a rough impression/structuring, the data to be archived are each allocated to a so-called data project. If you cannot set up any new archiving order, you will not have any write permissions in any of the existing data projects. In that case, please contact an administrator of an existing data project or wkdv-datamanagement@ufz.de for setting up a new data project.

Also see here in the FAQ

I can select just one file only for archiving

That is intended. If you want to archive several files or entire directory structures, combine them first in a container file – like zip, rar or tar.

That provides the chance at the same time to structure the data and enrich them, in parallel, with other documentations, scripts, etc.

My archiving order failed, now what?

Failure of an archiving order can primarily have three reasons:

  1. The file to be archived could not be read by the archiving process. The reason might be that the file selected for archiving is no longer in its original directory or had been renamed. Or that you had lost the right to read the file since the selection of the file.
  2. The checksum you selected does not correspond with the checksum ascertained after archiving.
  3. An unexpected system error occurred during archiving.

In case of an error, the data management team will also be automatically notified. We will contact you in case of an unexpected system error. In the other cases, you can again initiate the archiving process yourself by ensuring your file access or selecting the proper checksum.

In any event, the failed archiving order can only be deleted or copied. Do the latter to prepare the same order once again.

Download

Where can archived files be downloaded?

Recommended is the download via the publicly accessible data research portal.

Alternatively, it is possible to download the data via a conventional file explorer:

If you are working in Windows, you will find the archived files via the ´Windows Explorer under Y:\Archive\. If you are working in Mac, Linux or Unix, you can peruse the files via the file explorer using the address smb://archive-gateway.intranet.ufz.de/archive/.

Within the directory, the archived files are once again grouped acording to their data project as well as the name of the data set. The data of the archive data set "Remote Sensing Data 2014" which is assigned to the data project "Tereno", would be found under Y:\Archive\Tereno\Fernerkundungsdaten 2014\, for example.

Note: You will see exclusively the data of successfully completed archivings which you are allowed to also download. Persons entitled to download an archive data set will be entered, within the archive data set, in the column entitled "The following group of persons is entitled to download the archived data". Please note that – after conclusion of archiving or after a change of access rights – it might take up to 30 minutes before the authorisations are effective for the download directory.

How can data be obtained via the EVE cluster?

Detailed information about it are found in EVE-Wiki.

Can archived data be made available to external partners?

Yes, that's possible. Within the data management portal, you may enter in the details of an archive data set who may download it. Deposit in the section "The following group of persons is entitled to download the data" the email addresses of the corresponding partners. They may subsequently request the data via the publicly accessible data research portal for downloading.

Metadata

What are metadata needed for?

In order to ensure that archived data can also be retrieved again, they should be described as precisely as possible by means of metadata. The archive component supports the DublinCore as the standard metadata catalogue. It has a high rate of distribution and thus makes it easier to find interfaces with other cooperation partner.

With increasing data volumes in the file archives, you will also profit directly from the entered metadata since they support the retrievability in the data management portal and the data research portal.

My data sets have similar metadata - are there no templates?

There are currently no templates for metadata.

However, you may copy any archive data set or archiving order (accessible to you in detail) and use it as the basis for a new archiving order. The metadata are also copied here so that you only need to adjust deviating data.

Autorisations

Who can perform archivings?

Archivings may be done by any UFZ employee.

Which archiving data sets / archiving orders may be accessed?

Principally all archiving orders and archive data sets may be accessed within the UFZ network. If one does not have any read access within a data project, the metadata of its archive data sets are exclusively visible.

Which prerequisites must be met for setting up new archive data sets?

Write authorisations are required within a data project in order to add new archive data sets.

Who may make changes to an archiving order or set it back to its draft status?

Exclusively the creator of the archiving order.

Who may delete an archiving order / archive data set and under which prerequisites?
  • The creator as long as the archiving order is in the draft status or the one-hour long processing period before beginning the archiving.
  • The creator if the archiving process failed.
Who may abort an archiving order and under which prerequisites which can thus be set back into the draft status?

The creator as long as the archiving order is in the one-hour long processing period prior to performing the archiving.

Who may copy an archiving order / archive data set and under which prerequisites?
  • Anyone who may access the archiving order/archive data set in a reading manner in detail and who has the write authorisation in at least one data project.
  • The copy may only be stored and allocated to a data project for which the copying user has write authorisations.
Who may download the archived data of an archive data set and under which prerequisites?

All users who had been allocated – in the settings of the archive data set – to the group of persons authorised to download and if the archive data set had been successfully archived.

Do you have any further questions?

If we were unable to help you there, please contact wkdv-datamanagement@ufz.de

You can establish contact to the users of the archive component of the data management portal via the dmp-archive-users@ufz.de mailing list.