Skip to content

Interface for automated archiving

In addition to the web interface, the archiving system provides an interface which can be used for automated archiving.

Basic procedure for setting up a new archiving order via the interface

The information required for archiving (what should be archived; who may download the data; metadata, etc.) is filed in an import file.

  • It may be taken from the specification of an import file which data an import file includes and how it can be structured.
  • Important: An import file does not include the data to be archived but only describes where they are. In this respect, the same data sources may be used as in the web interface (e.g. user and group directories)

So that the archiving system can process an import file, it must be filed within the directory share:

\\\\archive-gateway.intranet.ufz.de\\interface\\

  • There, every user has a directory available which is accessible only to that particular user in accordance with the name of his/her UFZ user account (e.g. \\archive-gateway.intranet.ufz.de\interface\username)
  • Linux and Mac users achieve the share via smb://archive-gateway/interface/

  • Users of the EVE cluster will find the directory on a front end server under /archive-interface/Benutzername/Benutzername/

  • When changing into the directory, it is automatically mounted

  • Prerequisite for it is that the connection to the cluster was established by means of a password and not by means of a public key
  • In case of questions or problems, please contact wkdv-cluster@ufz.de

The name of the import file must end on ".import" and should have a meaningful designation for recognition. The system inputs the import file and changes the designation of the file ending in ".validating".

  • When this happens will depend on how many import-files are waiting (user-specifically) for their processing. If the system is not fully utilised, the input should be effected within 10 minutes. Otherwise, the files are prioritised according to their last change date ("first come, first served").
  • Should another file with the same basic name exist already, the file will be additionally unambiguously named by adding a consecutive number to the basic name.

Subsequently, the system tries to create from the import file a new archiving order.

  • Should the import file be formally or substantitvely invalid, the file ending is renamed into ".invalidformat" or ".invaliddata" and the import proces aborted. The user of the import directory will be additionally informed about it by email. Error details are additionally written back into the import file if this is formally valid. Details may be taken from the specification of an import file.
  • If setting up the archiving order was successful, the file ending will be renamed, in turn, in ".processing".

The set up archiving order may now also be accessed via the web interface. There, the possibility exists for 1 hour to still manually delete the order again.

  • In that case, the system will rename the file ending of the import file into ".deletedbyuser" and inform the user of the import directory about it by email. This is done with a time delay and depending on the system utilisation.

If the archiving order was not prematurely deleted, it will be planned after 1 hour for archiving and processed by the system depending on the system utilisation. The current status can then be tracked via the status button of the web interface.

If archiving failed, the file ending of the import file will be renamed in ".error" and the user will receive information by email. If the cause of the failure is known, the error message is additionally written back to the import file. Details may be taken from the specification of an import file.

If archiving was successful, the file ending of the import file will be renamed in ".success" and the user will also receive information by email.

Important information and rules for handling import files

During the processing of an import order, the file ending of an import file is adjusted several times:

  • .import (initial status)
  • .validating (interim status)
  • .invalidformat (end status)
  • .invaliddata (end status)
  • .processing (interim status)
  • .deletedbyuser (end status)
  • .error (end status)
  • .success (end status)

It is important that the basic name of an import file is unambiguous so that a file during processing does not receive the same name by renaming as an already existing file. Although the import process ensures upon reading in a new import file that the basic name is unique (and will rename it, if necessary); but subsequently it cannot be prevented that files are set up by the user which will trigger conflicts.

For that reason, the following rules should be complied with:

  • Use a unique basic name for every import order
  • Do not put any files into the import directory which have the same basic name as an import file being processed
  • Do not delete any import file which is not yet in an end status
  • Do not change any import file which is not yet in an end status

Should you delete an import file whose processing is not yet completed, the import process actually continues to run but you will not receive any feedback via the file regarding the progress or conclusion of the import and archiving process.

It the system renames an import file and detects that, contrary to the rules, there is a file of the same name in the directory, this onw will be deleted regardless of the consequences and the renamed file takes its place.

Should the setup of the import files be done in an automated/scripted fashion, it is recommended to also remove the files in an end status in an automated/scripted manner after they are no longer needed.

Information on the interface directory

  • The user-specific interface directory \archive-gateway.intranet.ufz.de\interface\username\ is released via SMB/CIFS.
  • It is only accessible and writable by the respective user.
  • The directory accepts a maximum of 10 MB (since the import files only decribe the files to be archived and do not include them).

Further information