feature--SIONlib_with_transactions--create_this_plugin
This is the second iteration of the SIONlib plugin. It uses PDI's transaction feature to make sure that all variables that should be written to / read from a file are shared when the file is opened.
The configuration section for this plugin contains two entries, inputs
and outputs
(with the obvious meaning). Both contain entries (with arbitrary names that act as labels) which configure inlets and outlets. Both inlets and outlets have the following entries:
-
transaction
: data can move through this inlet/outlet every time this transaction is triggered, string, mandatory, not (re)evaluated -
path
: where on the filesystem data that moves through this inlet/outlet should go / come from, PDI string value, mandatory, (re)evaluated every timetransaction
is triggered -
select
: the usual, data movement only happens when this evaluates to true, PDI int value, optional, defaults to 1, (re)evaluated every timetransaction
is triggered -
n_files
: a number of physical files that the logical SIONlib file is split into on disk, PDI int value, optional, defaults to 1, (re)evaluated every timetransaction
is triggered -
vars
: a list of variable names contained in this inlet/outlet, these should be shared whentransaction
is triggered, list of strings, mandatory, not (re)evaluated
Variables are written to / read from files in the order they appear in the vars
list, lifting the one variable per file limitation of the first plugin.
This plugin could be improved by writing some metadata to the files (variable names, data types, sizes) and then checking this metadata for consistency when reading. Is this within the scope of a plugin, what are your thoughts?
For now I have marked this WIP, as I still want to write some tests specifically for this plugin (or at least port over the HDF5 plugin tests, again).
Beware: This branch is based on my feature--SIONlib--create_this_plugin branch, so merging it will also merge that branch.