Skip to content

feature--SIONlib_with_transactions--create_this_plugin

This is the second iteration of the SIONlib plugin. It uses PDI's transaction feature to make sure that all variables that should be written to / read from a file are shared when the file is opened.

The configuration section for this plugin contains two entries, inputs and outputs (with the obvious meaning). Both contain entries (with arbitrary names that act as labels) which configure inlets and outlets. Both inlets and outlets have the following entries:

  • transaction: data can move through this inlet/outlet every time this transaction is triggered, string, mandatory, not (re)evaluated
  • path: where on the filesystem data that moves through this inlet/outlet should go / come from, PDI string value, mandatory, (re)evaluated every time transaction is triggered
  • select: the usual, data movement only happens when this evaluates to true, PDI int value, optional, defaults to 1, (re)evaluated every time transaction is triggered
  • n_files: a number of physical files that the logical SIONlib file is split into on disk, PDI int value, optional, defaults to 1, (re)evaluated every time transaction is triggered
  • vars: a list of variable names contained in this inlet/outlet, these should be shared when transaction is triggered, list of strings, mandatory, not (re)evaluated

Variables are written to / read from files in the order they appear in the vars list, lifting the one variable per file limitation of the first plugin.

This plugin could be improved by writing some metadata to the files (variable names, data types, sizes) and then checking this metadata for consistency when reading. Is this within the scope of a plugin, what are your thoughts?

For now I have marked this WIP, as I still want to write some tests specifically for this plugin (or at least port over the HDF5 plugin tests, again).

Beware: This branch is based on my feature--SIONlib--create_this_plugin branch, so merging it will also merge that branch.

Merge request reports