evedata.scan.controllers.version_mapping module

Mapping SCML contents to the entities of the scan subpackage.

There are different versions of the schema underlying the SCML files. Hence, mapping the contents of an SCML file to the data model of the evedata package requires to get the correct mapper for the specific version. This is the typical use case for the factory pattern.

Users of the module hence will typically only obtain a VersionMapperFactory object to get the correct mappers for individual files. Furthermore, “users” basically boils down to the Scan class. Therefore, users of the evedata package usually do not interact directly with any of the classes provided by this module.

Overview

Being version agnostic with respect to eveH5 and SCML schema versions is a central aspect of the evedata package. This requires facilities mapping the actual SCML files to the data model provided by the entities technical layer of the scan subpackage. The File facade obtains the correct VersionMapper object via the VersionMapperFactory, providing a SCML resource object to the factory. It is the duty of the factory to obtain the “version” attribute from the SCML object.

../../../_images/evedata.scan.controllers.version_mapping.svg

Fig. 45 Class hierarchy of the evedata.scan.controllers.version_mapping module, providing the functionality to map different SCML file schemas to the data structure provided by the Scan and Station classes. The factory will be used to get the correct mapper for a given SCML file. For each SCML schema version, there exists an individual VersionMapperVxmy class dealing with the version-specific mapping. The idea behind the Mapping class is to provide simple mappings for attributes and alike that need not be hard-coded and can be stored externally, e.g. in YAML files. This would make it easier to account for (simple) changes.

For each SCML schema version, there exists an individual VersionMapperVxmy class dealing with the version-specific mapping. Currently, we assume major and minor version numbers to be relevant. Hence, the xmy suffix for the individual mapper classes. That part of the mapping common to all versions of the SCML schema takes place in the VersionMapper parent class. The idea behind the Mapping class is to provide simple mappings for attributes and alike that can be stored externally, e.g. in YAML files. This would make it easier to account for (simple) changes.

General aspects

Both, representing the SCML contents as well as mapping the information from a given SCML file to their representing entities do not strive to preserve the full information available. The evedata interface is in the luxurious situation to not require all information nor to perform scans. Hence, for the time being, only those parts necessary for interpreting an eveH5 file and providing the valuable abstractions for the users are obtained from an SCML file and mapped to the respective representations.

One particularly important aspect: Only those scan modules that are actually connected (either to the root node or to another scan module) are actually mapped and contained in the list of scan modules.

Mapping tasks for SCML schema up to v9.2

Currently, the structure of the entities in the evedata.scan.entities subpackage is not stable, and furthermore, there is no overview how much the SCML schema has changed for the different versions. Hence, the tasks described here will definitely change and evolve over time.

  • Map SCML metadata (version, location) ✓

  • Map scan – if it exists

    • Map scan metadata (repeat_count, comment, description, …) (✓)

    • Map scan modules (✓)

      • Map basic metadata ✓

      • Map parent, appended, nested, is_nested ✓

      • Distinguish types of scan modules: “classic” vs. snapshot ✓

      • Extract list of detector channels and motor axes ✓

      • Map pre- and post-scans ✓

      • Map positionings ✓

      • Calculate number of positions per pass/total and actual positions ✓

      • Map plots? ✗

      • Map remaining information? ✗

  • Map all devices ✗

    • Map detectors

    • Map motors

    • Map devices

Todo

Implement mapping of positions for axes using plugins as step function. The reason for not yet having implemented this case: these axes need to have access to another reference axis, and the current implementation cannot guarantee this axis being present.

Currently, there is a need to map at least basic information about the scan modules, to proceed with several controllers from the evedata.evefile.controllers subpackage: mpskip, separating datasets of redefined channels, obtain set values for axes. For details, see the respective section of the Architecture document.

Important

Due to the reasons mentioned above (urgent need to map only basic information about the scan modules), the mapping of the SCML file contents to the entities from the evedata.scan.entities subpackage is only rudimentary. Therefore, both mapping and internal handling of the SCML/XML tree will probably change in the future.

Distinguishing types of scan modules

Currently, only two types of scan modules are distinguished: “classical” modules and snapshot modules. The distinction is based on the existence of the tag <classic>: if present, we have a “classical” scan module, if not, we have a snapshot module.

For “classical” modules, both axes and channels are mapped, for static axis or channel snapshot modules, axes or channels, respectively.

Creating positions for individual scan modules

Currently, (up to eveH5 schema version 7), the storage layer does not resemble nor preserve the structure of a scan, particularly its composition of distinct scan modules. To allow for sensible access to the data, the evedata interface tries to reconstruct this structure, at least as long as the SCML is available from the eveH5 file. This means that for each individual HDF5 dataset, the positions belonging to the individual scan modules need to be known, as one and the same channel or axis can be used in different scan modules. Hence, the mapper tries to calculate the number of positions per pass for each scan module, the number of total positions for each scan module, and finally the actual position (count)s for each scan module.

With the current way the engine stores the data, determining the actual position counts belonging to each individual scan module is not possible in full generality. Situations that may corrupt the algorithm to determine the positions from the scan description or even make it impossible to proceed contain:

  • Axes positions defined as lists in external files.

    Axes positions can be loaded from external files. However, only the file name (and path) are stored in the SCML. There is no guarantee at all that the file contents are the same as they were during scan execution. Besides that, the evedata interface usually cannot access the file containing the positions. In case the axis whose positions are defined using an external file has more positions set than any other axis in the scan module (the same applies if it is the only axis in the scan module), the algorithm calculating the (number of) positions for the scan module will fail.

  • External events leading to skipping execution of parts of a scan module.

    Generally, users can skip parts of a scan module. Similarly, events can be defined in scans that skip parts of a scan module given a specific condition. In both cases, these events will not be saved in any way in the resulting eveH5 file. Hence, there is no way in determining that an event occurred. This most probably leads to a full corruption of the positions determined by the algorithm.

  • Scans containing the MPSKIP detector

    This is a special situation dealt with by the mpskip module. Intrinsically, the positions cannot be determined algorithmically. However, given certain assumptions discussed in the documentation of the mpskip module, mapping should be possible.

Besides these general problems with accurately determining the (number of) positions per scan module, the algorithm generally proceeds as follows, assuming a list of those scan modules to be available that are part of the actual scan, together with the information on nesting and appending:

  • For each scan module, obtain all axes, and for each axis, determine the number of positions. The maximum of the number of positions of all axes within a scan module defines the number of positions per pass of a scan module, stored in the ScanModule.number_of_positions_per_pass attribute.

  • For each scan module, check the “number of measurements” parameter (known also as “measurements per point”, MPP, or “valuecount” in the SCML schema), and multiply the number of positions per pass with its value, to obtain the correct number of positions per pass.

  • For each scan module, determine whether a positioning takes place. Positionings get their own position (count) after all positions in the scan module have been accessed. In case of nested scan modules, the nested scan modules are executed for each individual position of the next-higher scan module. Hence, the parent scan module of a (series of) nested scan module(s) acts as a “for loop”.

  • For each scan module, determine whether it is nested, and if so, multiply the number of positions per pass by the number of positions of the next-higher scan module, stored in the ScanModule.number_of_positions attribute.

  • For each scan module, obtain the actual position counts, starting with the first scan module with parent 0 (defined as root node and start of the scan) and with 1 as first position (count), stored in the ScanModule.positions attribute.

A general plausibility check is to compare the number of positions calculated by this algorithm with the number of positions created during the actual scan. The latter can be easily obtained using the special HDF5 dataset containing the mapping of position (count)s to timestamps. Furthermore, obtaining the shape of the data in an HDF5 dataset is a “cheap” operation not requiring reading the data themselves, hence, not breaking with the general philosophy of the evedata interface to read data usually only on demand.

Fundamental change of SCML schema with v10

Most probably, the SCML schema will be changed quite substantially in the future, based on all the experience with developing the evedata package and the data models implemented therein.

Depending on how dramatic this changes will be, the mappers for versions up to the current one (v9.x) and those of the next major version may differ substantially.

Module documentation

class evedata.scan.controllers.version_mapping.VersionMapperFactory

Bases: object

Factory for obtaining the correct version mapper object.

There are different versions of the schema underlying the SCML files. Hence, mapping the contents of an SCML file to the entities of the scan subpackage requires to get the correct mapper for the specific version. This is the typical use case for the factory pattern.

scml

Python object representation of an SCML file

Type:

evedata.scan.boundaries.scml.SCML

Raises:

ValueError – Raised if no SCML object is present

Examples

Using the factory is pretty simple. There are actually two ways how to set the scml attribute – either explicitly or when calling the get_mapper() method of the factory:

factory = VersionMapperFactory()
factory.scml = scml_object
mapper = factory.get_mapper()

factory = VersionMapperFactory()
mapper = factory.get_mapper(scml=scml_object)

In both cases, mapper will contain the correct mapper object, and scml_object contains the Python object representation of an SCML file.

get_mapper(scml=None)

Return the correct mapper for a given SCML file.

For convenience, the returned mapper has its VersionMapper.source attribute already set to the scml object used to get the mapper for.

Parameters:

scml (evedata.scan.boundaries.scml.SCML) – Python object representation of an SCML file

Returns:

mapper – Mapper used to map the SCML file contents to evedata structures.

Return type:

VersionMapper

Raises:

ValueError – Raised if no scml object is present

class evedata.scan.controllers.version_mapping.VersionMapper

Bases: object

Mapper for mapping the SCML file contents to data structures.

This is the base class for all version-dependent mappers. Given that there are different versions of the SCML schema, each version gets handled by a distinct mapper subclass.

To get an object of the appropriate class, use the VersionMapperFactory factory.

source

Python object representation of an SCML file

Type:

evedata.scan.boundaries.scml.SCML

destination

High(er)-level data structure representing an SCML file

Can alternatively be an evedata.scan.boundaries.scan.Station object.

Type:

evedata.scan.boundaries.scan.Scan

Examples

Although the VersionMapper class is not meant to be used directly, its use is prototypical for all the concrete mappers:

mapper = VersionMapper()
mapper.map(source=scml, destination=scan)

Usually, you will obtain the correct mapper from the VersionMapperFactory. In this case, the returned mapper has its source attribute already set for convenience:

factory = VersionMapperFactory()
mapper = factory.get_mapper(scml=scml)
mapper.map(destination=scan)
map(source=None, destination=None)

Map the SCML file contents to data structures.

Parameters:
Raises:

ValueError – Raised if either source or destination are not provided

class evedata.scan.controllers.version_mapping.VersionMapperV9m0

Bases: VersionMapper

Mapper for mapping SCML v9.0 file contents to data structures.

Note

Currently, most mapping is implemented in this class, and needs to be moved upwards in the inheritance hierarchy once this hierarchy exists.

Furthermore, only a minimal set of mappings is currently performed, far from being a complete mapping of the contents of an SCML file.

Note

Only those scan modules that are actually connected (either to the root node or to another scan module) are actually mapped and contained in the list of scan modules.

source

Python object representation of an SCML file

Type:

evedata.scan.boundaries.scml.SCML

destination

High(er)-level data structure representing an SCML file

Type:

evedata.scan.boundaries.scan.Scan

Examples

Mapping a given SCML file to the evedata structures is the same for each of the mappers:

mapper = VersionMapperV9m0()
mapper.map(source=scml, destination=scan)

Usually, you will obtain the correct mapper from the VersionMapperFactory. In this case, the returned mapper has its source attribute already set for convenience:

factory = VersionMapperFactory()
mapper = factory.get_mapper(scml=scml)
mapper.map(destination=scan)
map(source=None, destination=None)

Map the SCML file contents to data structures.

Parameters:
Raises:

ValueError – Raised if either source or destination are not provided

class evedata.scan.controllers.version_mapping.VersionMapperV9m1

Bases: VersionMapperV9m0

Mapper for mapping SCML v9.1 file contents to data structures.

Note

SCML schema versions 9.0 and 9.1 differ only in their version number. Hence, this is an empty class delegating everything to its parent.

source

Python object representation of an SCML file

Type:

evedata.scan.boundaries.scml.SCML

destination

High(er)-level data structure representing an SCML file

Type:

evedata.scan.boundaries.scan.Scan

Examples

Mapping a given SCML file to the evedata structures is the same for each of the mappers:

mapper = VersionMapperV9m1()
mapper.map(source=scml, destination=scan)

Usually, you will obtain the correct mapper from the VersionMapperFactory. In this case, the returned mapper has its source attribute already set for convenience:

factory = VersionMapperFactory()
mapper = factory.get_mapper(scml=scml)
mapper.map(destination=scan)
map(source=None, destination=None)

Map the SCML file contents to data structures.

Parameters:
Raises:

ValueError – Raised if either source or destination are not provided

class evedata.scan.controllers.version_mapping.VersionMapperV9m2

Bases: VersionMapperV9m0

Mapper for mapping SCML v9.2 file contents to data structures.

Note

SCML schema versions 9.0 and 9.2 differ only in their version number. Hence, this is an empty class delegating everything to its parent.

source

Python object representation of an SCML file

Type:

evedata.scan.boundaries.scml.SCML

destination

High(er)-level data structure representing an SCML file

Type:

evedata.scan.boundaries.scan.Scan

Examples

Mapping a given SCML file to the evedata structures is the same for each of the mappers:

mapper = VersionMapperV9m2()
mapper.map(source=scml, destination=scan)

Usually, you will obtain the correct mapper from the VersionMapperFactory. In this case, the returned mapper has its source attribute already set for convenience:

factory = VersionMapperFactory()
mapper = factory.get_mapper(scml=scml)
mapper.map(destination=scan)
map(source=None, destination=None)

Map the SCML file contents to data structures.

Parameters:
Raises:

ValueError – Raised if either source or destination are not provided