Data Processing and Instrument Data Standardization

class fluxEngine.HSICube_StorageOrder(value)

Hyperspectral data cube storage order

Hyperspectral cubes consist of three dimensions, and the storage order defines how these dimensions are mapped into linear memory.

The introductory documentation also contains a visual depiction of the various storage orders of HSI cubes.

BIL = 1

Band Interleaved by Line Storage Order

In this storage order all pixels of a line are next to each other in memory, and wavelengths are grouped by line. This means that input arrays have the shape (y, λ, x).

BIP = 0

Band Interleaved by Pixel Storage Order.

In this storage order all wavelengths of each pixel are next to each other in memory. This means that input arrays have the shape (y, x, λ).

BSQ = 2

Band Sequential Storage Order

In this storage order all pixels of an individual band are next to each other in memory, and wavelengths are grouped by image. This means that input arrays have the shape (λ, y, x).

class fluxEngine.PushBroomFrame_StorageOrder(value)

Hyperspectral PushBroom frame storage order

A PushBroom camera is a hyperspectral camera that uses a 2D sensor to image a single line, where the optics project different wavelengths of the incoming light onto one of the sensor dimensions. The other sensor dimension is used to spatially resolve the line that is being imaged.

There are two possible orientations of the optics: the wavelengths could be mapped onto the x- or the y-direction of the camera sensor. This enumeration allows the user to select which of these storage orders is actually used.

LambdaX = 0

Wavelengths are in X-direction

The y direction of the frame contains the spatial information.

LambdaY = 1

Wavelengths are in Y-direction

The x direction of the frame contains the spatial information.

class fluxEngine.ValueType(value)

The value type of a given input

Determines how to interpret the input values the user has provided.

Intensity = 0

Intensities

The data supplied by the user are raw intensities. If the model is set to process reflectances and/or absorbances reference data must be provided before processing can occur.

Reflectance = 1

Reflectances

The data supplied by the user are reflectances.

class fluxEngine.ReferenceInfo(valueType, whiteReference=None, darkReference=None)

Information about references

This information structure must be supplied when creating a processing context. It specifies the input value type of the processing context, as well as any references.

There are three primary ways to handle referencing of data:

  • The source in the model is set to raw intensities, and raw intensities are supplied by the user for the input data of the model while processing. In that case any references provided will be ignored

  • The source in the model is set to reflectances or absorbances, and the user provides reflectances for the input data of the model while processing. In that case any references provided will be ignored

  • The source in the model is set to reflectances or absorbances, and the user provides raw intensities for the input data of the model while processing. In that case a white reference must be provided to automatically reference the input data, and optionally a dark reference may be provided.

When referencing input data, if only a white reference is provided, reflectances are calculated with the following formula:

reflectance = intensity / white

If a dark reference is also present, reflectances are calculated with the following formula:

reflectance = (intensity - dark) / (white - dark)
valueType

The value type of the input data, determining how to preprocess the data (if at all) before using it for the model

Type

ValueType

whiteReference

The white reference data. This must be a numpy array that contains the white reference that will be used in conjunction with the input data.

Since it is advantageous to average multiple reference measurements, this array has to have an additional dimension to denote a list of input frames.

  • For HSI cubes in BIP order this means that the shape of the reference array has to be (N, height, width, bands).

  • For HSI cubes in BIL order this means that the shape of the reference array has to be (N, height, bands, width).

  • For HSI cubes in BSQ order this means that the shape of the reference array has to be (N, bands, height, width).

  • For PushBroom frames in LambdaX order the shape of the reference array has to be (N, width, bands)

  • For PushBroom frames in LambdaX order the shape of the reference array has to be (N, bands, width)

The number of averages, N, may be 1, indicating that no average is to be calculated. In that case numpy.expand_dims may be used to convert a single measurement to the desired format:

info.whiteReference = numpy.expand_dims(white_ref_cube, axis=0)

If no white reference is present, the user may specify None.

Type

numpy.array

darkReference

The dark reference data. This must have the same structure as the white reference data.

If no dark reference is present, the user may specify None.

Type

numpy.array

class fluxEngine.OutputSinkTensorStructure(dataType, maxSizes, fixedSizes)

Tensor Structure of an output sink

An object of this type will be assigned to the structure member of the OutputSinkInfo object when querying an output sink that returns tensor (numpy array) data. It will contain information about what type of data the output sink returns.

dataType

The numpy data type of the array that will be returned

Type

numpy.dtype

maxSizes

The maximum shape of the data that will be returned. Any data returned from this output sink will always be an array that has a shape with the same number of dimensions, but each dimension will be at most the corresponding entry in this tuple.

Type

tuple

fixedSizes

The fixed shape of the data that will be returned. The number of entries in this tuple will indicate to the user the number of dimensions of the data that is being returned. If a specific dimension is dynamic, i.e. the number of elements that are returned may change, the corresponding entry will be -1, otherwise it will be the size that will be returned in all cases.

Type

tuple

class fluxEngine.OutputSinkObjectListStructure(maxObjectCount, additionalDataSize=0, additionalDataType=None)

Output Sink Object List Structure

An object of this type will be assigned to the structure member of the OutputSinkInfo object when querying an output sink that returns a list of objects.

maxObjectCount

The maximum number of objects that may be returned during a single iteration of processing. This number is typically very high and will be an indication of the theoretical maximum that could be returned, but the actual number will in practice be quite a bit lower.

Type

int

additionalDataSize

If additional per-object data is returned in form of a numpy array, this indicates the size of the data returned. Additional data will always have only one dimension, and will always be of a fixed size. If this is 0 no additional data will be returned together with the objects.

Type

int

additionalDataType

If additional per-object data is returned this will contain the numpy datatype of the additional data.

Type

numpy.dtype or None

statisticsDataType

If statistics data is returned this will contain the numpy datatype of the statistical quantities (mean, standard deviation, etc.).

Type

numpy.dtype or None

statisticsDimensions

If statistics data is returned this will contain the numpy array shape of the statistics data.

Type

tuple(int) or None

qualityDataType

If quality data is returned this will contain the numpy datatype of the quality data.

Type

numpy.dtype or None

qualityCount

If quality data is returned this will contain the number of quality parameters that are being returned.

Type

int or None

class fluxEngine.OutputSinkInfo(outputId, name, structure, delay)

Output Sink Information

Objects of this type will be returned when querying information about output sinks of a model.

To determine whether an output sink returns tensor data (in the form of numpy arrays) or lists of objects, the following code may be used:

if isinstance(info.structure, fluxEngine.OutputSinkTensorStructure):
    print("output sink returns tensor data")
elif isinstance(info.structure, fluxEngine.OutputSinkObjectListStructure):
    print("output sink returns object list")
else:
    print("output sink returns unknown data")
outputId

The output id of the given output sink. This may be assigned by the user in fluxTrainer.

Type

int

name

The name the user has assigned to the output sink in fluxTrainer.

Type

str

structure

This may either be of type OutputSinkTensorStructure or of type OutputSinkObjectListStructure, and will contain further information about the output sink.

Type

object

delay

The input delay of the output sink.

This is only relevant when processing sequences of PushBroom frames, please see the advanced topics chapter in the documentation for further details.

Type

int

class fluxEngine.OutputObject(boundingBox, gravityCenter, area, mask, primaryClass, additionalData)

An object that is output

If an output sink is configured to output object data, it will return a list of objects of this type, containing the information related to each object.

This class provides some helper methods that perform some common operations.

boundingBox

The bounding box of the object, in form of a tuple with the elements (x, y, width, height). Each member of the tuple is of type int.

Type

tuple

gravityCenter

The center of gravity of the object, in form of a tuple with the elements (x, y). Each member of the tuple is of type float.

Type

tuple

area

The area of the object in pixels

Type

int

mask

The mask of the object within the bounding box. The shape of this array will always be (height, width), where height and width are taken from the bounding box. A value of 0 indicates that a given pixel belongs to the object, while a value of -1 indicates that a given pixel does not belong to the object.

It may be the case that this is not present and None.

There are helper methods, isObjectPresentAt() and isObjectPresentAtAbsolute(), that may be used to determine whether a given pixel belongs to this object.

Type

numpy.array or None

primaryClass

The primary class of the object. If this is None it indicates that no classifier was applied to this object in the processing chain. If this is of type int it indicates that the object was subject to a classifier, but it may still be that the classifier couldn’t find a matching class for this object, in which case the value will be -1. If the value is non-negative it corresponds to the group id of the class within the model.

Important: since 0 is a valid class, the user should take care to use the check if object.primaryClass is not None: instead of a simple if object.primaryClass to test for the presence of a primary class.

Type

int

additionalData

Any additional data that was provided in conjunction with the object. This will be a numpy array and may be something like and average spectrum, or a PLS value associated with the object. (In either case this will be a numpy array, typically with one dimension.)

It may be the case that this is not present and None.

Type

numpy.array or None

statistics

Statistics data for the given object. Each of these are numpy arrays that have the same shape.

It may be the case that this is not present and None.

Type

OutputObjectStatistics or None

quality

The quality data associated with the object.

It may be the case that this is not present and None.

Type

numpy.array or None

isObjectPresentAt(x, y)

Determine if a given pixel is part of the detected object

For a given set of coordinates for a pixel that are measured relative to the top-left corner of the bounding box this method will determine if that pixel belongs to the object or not. If the object does not contain any mask information, this will always return false.

Parameters

  • x (int) – The x coordinate relative to the top-left corner of the bounding box of the object to check

  • y (int) – The y coordinate relative to the top-left corner of the bounding box of the object to check

Returns

Whether a pixel is part of the object

Return type

bool

isObjectPresentAtAbsolute(x, y)

Determine if a given pixel is part of the detected object

For a given set of coordinates for a pixel that are measured absolutely to the start of the cube this method will determine if that pixel belongs to the object or not. For sequences of PushBroom frames that are being processed the y coordinate is measured in the number of frames since the last reset via ProcessingContext.resetState(). If the object does not contain any mask information, this will always return false.

Parameters

  • x (int) – The absolute x coordinate

  • y (int) – The absolute y coordinate

Returns

Whether a pixel is part of the object

Return type

bool

class fluxEngine.ProcessingContext(model, contextType, **kwargs)

Processing Context

This class wraps a processing context, the main interface to processing data with fluxEngine.

Parameters

  • model (Model) – The model to create the processing context for. For instrument device recording or preview contexts this should be set to None.

  • contextType (int) – The type of processing context to create. This must be either ProcessingContext.HSICube, ProcessingContext.PushBroomFrame, ProcessingContext.InstrumentPreview, ProcessingContext.InstrumentHSIRecording, ProcessingContext.InstrumentProcessing, or ProcessingContext.MeasurementProcessing. Depending on this value specific keyword arguments may be supplied.

Keyword Arguments

  • storageOrder (HSICube_StorageOrder or PushBroomFrame_StorageOrder) – (Non-device processing) The storage order of the input data

  • dataType (numpy.dtype) – (Non-device processing) The scalar data type of the input data of the processing context. Not all possibly numpy types are supported, the following are allowed: int8, int16, int32, int64, uint8, uint16, uint32, uint64, float32, float64.

  • maxHeight (int) – (Non-device HSI Cubes) The maximum height of a cube that will be processed using this processing context

  • height (int) – (Non-device HSI Cubes only) Specify -1 here to leave the cube height dynamic (which might not be as efficient at runtime for some models), or the same value as maxHeight to fix the height and indicate it will always be the same for every cube that is being processed

  • maxWidth (int) – (Non-device HSI Cubes) The maximum width of a cube that will be processed using this processing context

  • width (int) –

    (Non-device HSI Cubes) Specify -1 here to leave the cube width dynamic (which might not be as efficient at runtime for some models), or the same value as maxWidth to fix the width and indicate it will always be the same for every cube that is being processed

    (Non-device PushBroom Frames) The spatial dimension of each PushBroom frame that is supplied (if the storage order indicates that wavelengths are across the x direction of the frame, this indicates the size of the frame in the y direction, and vice-versa)

  • wavelengths (list or numpy.array) –

    (Non-device processing) a 1d list of wavelengths that determine the bands of the input data.

    (Instrument HSI recording) A list of floating point values indicating the wavelength grid onto which to interpolate the data. This may be None to indicate that no interpolation should occur.

  • referenceInfo (ReferenceInfo) – (Non-device processing) how the data should be referenced. See the documentation of the ReferenceInfo type for further details

  • device (InstrumentDevice) – (Instrument preview/recording/processing) The device to create the processing context for. This must always be specified when creating contexts for processing instrument data directly.

  • valueType (ValueType) – (Instrument HSI recording) Specifies the value type of the data that is to be recorded. If ValueType.Reflectance is specified a white reference must typically be provided, unless the driver directly returns reflectance values. (Which is only the case for virtual drivers reading reflectance data.)

  • instrumentParameters (InstrumentParameters) – (Instrument HSI recording and instrument processing) Specifies the white and dark references required to standardize the data

  • measurementList (MeasurementList) – (Measurement contexts only) The measurement list to take the measurement from which to create the processing context

  • index (int) – (Measurement contexts only) The index of the measurement within the measurement from which to create the processing context. The first measurement in the list has index 0. If -1 is specified for the index the context will be created so it is possible to process all measurements within the list. If the measurements are mutually incompatible (different data types, different value types, etc.) creating such a processing context will fail.

  • flags (MeasurementProcessingContextFlag or 0) – (Measurement contexts only) Flags that will influence the creation of such a processing context. This should either be 0 or a bitwise or of various values of the MeasurementProcessingContextFlag enumeration.

  • processingQueueSet (ProcessingQueueSet or None) – If provided and not None this will be used as the underlying processing queue set of the context. Otherwise the context will be created with the default processing queue set of the handle.

Notes

For ProcessingContext.HSICube type contexts the following keyword arguments are required (and the others should not be specified):

  • storageOrder

  • dataType

  • maxHeight

  • height

  • maxWidth

  • width

  • wavelengths

  • referenceInfo

  • processingQueueSet

For ProcessingContext.PushBroomFrame type contexts the following keyword arguments are required (and the others should not be specified):

  • storageOrder

  • dataType

  • maxWidth

  • width

  • wavelengths

  • referenceInfo

  • processingQueueSet

For ProcessingContext.InstrumentPreview type contexts the following keyword arguments are required (and the others should not be specified):

  • device

  • processingQueueSet

For ProcessingContext.InstrumentHSIRecording type contexts the following keyword arguments are required (and the others should not be specified):

  • device

  • valueType

  • instrumentParameters

  • wavelengths (optional)

  • processingQueueSet

For ProcessingContext.InstrumentProcessing type contexts the following keyword arguments are required (and the others should not be specified):

  • device

  • instrumentParameters

  • processingQueueSet

For ProcessingContext.MeasurementProcessing type contexts the following keyword arguments are required (and the others should not be specified):

  • measurementList

  • index

  • flags

  • processingQueueSet

HSICube = 0

HSI Cube Context Type

This value may be supplied to the constructor to indicate that a processing context is to be created for processing entire HSI cubes.

InstrumentHSIRecording = 3

Instrument HSI Recording Context Type

This value may be supplied to the constructor to indicate that a processing context is to be created for recording HSI data from a device. The model parameter must be set to None and a keyword parameter device must be specified to indicate the device to generate the context for.

InstrumentPreview = 2

Instrument Preview Context Type

This value may be supplied to the constructor to indicate that a processing context is to be created for generating a standardized preview of the device data. The model parameter must be set to None and a keyword parameter device must be specified to indicate the device to generate the context for.

InstrumentProcessing = 4

Instrument Processing Context Type

This value may be supplied to the constructor to indicate that a processing context is to be created for processing data directly from a device. The model parameter must be set to the requested model and a keyword parameter device must be specified to indicate the device to generate the context for.

MeasurementProcessing = 5

Measurement Processing Context Type

This value may be supplied to the constructor to indciate that a processing context is to be created from a loaded measurement. If the measurement is a HSI cube the context will be equivalent to that created manually of the ProcessingContext.HSICube type, but with the structure (scalar type, wavelengths, etc.) automatically taken from the loaded measurement.

PushBroomFrame = 1

PushBroom Frame Context Type

This value may be supplied to the constructor to indicate that a processing context is to be created for processing a sequence of PushBroom frames.

abort()

Abort processing.

This method may be called from a different thread while processing is currently active. It will signal the processing context to abort processing. This method will return immediately, but the processing context is likely still active. Use the wait() method to wait until the processing context is no longer active. After a call to this methgod the processing context needs to be reset via the method resetState() before it may be used again.

findOutputSink(outputId)

Find the index of an output sink with a given output id

If there is exactly one output sink in the model with a given output id, this will return the index of that sink. If there are no output sinks with that id, or that output id is used multiple times, this will throw an error.

Returns

The index of the sink

Return type

int

hsiRecordingResultInfo()

Get the HSI recording result information

This only applies to processing contexts for Instrument HSI recordings. This will additional inforamtion that the user may need to properly interpret the recording data.

On other processing contexts None will be returned.

numOutputSinks()

Get the number of output sinks

Returns

The number of output sinks

Return type

int

outputSinkData(index)

Get the resulting output sink data after processing

This may only be called after processNext() has been called at least once and resetState() has not been called since the last call to processNext().

Returns the data that arrived at the output sink during the last processing execution. For output sinks that return tensor data a numpy.array will be returned, while for output sinks that return lists of objects a list of OutputObject objects will be returned.

Note: at the moemnt a copy of the data returned by fluxEngine is made before it is passed back to the user, as the internal memory management of fluxEngine is not easily mapped onto the memory ownership model of numpy or Python. This might change in a future version of fluxEngine.

Parameters

index (int) – The index of the output sink to obtain the data from

Returns

The output sink data, depending on the output sink type

Return type

object

outputSinkInfo(index)

Get information about a specific output sink

Parameters

index (int) – The index of the output sink

Returns

The information about the given output sink

Return type

OutputSinkInfo

outputSinkInfos()

Get information about all output sinks in the model

The position in the list will indicate the index of the sink. For example, the first entry of the list will correspond to an output sink index of 0, the second to 1, etc.

Returns

A list of OutputSinkInfo objects for each output sink

Return type

list

processNext()

Perform data processing

After setSourceData() has been called process that data with fluxEngine. The results may be queried via the outputSinkData() method.

resetState()

Reset the state of the processing context

This method may only be called in between calls of processNext().

When the data to be processed is in the form of entire HSI cubes, that is, the context was created via the variant of the constructor that indicates HSI cubes should be processed, this method will have no effect. (Unless processing was aborted via abort(), in which case this must be called to clean up the state.)

When the data to be processed is in the form of consecutive PushBroom frames, that is, the context was created via the overload of the constructor that indicates PushBroom frames should be processed, this method will reset the internal state and make the context appear as if it had been freshly created. This means that any operation that remembers past state to gain spatial information in the y direction will be reset to the beginning. This affects mostly object-based operations.

This would typically be called when a system with a PushBroom camera is started up again after a pause, and the previously processed data has no direct relation to the data to be processed from this point onwards.

setSourceData(data, index=-1, **kwargs)

Set the source data for the next processing iteration

In contrast to the C/C++ APIs this method must be called before each call to processNext(), otherwise the same data will be processed each time.

HSI Cubes

If the input cube size was fixed during the creation of the processing context, the height and width parameters must match the height and width specified during creation of the context, or an error will be thrown.

If the input cube size was variable during the creation of the processing context, the height and width parameters must be smaller than or equal to the maximum size specified during the creation of the context.

The storage order of the cube that has been specified during the creation of the processing context will be used. This means that the height and width parameters may refer to different dimensions of the cube depending on the storage order:

  • For a BIP cube, the cube will be indexed via (y, x, band), meaning the height parameter referes to the dimension 0, the width parameter to dimension 1 and the wavelength count supplied during creation of the cube to the dimension 2 of the cube. This means the supplied data must have the shape (height, width, bands)

  • For a BIL cube, the cube will be indexed via (y, band, x), meaning the height parameter referes to the dimension 0, the width parameter to dimension 2 and the wavelength count supplied during creation of the cube to the dimension 1 of the cube. This means the supplied data must have the shape (height, bands, width)

  • For a BSQ cube, the cube will be indexed via (band, y, x), meaning the height parameter referes to the dimension 1, the width parameter to dimension 2 and the wavelength count supplied during creation of the cube to the dimension 0 of the cube. This means the supplied data must have the shape (bands, height, width)

PushBroom Frames

The input PushBroom frame size had to be fixed during the creation of the processing context, and the size of the frame must be equal to width and bands.

The storage order of the cube that has been specified during the creation of the processing context will be used. The supplied frame must be a 2D image with the following dimensions:

  • For LambdaX storage order, the width of the image must be equal to the wavelength count specified during the creation of the processing context, while the height of the image must be equal to the specified spatial width.

  • For LambdaY storage order, the height of the image must be equal to the wavelength count specified during the creation of the processing context, while the width of the image must be equal to the specified spatial width.

Note: at the moment a copy of the source data will be made as soon as this method is called, as the internal memory management of fluxEngine is not easily mapped onto the memory ownership model of numpy. This might change in a future version of fluxEngine.

Devices

A BufferInfo or PersistentBufferInfo object must be supplied that was retrieved from the device.

Measurement

A MeasurementList must be supplied as the first argument, and the index into the measurement list as the second argument.

Sequence Id Handling

The sequence id is a number that has to increase between individual buffers that are being processed. In an ideal world the sequence id will be incremented by 1 between each buffer that is being processed. If the sequence id increases by more than one, the processing logic assumes that buffers have been skipped (because they were lost, for example), and will act accordingly. (This only affects processing when state is tracked between invocations, such as when processing data from pushbroom cameras, which allows filters such as object detectors to work.)

If the sequence id increases by a large amount between buffers (on the order of 100) the processing logic may implicitly reset any internal state that it keeps between buffers.

Supplying a sequence id that is lower than a previous sequence id may lead to undefined results.

And optional keyword argument sequenceId may be supplied that overrides the default sequence id behavior. By default, for buffers and persistent buffers the sequence id will be the buffer number stored in the buffer information structure. For manually supplied PushBroom frames the sequence id will be given by the last processed sequence id plus one. If a non-negative number is specified for the sequence id though, that will be used instead of the default behavior.

Specifying a sequence id when processing entire HSI cubes at once has no effect at the moment, as the sequence id is not considered by the processing logic at the current time for these cases. (This may change in the future.)

Parameters

  • data (numpy.array or BufferInfo or MeasurementList) – The new source data to set

  • index (int) – The index of the measurement to use (0 is the first measurement in the measurement list). This needs to be specified only when supplying a MeasurementList for the first argument, and will be ignored otherwise.

Keyword Arguments

sequenceId (int) – The sequence id to use instead of the default sequence id. Do not specify, or specify a negative number to indicate that the default behavior is to be used.

wait()

Wait until processing or an abort is complete.

This method may be called from a different thread while processing is currently active. It will wait until the processing context is not in use anymore, either because processing has completed in the mean time, or an abort was requested and the abort has completed.

Note that ProcessingContext::processNext() already blocks and this method must only be used from different threads that also want to wait for the processing of a specific context to complete.