File I/O

fluxEngine provides the functionality to load or save recordings from disk. At the moment this is restricted to the ENVI format for HSI cubes.

Loading ENVI cubes

To load an ENVI cube from disk, one may use the loadMeasurementList() function. The user is required to specify the format, which must be "ENVI" in this case (the format is case sensitive!), and the file name.

For example, assuming the cube has been stored in a file cube.hdr (accompanied by its corresponding .bin file), the following code would load that file:

fluxEngine::MeasurementList cube = fluxEngine::loadMeasurementList(handle, "ENVI", inputCubeFileName);

The result of this method is a measurement list. In the case of ENVI this will always consist of exactly one measurement, but other formats that may be supported in the future might contain more than one measurement per file. The count() method will return the number of measurements in a measurement list, which in this case will be 1.

The following code shows how to extract information about the measurement:

 1 fluxEngine::ValueType valueType;
 2 if (!cube.getValueType(0, valueType))
 3     valueType = fluxEngine::ValueType::Intensity;
 4 std::vector<double> wavelengths = cube.wavelengths(0);
 5
 6 fluxEngine::TensorData data = cube.tensorData(0);
 7 fluxEngine::TensorData whiteData, darkData;
 8 if (cube.hasReference(0, "WhiteReference"))
 9     whiteData = cube.referenceTensorData(0, "WhiteReference");
10 if (cube.hasReference(0, "DarkReference"))
11     darkData = cube.referenceTensorData(0, "DarkReference");

Note that the white and dark references may not be present. The ENVI parser of fluxEngine will look for files thast have the name cube_White.hdr and WHITEREF_cube.hdr to find the white reference cube in the same directory as the specified file, and cube_Dark.hdr and DARKREF_cube.hdr to find the dark reference cube. If these exist they may be returned by the referenceTensorData() method, otherwise only the cube itself will be loaded.

The information returned here may be used to initialize a processing context, see below for details.

Note

After loading an ENVI cube the storage order of the loaded measurement will always be BIP, regardless of how the cube was stored on disk. This is because fluxEngine internally works in the BIP storage order and automatically converts data when it is loaded by the user.

Creating an ENVI cube from camera recordings

When recording data from a PushBroom camera it is possible to use that to create an ENVI cube. The user must have created a instrument recording processing context, see Recording HSI Data for details. The output of that processing context must then have been stored in one or more buffer containers. These may then be used to create a measurement that then may be saved as an ENVI cube.

First the user must fill a MeasurementHSICubeBufferInput structure with the metadata required to create the cube.

The following fields are part of the structure and may be set:

Finally the user may specify the data itself as a list of pointers to BufferContainer objects. The list is stored in bufferContainers. The contents of the buffer containers will be concatenated in the order specified by the user to make up the cube.

This information must be provided to the createMeasurementHSICube() function, which will create a MeasurementList object. That may then be saved to disk as an ENVI cube via the saveMeasurementList() function.

Following up the example that performed the actual measurement in Recording HSI Data, the following code will store the measurement in an ENVI cube.

 1 // Specify the file name (Windows)
 2 std::wstring fileName = L"C:\\some_path.hdr";
 3 // (other operating systems:)
 4 // std::string fileName = "/some/path.hdr";
 5 // The following were created previously:
 6 fluxEngine::ProcessingContext::HSIRecordingResult ctxAndInfo;
 7 fluxEngine::ProcessingContext ctx;
 8 fluxEngine::BufferContainer recordedData;
 9 try {
10     fluxEngine::MeasurementHSICubeBufferInput input;
11     input.name = "Example cube";
12     input.valueType = fluxEngine::ValueType::Intensity;
13     input.wavelengths = ctxAndInfo.wavelengths;
14     input.whiteReference = ctxAndInfo.whiteReference;
15     input.darkReference = ctxAndInfo.darkReference;
16     input.calibrationInfo = &ctxAndInfo.calibrationInfo;
17     input.bufferContainers.push_back(&recordedData);
18
19     auto cube = fluxEngine::createMeasurementHSICube(handle, input);
20     fluxEngine::saveMeasurementList(handle, cube, "ENVI", fileName, true);
21 } catch (std::exception& e) {
22     std::cerr << "An error occurred: " << e.what() << std::endl;
23     exit(1);
24 }

Here the saveReferences parameter to saveMeasurementList() is True, so the references will be saved next to the ENVI file itself. Otherwise only the data itself will be saved, even if the measurement contains white reference data. The white reference will be stored in the form cube_White.hdr. (The ENVI reader in fluxEngine will also try to look for files of the pattern WHITEREF_cube.hdr for compatibility with other software, but the ENVI writer will only write out white references in the other file name pattern.)

Note

While creating a cube with createMeasurementHSICube() a copy of all of the data of the cube is made, so the user needs to have a sufficient amount of RAM to perform this operation.

Creating an ENVI cube from user-supplied data

To manually create an ENVI cube the user must provide a three-dimensional tensor that contains the cube data in any storage order, as well as a single array containing the wavelengths associated with that cube.

That information has to be stored in a MeasurementHSICubeInput structure that the user must provide to createMeasurementHSICube().

It has the largely the same fields as the MeasurementHSICubeBufferInput structure, with the following differences:

  • Instead of the bufferContainers field to specify the data, the following fields exist, instead:

    • storageOrder to indicate the storage order the cube is stored in.

    • lines to specify the height of the cube. (This will correspond to a dimension of the cube, which one will depend on the storage order.)

    • samples to specify the width of the cube. (This will correspond to a dimension of the cube, which one will depend on the storage order.)

    • bands to specify the number of bands of the cube. (This will correspond to a dimension of the cube, which one will depend on the storage order.) This must be identical to the number of entries in the wavelengths list.

    • strides to specify the stride structure the data is present in memory.

    • data to specify a pointer to the first element of the cube.

  • The whiteReference and darkReference fields are pointers to MeasurementHSICubeInput structures (they may be nullptr) that define the cubes of the references.

The following example shows how such a cube could be created and stored on disk:

 1 // Specify the file name (Windows)
 2 std::wstring fileName = L"C:\\some_path.hdr";
 3 // (other operating systems:)
 4 // std::string fileName = "/some/path.hdr";
 5 // This must be present and filled with the data to save:
 6 void const* whiteReferenceData;
 7 void const* cubeData;
 8 try {
 9     fluxEngine::MeasurementHSICubeInput input, whiteReference;
10     whiteReference.name = "White reference of Example cube";
11     whiteReference.valueType = fluxEngine::ValueType::Intensity;
12     whiteReference.storageOrder = fluxEngine::HSICube_StorageOrder::BIP;
13     whiteReference.lines = 5;
14     whiteReference.samples = 100;
15     whiteReference.bands = 13;
16     whiteReference.strides[0] = 1300;
17     whiteReference.strides[1] = 13;
18     whiteReference.strides[2] = 1;
19     whiteReference.data = whiteReferenceData;
20     whiteReference.wavelengths = { 400, 450, 500, 550, 600, 650, 700, 750, 800, 850, 900, 950, 1000 };
21
22     input.name = "Example cube";
23     input.valueType = fluxEngine::ValueType::Intensity;
24     input.storageOrder = fluxEngine::HSICube_StorageOrder::BIP;
25     input.lines = 100;
26     input.samples = 100;
27     input.bands = 13;
28     input.strides[0] = 1300;
29     input.strides[1] = 13;
30     input.strides[2] = 1;
31     input.data = cubeData;
32     input.wavelengths = { 400, 450, 500, 550, 600, 650, 700, 750, 800, 850, 900, 950, 1000 };
33     input.whiteReference = &whiteReference;
34     // no dark reference
35     // no calibration info
36
37     auto cube = fluxEngine::createMeasurementHSICube(handle, input);
38     fluxEngine::saveMeasurementList(handle, cube, "ENVI", fileName, true);
39 } catch (std::exception& e) {
40     std::cerr << "An error occurred: " << e.what() << std::endl;
41     exit(1);
42 }

Note

It is currently not possible to create an ENVI cube in a storage order other than BIP, which is what fluxEngine uses internally. When creating the measurement list via the createMeasurementHSICube() method the storage order is converted internally. The storage order attribute is only to ensure that fluxEngine knows the storage order of the input.

Note

Since processing may be required when creating a measurement from manually specified HSI cube data (in order to normalize the storage order, for example), the user may use an overload of createMeasurementHSICube() to specify a processing queue set where the processing occurs. If no processing queue set is specified, the default queue set of the handle will be used, and createMeasurementHSICube() may block if that is currently in use from a different thread.

Processing an ENVI cube

It is also possible to create a processing context that may be used to process data from a HSI cube that was loaded. This is done via the ProcessingContext::createMeasurementProcessingContext() static method. It takes the following arguments:

  • The measurement list to create the processing context for.

  • The index of the measurement to use. For ENVI cubes this will always be 0.

  • The model to process the cube with.

  • Flags to use to influence how the context is set up. See the MeasurementProcessingContextFlag enumeration for details on the available flags. By default the user may specify 0 here to indicate that the default behavior is chosen.

  • Optionally: a processing queue set that applies to the context.

The following example shows how to process a HSI cube that was loaded from disk with a given model:

 1 // These were loaded beforehand
 2 fluxEngine::MeasurementList cube;
 3 fluxEngine::Model model;
 4 try {
 5     auto context = fluxEngine::createMeasurementProcessingContext(cube, 0, model, 0);
 6     context.setSourceData(cube, 0);
 7     context.processNext();
 8     // At this point the data may be extracted from the context via
 9     // contest.outputSinkData().
10 } catch (std::exception& e) {
11     std::cerr << "An error occurred: " << e.what() << std::endl;
12     exit(1);
13 }