A Spatio-temporal-spectral Multi-dimensional Data Format (MDD) Tool: from MARS v1.0 to v2.03
Zhang, L. F.1 Sun, X. J.1* Zhang, X.1 Wang, N.1 Zhang, M. Y.1,2
Lin, Y. K.1,2 Huang, H.1 Cen, Y.1 Huang, C. P.1 Yang, H.1
Zhang, H. M.1 Liu, J.1 Tong, Q. X.1
1. Institute of Remote Sensing and Digital Earth, Chinese Academy of Sciences, Beijing 100101, China;
2. University of Chinese Academy of Sciences, Beijing 100049, China
Abstract: The MARS-Multi-dimensional Analysis of Remote Sensing V1.0 was published in June 2017, based on which the updated version MARS v2.03 is now released. The system of this version can process the spatio-temporal-spectral multi-dimensional dataset, namely MDD—Multi- Dimensional Data Format, which includes Temporal Sequential in Band (TSB), Temporal Sequential in Pixel (TSP), Temporal Interleaved by Band (TIB),Temporal Interleaved by Pixel (TIP), and Temporal Interleaved by Spectrum (TIS). It can deal with tasks of input, storage, analysis, output, format transform, data preprocessing, spectral analysis, classification, vegetation index calculation, etc. with data format of .mdd. Also the redundancy and mistakes have been corrected to make the system run efficiently. The renewed version keeps the original DOI.
Keywords: MDD; data format; multi-dimensional dataset; data update; MARS2.03
1 Multi-Dimensional Dataset (MDD)
1.1 MDD Data Format
MDD is a multi-dimensional data format that can integrate the temporal, spatial, and spectral features of remote sensing data[1]. A MDD consists of spectral data cube of series of different phases. From the prospect of dataset, MDD includes temporal information compared to remote sensing dataset with only spatial and spectral information and changes into a spatio-temporal-spectral 4-dimensional remote sensing dataset with spatial dimension of 2 and spectral dimension of 1, and at the same time it has the ability to extending to 5-dimensional dataset with spatial dimension of 3 and spectral dimension of 1[2].
1.2 Physical Storage Structure of the MDD
MDD dataset consists of a header file and a data file.
The header file records information regarding the image data itself as well as descriptive information. It includes the size of each dimension, interleave, and data type. The data file includes the coordinate projection, affine transform coefficients, file name, file type, and data offset values. Data files are actually stored image data files, and are divided into five data storage structures (TSB, TSP, TIB, TIP, and TIS) according to the storage order of the data.
1.2.1 MDD Header File
The MDD header file records all metadata information used to express and read the image data file, as shown in Table 1.
Table 1 Definition of the fields in the header file
|
Field
|
Definition
|
|
Samples
|
The number of samples (pixels) per image line for each band
|
|
Lines
|
The number of lines per image for each band
|
|
Bands
|
The number of bands per image file
|
|
Time
|
The number of phases per image file
|
|
Header offset
|
Offset values for each band. When a file is read, these bytes are skipped
|
|
File type
|
The MDD-defined file type, e.g., MDD Standard
|
|
Data type
|
The type of data representation: 1 = 8-bit byte, 2 =16-bit signed integer, 3 = 32-bit signed long integer, 4 = 32-bit floating point, 5 = 64-bit double-precision floating point, 6 = 2 × 32-bit complex, real-imaginary pair of double precision, 9 = 2 × 64-bit double-precision complex, real-imaginary pair of double precision, 12 =16-bit unsigned integer, 13 = 32-bit unsigned long integer, 14 = 64-bit signed long integer, 15 = 64-bit unsigned long integer
|
|
Interleave
|
Refers to whether the data are TSB, TSP, TIB, TIP, or TIS
|
|
Sensor type
|
The type or name of sensor, e.g., Landsat TM, SPOT, MODIS, etc.
|
|
Byte order
|
The order of the bytes in an integer, long integer, 64-bit integer, unsigned 64-bit integer, floating point, double-precision, and complex data types. One of the following is used:
Byte order = 0 (Host (Intel) in the Header Info dialog) is least significant byte first (LSF) data (DEC and MS-DOS systems)
Byte order = 1 (Network (IEEE) in the Header Info dialog) is most significant byte first (MSF) data (all other platforms)
|
|
Map info
|
Lists geographic coordinate information in the order of projection name (UTM), reference pixel x location (in file coordinates), pixel y, pixel easting, pixel northing, x pixel size, y pixel size, projection zone, and north or south (UTM only)
In MDD, pixel values always refer to the upper-left corner of the pixel. Map coordinates also typically refer to the upper-left corner of the pixel
|
|
Coordinate system string
|
A coordinate system string field to the header file. It lists the parameters used for a geographic coordinate system or projected coordinate system
|
|
Band names
|
Specific names for each band of an image
|
|
Time names
|
Specific names for each phase of an image
|
1.2.2 MDD Data File
The MDD data file stores the image data. The raster image data can be stored in one of the TSB, TSP, TIB, TIP, or TIS storage structure and as with the binary data stream, the five data structures can be converted to each other. The data format has been fully demonstrated in previous publication[1].
2 MARS[3] System Update
2.1 Descriptions and Existing Problems of MARS v1.0.
2.1.1 Descriptions of MARS v1.0
Version naming rules:
Major_Version_Number.Minor_Version_Number[Revision_Number[.Build_Number]]
Version management rules:
(1) The version number of the first version of the software is 1.0 or 1.00.
(2) When the software undergoes modifications or debug, the major version number and minor version number will not be changed. Revision number adds 1.
(3) When only minor applications were added to or removed from the original software, the major version number will not be changed, the minor version number adds 1 and the revision number returns to 0.
(4) When the software has been significantly modified or too many modifications resulting in overall changes, the major version number adds 1.
2.1.2 Existing Problems of MARS v1.0
After one and half years’ applications, we found some functions in MARS v1.0 need to be improved. The updates in new version (MARS v2.03) include more precise and complete settings of system, further improvements on design and usage, and some bugs to be corrected.
2.2 Updates of MARS v2.03
Table 2 Deleted parts of MARS v2.03 from MARS v1.0
|
Number
|
Deleted parts
|
|
1
|
Deleting Processing from menu
|
|
2
|
Deleting Spectral from menu
|
|
3
|
Deleting Save As from menu
|
|
4
|
Deleting Open Vector from menu
|
|
5
|
Deleting Temporal Filter from menu
|
|
6
|
Deleting Statistics from menu
|
|
7
|
Deleting PCA from menu
|
2.2.1 Deleted Parts
The following parts (Table 2) have been deleted in the updated version to make usage easier.
2.2.2 Added Parts
There are calculation functions in the updated version including vegetation index calculation (Table 3).
2.2.3 Modified Parts
Some functions have been modified in the updated version (Table 4).
Table 3 Added parts of MARS v2.03 based on MARS v1.0
|
Number
|
Added parts
|
|
1
|
Vegetation Index calculation operation, including calculating UNVI and NDVI
|
|
2
|
Temporal classification and Temporal_Spectral Classification in menu
(To be developed)
|
2.2.4 Debug
During the usage of MARS v1.0, we found some bugs in the system. And in the updated version we corrected these bugs, as shown in Table 5.
Table 4 Modified parts of MARS v2.03 based on MARS v1.0
|
Number
|
Modified parts
|
|
1
|
Moving MDD Open from the menu of MDD Analysis to the menu of File and renaming it Open MDD
|
|
2
|
Moving MDD Export from the menu of MDD Analysis to the menu of File
|
|
3
|
Substituting MDD Builder in the menu of MDD Analysis with Processing and putting MDD Append, MDD Merge and MDD Split under it
|
|
4
|
Moving MDD Math from MDD Analysis to Basic Tools and combining it with Band math in Data Math
|
|
5
|
Moving all sub-items from Classification to MDD Analysis
|
Table 5 Debug of MARS v2.03 based on MARS v1.0
|
Number
|
Debug parts
|
|
1
|
During MDD construction, when dealing with large amount of data the speed is low and the software breaks down
|
|
2
|
The version of the third party QT used by Image Format Transformation is too old, which makes the software breaks down when applying the operation
|
|
3
|
The code of MDD Append is not fully logical
|
|
4
|
The speed of MDD Merge is low
|
|
5
|
In the operation of Masking, the mask file cannot be loaded
|
2.3 Main Functions of MARS v2.03
The main functions of the updated MARS v2.03 include 7 categories and 27 applications, which consists of: File, including functions of inputs, outputs, save, exit and etc.; MDD Builders transforming data format from Landsat[4–8] to MDD, or from HDF [9–10] or IMG to MDD data format; MDD Construction operations, including appending, merging and splitting; Basic Tools, including building 3D Cube, rotating image, resizing image, spatial filtering, spectral filtering, data math and masking; Transform, including MDD conversion and image format transformation; MDD Analysis, including classification and vegetation index calculation; Windows and Help, etc.
Table 6 Main functions of MARS v2.03
|
Number
|
Menu
|
Function
|
Description
|
|
1
|
File
|
Open MDD
|
Opening MDD files
|
|
2
|
Open Image
|
Opening image files
|
|
3
|
Edit Image Header
|
Editing header file of Images
|
|
4
|
MDD Export
|
Exporting MDD files
|
|
5
|
Save as ASCII
|
Saving files as ASCII data
|
|
6
|
Exit
|
Exiting
|
|
7
|
MDD Builder
|
From Landsat
|
Constructing MDD data from Landsat data
|
|
8
|
From MODIS hdf
|
Constructing MDD data from MODIS hdf data
|
|
9
|
From MODIS img
|
Constructing MDD data from data with format of .img
|
|
10
|
From ENVI Image
|
Constructing MDD data from ENVI Image
|
|
11
|
MDD Append
|
Appending MDD files
|
|
12
|
MDD Merge
|
Merging MDD files
|
|
13
|
MDD Split
|
Splitting MDD files
|
|
14
|
Basic Tools
|
Build 3D Cube
|
Building 3D image cube
|
|
15
|
Rotate Image
|
Rotating image
|
|
16
|
Resize Image
|
Resizing image
|
|
17
|
Spatial Filter
|
Filtering in the spatial dimension
|
|
18
|
Spectral Filter
|
Filtering in the spectral dimension
|
|
19
|
Data Math
|
Calculating
|
|
20
|
Masking
|
Masking the image file
|
|
21
|
Transform
|
MDD Conversion
|
Data format conversion of MDD
|
|
22
|
Image Format Transformation
|
Transforming image format
|
|
23
|
MDD Analysis
|
Classification
|
Classification operation
|
|
24
|
Vegetation Index
|
Calculating vegetation index
|
|
25
|
Windows
|
Available Data List
|
Showing available data list
|
|
26
|
Help
|
User Manual
|
Showing user manual
|
|
27
|
About MARS v2.03
|
Showing information about MARS v2.03
|
2.4 Comparison of Main Interface and Menu of MARS v1.0 and v2.03
For the main interface, compared with MARS v1.0 (Figure 1), the main interface of the updated MARS v2.03 (Figure 2) is more concise.
MARS v2.03 merges, deletes and adds some menu function items based on MARS v1.0. For example, “MDD Conversion” and “Image Format Transformation” are added into the menu item of “Transform dish”, redundant menu items for processing and analyzing the MDD dataset such as “Preprocessing” are deleted, it Added menu items such as “Vegetation Index” calculation for MDD dataset. The new version will make the generation, operation and analysis of the MDD dataset more purposeful, make the software operation more reasonable, and repair the system vulnerabilities existing in the original version, and thus the software will run more smoothly.
Figure 1 Main interface and its menus of MARS v1.0
Figure 2 Main interface and its menus of MARS v2.03
3 Intellectual Property Declaration
MARS v2.03 is developed based on MARS v1.0. Despite of the difference in these two versions, the principles of academic achievements are the same and they are correlated and successive. Therefore, MARS v2.03 succeeds the DOI from MARS v1.0 as DOI: 10.3974/geodb.2017.02.20.V1. The research and development team (the authors, corresponding author, co-authors with author sequences and institute), and funding project are not changed compared to that of MARS v1.0. Thus the intellectual property of MARS v2.03 and MARS v1.0 are not changed.
References
[1] Zhang, L. F., Chen, H., Sun X. J., et al. Designing spatial-temporal-spectral integrated storage structure of multi-dimensional remote sensing images [J]. Journal of Remote Sensing, 2017, 21(1): 62-73.
[2] Zhang, L. F., Sun, X. J., Zhang, X., et al. New version for the multi-dimensional data format (MDD) analysis tool (MARS v2.03) [DB/OL]. Global Change Research Data Publishing & Repository, 2017. DOI: 10.3974/geodb.2017.02.20.V1.
[3] Zhang, L. F., Sun, X. J., Zhang, X., et al. A multi-dimensional data format (MDD) and analysis tool [J]. Journal of Global Change Data & Discovery, 2017, 1(2): 121-135. DOI: 10.3974/geodp.2017.02.01.
[4] Kennedy, R. E., Yang, Z., Cohen, W. B. Detecting trends in forest disturbance and recovery using yearly Landsat time series: 1. LandTrendr—temporal segmentation algorithms[J]. Remote Sensing of Environment, 2010, 114(12): 2897-2910.
[5] Huang, C., Goward, S. N., Masek, J. G., et al. An automated approach for reconstructing recent forest disturbance history using dense Landsat time series stacks [J]. Remote Sensing of Environment, 2010, 114(1): 183-198.
[6] Powell, S. L., Cohen, W. B., Healey, S. P., et al. Quantification of live aboveground forest biomass dynamics with Landsat time-series and field inventory data: a comparison of empirical modeling approaches [J]. Remote Sensing of Environment, 2010, 114(5): 1053-1068.
[7] Margono, B. A., Potapov, P., Hansen, M. C. Mapping and monitoring deforestation and forest degradation in Sumatra (Indonesia) using Landsat time series data sets from 1990 to 2010 [J]. Environmental Research Letters, 2012, 7(3): 034010.
[8] Townsend, P. A., Helmers, D. P., Kingdon, C. C., et al. Changes in the extent of surface mining and reclamation in the Central Appalachians detected using a 1976-2006 Landsat time series [J]. Remote Sensing of Environment, 2009, 113(1): 62-72.
[9] Galford, G. L., Mustard, J. F., Melillo, J., et al. Wavelet analysis of MODIS time series to detect expansion and intensification of row-crop agriculture in Brazil [J]. Remote Sensing of Environment, 2008, 112(2): 576-587.
[10] Sakamoto, T., Yokozawa, M., Toritani, H., et al. A crop phenology detection method using time-series MODIS data [J]. Remote Sensing of Environment, 2005, 96(3): 366-374.