Dataset of Aircraft Classification by Remote

Sensing Image

Chen, J. Y.^1,2  Li, H. W.^1*  Zhang, G.¹  Wang, S.¹  Chen, T. Q.¹

1. Xi'an Institute of Optics and Precision Mechanics, Chinese Academy of Sciences, Xi??an 710119, China;

2. University of Chinese Academy of Sciences, Beijing 100049, China

Abstract: The aircraft is an important target in remote sensing domain. However, public classification datasets of aircraft types are insufficient, which limits the research and application of large-scale extraction of aircraft types information by using remote sensing images. This paper solves the problem of data shortage in classification of aircraft types. First, we obtain high-resolution remote sensing images containing various airports and plane types information around the world from public data sources and select 3,594 pieces of valid aircraft images. Then, we choose the feature of wing and propeller in remote sensing satellite and divide the aircraft types into seven level I classifications by employing the attention mechanism including (1) Swept-back wing air-craft; (2) Swept-back aircraft with leading edge; (3) Forward-swept wing airplane with trailing edge; (4) Delta-wing aircraft; (5) Flat-wing aircraft; (6) Propeller aircraft; (7) Helicopter. Next, 11 level II classifications are obtained according to the fuselage color and so on. All images are compiled into a classification dataset of aircraft types based on remote sensing images (OPT-aircraft _v1.0), which is stored in .png format and consists of 3,594 data files with a size of 69.3 MB. The datasets not only identify aircraft in a fine-grained manner and simulate different types of aircraft but also provide experimental data for aircraft image classification in remote sensing field.

Keywords: remote sensing image classification; the classification dataset of aircraft types; Google Earth; attention mechanism

Dataset Available Statement:

The dataset supporting this paper was published at: Chen, J. Y., Li, H. W., Zhang, G., Wang, S., Chen, T. Q. Dataset of identifying aircraft groups by remote sensing images [J/DB/OL]. Digital Journal of Global Change Data Repository, 2020. DOI: 10.3974/ geodb.2020.03.25.V1.

1 Introduction

Aircraft is a kind of important target in remote sensing field. It is one of the hotspots to identify the position, type, and number quickly and accurately during the detection of aircraft target. The existing fine-grained visual classification dataset of aircraft is FGVC-aircraft^[1], which is collected 120 aircraft types. However, the FGVC-aircraft dataset collects aircraft photos at close range, which cannot be applied to identification of aircraft types directly in remote sensing domain. In order to identify and detect abnormal aircraft, the reference data of aircraft types in remote sensing observation environment is needed.

To speed up the development of remote sensing aircraft recognition technology, it is very important to obtain aircraft classification dataset. The classification dataset of aircraft types based on remote sensing images (OPT-Aircraft _v1.0) is established from several airports around the world and corresponding classification criteria to provide a reference for classification of aircraft types.

2 Metadata of the Dataset

The metadata of the ??Dataset of identifying aircraft groups by remote sensing images??^[2] is summarized in Table 1. It includes the dataset full name, short name, authors, data format, data size, data files, data publisher, and data sharing policy, etc.

Table 1  Metadata summary of the dataset

3 Methods

3.1 Algorithmic Rationale

(1) Characteristics of aircraft types in natural scene

Aircraft classification photos in natural scenes (e.g., FGVC-aircraft^[1]) are obtained from the side of airframe and the shooting distance is close so that we could see the structure clearly and get category information.

The remote sensing aircraft images are obtained from the aerial view of remote sensors. The shooting distance is relatively long and there??s a lot of noise. In the absence of prior knowledge, we can??t distinguish the type of aircraft and relevant information directly.

Therefore, when remote sensing data is used for aircraft type classification, we neither use natural scene aircraft photos as reference nor use natural scene plane type classification system. We should stablish aircraft data and aircraft type classification system suitable for remote sensing data.

(2) Classification of aircraft types system for remote sensing data

The aircraft photos in natural scene cannot be applied to classification of remote sensing aircraft types directly. However, after connecting airports, aircraft types and high-resolution remote sensing images with GPS positioning information, remote sensing features of various aircraft types can be obtained indirectly by using interpretation or machine learning methods. These characteristics can be used as reference data for a wider range of aircraft type classification. Firstly, the airport images are obtained from the public high-resolution remote sensing images. Next, we cut and sift out high-quality aircraft data from airport images. And then, the aircraft types are divided according to the characteristics of the screened aircraft remote sensing data. Finally, we augment the data and obtain classification dataset of aircraft types based on remote sensing images (OPT-Aircraft _v1.0).

3.2 Data Collection and Processing

The flowchart of the dataset processing is shown in Figure 1, which includes seven parts: remote sensing data collection containing aircraft information, screening of remote sensing data, purification of remote sensing data, unified scale, level I classification of aircraft types, level II classification of aircraft types, and data augmentation.

3.2.1 Collection of Remote Sensing Data Containing Aircraft Information

We collect a lot of airport images from public datasets, including DIOR^[4], UCAS_AOD^[5], NWPU VHR-10^[6-8], DOTA^[9-10] and Google Earth. Specifically, for the Google Earth, we select some airport images around the world such as Beijing Capital International Airport, Amsterdam Airport, Phoenix Sky Harbor International Airport and so on. The collected images come from different sensors, times, seasons, and light so that these data have diversity within the class.

3.2.2 Screening of Remote Sensing Data

For the airport images from public dataset, we combine the category labels to get aircraft data. For the airport images from Google Earth, we obtain the aircraft target by employing professionals to annotate the aircraft manually.

3.2.3 Purification of Remote Sensing Data

Four kinds of problems in data are identified and eliminate.

(1) The interference information includes stepladder, trucks, personnel, and adjacent aircraft.

(2) Aircraft background noises include white zebra crossing, bright stripe, and shadows of surrounding buildings.

(3) The aircraft is located at the edge.

(4) Bad data with too strong exposure, weak exposure, fog, and too small target.

3.2.4 Unified Scale

After filtering the remote sensing aircraft data through human computer interaction, the data size is normalized to 96 pixels ?? 96 pixels.

3.2.5 Level I Classification of Aircraft Types

According to 15 research results and the visual attention mechanism, the aircraft could be classified to the seven level I types according to the wing and the propeller: (1) Swept-back wing aircraft; (2) Swept-back aircraft with leading edge; (3) Forward-swept wing airplane with trailing edge; (4) Delta-wing aircraft; (5) Flat-wing aircraft; (6) Propeller aircraft; (7) Helicopter.

3.2.6 Level II Classification of Aircraft Types

Four level I types are classified further and obtains 11 level II classifications according to engine and color.

(1) The swept-back wing aircraft: It can be divided into three types according to the width of wings, the position of engine and the fuselage color. The swept-back wing aircraft with narrow wings, bright colored airframe (white, blue, etc.) and engines at empennage is named the swept-back wing aircraft I. The swept-back wing aircraft with narrow wings, bright color airframe (white, blue, etc.) and engines under wings is named the swept-back wing aircraft II. The swept-back wing aircraft with wider wings and dark colored airframe (grey, dark green, etc.) is named the swept-back wing aircraft III.

(2) Swept-back aircraft with leading edge: It can be divided into two categories according to the position of the engine. The engine at the tail is named as swept-back aircraft with leading edge-I. The engine is not located at the tail and this type of aircraft is named as the leading-edge swept wing II.

(3) Flat-wing aircraft. The engine at the tail is named flat-wing aircraft-I. The engine is not located at the tail and this type of aircraft is named flat-wing aircraft-II.

(4) The propeller aircraft: It can be divided into four categories according to the ratio of propeller to airframe and color. The proportion of smaller propeller and the fuselage with bright color (white, color, etc.) is named propeller aircraft I. The proportion of smaller propeller and the fuselage with dark colored airframe (grey, dark green, etc.) is named propeller aircraft II. The proportion of larger propeller and bright colored airframe (white, color, etc.) is named propeller aircraft III. The proportion of larger propeller and dark colored airframe (grey, dark green, etc.) is named propeller aircraft IV.

3.2.7 Data Augmentation

According to different experimental requirements, the data for the OPT-Aircraft _v1.0 can be augmented. For example, we can mirror the data and rotate it by 45??, 90??, 135??, 180??, 270??, etc. The data can be amplified by 3 to 7 times according to the experimental conditions so that we can improve the experimental accuracy.

4 Dataset Results and Validation

4.1 Data Composition

The classification dataset of aircraft types based on remote sensing images (OPT-aircraft _V1.0) collects a total of 3,954 valid aircraft, which is divided into 7 level I classifications and 11 level II classifications.

The 7 level I classifications are: 1. Swept-back wing aircraft, 2. Swept-back aircraft with leading edge, 3. Forward-swept wing aircraft with trailing edge, 4. Delta-wing aircraft, 5. Flat wing aircraft, 6. Propeller aircraft, 7. Helicopter.

The naming rules of the 11 level II classifications are separated by the decimal point. The number before the decimal point represents the level I classifications, and the number after the decimal point represents the level II classification of the same category. Table 2 shows the details of the specific classification, the English name of the aircraft types and the corresponding data folder??s name.

Table 2  Classification Dataset of Aircraft Types based on Remote Sensing Images (OPT-Aircraft _v1.0)

4.2 Data Products

Figure 2 shows the 11 level II classifications in the dataset of OPT-Aircraft _v1.0. Each row displays 10 images randomly in the same category.

We used t-SNE^[11] algorithm to reduce the dimension of the OPT-Aircraft _v1.0 dataset. The t-SNE^[11] (t-distributed stochastic neighbor embedding) algorithm maps the sample data points to the probability distribution through affine transformation so as to make the two probability distributions in high spaces and low dimensional spaces are as similar as possible. Figure 3 shows the two-dimensional space representation of the 5 level II classifications (1.3 Swept-back wing aircraft III, 2.2 Swept-back aircraft with leading edge II, 4 Delta-wing aircraft, 5.2 Flat-wing aircraft II, 6.2 Propeller aircraft II) by using the t-SNE algorithm^[11]. The same color represents the same type. The horizontal and vertical coordinates represent the mapping values of two dimensions in a low-dimensional space. For these five level II classifications, we can see that the distance between the same class is small and the distance between different classes is large. The reduced spaces can be separated.

Figure 2  Part of images from the dataset of OPT-Aircraft _v1.0

Combining Figure 2 and 3, we can see that each of the 11 level II classifications has its own characteristics. This method of classification can cover different aircraft remote sensing data. More importantly, we can distinguish 11 level II classification further based on the color, engine position and so on. According to the rule of classification in 3.2, we use the remote sensing image of the aircraft in TGRS-HRRSD-Dataset^[12] to verify. It can be found that we can classify the aircraft of TGRS-HRRSD-Dataset into 11 level II classification accurately. Finally, the dataset is stored in .png format with a data size of 69.3 MB.

5 Discussion and Conclusion

In remote sensing data processing, the fine-grained recognition is still in the early stage. The shortage of aircraft classification data has difficulties in researching the scientific issues. This paper solves the above problems. First, the dataset of OPT-Aircraft _v1.0 is screened aircraft images from public datasets and high-resolution remote sensing images in Google Earth. Then, we classify the aircraft images according to the classification criteria. This classification criteria are according to the wing shape, engines, airframe??s color and so on. Finally, we divide the remote sensing images of aircraft into 7 level I classifications and 11 level II classifications. It covers aircraft with different characteristics in the field of remote sensing and complete a data division criterion.

By using the t-SNE, we can see the separability of the 11 level II classifications in low dimensional space. This is helpful to improve the accuracy of fine-grained recognition and classification of aircraft images. What??s more, the dataset of OPT-Aircraft _v1.0 is marked with different labels. In the simulation of remote sensing targets, the generative precision of different types of aircraft can be improved by using the generative adversarial networks (GAN).

In order to promote the accuracy of simulation and recognition in remote sensing scenes, the next task plan to complete the realistic simulation of remote sensing aircraft and fine-grained recognition based on OPT aircraft_ V1.0. This is very beneficial to accurately predict the risk and cost in the stage of load development and identifying specific targets. Of course, the dataset of OPT-Aircraft _v1.0 can also be improved. In the following stage, we can continue to collect and process more aircraft images according to the classification criteria in this paper. At the same time, we use the network of GAN to generate the required data to supplement the dataset of OPT-Aircraft _v1.0 to promote the precision of fine-grained recognition, which is a process of mutual promotion between them.

Author Contributions

Li, H. W. designed the algorithms of dataset. Chen, J. Y. and Chen, T. Q. contributed to the data processing and analysis. Li, H. W. and Chen, J. Y. designed the model and algorithm. Zhang, G., Wang, S. and Chen, J. Y. verified the data. Chen, J. Y. wrote the data paper.

References                                                       

[1]     Maji, S., Rahtu, E., Kannala, J., et al. Fine-grained visual classification of aircraft [OL]. arXiv, 2013. http://arxiv.org/abs/1306.5151.

[2]     Chen, J. Y., Li, H. W., Zhang, G., et al. Dataset of identifying aircraft groups by remote sensing images [J/DB/OL]. Digital Journal of Global Change Data Repository, 2020. DOI: 10.3974/geodb.2020.03.25.V1.

[3]     GCdataPR Editorial Office. GCdataPR Data Sharing Policy [OL]. DOI: 10.3974/dp.policy.2014.05 (Updated 2017).

[4]     Li, L., Xin, X. Z., Zhang, H. L., et al. A method for estimating hourly photosynthetically active radiation (PAR) in China by combining geostationary and polar-orbiting satellite data [J]. Remote Sensing of Environment, 2015, 165: 14?C26. DOI: 10.1016/j.rse.2015.03.034.

[5]     Zhu, H., Chen, X., Dai, W., et al. Orientation robust object detection in aerial images using deep convolutional neural network [C]. IEEE International Conference on Image Processing (ICIP), 2015, DOI: 10.1109/ICIP.2015.7351502.

[6]     Cheng, G., Han, J., Zhou, P., et al. Multi-class geospatial object detection and geographic image classification based on collection of part detectors [J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2014, 98: 119?C132. DOI: 10.1016/j.isprsjprs.2014.10.002.

[7]     Cheng, G., Han, J. A survey on object detection in optical remote sensing images [J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2016, 117: 11?C28. DOI: 10.1016/j.isprsjprs.2016.03.014.

[8]     Cheng, G., Zhou, P., Han, J. Learning rotation-invariant convolutional neural networks for object detection in vhr optical remote sensing images [J]. IEEE Transactions on Geoscience and Remote Sensing, 2016, 54(12): 7405?C7415. DOI: 10.1109/TGRS.2016.2601622.

[9]     Xia, G. S., Bai, X., Ding, J., et al. DOTA: a large-scale dataset for object detection in aerial images [OL]. arXiv, 2019. http://arxiv.org/abs/1711.10398.

[10]   Ding, J., Xue, N., Long, Y., et al. Learning RoI transformer for detecting oriented objects in aerial images [OL]. arXiv, 2018. http://arxiv.org/abs/1812.00155.

[11]   Der Maaten, L. V., Hinton, G. E. Visualizing data using T-SNE [J]. Journal of Machine Learning Research, 2008, 9: 2579?C2625.

[12]   Zhang, Y., Yuan, Y., Feng, Y., et al. Hierarchical and robust convolutional neural network for very high-resolution remote sensing object detection [J]. IEEE Transactions on Geoscience and Remote Sensing, 2019, 57(8): 5535?C5548. DOI: 10.1109/TGRS.2019.2900302.