Accuracy Assessment of GlobeLand30 (2010)
over China with a Landscape Shape Index-based
Sampling Approach

Wang, Y.^1*  Zhang, J. X.²  Liu, D.²  Yang, W. J.¹  Zhang, W. L.¹

1. School of Remote Sensing and Information Engineering, Wuhan University, Wuhan 430079, China;

2. School of Geodesy and Geomatics, Wuhan University, Wuhan 430079, China

Abstract: Large-scale land cover data is vital for the study of earth science, particularly in the fields of earth surface processes research, ecosystem assessment, and environmental modeling. The assessment of existing land cover datasets is instructive for dataset use and new dataset production. The GlobeLand30 (2010) dataset over China was evaluated in this study. To obtain more appropriate sample sizes for each class, especially sufficient sample numbers for rare classes, 1,000 validation sample units based on a landscape shape index (LSI) were selected, and the classification accuracy of the GlobeLand30 (2010) dataset was evaluated using Google Earth high-resolution satellite images. The overall accuracy of the GlobeLand30 (2010) dataset over China was 77.90%, which indicated that the consistency between the GlobeLand30 (2010) dataset and the true land cover was high, and thus it could accurately reflect the distribution of different land cover types in China.

Keywords: GlobeLand30 (2010); accuracy assessment; sampling approach

1 Introduction

Large-scale land cover datasets are widely used in earth surface processes research, ecosystem assessments, environmental modeling, and sustainable development planning. GlobeLand30 (2010) was developed by the National Geomatics Center of China during 2009-2014 and is the first global land cover product at 30 m resolution. The product has been available worldwide since September 2014^[1]. The database categorizes land use into ten classes (water bodies, wetland, permanent snow and ice, artificial surfaces, cultivated land, bare land, forest, shrub land, grassland, and tundra) in accordance with a hierarchical classification method based on pixel classification, object optimization, and proof of knowledge^[2].

To verify the accuracy of this product, a preliminary accuracy assessment was conducted by third-party experts based on a two-rank sampling strategy. During the first-rank sampling, map sheet samples were selected from global map sheets, while the second-rank sampling selected feature samples of each land cover type within each of the selected map sheets^[3]. Eighty map sheet samples were selected from a total of 847 map sheets in the first-rank sampling, and these samples were spatially distributed into five continents based on the proportional area of each land class. A total of 159,874 pixel samples were selected for the assessment of GlobeLand30 in the baseline year of 2010, and the overall accuracy of GlobeLand30 (2010) data was 80.33% ± 0.2%^[4].

There have only been a few studies of GlobeLand30 (2010) product validation in China. To compare and assess large-scale land cover datasets in China, Yang et al. (2017) resized the GlobeLand30 (2010) product to a scale of 300 m, and manually selected 1,063 sample points from homogenous regions. It was found that GlobeLand30 (2010) was the most accurate of the datasets investigated, with an overall accuracy of 82.39% in China^[5]. Lu et al. (2016) compared the accuracy of five global cultivated land datasets for the base-year of 2010. They found that the accuracy of cultivated land of GlobeLand30 (2010) was higher than for the other four products, with an overall accuracy of 79.61%^[6]. Obviously, different sampling designs lead to different verification results. The aim of this study was to assess the GlobeLand30 (2010) dataset over China with a landscape shape index (LSI)-based sampling approach, to obtain a creditable accuracy assessment result, especially for the easily misclassified heterogeneous regions.

2 Metadata of Dataset

The validation dataset has both a full name and a short name^[7]. It contains details of the corresponding author, authors, geographical regions covered by the content, year that data were acquired, the spatial and temporal resolutions of data, the number of dataset tiles, data format and size, data files, the foundation that owns the data, data publisher, data access and services platform, and data sharing policy. Table 1 summarizes the main metadata elements of the dataset.

3 Methods

Different sampling designs lead to different verification results. To take account of the spatial heterogeneity of the land cover, this study adopted an LSI-based sampling approach and used Google Earth high-resolution historical images to evaluate the accuracy of the GlobeLand30 (2010) dataset over China.

3.1 Sampling Design

Accuracy assessment is a key step in land cover mapping and is implemented by selecting samples for which reference data will be collected and used to estimate the overall and class-specific accuracies in the target region. Many sampling approaches (cluster sampling, stratified random sampling, and systematic sampling) have been developed to determine the sample size and spatial distribution according to several fundamental criteria, such as probability, cost effectiveness, and spatial balance. However, most traditional methods ignore the spatial heterogeneity of land cover in large areas and may not obtain credible spatial samplings, which include sample size and sample distribution.

Table 1  Metadata summary of validation of globeLand30 in China with a LSI-based sampling approach

To solve this problem, Chen et al. (2016) proposed an LSI-based sampling approach^[9]. This method needs to calculate LSI to characterize the spatial heterogeneity of land cover at three-levels. At the first level, a regional LSI can be calculated in the target regions and used to estimate the sample size per region. Second, a class-level LSI must be derived for allocating the sample size into classes. Finally, because the samples allocated into each class should be distributed into geographical units according to their spatial heterogeneities, the LSI per class must be calculated in each geographical unit. It has been proven that the accuracy of this method is closer to the true accuracy, with the accuracy assessment of rare classes being significantly improved^[9].

Table 2 shows that the distribution of land cover in China is not uniform and the spatial heterogeneity of land cover can be easily observed from the GlobeLand30 map. The areas of cultivated land, forest, grassland, and bare land are similar and together account for 94.07% of the Chinese land area. The area occupied by the remaining categories of land cover is very small, accounting for about 6% of the Chinese land area. If the traditional sampling approaches were to be adopted, the proportion of sampling points in the rare classes would be very low, and the sample sizes of rare classes allocated from the limited total sample size would be relatively small, which would lead to an unreliable accuracy assessment. The LSI-based sampling approach was therefore adopted in this study. The sample sizes and distribution of the validation points are shown in Figure 1 and Table 2.

As shown in Table 2, an LSI-based sampling approach considers the distribution of rare classes in China (shrub land, wetland, water bodies, artificial surfaces and permanent snow and ice) and obtains sufficient sample numbers for rare classes and optimal sample distributions in geographical space. Thus, a more precise accuracy is achieved for rare classes using the LSI approach. It should be noted that the tundra area in China is very small, and due to its limited distribution it was not included in the validation.

3.2 Sampling Evaluation

The land cover types of the sample point or polygon were evaluated by validation experts. The evaluation of the properties of the validation samples were mainly based on Google Earth high-resolution images. Images of the 2010 growing season were preferred. The basic approach taken in the evaluation was as follows: the collection of multi-source reference data, the collection of interpretation elements (shape, size, pattern, shadow, tone, texture, site, temporal serial, spatial serial, and altitude), and sample integration. The sample assessment unit

 (SAU) for a GlobeLand30 validation sample was the region corresponding to a 30×30 m pixel. For implementation, the SAU is a square polygon in Google Earth format (.kmz) and can be displayed over Google Earth images for interpretation. Considering that there is no dominant land cover type in the SAU or the SAU can sometimes be isolated, a polygon of 300 m×300 m was also used to assist the interpretation (Figure 2).

4 Results

Accuracy assessment is one of the key steps in land cover mapping and is implemented by selecting samples for which reference data is collected. From the classification results and “real results”, an error matrix was constructed, and thus accuracy indicators, such as overall accuracy, user’s accuracy, producer’s accuracy, and the Kappa

coefficient, could be calculated^[10]. The error matrix, also called a confusion matrix, is a comparison array for representing the number of cells classified as a certain category and the number actually verified as being within that category.

In general, the columns in the array represent reference data; the rows represent the category data classified by a remote sensing approach. There are two forms of error matrix: one uses the number of pixels, while the other uses the percentage. Error matrix results for 1,000 GlobeLand30 (2010) sampling points in China are shown in Table 3.

Table 3  The error matrix of GlobeLand30 (2010) sampling results

The error matrix is the basis for evaluating the accuracy of a thematic map. It is often used to calculate the following four accuracy indicators.

(1) Overall accuracy: a comprehensive description of the entire land cover product. The overall accuracy can be expressed as the ratio of the sum of the pixels correctly classified to the total sample size.

(2) User’s accuracy: this refers to the ratio of the number of pixels in the actual type of ground surface to the number of pixels in the map classification result. Therefore, it is the row total in the error matrix divided by the number of correctly classified pixels.

(3) Producer’s accuracy: this is the ratio of the number of pixels of a certain category correctclassified pixels in the error matrix.