“格网索引+MapReduce”策略下的地理国情统计分析研究

地理国情统计分析是深度研究地理国情普查数据的首要前提.针对现有单机集中式数据存储与处理方式存在耗时长、效率低甚至不支持的问题,设计了"格网索引+MapReduce"策略,基于规则格网设计普查数据文件的分块组织与分布式存储方式,研制了格网索引与空间分析相结合的双层过滤机制,构建基于MapReduce的地理国情并行统计算法.最后,与无索引MapReduce、ArcGIS平台进行性能对比测试,结果表明:"格网索引+MapReduce"方法的统计效率远高于ArcGIS平台,对无索引MapReduce方法亦有明显的效率优势,研究拟为地理国情普查数据的高性能、多类型、大批量统计分析提供优选方案.

Research on the analysis and statistic of geographical conditions based on the strategy of “Grid Index + MapReduce”

The statistic of geographical conditions is the primary premise for the deep excavation and application of geographical data. However, the traditional centralized data storage and processing method based on a single computer are time-consuming, inefficient and even unsupported. This paper creates a strategy called "Grid Index + MapReduce" to solve these problems. Firstly, we design a blocking file organization and distributed storage mode of the census data of geographical situation based on the regular square grid, and then make a double layer filtering method which combines the grid index and the accurate analysis. Lastly, we build a parallel processing algorithm of statistic of the geography conditions based on MapReduce. The test results of performance comparison of the strategy of "Grid Index + MapReduce", the indexless MapReduce and ArcGIS software show that the method of "Grid Index + MapReduce" is much more efficient than the ArcGIS software, and also has obvious efficiency advantages for the indexless MapReduce method. The study tries to provide an optimal scheme for the high-performance, multi-type and high-volume statistic and analysis method for the data of geographical condition survey.