一种高维大数据全k近邻查询算法

全k近邻(all k-nearest neighbor,AkNN)查询,是k近邻查询的一个变型,旨在在一个查询过程中为给定数据集的每个对象确定k个最近邻.提出了一种在Hadoop分布式平台下处理高维大数据的AkNN查询算法.首先使用行条化思想结合p-stable LSH算法将高维数据对象降维,然后结合空间填充曲线Z-order的优良特性,把降维后的数据嵌入一维空间中,接着进行范围查询.整个过程使用MapReduce框架分布式并行处理.实验结果表明,所提出的算法可以高效处理高维大数据的AkNN查询.     

基于动态域划分的 MapReduce 安全冗余调度策略简

MapReduce现有调度策略无法实现云环境中多租户作业的安全隔离。提出一种基于动态域划分的安全冗余调度策略：通过引入冲突关系、信任度、安全标签等概念,建立一种动态域划分模型,以将待调度节点划分为与不同租户作业关联的冲突域、可信域或调度域;结合冗余方式,将租户作业同时调度到其可信域节点和调度域节点(但不允许为其冲突域节点),通过二者执行环境和部分计算结果的一致性验证决定是否重新调度。实验分析了其有效性和安全性。     

MapReduce optimization algorithm based on machine learning in heterogeneous cloud environment

We present an approach to optimize the MapReduce architecture, which could make heterogeneous cloud environment more stable and efficient. Fundamentally different from previous methods, our approach introduces the machine learning technique into MapReduce framework, and dynamically improve MapReduce algorithm according to the statistics result of machine learning. There are three main aspects： learning machine performance, reduce task assignment algorithm based on learning result, and speculative execution optimization mechanism. Furthermore, there are two important features in our approach. First, the MapReduce framework can obtain nodes＇ performance values in the cluster through machine learning module. And machine learning module will daily calibrate nodes＇ performance values to make an accurate assessment of cluster performance. Second, with the optimization of tasks assignment algorithm, we can maximize the performance of heterogeneous clusters. According to our evaluation result, the cluster performance could have 19% improvement in current heterogeneous cloud environment, and the stability of cluster has greatly enhanced.     

Hadoop Performance Prediction Model Based on Random Forest

Map Reduce is a programming model for processing large data sets,and Hadoop is the most popular open-source implementation of MapReduce.To achieve high performance,up to 190 Hadoop configuration parameters must be manually tunned.This is not only time-consuming but also error-pron.In this paper,we propose a new performance model based on random forest,a recently developed machine-learning algorithm.The model,called RFMS,is used to predict the performance of a Hadoop system according to the system’s configuration parameters.RFMS is created from 2000 distinct fine-grained performance observations with different Hadoop configurations.We test RFMS against the measured performance of representative workloads from the Hadoop Micro-benchmark suite.The results show that the prediction accuracy of RFMS achieves 95% on average and up to 99%.This new,highly accurate prediction model can be used to automatically optimize the performance of Hadoop systems.     

基于Jaccard的移动终端自动识别并行算法及其MapReduce实现

The ability of accurate and scalable mobile device recognition is critically important for mobile network operators and ISPs to understand their customers’ behaviours and enhance their user experience. In this paper, we propose a novel method for mobile device model recognition by using statistical infor-mation derived from large amounts of mobile network traffic data. Specifically, we create a Jaccard-based coefficient measure method to identify a proper keyword representing each mobile device model from massive unstruc-tured textual HTTP access logs. To handle the large amount of traffic data generated from large mobile networks, this method is designed as a set of parallel algorithms, and is imple-mented through the MapReduce framework which is a distributed parallel programming model with proven low-cost and high-efficiency features. Evaluations using real data sets show that our method can accurately recognise mobile client models while meeting the scalability and pro-ducer-independency requirements of large mobile network operators. Results show that a 91.5% accuracy rate is achieved for rec-ognising mobile client models from 2 billion records, which is dramatically higher than existing solutions.     

Efficiently predicting large-scale protein-protein interactions using MapReduce

With a rapid development of high-throughput genomic technologies, a vast amount of protein-protein interactions (PPIs) data has been generated for difference species. However, such set of PPIs is rather small when compared with all possible PPIs. Hence, there is a necessity to specifically develop computational algorithms for large-scale PPI prediction. In response to this need, we propose a parallel algorithm, namely pVLASPD, to perform the prediction task in a distributed manner. In particular, pVLASPD was modified based on the VLASPD algorithm for the purpose of improving the efficiency of VLASPD while maintaining a comparable effectiveness. To do so, we first analyzed VLASPD step by step to identify the places that caused the bottlenecks of efficiency. After that, pVLASPD was developed by parallelizing those inefficient places with the framework of MapReduce. The extensive experimental results demonstrate the promising performance of pVLASPD when applied to prediction of large-scale PPIs.     

基于MapReduce的大规模流量分析系统的设计

随着用户和网络规模的快速扩大以及精细化运营需求的增加,网络流量分析系统面临的数据规模及分析深度要求都在快速发展,针对传统技术在系统扩展性、建设成本以及分析深度方面已经很难满足目前需求这一问题,提出了一种基于MapReduce 技术的大规模流量分析系统技术方案,对数据存储、数据分析全部并行化处理,消除传统系统存在的若干瓶颈。     

基于结构相似度的大规模社交网络聚类算法

针对社交网络的有向交互性和大规模特性,该文提出一种基于结构相似度的有向网络聚类算法(DirSCAN),以及相应的分布式并行算法(PDirSCAN)。考虑社交网络中节点间的有向交互性,将行为结构相似的节点聚集起来,并进行节点功能分析。针对社交网络规模巨大的特点,提出MapReduce框架下的分布式并行聚类算法,在确保聚类结果一致的前提下,提高处理性能。大量真实数据集上的实验结果表明,DirSCAN比无向网络聚类算法(SCAN)在F1上可提高2.34%的性能,并行算法PDirSCAN比DirSCAN运行速度提升1.67倍,能够有效处理大规模的有向网络聚类问题。     

基于MapReduce模型的并行科学计算

随着多核处理器日渐普及,开发高效易用的并行编程模型成为新的挑战,MapReduce是Google开发的一种并行分布式计算模型,在其搜索业务中获得了巨大的成功,将MapReduce模型引入科学计算领域,并结合实例阐述了如何使用面向高性能计算的HPMR/HPMR-s系统在分布式或共享存储系统中采用统一的方式描述并实现并行科学计算.