基于HDFS的云存储服务系统的实现

本文通过实现大数据场景下基于HDFS的云存储服务系统,以缓解甚至解决目前海量数据快速增长造成的现有存储容量和技术：网络带宽能力等面临的挑战。     

探讨网络备份中重复数据删除技术研究

随着计算机技术和网络技术的不断发展,带动了社会经济水平的高速提高,信息化技术给我们的生活、学习和工作带来了极大便利,但是在我们使用信息化技术的同时也会产生大量数据,储存和备份这些数据可能需要千万亿字节的存储空间,可以说我们正生活在信息和数据爆炸的时代,但是研究表明,在信息处理和存储的各个环节中,会有大量的重复数据产生,例如邮件附件、计算机资源管理器、应用软件和操作系统等。诸如数据备份、文件快照等传统数据备份形式,也会产生大量的重复数据,占据了大量的网络带宽,浪费了宝贵的存储空间。本文将就网络备份中重复数据删除技术展开探讨。     

基于HDFS的网络教学资源归并云存储模型研究

随着云计算时代的到来,远程教育作为现代教育技术的重要手段,在教育教学中起着越来越重要的作用.但是网络教学资源的重复建设、缺乏共享等问题遏制了其发展,所以需要一种高效的存储技术对资源实行统一管理.本文分析了基于HDFS的云存储模型,并在此基础上搭建了一个基于云存储技术的网络教学资源整合模型,期望能够克服现行存储方式存在的不足,解决海量教学数据在存储当中存在的问题,并且能够更好地提高存储效率.     

基于HDFS的分布式存储策略分析

本文重点围绕HDFS机架感知和副本存放策略方面对HDFS分布式存储进行剖析.副本存放策略和机架感知主要通过Datanode节点形成的树状网络拓扑图来让Namenode节点获取,从而确定副本存放的位置,这种方式保证了对于数据的极高的容错性的同时也兼顾了数据本地化,即提高了数据在集群网络中的传输效率.基于此,提出一个设想,希望通过对副本存放策略的深入挖掘,根据Datanode数据节点的实时状态信息,实现对于数据块副本的定向存储,再由数据驱动任务分配,来为每一个Datanode数据节点分配更适合的任务,从而达到负载均衡提高资源利用率的作用.     

大数据场景下基于HDFS的云存储服务系统设计

本文通过构建大数据场景下基于HDFS的云存储服务系统,能够有效地存储海量数据,实现用户的文件数据进行分布式存储,并达到降低分布式文件系统成本的目的。     

通信运营商大数据量系统数据存储云化浅谈

基于上海电信实现无线网管系统数据存储云化的实践经验,简要介绍了该类系统当前特征和问题,详细论述了数据存储云化框架和动态扩容、map/reduce数据处理、重复数据删除、差分增量备份等关键技术。     

HDFS一致性管理的研究

HDFS是开源云计算项目Hadoop的存储系统,具有良好的容错性和扩展性,但是HDFS原有的一致性模型不能适用于实时性要求高,数据完整性要求不严格的任务。为了让HDFS能够满足云存储不同应用场景,在已有HDFS的总体架构和读写流程基础上,从数据一致性的角度剖析HDFS的局限和不足,提出解决方案,把NWR模型引入HDFS当中,并且将文件系统的读写锁机制改进为多粒度的锁机制。实验结果证明：通过以上措施,使用户可以根据自身需要配置云存储系统,平衡一致性和可用性。     

浅谈数据的存储备份

简要介绍了目前最新的数据存储技术和备份技术及备份策略,以及现流行的备份软件及其主要功能特点。     

基于HDFS的云存储在高校信息资源整合中的应用

研究如何将基于HDFS的云存储应用于整合高校信息资源,旨在通过合适的构建基于HDFS的云存储服务系统,解决高校的海量数据存储问题。通过对概念的解析,结合高校信息资源存储的特点,构建了一个低成本、高效率、高安全性的分布式信息资源整合解决方案,并给出了系统结构图。     

New method for file deduplication in cloud storage

Deduplication is widely used in cloud storage service to save bandwidth and storage resources,however,the security of client deduplication still flaws in an external attack to access a user’s private data.Xu-CDE,a deduplication solution of encrypting data for multi-client was first proposed,which could protect the privacy of data from the external attackers and honest but curious server,with favorable theoretical meaning and representativeness.However,in Xu-CDE,the user ownership authentication credentials were lack of instantaneity protection,which could not resist replay attack.As an improvement to the flaw,the protocol MRN-CDE (MLE based and random number modified client-side deduplication of encrypted data in cloud storage) was proposed,adding random number in order to ensure the instantaneity of the authentication credentials,and using the algorithm of MLE-K_Pto extract key from original file to replace the file itself as an encryption key.As a consequence,the new protocol improved security while significantly reduced the amount of computation.After the safety analysis and the actual tests,results show that based on Xu-CDE,the proposed protocol MRN-CDE has stronger security of ownership,and improves time efficiency.Specially,the new protocol works better on large files in cloud with a certain value.     

Key-exposure resilient integrity auditing scheme with encrypted data deduplication

For the problems of key-exposure,encrypted data duplication and integrity auditing in cloud data storage,a public auditing scheme was proposed to support key update and encrypted data deduplication.Utilizing Bloom filters,the proposed scheme could achieve client-side deduplication,and guaranteed that the key exposure in one time period didn’t effect the users’ private key in other time periods.The proposed scheme could solve the conflict between key-exposure resilient and encrypted data deduplication in public auditing scheme for the first time.Security analysis indicates that the proposed scheme is strong key-exposure resilient,confidentiality,detectability,and unforgeability of authentication tags and tokens under the computation Diffie-Hellman hardness assumption in the random oracle model.     

Data access control policy of encrypted deduplication system

To solve the problem that convergent encryption was commonly used in existing encrypted deduplication systems in cloud storage and data owner couldn’t effectively enforce access control on their outsourced data,an encrypted deduplication system was proposed to support access control functions such as identity authentication,authorization deduplication and the update of access control policy.The outsourced data was only deduplicated with the authorized users,and the unauthorized users couldn’t obtain any data information.CP-ABE and the partition of the ElGamal private key were used to update the access control policy of data.Self-control objects was used to encapsulate user’s data and its access policy,providing authentication for data visitors and ensuring the access control policies enforced effectively.Security analysis and simulation results demonstrate that the proposed system enables data access control and executes efficiently.     

Stochastic algorithm for HDFS data theft detection based on MapReduce

To address the problems of big data efficient analysis and insider theft detection in the data theft detection of distributed cloud computing storage,taking HDFS (hadoop distributed file system) as a case study,a stochastic algorithm for HDFS data theft detection based on MapReduce was proposed.By analyzing the MAC timestamp features of HDFS generated by folder replication,the replication behavior’s detection and measurement method was established to detect all data theft modes including insider theft.The data set which is suitable for MapReduce task partition and maintains the HDFS hierarchy was designed to achieve efficient analysis of large-volume timestamps.The experimental results show that the missed rate and the number of mislabeled folders could be kept at a low level by adopting segment detection strategy.The algorithm was proved to be efficient and had good scalability under the MapReduce framework.     

PoSW:novel secure deduplication scheme for collaborative cloud applications

In order to solve the problems of secure access and deduplication to the shared file in the cloud environment,a novel notion called proof of shared ownership (PoSW) was formalized,and a formal definition of the PoSW was given.Furthermore,a PoSW scheme and an enhanced version of that were proposed.In the PoSW scheme,secure shared file dispersal,convergent encryption and secret sharing algorithm were employed to transform the shared file realize the sharing and authorization for the shared file,and then a novel challenge-response protocol was proposed to achieve the proof of shared ownership and the foundation for the secure deduplication of the shared file was provided.An enhanced PoSW scheme was designed to improve the availability and reliability for different kinds of the shared files by introducing the multi-cloud server providers and using the strategies of both data duplication and secret file dispersal.Security analysis and performance evaluation show the security and efficiency of the proposed scheme.     

基于确定/概率性文件拥有证明的机密数据安全去重方案

为解决云存储系统中机密数据去重面临的密文重复性检测与拥有性证明、针对数据机密性的攻击等难题,提出了基于Merkle散列树的MHT-Dedup方案和基于同态MAC的hMAC-Dedup方案。两者均通过对密文文件的拥有证明进行跨用户文件级重复性检测,并通过检查数据块明文的摘要进行本地数据块级重复性检测,避免了跨用户文件级客户端重复性检测中hash-as-a-proof方法存在的安全缺陷。MHT-Dedup方案通过数据块密文的标签生成的验证二叉树提供确定性的文件拥有证明,具有较低的计算和传输开销,而hMAC-Dedup方案则通过对抽样数据块密文和其标签进行同态MAC运算提供概率性的文件拥有证明,具有较低的额外存储开销。分析与比较表明,本方案在同时支持两级客户端机密数据安全去重和抵抗对数据块的暴力搜索攻击方面具有明显优势。     

基于云计算的作战数据存储系统研究

为了解决部队战斗时产生的海量数据的存储问题,做了一种基于云计算的作战数据存储系统的实验。通过实际的应用,该方案可以很好地克服现行的存储方式存在的不足,有效提高部队战斗效率。     

基于压缩近邻的查重元数据去冗算法设计

随着重复数据删除次数的增加,系统中用于存储指纹索引的清单文件等元数据信息会不断累积,导致不可忽视的存储资源开销。因此,如何在不影响重复数据删除率的基础上,对重复数据删除过程中产生的元数据信息进行压缩,从而减小查重索引,是进一步提高重复数据删除效率和存储资源利用率的重要因素。针对查重元数据中存在大量冗余数据,提出了一种基于压缩近邻的查重元数据去冗算法Dedup2。该算法先利用聚类算法将查重元数据分为若干类,然后利用压缩近邻算法消除查重元数据中相似度较高的数据以获得查重子集,并在该查重子集上利用文件相似性对数据对象进行重复数据删除操作。实验结果表明,Dedup2可以在保持近似的重复数据删除比的基础上,将查重索引大小压缩50%以上。