基于词向量和EMD距离的短文本聚类

短文本聚类在数据挖掘中发挥着重要的作用,传统的短文本聚类模型存在维度高、数据稀疏和缺乏语义信息等问题,针对互联网短文本特征稀疏、语义存在奇异性和动态性而导致的短文本聚类性能较差的问题,提出了一种基于特征词向量的文本表示和基于特征词移动距离的短文本聚类算法。首先使用Skip-gram模型(Continuous Skip-gram Model)在大规模语料中训练得到表示特征词语义的词向量;然后利用欧式距离计算特征词相似度,引入EMD(Earth Movers Distance)来计算短文本间的相似度;最后将其应用到Kmeans聚类算法中实现短文本聚类。在3个数据集上进行的评测结果表明,效果优于传统的聚类算法。

Short text clustering based on word embeddings and EMD

Short text clustering plays an important role in data mining. The traditional short text clustering model has some problems, such as high dimensionality、sparse data and lack of semantic information. To overcome the shortcomings of short text clustering caused by sparse features、semantic ambiguity、dynamics and other reasons, this paper presents a feature based on the word embeddings representation of text and short text clustering algorithm based on the moving distance of the characteristic words. Initially, the word embeddings that represents semantics of the feature word was gained through training in large-scale corpus with the Continous Skip-gram Model. Furthermore, use the Euclidean distance calculation feature word similarity. Additionally, EMD(Earth Movers Distance)was used to calculate the similarity between the short text. Finally, apply the similarity between the short text to Kmeans clustering algorithm implemented in the short text clustering. The evaluation results on three data sets show that the effect of this method is superior to traditional clustering algorithms.