Improving Document Similarity Calculation Using Cosine-Similarity Graphs

Abstract

Data mining information using various indices and determining candidates that can be judged as having the same tendency based on similarity between documents is common. The accuracy of similarity largely depends on a sufficient amount of data and requires advanced analysis using natural language processing. In this paper, we present an approach for filtering based on the cosine similarity graph and clustering between candidates. For filtering candidates, we focus on the inflection point of the graphs when plotting by sorting similarities in descending order. By clustering among higher similarities, we aim at filtering candidates that cannot be removed without analyzing advanced natural language processing. The proposed method was applied to movie reviews and sightseeing location reviews written in Japanese. Although this study is a work in progress, it shows that candidates can be recommended without having to manually apply natural language processing, such as preparing stopwords for each category.