An NLP and clustering approach for information retrieval and sentiment analysis in textual documents

Abstract

In recent years, information has been overloaded and widely available because of the rapid growth of the internet, and that is not different with scientific papers. An NLP and clustering approach can be used to deal with an increased amount of documents for information retrieval, creating the most relevant clusters for a user and answering questions about the specific papers being analyzed. Information retrieval needs to scan all documents found in a database, give scores according to relevance degree to the user, then rank all results and present them to the user. Thus, information retrieval requires a long runtime to scan all documents. The cluster analysis tool plays the primary role in information retrieval to improve its performance by reducing the search time and preventing results from being irrelevant. In this paper, a K-means clustering approach is proposed using a TF-IDF sentence embedding, and it is also proposed a Question Answering model, fine-tuned with a dataset composed of the abstracts of scientific papers, using an SBERT sentence embedding, to answer questions to researchers and help them retrieve relevant information in a more efficient manner. Additionally, this work includes a practical case study applying FinBERT, to analyze earnings call transcripts. Through this analysis, the study explores the effectiveness and limitations of textual sentiment analysis for predicting short-term stock market reactions, providing important insights into the complexities of financial communication and market behavior.