An Anomaly Detection Framework for Twitter Data

Title

An Anomaly Detection Framework for Twitter Data

Creator

Kumar S.; Khan M.B.; Hasanat M.H.A.; Saudagar A.K.J.; AlTameem A.; AlKhathami M.

Description

An anomaly indicates something unusual, related to detecting a sudden behavior change, and is also helpful in detecting irregular and malicious behavior. Anomaly detection identifies unusual events, suspicious objects, or observations that differ significantly from normal behavior or patterns. Discrepancies in data can be observed in different ways, such as outliers, standard deviation, and noise. Anomaly detection helps us understand the emergence of specific diseases based on health-related tweets. This paper aims to analyze tweets to detect the unusual emergence of healthcare-related tweets, especially pre-COVID-19 and during COVID-19. After pre-processing, this work collected more than 44 thousand tweets and performed topic modeling. Non-negative matrix factorization (NMF) and latent Dirichlet allocation (LDA) were deployed for topic modeling, and a query set was designed based on resultant topics. This query set was used for anomaly detection using a sentence transformer. K-means was also employed for clustering outlier tweets from the cleaned tweets based on similarity. Finally, an unusual cluster was selected to identify pandemic-like healthcare emergencies. Experimental results show that the proposed framework can detect a sudden rise of unusual tweets unrelated to regular tweets. The new framework was employed in two case studies for anomaly detection and performed with 78.57% and 70.19% accuracy. 2022 by the authors.

Source

Applied Sciences (Switzerland), Vol-12, No. 21

Date

2022-01-01

Publisher

MDPI

Subject

anomaly detection; outlier detection; sentiment analysis; social media analytics; topic modeling

Coverage

Kumar S., Department of Computer Science and Engineering, CHRIST (Deemed to be University), Bangalore, 560074, India; Khan M.B., Information Systems Department, College of Computer and Information Sciences, Imam Mohammad Ibn Saud Islamic University (IMSIU), Riyadh, 11432, Saudi Arabia; Hasanat M.H.A., Information Systems Department, College of Computer and Information Sciences, Imam Mohammad Ibn Saud Islamic University (IMSIU), Riyadh, 11432, Saudi Arabia; Saudagar A.K.J., Information Systems Department, College of Computer and Information Sciences, Imam Mohammad Ibn Saud Islamic University (IMSIU), Riyadh, 11432, Saudi Arabia; AlTameem A., Information Systems Department, College of Computer and Information Sciences, Imam Mohammad Ibn Saud Islamic University (IMSIU), Riyadh, 11432, Saudi Arabia; AlKhathami M., Information Systems Department, College of Computer and Information Sciences, Imam Mohammad Ibn Saud Islamic University (IMSIU), Riyadh, 11432, Saudi Arabia

Rights

All Open Access; Gold Open Access

Relation

ISSN: 20763417

Format

Online

Language

English

Type

Article

Identifier

https://doi.org/10.3390/app122111059

https://www.scopus.com/inward/record.uri?eid=2-s2.0-85141860286&doi=10.3390%2fapp122111059&partnerID=40&md5=4c87807537ebd56110bb7cf8419644f1