Leveraging Large Language Models for Trend Detection in Arabic Text
| Author | Al-Hassabi, Fatema Hasan | |
|---|---|---|
| metadata.dc.contributor.department | University of Bahrain, College of Information Technology, Information systems | |
| metadata.dc.date.accessioned | 2026-09-20T15:51:21Z | |
| Publication date | 2025 | |
| Degree level | Master | |
| Abstract | Abstract : The Tawasul platform serves as Bahrain's centralized mechanism for residents to submit complaints and inquiries to government entities. This research focuses on analyzing approximately 45,000 complaints related to the Ministry of Education over five years. The dataset presents significant challenges, including linguistic diversity: primarily Arabic but also English, making it bilingual in nature. However, the performance of LLMs in this research is specifically evaluated for Arabic, as it is the dominant language in the dataset, while ensuring the models maintain the ability to handle both languages. Additionally, as a government dataset, Tawasul's data is highly sensitive and restricted from public access, creating challenges in utilizing public cloud-based LLMs and infrastructures. A key objective of this research is to explore privacy-preserving approaches by running LLMs on local infrastructure, ensuring compliance with data protection policies while maintaining model efficiency. To address these challenges, we evaluated two complaint analysis methods: (1) LLAMA 2 (7B) was used for feature extraction, followed by Long Short-Term Memory (LSTM) classification, achieving an accuracy of 0.496. (2) LLAMA 3.1 (8B) with Retrieval-Augmented Generation (RAG) was employed for direct classification, significantly improving performance, with an accuracy of 0.78. An iterative refinement of LLM prompts was implemented to extract high-quality summaries, keywords, and complaint categories. Additionally, entity recognition techniques were applied to identify school names, and advanced visualizations were developed to gain insights into complaint trends. Our findings reveal that Method (2), LLAMA 3.1 with RAG, significantly outperformed Method (1), achieving higher accuracy, precision, and efficiency in classifying complaints and extracting relevant insights. The integration of RAG enhanced contextual alignment, reduc-ing reliance on inconsistent human labels. Moreover, prompt engineering techniques played a crucial role in minimizing hallucinations and improving overall output quality. This result compares to human evaluation accuracy of 0.72, showing that the model performs relatively well, especially considering that human classification often includes errors. This research makes several key contributions: (1) Demonstrating the effectiveness of LLMs in handling complex, multilingual complaint data, particularly for Arabic, using LLAMA 3.1 with RAG, achieving high accuracy (0.91) and precision. (2) Developing refined prompt engineering techniques that reduce hallucinations and enhance the reliability of generated outputs. (3) Employing a hybrid entity recognition approach, leveraging LLM-based ex- traction and traditional matching methods to ensure accuracy. (4) Integrating advanced visualizations to provide actionable insights, enabling government stakeholders to identify trends, allocate resources, and improve public service delivery. By showcasing the potential of LLMs in government applications, this study highlights their role in enhancing citizen satisfaction through more responsive and insightful complaint management systems, while addressing privacy concerns through localized model deployment. | |
| metadata.dc.identifier.other | Drupal node 4620 | |
| metadata.dc.identifier.other | https://digitalrepository.uob.edu.bh/en/dar/leveraging-large-language-models-trend-detection-arabic-text | |
| metadata.dc.identifier.uri | http://165.227.56.122/handle/123456789/7138 | |
| Language | en | |
| Link | https://digitalrepository.uob.edu.bh/en/_flysystem/fedora/2026-02/002477_leveraging_large_language_models_for_trend_detection_in_arabic_text.pdf | |
| Title | Leveraging Large Language Models for Trend Detection in Arabic Text | |
| Type | Master's Thesis |
Files
Original bundle
1 - 1 of 1
Loading...
- Name:
- node-4620-1-002477_leveraging_large_language_models_for_trend_detection_in_arabic_text.pdf
- Size:
- 5.45 MB
- Format:
- Adobe Portable Document Format