Evaluating Precision and Semantic Similarity in a Retrieval-Augmented Generation-Based Question Answering System for Shafi'i Prayer Fiqh

Authors

  • Thoriq Wijaya Universitas Malikussaleh
  • Defry Hamdhana Universitas Malikussaleh
  • Fajriana Universitas Malikussaleh
  • Mukti Qamal Universitas Malikussaleh

DOI:

https://doi.org/10.61255/decoding.v4i3.1840

Keywords:

Fiqh of prayer, Large language model, Question answering system, Retrieval-augmented generation, Semantic retrieval

Abstract

Purpose – This study evaluates retrieval precision and answer similarity in a closed-domain question-answering system for Shafi'i prayer fiqh using retrievalaugmented generation (RAG).
Methods – Indonesian translations of three classical fiqh texts were segmented into 251 chunks using a 1000-character chunk size and 200-character overlap. LangChain coordinated multilingual-e5-base embeddings, ChromaDB retrieval, and local Qwen2.5:3b generation. The retriever returned up to four eligible chunks. Twenty predefined queries were evaluated using Precision@4, cosine similarity to researcher-prepared reference answers, and qualitative assessment of required answer components.
Findings – Mean Precision@4 was 70.00%, and mean cosine similarity was 0.9197 (91.97%). Retrieval varied across queries, with Query 19 returning only one relevant chunk. Queries 8 and 19 produced incomplete answers despite high semantic similarity. The system returned a predefined fallback response when no retrieved passages passed the distance threshold.
Research Implications – The results support preliminary prototype feasibility but do not establish expert-verified correctness of the rulings. The small query set and absence of a quantitative non-RAG baseline limit generalization and prevent estimation of any reduction in hallucination.
Originality – The study evaluates retrieval and generation separately within a constrained Indonesian fiqh corpus and illustrates the distinction between semantic similarity and answer completeness.

Abstract views: 27 , PDF downloads: 32

Downloads

Download data is not yet available.

References

S. Suaidi, F. Faridi, and S. Sunarto, "Technology-Based Digitalization of Islamic Religious Education," Halaqa: Islamic Education Journal, vol. 9, no. 2, 2025.

Khairuddin, "Sistematika Penulisan Fiqh Syafi'i (Analisa Buku Fiqh Al-Manhaji 'Ala Mazhab Imam Al'syafi'i) [Systematics of Shafi'i Fiqh Writing]," Sangaji: Jurnal Pemikiran Syariah dan Hukum, vol. 8, no. 2, pp. 288-298, 2024.

D. Suhartono, M. R. N. Majiid, and R. Fredyan, "Towards automatic question generation using pre-trained model in academic field for Bahasa Indonesia," Education and Information Technologies, vol. 29, no. 16, 2024.

K. Shuster, S. Poff, M. Chen, D. Kiela, and J. Weston, "Retrieval Augmentation Reduces Hallucination in Conversation," Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 3784-3803, 2021.

O. Ram, Y. Levine, I. Dalmedigo, D. Muhlgay, A. Shashua, K. Leyton-Brown, and Y. Shoham, "In-Context Retrieval-Augmented Language Models," Transactions of the Association for Computational Linguistics, vol. 11, pp. 1316-1331, 2023.

P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W. T. Yih, T. Rocktäschel, S. Riedel, and D. Kiela, "Retrieval-augmented generation for knowledge-intensive NLP tasks," Advances in Neural Information Processing Systems, vol. 33, pp. 9459-9474, 2020.

K. Fathoni, A. Sudaryanto, I. Syarif, Z. M. E. Darmawan, A. A. Yunanto, Prasetyo, R. S. Aji, A. I. Fauzy, and R. Y. Hakkun, "EMasjid, Islamic Application with RAG (Retrieval-Augmented Generation)," Informatics, Electrical and Electronics Engineering (Infotron), vol. 5, no. 1, pp. 49–58, 2025.

M. A. Asl and B. Minaei-Bidgoli, "FARSIQA: Faithful and Advanced RAG System for Islamic Question Answering," arXiv preprint arXiv:2510.25621, 2025.

AA. Y. Alan, E. Karaarslan, and O. Aydin, "A RAG-based Question Answering System Proposal for Understanding Islam: MufassirQAS LLM," TechRxiv, 2024. DOI: 10.36227/techrxiv.170723304.41988020/v2.

M. Ichsan and A. G. Nusantara, "Perancangan Chatbot Fiqih Berbasis Retrieval-Augmented Generation Kitab Safinatun Najah [Designing a Fiqh Chatbot Based on Retrieval-Augmented Generation of the Safinatun Najah Book]," Seminar Nasional Teknologi & Sains (STAINS), vol. 5, no. 1, pp. 604-610, 2026.

A. Farisi, Chriestina, D. Dafid, and A. Jansa, "Penerapan Model AI Gemini pada Pengembangan Chatbot Tanya Fiqih Haji dan Umrah [Implementation of the Gemini AI Model in the Development of a Hajj and Umrah Fiqh QA Chatbot]," INSOLOGI: Jurnal Sains Dan Teknologi, vol. 4, no. 6, pp. 1778–1787, 2025.

S. Anwar, F. Bawazir, R. Sakina, M. Lukita, N. Hernata, M. Miranda, and M. Ridwan, "Mazhab Syafi’i sebagai Paradigma dalam Pemikiran dan Penetapan Hukum Islam Di Indonesia," Varia Hukum, vol. 5, no. 2, pp. 101–123, 2023.

S. Es, J. James, L. Espinosa-Anke, and S. Schockaert, "RAGAS: Automated Evaluation of Retrieval Augmented Generation," 18th Conference of the European Chapter of the Association for Computational Linguistics (EACL 2024), pp. 1-16, 2024.

L. Wang, N. Yang, X. Huang, L. Yang, R. Majumder, and F. Wei, "Multilingual E5 Text Embeddings: A Technical Report," arXiv preprint arXiv:2402.05672, 2024.

Chroma Core Team, "Chroma: The AI-native Open-source Embedding Database," Official Documentation, 2024. [Online]. Available: https://docs.trychroma.com.

Mozilla Developer Network, "Progressive Web Apps (PWA)," MDN Web Docs, 2024. [Online]. Available: https://developer.mozilla.org/en-US/docs/Web/Progressive_web_apps.

S. Barnett, S. Kurniawan, S. Thudumu, Z. Brannelly, and M. Abdelrazek, "Seven Failure Points When Engineering A Retrieval Augmented Generation System" in Proceedings - 2024 IEEE/ACM 3rd International Conference on AI Engineering, 2024.

N. Thakur, N. Reimers, A. Rücklé, A. Srivastava, and I. Gurevych, "BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models," 35th Conference on Neural Information Processing Systems (NeurIPS 2021), 2021.

Qwen Team, "Qwen2.5 Technical Report," arXiv preprint arXiv:2412.15221, 2024.

A. Mallen, A. Asai, V. Zhong, R. Das, D. Khashabi, and H. Hajishirzi, "When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories," Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, pp. 9802-9822, 2023.

P. Xu, W. Ping, X. Wu, L. McAfee, C. Zhu, Z. Liu, S. Subramanian, E. Bakhturina, M. Shoeybi, and B. Catanzaro, "Retrieval Meets Long Context Large Language Models," 12th International Conference on Learning Representations, ICLR 2024, 1–16, 2024.

W. Shi, S. Min, M. Yasunaga, M. Seo, R. James, M. Lewis, L. Zettlemoyer, and W. Yih, "REPLUG: Retrieval-Augmented Black-Box Language Models," arXiv preprint arXiv:2301.12652, 2023.

A. Asai, Z. Wu, Y. Wang, A. Sil, and H. Hajishirzi, "Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection," arXiv preprint arXiv:2310.11511, 2023.

G. Izacard, P. Lewis, M. Lomeli, L. Hosseini, F. Petroni, T. Schick, J. Dwivedi-Yu, A. Joulin, S. Riedel, and E. Grave, "Few-shot Learning with Retrieval Augmented Language Models," Journal of Machine Learning Research, vol. 24, pp. 1-43, 2023.

S. Siriwardhana, R. Weerasekera, E. Wen, T. Kaluarachchi, R. Bhowmik, and S. Nanayakkara, "Improving the Domain Adaptation of Retrieval Augmented Generation (RAG) Models for Open Domain Question Answering," Transactions of the Association for Computational Linguistics, vol. 11, pp. 1-17, 2023.

Downloads

Published

2026-09-10

How to Cite

Wijaya, T., Defry Hamdhana, Fajriana, & Mukti Qamal. (2026). Evaluating Precision and Semantic Similarity in a Retrieval-Augmented Generation-Based Question Answering System for Shafi’i Prayer Fiqh. Journal of Deep Learning, Computer Vision and Digital Image Processing, 4(3), 169–182. https://doi.org/10.61255/decoding.v4i3.1840

Issue

Section

Articles