Document Type
Article
Publication Date
Winter 10-20-2026
Abstract
Industrial safety question answering requires access to the exact source material that supports an answer, not merely a related document. This study evaluates the retrieval stage of a local retrieval-augmented generation (RAG) system at the source page level. Eighteen documents on lockout/tagout, machine guarding, and personal protective equipment were prepared in raw, cleaned, and metadata-enriched forms. Best Matching 25 (BM25), latent semantic analysis (LSA), and reciprocal rank fusion (RRF) hybrid retrieval were tested on 54 questions: 36 answerable, 12 safety-related but unsupported, and 6 outside the covered topics. Raw BM25 achieved the highest mean reciprocal rank (0.492) and Hit@5 (0.722), cleaned BM25 achieved the highest Hit@3 (0.556) and answerability AUROC (0.765), and metadata-enriched BM25 achieved the highest Hit@10 (0.833). For raw text, the retrieval method affected reciprocal rank (Friedman χ2 = 7.13, p = 0.028). Hybrid retrieval exceeded LSA after Holm correction (p = 0.011), whereas BM25 and hybrid retrieval did not differ (p = 0.302). Text preparation had no overall effect within BM25 (p = 0.220). The results show that document-level evaluation can overstate access to exact evidence. The study provides a repeatable procedure for source tracking, page-level relevance labeling, separate retrieval and generation evaluation, and refusal analysis. These design principles are also relevant to construction safety knowledge systems that combine regulations, accident reports, project data, and BIM-linked information. Because the relevance set is a model-assisted silver standard, the findings remain exploratory rather than expert-certified absolute retrieval accuracy. A controlled same-run extension with all MiniLM L6 v2 and BM25 + MiniLM RRF is reported separately; its observed fusion gains did not reach statistical significance on the 36 answerable questions. The work is intended to support safety engineers, researchers, and developers designing evidence-grounded industrial and construction safety information systems.
Recommended Citation
Sorour, S. S., El Mufti, A., Nabawy, M. A., & Sahbel, A. (2026). Source Page Evaluation of Local RAG Retrieval for Engineering Safety: Implications for Mechanical and Civil Engineering. Technologies, 14(10), 682. https://doi.org/10.3390/technologies14100682
Included in
Applied Mechanics Commons, Computer-Aided Engineering and Design Commons, Construction Engineering and Management Commons