Web Server Log Analysis Using Retrieval-Augmented Generation (RAG) and Large Language Model (LLM) for AI Chatbot Assistant

Web server logs are essential for monitoring performance, analyzing user behavior, and detecting security threats, yet the growing volume and semi-structured format of these logs make manual analysis impractical and error-prone. Existing automated approaches, including machine learning-based anomaly detection, deep learning frameworks such as LogEDL and SuperLog, and log summarization tools like LogAssist and LogSummary, remain limited by reliance on labeled training data, poor generalization to unseen patterns, lack of interactive querying, and high retraining costs. This research develops an AI chatbot assistant for NGINX log analysis using Retrieval-Augmented Generation (RAG) with a pre-trained Large Language Model (LLM), enabling contextual and accurate analysis through a conversational interface without domain-specific fine-tuning. A Cross-Encoder re-ranking mechanism is integrated after initial retrieval to ensure only the most relevant documents are passed to the LLM for response generation.