LogRadar

Overview

LogRadar is a specialized security tool designed to monitor HTTP access logs in real-time. It acts as a passive detection layer, ingesting log streams from web servers like Nginx or Apache, identifying potential attack patterns, and visualizing threats through a terminal-based interface (TUI).

The system is engineered for high throughput on commodity hardware, capable of processing over 50,000 log lines per second with sub-millisecond detection latency. It serves as both a forensic analysis tool and a live monitoring dashboard.

Features

• High-Performance Analysis Engine: utilizing a worker pool architecture to parallelize log processing.
• Multi-Layered Detection:
  • Signature-Based: Identifies known attack vectors including SQL Injection, XSS, Path Traversal, and Log4Shell using optimized regular expressions.
  • Behavioral Analysis: Detects anomalies such as brute force attempts, high-rate DoS attacks, and User-Agent rotation.
  • Threat Intelligence: cross-references IP addresses against known malicious sources using Bloom filters.
• Interactive TUI: A terminal user interface built with the Bubble Tea framework, providing real-time visibility into top attacking IPs, alert streams, and traffic metrics.
• Hot Reload Configuration: Updates detection rules and thresholds without service interruption using atomic swaps.
• Observability: Exposes key Prometheus metrics integrated with a Grafana dashboard.
• Deep Inspection: Parses JSON logs to analyze POST bodies, HTTP headers, and cookies for hidden threats.

Technical Details

The application is built in Go (1.23+) and follows a Clean Architecture (Hexagonal) pattern to separate core business logic from external interfaces.

Architecture

The system is divided into three main layers:

• Domain Layer: Defines core entities (LogEntry, Alert) and business rules. It has no external dependencies.
• Application Layer: Orchestrates the data flow. The Analyzer service manages the worker pool and coordinates the detection pipeline.
• Adapter Layer: Handles input (Tail/File reading), output (TUI/JSON), and detection strategies.

Performance Optimizations

To achieve the target throughput, several low-level optimizations were implemented:

• Zero-Allocation Hot Path: The core event loop uses fixed-size ring buffers (EventRingBuffer) for tracking behavioral events. This avoids garbage collection overhead during steady-state operation.
• Fast Hashing: The standard crypto/sha256 was replaced with hash/maphash for non-cryptographic operations like IP sharding and User-Agent tracking, resulting in significant CPU cycle savings.
• Aho-Corasick Pre-filtering: Before running expensive regex matching, a multi-pattern search algorithm filters out benign traffic. Regex engines are only invoked if potentially malicious keywords are present.
• Sharded Concurrency: The BehavioralDetector partitions state across multiple shards (default 16) based on IP address. This minimizes mutex contention in highly concurrent environments.

Terminal UI Implementation

The TUI is implemented using the Model-View-Update (ELM) architecture via Bubble Tea. It manages high-frequency updates by aggregating alerts into batch messages, preventing UI thread blocking. The “Top Attacking IPs” view uses a specialized ipMaxHeap data structure to maintain a sorted list of threats in O(log n) time.

Challenges and Solutions

High-Volume Log Truncation

Processing logs at high speed presents a risk of partial reads or line truncation, which can break parsers.

Solution: Implemented a custom buffering wrapper around the tailing library to ensure complete line reads, even during log rotation bursts.

The “Cookie Bomb” Vulnerability

Initial stress testing revealed that large cookie headers could cause excessive memory allocation and regex backtracking (ReDoS).

Solution: Enforced strict length limits on all analyzed fields and implemented a linear-scan pre-validator before passing suspect strings to the regex engine.

Memory Leak in Behavioral Tracking

Long-running instances initially showed slow memory growth due to stale IP entries in the behavioral tracker.

Solution: A “Stop-the-World” cleanup was too disruptive. Instead, a sharded LRU (Least Recently Used) eviction policy was added to each IP shard, combined with a background ticker that incrementally prunes expired records without locking the entire detector.

Lessons Learned

This project demonstrated the importance of memory layout in high-performance Go applications. Using slice-based buffers instead of linked lists reduced GC pressure by orders of magnitude.

Additionally, the complexity of TUI rendering proved to be a significant bottleneck. Decoupling the data ingestion rate from the screen refresh rate was essential to maintain a responsive interface under load.