In modern enterprise Security Operations Centers (SOC), security information and event management (SIEM) is the beating heart of incident detection, compliance auditing, and threat response. Every firewall packet drop, SSH authentication attempt, Windows Active Directory Kerberos ticket request, Kubernetes audit log, and endpoint process creation event must be collected, parsed, indexed, and analyzed in real time. For a mid-market enterprise managing 500 to 1,500 endpoints, daily log volume easily exceeds 250 to 500 gigabytes per day—accumulating to 10 to 15 terabytes of raw security telemetry every month.

When security architects design a SIEM to ingest that firehose of data without paying multi-million-dollar annual SaaS licensing fees to Splunk or Microsoft Sentinel, two open-source giants dominate the architectural discussion: Elastic SIEM (Elastic Security) and Wazuh. Both platforms promise to deliver enterprise threat hunting, automated MITRE ATT&CK mapping, and compliance dashboards without crushing software license bills.

Yet behind the open-source banners lies a brutal engineering reality: storage physics and JVM memory consumption. If you do not understand the difference between Elasticsearch’s Lucene inverted index architecture and Wazuh’s specialized Host-based Intrusion Detection System (HIDS) decoding engine, an unoptimized SIEM cluster will consume tens of thousands of dollars in high-speed NVMe storage, drown in JVM garbage collection pause deadlocks, and collapse under unmanageable cluster shard bloat. To build an unyielding SOC on a budget, you must master the forensic economics of log ingestion pipelines, Index Lifecycle Management (ILM), and hot-warm-cold storage tiering.

Which SIEM Is More Cost-Effective: Elastic Security or Wazuh?

The Lucene Inverted Index Storage Penalty: Why Security Logs Explode

To understand why SIEM storage costs spiral out of control, we must analyze how search engines store text data. Both Elastic SIEM and Wazuh rely on distributed search engines rooted in the Apache Lucene library (Elasticsearch for Elastic SIEM; Wazuh Indexer, which is a fork of OpenSearch, for Wazuh).

When an endpoint emits a 1-kilobyte JSON log entry (such as a Windows Sysmon Event ID 1: Process Creation), a traditional relational database writes that 1 KB record to disk with minimal overhead. Lucene, however, is not a simple database; it is a full-text search engine. To allow an analyst to search 100 million logs in sub-seconds, Lucene constructs an Inverted Index for every indexed field in the document (process name, parent process, command line arguments, hashes, user SID, IP address, port numbers).

Constructing an inverted index involves building extensive term dictionaries, posting lists, doc values for column-oriented aggregation, and term frequency arrays. Under default mapping configurations, a 1-gigabyte raw text log file expands into 1.4 to 1.8 gigabytes of disk footprint once Lucene finishes building its indexes. When you factor in primary-replica replication (mandatory for cluster high availability so a single drive failure does not lose audit data), 10 terabytes of raw logs instantly morph into 28 to 36 terabytes of active high-speed NVMe storage every month.

The Cluster Shard Bloat Trap: Why Clusters Deadlock

The most pervasive technical disaster in open-source SIEM deployments is shard overallocation. When junior engineers set up Elasticsearch or OpenSearch, they frequently leave the default index template configured to generate a new index every single day with five primary shards and one replica.

Consider the mathematical result after six months of operation across multiple data streams (Windows, Linux, firewall, DNS, web proxy):

1. Ten distinct data streams × 180 days = 1,800 individual daily index tables.
2. 1,800 indices × 5 primary shards × 2 (primary + replica) = 18,000 active cluster shards.
3. Each Lucene shard maintains internal memory overhead inside the JVM heap (term dictionaries, field caches, Lucene segment readers), consuming 15 to 30 megabytes of RAM per shard regardless of whether it holds data.
4. 18,000 shards consume over 360 gigabytes of JVM memory purely for index metadata. The 32 GB JVM heap maximum is instantly overwhelmed.
5. Garbage collection cycles freeze the cluster for 30 to 60 seconds (Stop-the-World pauses). Nodes drop out of the cluster, ingestion queues back up, and the SIEM crashes permanently.

To avoid this, best-in-class architectures enforce Index Lifecycle Management (ILM) rollover policies based on size, NOT time: shards roll over only when they reach 30 to 50 gigabytes, keeping total cluster shard count below 500 shards per 32 GB node.

Forensic 3-Year TCO Comparison: 10TB Monthly Log Retention

Let us construct a rigorous financial model for an enterprise ingesting 350 gigabytes of security telemetry per day (~10.5 terabytes per month), requiring 90 days of hot/warm searchable retention and 1 year of cold compliance archive:

Scenario A: Elastic Security (Self-Hosted on Enterprise Hardware)

• Hot Tier Bare-Metal Nodes (3 Nodes, 2x 7.68TB Enterprise NVMe U.3 each): $24,500
• Warm Tier Storage Nodes (2 Nodes, 4x 16TB SATA SSD each): $16,800
• Cold Archive (MinIO / S3 Object Storage, 120TB Raw): $4,800 / year ($14,400 over 3 yrs)
• Software Licensing: $0 (Elastic Basic free tier, no ML) or $45,000/yr (Platinum for ML/Endpoint)
• Engineering Maintenance Hours (0.5 FTE): $65,000 / year ($195,000 over 3 yrs)
• Estimated 3-Year Pure Hardware TCO (Free Tier): $55,700
• Estimated 3-Year Enterprise TCO (With Platinum License): $190,700

Scenario B: Wazuh Architecture (Self-Hosted on Bare-Metal Hardware)

• Wazuh Master / Worker Nodes (2x Server Nodes, 64GB RAM): $8,400
• Wazuh Indexer Nodes (3x High-Density NVMe Nodes, 1x 15.36TB NVMe each): $18,200
• Cold Archive (Compressed JSON rotated to local ZFS pool or S3): $2,400 / year ($7,200 over 3 yrs)
• Software Licensing: $0 (100% Free Apache 2.0 forever, full features included)
• Engineering Maintenance Hours (0.35 FTE): $45,500 / year ($136,500 over 3 yrs)
• Estimated 3-Year Total Cost of Ownership: $33,800 (Hardware/Storage Only)

Wazuh delivers an immediate $22,000 hardware/storage advantage and saves up to $135,000 in commercial software licenses compared to commercial Elastic Platinum deployments, while delivering out-of-the-box FIM (File Integrity Monitoring), vulnerability scanning, and SCA (Security Configuration Assessment) that Elastic gates behind higher tiers.

Logstash vs Fluentbit: Pipeline Processing Overhead & Buffer Memory

In high-throughput SIEM architectures ingesting 10 terabytes monthly, the bottleneck that causes log loss is rarely the database itself; it is the log collection and forwarding pipeline. When thousands of endpoints emit high-velocity telemetry, an ingestion buffer must ingest, queue, parse, and forward records without dropping packets.

Consider the architectural battle between traditional Logstash and modern Fluentbit:

• Logstash (The Legacy Heavyweight): Logstash is built on the Java Virtual Machine (JRuby). To parse complex syslog formats and dissect CEF/LEEF headers, a single Logstash instance routinely consumes 4 to 8 gigabytes of RAM and requires heavy multi-core CPU scheduling. Under sudden event spikes (such as a corporate-wide phishing simulation or brute-force attack), Logstash’s persistent queue disk buffers fill up, backpressure stalls network sockets, and Windows event loggers drop unforwarded security events on the floor.
• Fluentbit (The High-Performance C Engine): Fluentbit is written in pure C with zero runtime dependencies. A production Fluentbit collector consumes less than 25 megabytes of RAM and executes regex parsing, field transformations, and JSON serialization with sub-millisecond latency. In high-density environments, replacing Logstash with Fluentbit pipelines cuts ingestion server CPU load by over 65% while providing bulletproof memory-backed ring buffers that withstand massive telemetry bursts.
• Wazuh AnalysisD Pipeline Efficiency: Wazuh bypasses external log shippers entirely for endpoint data. The native Wazuh Agent maintains a local, persistent sqlite queue on the endpoint. If the network drops or the SIEM cluster experiences an ingestion spike, the agent buffers up to 100,000 events locally on the client machine’s disk, resuming transmission with strict flow control once cluster capacity recovers.

The Elasticsearch Frozen Tier & Searchable Snapshots Architecture

For organizations compelled to maintain 365 days of active audit retention for SOC 2 Type II or PCI-DSS compliance, holding 120 terabytes of data on hot or warm NVMe/SSD storage represents a prohibitive financial burden. This is where Elastic SIEM’s Searchable Snapshots and Frozen Tier deliver true architectural dominance over OpenSearch and Wazuh:

In Elastic 8.x, once indices age past 90 days, an automated Index Lifecycle Management (ILM) policy executes a snapshot directly to low-cost cloud object storage (such as AWS S3 Standard-Infrequent Access or on-premises MinIO). Once the snapshot is verified, the local index is deleted from high-speed SSDs. However, unlike traditional backups that must be restored to disk before querying, the Frozen Tier mounts the snapshot as a live, searchable index directly from S3, caching only small block segments in a tiny local NVMe read cache. When an auditor or incident responder executes a historical hunt against 9-month-old logs, the query completes in 15 to 45 seconds while reducing monthly storage costs by more than 85%.

The 4:00 PM Resolution: The 10TB Log Ingestion Blueprint

When you engineer your SIEM around storage physics and strict index lifecycle hygiene, 10 terabytes of security data ceases to be a budget-destroying liability. It becomes an affordable, lightning-fast threat detection asset that shields your enterprise from catastrophic breaches.

Related analysis: For open-source intrusion detection and network monitoring comparisons, read our audit of Wazuh vs Zeek vs Suricata architectures.