This article compares the performance and storage of standard Elasticsearch indices and LogsDB. We used Elasticsearch Rally (esrally) to benchmark data ingestion and storage size.
Elasticsearch version 8.17 introduced LogsDB. This index mode optimizes log data storage through index sorting, advanced compression, and synthetic _source reconstruction. These features improve storage efficiency for large log datasets.
Benchmark Setup
We ran Elasticsearch Rally to simulate real-world data ingestion. We tested both standard and LogsDB indices.
Tools and Environment
- Elasticsearch Rally for ingestion benchmarks
- Machine: Ubuntu VM with DockerGCP Instance: e2-standard-16
- 16 vCPUs
- 64 GB RAM
- 250 GB storage
- Elastic Cloud Cluster: Logsdb-logs
- Version: 8.17.3
- Hot Storage: 720 GB
- Hot Memory: 16 GB
- Kibana Memory: 1 GB
Configure Elasticsearch Rally
Execute the following steps to prepare the environment:
Install Docker
Retrieve the Rally Image
- Verify Docker is installed and active on the virtual machine.
| docker pull elastic/rally |
|---|
- View Available Tracks
| sudo docker run elastic/rally list tracks |
|---|
Create a Persistent Volume
- This step reuses data across runs.
| docker volume create rally_data |
|---|
Run the Benchmark
- Use the tsdb track with 123 GB of data.
- This track uses Kubernetes cluster metrics from Metricbeat. It includes pod performance, node stats, and container telemetry. The data is time-series. Es Rally ingests it into the Elasticsearch index.
| docker run -d --rm -v rally_data: /rally/.rally elastic/rally race \ --track=tsdb \ --pipeline=benchmark-only \ --target-hosts=<es-url> :9243 \ --client-options= "use_ssl:true,verify_certs:true,basic_auth_user:'elastic',basic_auth_password:'<password>'" \ --track-params= '{ "bulk_indexing_clients" :25, "ingest_percentage" :100}' \ |
|---|
- The setup creates 116,633,698 documents in Elasticsearch.
- We run this process for the standard index and logsdb.
We prepare two index templates:
- One contains standard index settings.
- One index with Logs DB settings (see below);
Update Index Settings
Benchmark setup required updated index settings to optimize performance for standard indices and LogsDB. Applied configurations:
An Elasticsearch index template applied these settings. All indices matching the pattern inherit these configurations automatically.
Standard Index Settings
| { "settings": { "index": { "mode": "standard" , "codec": "default" , "mapping": { "synthetic_source_keep": "none" }, "sort": { "field": [], "order": [], "mode": [], "missing": [] } } }} |
|---|
LogsDB Index Configuration
| { "index": { "mode": "logsdb" , "mapping.synthetic_source_keep": "arrays" , "sort.field": [ " host.name " , "@timestamp" ], "sort.order": [ "desc" , "desc" ], "sort.mode": [ "min" , "min" ], "sort.missing": [ "_first" , "_first" ], "codec": "best_compression" , "mapping.ignore_malformed": true , "mapping.ignore_above": 8191 , " mapping.total _fields.ignore_dynamic_beyond_limit": true }} |
|---|
LogsDB Optimization Parameters
LogsDB uses three techniques for efficient performance. The Elasticsearch index settings show these techniques:
Optimized Index Sorting
Enhanced Compression
"codec": "best_compression" applies optimized compression (Zstandard, delta encoding) to reduce storage. Performance remains stable.
synthetic_source
"mapping.synthetic_source_keep": "arrays" retains essential arrays. It discards the full _source and reconstructs data only when needed. This reduces storage usage.
- "sort.field": ["host.name", "@timestamp"] groups logs by host and timestamp. This reduces fragmentation.
- "sort.order": ["desc", "desc"] places the latest data first. This accelerates retrieval.
Additional Performance Adjustments
- Set "mapping.ignore_malformed" to true to skip invalid data. This prevents ingestion errors.
- Set "mapping.ignore_above" to 8191 to limit field size. This improves indexing speed.
- Set "mapping.total_fields.ignore_dynamic_beyond_limit" to true. This prevents index bloat from excessive dynamic fields.
Data Ingestion and Storage Metrics
Benchmark Results
Each Rally run ingests 116,633,698 documents. To complete the dataset, we ran 8 cycles in standard mode and 8 cycles with the LogsDB index.
Standard Index
- Total Documents: 933,069,584
- Total Storage Size: 381.82 GB
- Average Ingestion Time: 1 hour 15 minutes to 1 hour 30 minutes for one es rally run in standard mode.
LogsDB Index
- Total Documents: 933,069,584 (identical document count)
- Total Storage Size: 206.06 GB
- Average Ingestion Time: 1 hour 45 minutes to 2 hours for one es rally run in logsdb mode
Storage Reduction: LogsDB reduced storage by 46.01% compared to standard mode.
Key Takeaways
- LogsDB reduced storage by 46.01% without losing documents.
- Elastic Cloud Benefits: Reduced storage lowers costs and speeds up snapshots.
Evaluate LogsDB if you scale log ingestion in Elastic.