Selecting Elasticsearch Indexing Strategies: LogsDB vs. Standard Index
Selecting the correct Elasticsearch indexing strategy balances speed, storage, and stability. This guide compares LogsDB and Standard Indexes using Elastic Rally benchmarks. It also examines how ES|QL full-text search enhances LogsDB for log analytics.
In the Rally full-text search test, Logs DB reached a mean throughput of 14.12 ops/s against 13.96 ops/s for the standard index.
To create a custom AI agent, see this guide:
LogsDB vs. Standard Index: Performance Comparison
Elasticsearch provides two indexing methods.
- Logs DB Index: Optimized for log data. Provides efficient storage and stable performance.
- Standard Index: A general-purpose option. Supports diverse workloads.
This article tests both options under real-world conditions. It evaluates their suitability for log-heavy applications.
Testing Methodology: Benchmark Approach
We benchmarked both indexing strategies using Elastic Rally and realistic log datasets. We measured the following key performance metrics:
- Indexing Performance: Time to add new data.
- Query Speed: Time to return search results.
Full-Text Query Used for Testing
We ran a MATCH query to test Elasticsearch full-text search. The test query was:
```json
{
"query": {
"bool": {
"must": [
{
"match": {
"message": "Database connection failed"
}
}
]
"should": [
{
"match": {
"details": "359+characters.",
},
},
]
}
}
}
```
Test Parameters (Rally Config):
- Iterations: 1,000
- Client count: 5
- Warmup runs: 10
- Workload: Simulated real-world log queries in a Kubernetes environment
LogsDB benchmark results: LogsDB index versus standard index
| Metric | Logs DB index | Standard index | Result |
|-------------------|----------------|------------------|------------|
| Min Throughput | 11.88 ops/s | 10.55 ops/s | Logs DB |
| Mean Throughput | 14.12 ops/s | 13.96 ops/s | Logs DB |
| Max Throughput | 15.01 ops/s | 14.86 ops/s | Logs DB |
Logs DB sustains higher throughput in full-text search tests. Select Logs DB for log-based search workloads.
ES|QL Optimizes Logs DB Full-Text Search
Logs DB stores, retrieves, and analyzes logs. Search performance is critical. ES|QL (Elasticsearch Query Language) optimizes and accelerates full-text search. This improves performance for log-heavy workloads.
ES|QL enhances Logs DB performance in these ways:
- MATCH function: Executes fast, Lucene-style full-text search.
Example Query
```json
GET logs/_search?size=1
{
"query": {
"match": {
"message": "error"
}
},
"sort": [
{ "timestamp": "desc" }
]
}
```
- QSTR function: Filters data using Lucene query syntax.
Example query
```json
GET logs/_search?size=1
{
"query": {
"query_string": {
"query": "message:\"server error\" OR status:500"
}
},
"sort": [
{ "timestamp": "desc" }
]
}
```
Relevance to Logs DB
- Pre-indexed fields enable immediate search results without runtime latency. Use `GET /your-index/_mapping` to verify field status. Indexed fields display `"index": true` in the mapping.
- Geo-search: Use location filters to query log data.
Application Example
When logs contain location data (e.g., IP addresses mapped to coordinates):
- Isolate logs from a defined region.
- Rank logs by distance from a point.
Select an Index
Index selection depends on workload requirements and priorities. Table 1 lists criteria.
Use Logs DB Index if:
- Storage Efficiency: Reducing infrastructure costs is a primary goal.
- Log Analytics & Search: Logging is the primary application use case.
Choose the standard index when:
- Fast indexing is the primary goal.
- Flexibility is required to process data types other than logs.
- Latency spikes are acceptable due to potential performance variability.
Understanding these differences allows you to align your Elasticsearch configuration with specific use cases and performance requirements.
Key Takeaways
The Logs DB Index offers a high-performance, low-cost solution for log analytics. It integrates with ES|QL to provide advanced search capabilities. While the Standard Index serves as a reliable general-purpose option, it faces performance limits with high-volume logging workloads.
Recommendation: Run custom Elastic Rally tests to validate performance against your specific data patterns.
Related reading: Benchmarking Elastic Cloud on AWS, GCP and Azure and what Elastic Observability covers.
Note: These benchmarks use Elastic Rally. Actual results vary based on workload and cluster configuration. Implement enterprise-scale search capabilities by contacting our team.
Contact Us