Benchmarking Full-Text Search Performance when using LogsDB mode
- SquareShift Engineering Team
- Mar 26
- 3 min read
Updated: Aug 4
Choosing the Right Indexing Strategy in Elasticsearch: Logs DB vs. Standard Index
Selecting an appropriate indexing strategy in Elasticsearch is crucial. It is essential for balancing speed, storage, and stability. In this guide, we compare Logs DB Index and Standard Index with real-world benchmarks using Elastic Rally. This is the official performance testing tool from Elastic. Additionally, we will explore how ES|QL's full-text search capabilities enhance Logs DB, making it an effective choice for log analytics.
To learn to create a Custom AI for yourself, Check this blog out:
Why Compare Logs DB vs. Standard Index?
Elasticsearch offers two main indexing approaches:
Logs DB Index: Specially optimized for log data, focusing on efficient storage and stable performance.
Standard Index: A general-purpose option that offers flexibility across different workloads.
You may wonder which is better for log-heavy applications. To answer this question, we tested both options under real-world conditions to find out!
How We Tested (Benchmarking Methodology)
To compare the two indexing strategies, we utilized Elastic Rally with realistic log datasets. We measured key performance metrics, including:
Indexing Performance: Speed of adding new data.
Query Speed & Efficiency: How quickly searches return results.
Full-Text Query Used for Testing
For the benchmarking process, we executed a MATCH query. This utilized Elasticsearch’s full-text search capabilities. The test query was as follows:
```json
{
"query": {
"bool": {
"must": [
{
"match": {
"message": "Database connection failed"
}
}
],
"should": [
{
"match": {
"details": "359+characters."
}
}
]
}
}
}
```
Test Parameters [Rally config]:
Iterations: 1000
Clients: 5
Warmup Iterations: 10
Workload: Simulated real-world log searches in a Kubernetes environment
Performance Comparison: Logs DB Index vs. Standard Index
| Metric | Logs DB Index | Standard Index | Winner |
|-------------------|----------------|------------------|------------|
| Min Throughput | 11.88 ops/s | 10.55 ops/s | Logs DB |
| Mean Throughput | 14.12 ops/s | 13.96 ops/s | Logs DB |
| Max Throughput | 15.01 ops/s | 14.86 ops/s | Logs DB |
Takeaway: Logs DB performs better in full-text search across all throughput metrics, making it a better choice for log-based searches.
How ES|QL Enhances Logs DB's Full-Text Search
While Logs DB is designed for efficient log storage, retrieval, and analytics, search performance is equally important. This is where ES|QL (Elasticsearch Query Language) comes in. It optimizes and accelerates full-text searches in Logs DB, making it highly effective for log-heavy workloads.
Here’s how ES|QL enhances Logs DB’s performance:
MATCH Function: Enables fast, Lucene-style full-text search.
Example Query
```json
GET logs/_search?size=1
{
"query": {
"match": {
"message": "error"
}
},
"sort": [
{ "timestamp": "desc" }
]
}
```
QSTR Function: Allows advanced filtering using Lucene query syntax.
Example Query
```json
GET logs/_search?size=1
{
"query": {
"query_string": {
"query": "message:\"server error\" OR status:500"
}
},
"sort": [
{ "timestamp": "desc" }
]
}
```
Why This Matters for Logs DB
Pre-indexed Fields: This means instant searches without runtime slowdown. Use `GET /your-index/_mapping` to check which fields are indexed. Indexed fields will have `"index": true` in their mapping.
Improved Geo-searching: Optimize log data with location-based filters.
Example Use Case
If your logs contain location-based data (e.g., IP addresses mapped to latitude/longitude), you can:
Filter logs from a specific region (e.g., "Show logs from New York only").
Sort logs by proximity (e.g., "Find the nearest server experiencing errors").
Which One Should You Use?
Choosing the right index depends greatly on your workload and priorities. Here’s a breakdown:
Logs DB Index if:
Storage Efficiency: A clear priority to reduce infrastructure costs.
Log Analytics & Search: The primary use case of your application.
Standard Index if:
Fast Indexing: Is your utmost priority.
Flexibility: Needed to handle various data types beyond logs.
Latency Spikes: Are acceptable as Standard Index may experience performance variability.
By understanding these differences, you can optimize your Elasticsearch setup to better match your use case and performance needs.
Final Thoughts
For log analytics, the Logs DB Index is a high-performance, cost-effective solution. It is especially effective when combined with ES|QL’s powerful search features. Although the Standard Index remains a solid general-purpose option, it can struggle with high-volume logging.
Pro Tip: Run your own Elastic Rally tests to see what works best for your data!
Note: These benchmarks were performed using Elastic Rally. Your results may vary depending on workload and cluster configurations.
To Unlock the super search in your enterprise.
Comments