Modernizing News Ingestion and Search Platforms with Elastic Stack for a Global Trading Firm
Client
The client is a global proprietary trading firm operating across major financial markets. With a strong emphasis on real-time information and data-driven strategies, their infrastructure relies on fast and reliable news ingestion and search capabilities to support high-frequency trading decisions.
Project Context
The client managed two custom-built applications: a Node.js-based news ingestion service handling thousands of items per second, and a proprietary search interface. They wanted to modernize this system using Elastic Stack components for better scalability and easier maintenance.
Project Objectives
- Replace the custom Node.js ingestion app with a scalable Logstash pipeline
- Replace the proprietary search interface with Kibana dashboards
- Deploy the full solution on Kubernetes for simplified operations
Challenges
- Maintain high throughput (2,000 news items/sec)
- Replicate advanced search features in Kibana
- Ensure fault tolerance and role-based access control
Solution
- Replaced ingestion app with Logstash, using persistent queues and retry logic
- Rebuilt the search experience in Kibana with Lucene query support, highlights, and filtering
- Deployed Kibana and Logstash on Kubernetes for better resource management
