Modernizing News Ingestion and Search Platforms with Elastic Stack for a Global Trading Firm
Client
The client is a global proprietary trading firm operating across major financial markets. With a strong emphasis on real-time information and data-driven strategies, their infrastructure relies on fast and reliable news ingestion and search capabilities to support high-frequency trading decisions.
Project Context
The client managed two custom-built applications: a Node.js-based news ingestion service handling thousands of items per second, and a proprietary search interface. They wanted to modernize this system using Elastic Stack components for better scalability and easier maintenance.
Challenges
- Maintain high throughput (2,000 news items/sec)
- Replicate advanced search features in Kibana
- Ensure fault tolerance and role-based access control
Solution
- Replaced ingestion app with Logstash, using persistent queues and retry logic
- Rebuilt the search experience in Kibana with Lucene query support, highlights, and filtering
- Deployed Kibana and Logstash on Kubernetes for better resource management
Project Objectives
- Replace the custom Node.js ingestion app with a scalable Logstash pipeline
- Replace the proprietary search interface with Kibana dashboards
- Deploy the full solution on Kubernetes for simplified operations
Solution Delivery
SquareShift delivered a production-grade solution using Elastic Stack components. Legacy apps were decommissioned and replaced with fully managed, scalable services using Kubernetes, reducing maintenance overhead and improving performance.
Testimonial
SquareShift helped us modernize our ingestion and search stack without sacrificing performance or search flexibility.