Zero-Downtime Elasticsearch Migration for a Global Video Commerce Leader
Client
The client is a global video commerce company operating across live TV, e-commerce, and mobile platforms. Their observability stack, powered by Elasticsearch, supports log ingestion, real-time monitoring, and alerting workflows critical to ensuring performance and uptime across digital storefronts.
Project Context
The client managed an on-prem Elasticsearch cluster supporting Fleet, Logstash, and Kibana for monitoring and alerting. They opted for a selective migration of system indices to Elastic Cloud using dual writes to minimize cost, avoid migrating 52TB of historical data, and maintain operational continuity.
Project Objectives
- Migrate only essential system indices (.watcher, .kibana, etc.)
- Implement dual writes with Logstash and Fleet
- Fix snapshot and template issues
- Set up SAML and API key authentication
- Optimize ILM for high-ingestion workloads
Challenges
- Snapshot credential mismatches
- ILM misconfiguration under high ingestion
- API key decryption errors and SAML restrictions
- Template mismatches affecting indexing
Solution
- Manual verification and fix of snapshot access
- Tuned ILM for hot/frozen tiers
- Fixed API/SAML issues and restored dashboard access
- Migrated 4 Fleet policies and aligned templates
Solution Delivery
SquareShift executed a 3-phase plan: prep with snapshot testing, dual write cutover, and post-migration optimizations. Over 3,500 watchers were updated via automation. Downtime was avoided entirely, and redundant data transfer was skipped, saving costs and time.
Testimonial
SquareShift delivered a flawless Elastic Cloud migration, with zero downtime and major savings on data and effort.
