Search Moved With Zero Unplanned Downtime for a Global Market Research Group

The challenge

The group's on-premises Elasticsearch deployment was getting slower as search and ingestion volumes grew. Its capacity was fixed, so it could not grow with demand. Encryption and network access had been set up ad hoc, and data came in through a Python-based script.

The group set three goals. It wanted search on a multi-node cluster on Azure Kubernetes Service. It wanted one governed, secure entry point in front of Elasticsearch and Kibana. And it wanted fixed capacity replaced with scaling driven by real query and ingestion load.

What we delivered

SquareShift first documented the existing setup, its access roles and storage, and the query and ingestion patterns behind the slowness. That gave a current-versus-target architecture baseline before anything moved. The team then deployed a multi-node Elasticsearch cluster on AKS, run by Elastic's Kubernetes operator, ECK.

Azure Application Gateway for Containers became the single entry point for Elasticsearch and Kibana, on a wildcard TLS certificate. Getting it to work took three fixes. Two were strict rules in the gateway's TLS settings. The third was a health check policy, because the secured cluster rejected the gateway's own health checks.

Scaling now works at three levels. Coordinating and query nodes scale with the Horizontal Pod Autoscaler, the Vertical Pod Autoscaler right-sizes resources, and the AKS Cluster Autoscaler resizes node pools. The team also standardized index templates and sharding. It began moving ingestion from a Python script to Beats and Fleet agents, and set up snapshot backups to Azure Blob storage. To move Elasticsearch to Kubernetes, see our Elasticsearch migration services.

The outcome

Search now runs on a multi-node cluster whose capacity matches real query and ingestion load automatically, in place of fixed on-premises capacity. A scaling test confirmed this up to the agreed limits. The work ran in eight sequenced phases with zero unplanned downtime.

Five platform layers changed: the cluster, the search stack, ingress, scaling and backup. Search and dashboards now sit behind one governed, TLS-secured entry point. Production indices share standard templates and sharding, and snapshot restores from Azure Blob storage have been tested under a lifecycle policy.