Global IT Services Provider Cuts Peak Ingestion Lag From Hours to Minutes for Its Managed Observability Platform

The challenge

Data arrived 6–7 hours late, because one busy log source stalled every other source.

The cluster stayed yellow for months, with 95,640 rejected searches and no index clean-up.

Four on-premises regional sites fed a central 50-node Elasticsearch cluster on Azure Kubernetes Service. A gap analysis surfaced 30 findings, 13 of them critical.

What we did

SquareShift ran five workstreams. The engagement began with a data-flow-ordered gap analysis of all five clusters, then remediated in priority order.

Cluster health returned to GREEN. The YELLOW state traced to enrich indices that re-expanded replicas to every node.

Ingestion lag fell to minutes. The monolithic regional pipeline was split into three isolated per-source pipelines (syslog, network and infrastructure), each with its own queue. Each change was validated in UAT before production.

APM trace shards were blowing past the 50 GB rollover limit. Primary shards rose from 1 to 10, so size-based rollover fires as designed.

Frozen-tier search rejections were traced to high-frequency Synthetics UI queries against frozen data at 4× the recommended shard density. Elastic Support helped with tuning.

Index hygiene: 2,400+ empty indices purged, delete phases added to 3,037 indices, and 113 unmanaged indices brought under ILM.

Fleet-managed monitoring agents, 10 operational alerts and 2 Kibana dashboards now cover the platform itself.

A peer-reviewed upgrade runbook covers 8.18.3, 8.19.18 and 9.4.4 across ECK, air-gapped on-prem and edge environments. The operations team received coaching.

The results

Ingestion lag at peak fell from 6–7 hours to minutes, with isolated per-source pipelines in production.

Cluster health moved from YELLOW for months to GREEN, with the root cause fixed, not patched.

Delete phases are live, empty indices are purged and all indices sit under ILM. Allocation is balanced across all hot nodes.

The headline results: peak ingestion lag down from hours to minutes, 2,400+ empty indices purged, and data-copy errors cut from 44 to 0.

Why it worked

The team fixed each root cause at source, including why enrich replicas could not allocate and why size-based rollover silently failed.

Every change was replicated in UAT first, including the queue saturation pattern itself, then rolled into production in controlled windows.

The operations team runs its own alerting and dashboards, with a complete, peer-reviewed path to Elasticsearch 9.4.