Case study
1.5 Terabytes a Day Monitored for a National Digital ID Program
How SquareShift deployed a secure, on-premise Elasticsearch observability platform across two data centers for a national digital identity program, sized for 1.5TB of ingestion a day.
Book a session →
Client
One of the world's largest digital identity programs, a government initiative verifying and managing digital IDs for millions of residents across India.
Industry
Government
Services
Stack
Elasticsearch · Kafka (MirrorMaker) · Elastic APM · Elastic Security (SIEM) · Machine Learning (ML)
How did SquareShift deploy Elasticsearch observability across two data centers?
The challenge
The client runs one of the world's largest digital identity programs, verifying and managing digital IDs for millions of residents. Legacy tools could not scale to the program's data volumes or meet its security bar, and the in-house team needed expert support for capacity planning and deployment.
Given how sensitive the data is, the solution had to be delivered on-site under strict compliance and data-privacy requirements, with no shortcut to a faster, cloud-only build.
What we delivered
SquareShift deployed Elasticsearch with APM, machine learning, and SIEM features across two geographically separate data centers, sized for 1.5 terabytes of ingestion a day and 270 terabytes of retention. Kafka MirrorMaker handled geo-redundant ingestion between the two sites.
The team ran full high-availability configuration and stress testing to validate performance under load, delivering everything through on-site engagement rather than remote-only delivery, given the sensitivity of the data.
The outcome
The program now runs on-premise observability sized for 1.5 terabytes of ingestion a day and 270 terabytes of retention, with high availability spanning two data centers instead of a single point of failure.
Custom dashboards and rigorous benchmark testing give the client's own team the tools to operate the platform going forward, backed by an architecture built and validated for compliance from day one. To see how we build monitoring on Elastic, read about our Elastic services.
At national-identity scale, on-site delivery isn't a compliance checkbox — it's the only way to stress-test high-availability failover before sensitive data ever touches the cluster.
Platform & Software Engineering Practice Lead, SquareShift


