← All case studiesGovernment · Elastic consulting

Case study

1.5 Terabytes a Day Monitored for a National Digital ID Program

How SquareShift deployed a secure, on-premise Elasticsearch observability platform across two data centers for a national digital identity program, sized for 1.5TB of ingestion a day.

Book a session →
Illustration for the case study: 1.5 Terabytes a Day Monitored for a National Digital ID Program
2geographically separate data centres running active-active
270TB of retention across hot and warm storage tiers

Client

One of the world's largest digital identity programs, a government initiative verifying and managing digital IDs for millions of residents across India.

Industry

Government

Services

PePlatform & Software EngineeringStStrategy & Decision Intelligence

Stack

Elasticsearch · Kafka (MirrorMaker) · Elastic APM · Elastic Security (SIEM) · Machine Learning (ML)

How did SquareShift deploy Elasticsearch observability across two data centers?

01

The challenge

The client runs one of the world's largest digital identity programs, verifying and managing digital IDs for millions of residents. Legacy tools could not scale to the program's data volumes or meet its security bar, and the in-house team needed expert support for capacity planning and deployment.

Given how sensitive the data is, the solution had to be delivered on-site under strict compliance and data-privacy requirements, with no shortcut to a faster, cloud-only build.

02

What we delivered

SquareShift deployed Elasticsearch with APM, machine learning, and SIEM features across two geographically separate data centers, sized for 1.5 terabytes of ingestion a day and 270 terabytes of retention. Kafka MirrorMaker handled geo-redundant ingestion between the two sites.

The team ran full high-availability configuration and stress testing to validate performance under load, delivering everything through on-site engagement rather than remote-only delivery, given the sensitivity of the data.

03

The outcome

The program now runs on-premise observability sized for 1.5 terabytes of ingestion a day and 270 terabytes of retention, with high availability spanning two data centers instead of a single point of failure.

Custom dashboards and rigorous benchmark testing give the client's own team the tools to operate the platform going forward, backed by an architecture built and validated for compliance from day one. To see how we build monitoring on Elastic, read about our Elastic services.

“
At national-identity scale, on-site delivery isn't a compliance checkbox — it's the only way to stress-test high-availability failover before sensitive data ever touches the cluster.

Platform & Software Engineering Practice Lead, SquareShift

Related work