Audit Platform Costs Cut Around 70% for a Clinical Trial Logistics Provider

The challenge

The provider's shipment platform publishes every booking, shipment milestone, notification and API event to RabbitMQ, and stores them in Elasticsearch as an audit trail. The cluster behind it had become fragile and expensive. Queries timed out, memory safeguards tripped, nodes ran out of memory and restarted, and snapshots could not complete.

Ingestion ran on custom C# code whose developers had left the business. Nobody had documented it, and the storage credentials inside the cluster were unknown. Hundreds of message types with changing fields caused mapping conflicts. Volumes varied widely between types, and one very large message could block everything queued behind it. Years of history also had to move while live messages kept flowing.

What we delivered

SquareShift designed a new platform instead of patching the old cluster. It set up a right-sized Elastic Cloud deployment on Azure, in the provider's own subscription, with its index structure and retention rules defined from the start. Logstash pipelines running beside RabbitMQ replaced the C# service and send each message type to its own data stream.

SquareShift tuned mappings so changing message structures index cleanly, and kept key fields as text so engineers can search flexibly, not only by exact value. Messages that still fail go to a dead letter queue, where they stay visible and can be investigated. Lifecycle policies roll each data stream over by size or age, then move older data to the frozen tier.

Years of history, much of it already in low-cost searchable archives, moved to the new cluster through Azure Blob Storage while live ingestion continued. For very large messages, SquareShift tested a fix that lets them through without stalling the queue. It also defined how to split oversized documents and join them again later. Every step was documented for the provider's engineers. To rebuild an Elasticsearch platform on Elastic Cloud, see our Elasticsearch migration services.

The outcome

Monthly infrastructure cost fell by around 70%, and query timeouts and circuit-breaker errors stopped after the migration, leaving a stable cluster. Declarative Logstash pipelines replaced the custom C# code, so the provider no longer depends on departed developers or undocumented code to run a business-critical system. Its audit data now sits in its own Azure subscription and storage.

Failed messages are captured and can be investigated instead of disappearing, and far fewer messages now fail to index. Very large messages no longer block ingestion. Lifecycle policies handle rollover and retention automatically, and flexible text search helps engineers trace a shipment's history during incidents. The provider's engineers now own a fully documented platform they understand and can extend.