Case study
Analytics Cost Cut for a Global Learning and Talent Software Provider
How SquareShift re-engineered a learning and talent-management platform's BigQuery setup around persistent tables and incremental processing, so cost scales with data volume instead of against it.
Book a session →
Client
A global provider of cloud-based learning and talent-management software used by organizations to recruit, train, and engage employees.
Industry
Technology
Services
Stack
Google BigQuery · SQL · ETL Pipelines
How SquareShift delivered it.
The challenge
As the platform's data volume grew, so did its BigQuery bill, because every ETL job dropped and rebuilt large tables from scratch each time, even when most of the data hadn't changed.
What we delivered
SquareShift re-engineered the BigQuery setup around persistent tables, so historical data no longer gets dropped and rebuilt on every run. Incremental processing now fetches only the data that changed, while column pruning, join tuning, clustering, and partitioning sped queries up by reading only what each one actually needs. This is an example of our data engineering on Google Cloud.
The outcome
Processing cost came down because tables that were already correct stopped being rebuilt, and the environment now scales with data volume instead of scaling cost alongside it. Related reading: BigQuery cost and query performance tuning, reducing BigQuery cost and latency with dbt and Snowflake to BigQuery migration for a recruiting platform.
Rebuilding a table you already got right yesterday isn't thoroughness, it's waste. The cheapest query is the one you don't have to run again.
Data & Analytics Practice Lead, SquareShift
