All projects Consumer technology · location data

Geo-Data Delivery with SLA Observability

A large North American location-data company

Delivering location data to an external buyer every 30 minutes, where a missed SLA costs money.

AWS Athena AWS Glue AWS Lambda Amazon S3 Amazon CloudWatch PagerDuty

The team

  • 1 Director of Data Engineering
  • 1 Engineering Manager
  • 4 Data Engineers

Use cases

  • Metrics calculation
  • Reverse ETL: data delivery to an external client
  • Data quality metrics
  • Data observability with SLA enforcement and on-call

What was built

  • Athena SQL run from a Glue Python Shell job every 30 minutes
  • Results written as Parquet partitions into the client's landing zone
  • Lambda compliance checks before data leaves the account
  • CloudWatch metrics and alerts on every step
  • PagerDuty on-call rotation for the data team

What changed

  • About 90% of the engineering effort went into observability, not transformation — because an SLA breach meant a financial penalty.
  • Compliance checks ran inside the pipeline, so non-compliant data never reached the buyer.
  • A hard lesson: part of the SLA depended on upstream systems the data team did not own. Contracts and alerts must cover that.
Related service

Cloud Data Platforms

Architect and build cloud data platforms that scale to petabytes.

Explore service

Let's talk about your data platform

Tell us what you are working on. We reply within one business day, and we will tell you plainly if we are not the right fit.

Start a conversation Email us

Other projects

E-commerce marketplace · 400 employees

Oracle Data Warehouse to AWS Redshift

AbeBooks (an Amazon subsidiary)

Migrating a marketplace off Oracle, PL/SQL and Crystal Reports onto Redshift and Tableau.

Amazon Redshift Matillion ETL Tableau AWS EMR +4 more
Read the project Consumer technology · internal data products

Product Analytics for a Voice Assistant

Amazon Alexa — Natural Language Understanding

Replacing manual CSV extracts with an automated Redshift and Tableau product analytics stack.

Amazon Redshift Redshift Spectrum Tableau Amazon SageMaker +1 more
Read the project E-commerce · machine learning

Feature Engineering for Machine Learning

Amazon Retail

Building ML feature pipelines from a core data lake — with no warehouse, no data model and no dashboards.

Apache Spark Scala AWS EMR Amazon S3 +3 more
Read the project