All projects Video games · player analytics

Delta Lake for Gaming Analytics

Microsoft Xbox — The Coalition

Moving a game studio off on-premise SQL Server and Hive onto an Azure Databricks lakehouse.

Azure Databricks Delta Lake Power BI Azure Data Factory Azure Bicep Azure DevOps MLflow

The team

  • 2 Data Engineers
  • 1 BI Engineer
  • 2 Data Scientists
  • 1 Senior Manager, Analytics

Starting state

  • HDInsight and Hive for staging
  • On-premise SQL Server and SSIS for fact and dimension tables

Use cases

  • Player behaviour and engagement
  • Monetisation
  • Game balance
  • Skin, weapon and feature usage
  • Multiplayer balance

Architecture

Azure architecture: Xbox and PC telemetry into Databricks processing, Delta Lake bronze silver and gold, and Power BI for business users
Telemetry into Databricks, a bronze/silver/gold Delta Lake, and Power BI for five distinct business audiences.

What was built

  • Hourly batch from Azure Storage into Delta Lake
  • Bronze and silver layers in append mode, with minimal and then schema-level transformation
  • Gold fact tables built with MERGE, or INSERT plus DELETE
  • Power BI for reporting
  • Azure Bicep for infrastructure and Azure DevOps for CI/CD
  • MLflow and Azure Data Factory orchestration in the service layer

What changed

  • Staging and modelling consolidated onto one lakehouse instead of Hive plus an on-premise SQL Server.
  • The medallion split made the transformation cost explicit: minimal in bronze, schema work in silver, heavy logic in gold.
  • Append-mode bronze and silver left the door open to Structured Streaming without redesigning the pipeline.
  • Infrastructure and deployment moved into code with Bicep and Azure DevOps.
Related service

Databricks Consulting

Lakehouse architecture, Unity Catalog, and Spark pipelines built for production.

Explore service

Let's talk about your data platform

Tell us what you are working on. We reply within one business day, and we will tell you plainly if we are not the right fit.

Start a conversation Email us

Other projects

E-commerce marketplace · 400 employees

Oracle Data Warehouse to AWS Redshift

AbeBooks (an Amazon subsidiary)

Migrating a marketplace off Oracle, PL/SQL and Crystal Reports onto Redshift and Tableau.

Amazon Redshift Matillion ETL Tableau AWS EMR +4 more
Read the project Consumer technology · internal data products

Product Analytics for a Voice Assistant

Amazon Alexa — Natural Language Understanding

Replacing manual CSV extracts with an automated Redshift and Tableau product analytics stack.

Amazon Redshift Redshift Spectrum Tableau Amazon SageMaker +1 more
Read the project E-commerce · machine learning

Feature Engineering for Machine Learning

Amazon Retail

Building ML feature pipelines from a core data lake — with no warehouse, no data model and no dashboards.

Apache Spark Scala AWS EMR Amazon S3 +3 more
Read the project