All projects Financial technology · trading

Open Source Analytics for a FinTech

One of the largest North American FinTech trading companies

Moving a trading company from BigQuery to an Apache Iceberg and Trino lakehouse on Kubernetes.

Apache Iceberg Trino dbt Airbyte Redpanda Apache Airflow Metabase Kubernetes (GKE)

The team

  • 3 Data Engineers
  • 1 Analytics Engineer
  • 1 Data Director

Starting state

  • Google BigQuery as the warehouse

Use cases

  • BigQuery to Trino migration
  • Financial reporting
  • Product analytics from product logs and Amplitude
  • Reverse ETL to business tools

What was built

  • Apache Iceberg tables with Trino as the query engine
  • Airbyte for ingestion from PostgreSQL, market-maker quotes and product logs
  • Redpanda (Kafka-compatible) for streaming
  • dbt core for business logic
  • Airflow for orchestration, deployed with Helm charts on GKE
  • One Docker container per reverse ETL integration
  • Metabase for BI

What changed

  • A reminder that team size, not tooling, sets the ceiling: four people owning an open-source stack spend a meaningful share of their time on the stack itself.
  • Open source for data solutions tends to slow delivery and raise technical debt — it often means reading, modifying and fixing source code, which is not fast time to market.
Related service

Cloud Data Platforms

Architect and build cloud data platforms that scale to petabytes.

Explore service

Let's talk about your data platform

Tell us what you are working on. We reply within one business day, and we will tell you plainly if we are not the right fit.

Start a conversation Email us

Other projects

E-commerce marketplace · 400 employees

Oracle Data Warehouse to AWS Redshift

AbeBooks (an Amazon subsidiary)

Migrating a marketplace off Oracle, PL/SQL and Crystal Reports onto Redshift and Tableau.

Amazon Redshift Matillion ETL Tableau AWS EMR +4 more
Read the project Consumer technology · internal data products

Product Analytics for a Voice Assistant

Amazon Alexa — Natural Language Understanding

Replacing manual CSV extracts with an automated Redshift and Tableau product analytics stack.

Amazon Redshift Redshift Spectrum Tableau Amazon SageMaker +1 more
Read the project E-commerce · machine learning

Feature Engineering for Machine Learning

Amazon Retail

Building ML feature pipelines from a core data lake — with no warehouse, no data model and no dashboards.

Apache Spark Scala AWS EMR Amazon S3 +3 more
Read the project