What We Learned Building Our ML Platform
Six ideas about feature stores, AST-generated ETLs, content-addressed artifacts, and control planes — distilled from building a semantic layer for ML.
Read articleBali · Asia/Makassar · Available for fractional engagements
Airflow DAG Bundles: Managing DAGs Across Teams Without Helm Upgrades
Using S3 DAG bundles, a sidecar sync pattern, and a bundle watcher to onboard pipelines without Helm upgrades or downtime.
Read article
Building a Composable ETL Framework for Spark
Replacing bespoke PySpark scripts with a config-driven, hook-based framework inspired by Rust's composition model.
Read article
Taming S3 Shuffle at Scale
Fixing the GET request explosion, prefix throttling, and threading edge cases that emerge when S3 shuffle meets production scale.
Read article
Spark on Spot with S3 Shuffle
How S3 shuffle supports Spark executors on 100% spot capacity with Karpenter while cutting compute costs 70-85%.
Read article
Running Spark on Kubernetes: An Ops-First Approach
An ops-first pattern for production Spark on Kubernetes with one SparkApplication template, pre-baked images, and sub-10-second warm starts.
Read article
From 60 Minutes to 4: Optimizing Spark MERGE INTO on a 2 Billion Row Iceberg Table
Optimizing a daily Iceberg upsert from an hour to under 4 minutes with storage partition joins and shuffle hash hints.
Read article
Building Our Lakehouse with Apache Iceberg
A modern lakehouse architecture using Apache Iceberg, Lakekeeper, Spark, Trino, and AWS.
Read article