Real-time Feature Store

PYTHON · KAFKA · REDIS · AWS

Problem

Cut model-serving latency from 400ms to 30ms by rebuilding the feature pipeline on a streaming architecture.

Solution

Replaced the nightly batch job that materialized features into a warehouse table with a Kafka-backed streaming pipeline that writes directly into a low-latency Redis store, so the serving layer reads pre-computed features instead of joining raw events at request time.

Architecture

Event producers publish to Kafka topics partitioned by entity ID. A stream processor consumes each topic, applies windowed aggregations, and writes the resulting feature vectors into Redis with a TTL matched to feature freshness requirements. The serving API reads directly from Redis, falling back to the warehouse only for entities without a cached feature yet.

Challenges

The hardest part wasn't the streaming logic — it was reconciling feature values between the new real-time path and the existing batch path during the migration, since any drift would silently degrade model accuracy. Shadow-writing both paths and diffing outputs for two weeks caught several off-by-one windowing bugs before cutover.

Lessons Learned

Latency wins are cheap to demo and expensive to keep correct. I'd invest in the reconciliation tooling on day one next time, not as an afterthought once the streaming pipeline already worked.

Future Improvements

Add feature versioning so model rollbacks don't require redeploying the pipeline, and expand windowed aggregations to support multi-entity joins without a full warehouse round-trip.