Alex Rivera

Real-Time Inventory Sync Engine

Lead Platform Engineer · Apex Commerce · 2023–2024 · 82% less latency (12,000 transactions / sec)

The problem

Stock levels drifted between warehouses and the storefront because each warehouse system pushed nightly batch updates. Customers could buy items that were already gone, and support spent hours reconciling oversold orders.

Constraints

• Peak load of 12,000 inventory transactions per second during sales events
• No downtime for existing warehouse integrations while the new path rolled out
• Every stock change had to be auditable and replayable

What I did

• Replaced batch syncs with an event stream: each warehouse publishes stock changes to Kafka, partitioned by SKU so updates for one item stay in order
• Built idempotent Node.js consumers that write to PostgreSQL with per-SKU version checks, so retries and replays can never double-count
• Ran old and new paths side by side with a nightly diff report until the numbers matched, then switched traffic warehouse by warehouse

Outcome

Sync latency dropped by 82%, overselling during sales events fell to near zero, and any stock level can now be rebuilt by replaying the event log.

What I learned

Ordering guarantees are a design decision, not a library feature. Choosing the partition key early saved weeks of debugging later.

Pseudocode: Ordered, idempotent inventory events

Pseudocode distilled from the production system. Names and details are simplified, and nothing proprietary is shown.

# Warehouse side: publish every stock change as an event
on stock_changed(warehouse, sku, delta):
    event = {sku, warehouse, delta, version: next_version(sku)}
    kafka.publish("inventory", key=sku, value=event)  # same SKU, same partition

# Platform side: one consumer group, partitioned by SKU
for event in kafka.consume("inventory", group="inventory-sync"):
    with postgres.transaction() as tx:
        row = tx.lock_row("stock", event.sku, event.warehouse)
        if event.version <= row.version:
            continue  # a retry or replay: already applied
        tx.update("stock", qty=row.qty + event.delta, version=event.version)
        tx.insert("stock_log", event)  # audit trail, replayable
    kafka.commit(event.offset)  # only after the write is durable

# Storefront: one reconciled number per SKU
def available(sku):
    return postgres.sum("stock.qty", sku=sku)

1. A warehouse records a stock change and stamps it with that item's next version number.
2. The change goes to Kafka keyed by SKU, so every update to one item lands on the same partition, in order.
3. A consumer reads the stream and locks that item's stock row inside a transaction.
4. An old version means a retry or a replay, so it's skipped. That's what makes double-counting impossible.
5. The new quantity and an audit-log entry are written in the same transaction.
6. The offset is committed only after the write is durable, so a crash never loses an update.
7. The storefront reads one reconciled number instead of waiting for nightly batches.

Stack: Node.js, Apache Kafka, PostgreSQL

Multi-Region Distributed Cache Invalidator

Senior Backend Engineer · CloudScale Systems · 2022–2023 · 3 global regions (Multi-region cache invalidation)

The problem

Each region cached customer data independently. After a write in one region, the other two could serve stale data for minutes, which showed up as confusing account states for customers who moved between regions.

Constraints

• Three regions with real cross-region latency, so synchronous invalidation would slow every write
• Redis clusters per region could not be merged
• Invalidation had to survive a region becoming temporarily unreachable

What I did

• Designed a versioned invalidation protocol in Go: every write bumps a key version, and reads reject cached entries older than the latest known version
• Broadcast invalidations between regions over gRPC streams with acknowledgements and a durable retry queue
• Added a staleness metric and alert so drift became visible instead of anecdotal

Outcome

Stale cross-region reads were eliminated in normal operation, and during a regional outage the system degraded to slower, still-correct reads instead of wrong ones.

What I learned

Versioning beat deleting. Making staleness measurable changed the conversation from opinions to graphs.

Pseudocode: Versioned cross-region cache invalidation

Pseudocode distilled from the production system. Names and details are simplified, and nothing proprietary is shown.

# Region A: every write bumps the key's version
def write(key, value):
    version = db.next_version(key)
    db.put(key, value, version)
    cache.set(key, value, version)
    fanout.enqueue({key, version})  # durable: survives an outage

# Fan-out: stream invalidations to the other regions
for msg in fanout.stream():
    for region in other_regions():
        grpc(region).invalidate(msg.key, msg.version)  # retried until acked

# Regions B and C: remember the newest version seen
def on_invalidate(key, version):
    latest[key] = max(latest[key], version)

# Every region: reject cache entries older than that
def read(key):
    entry = cache.get(key)
    if entry and entry.version >= latest[key]:
        return entry.value
    metrics.count("stale_read_avoided")  # staleness is measured
    return refill_from_db(key)

1. A write in one region stores the value with a new version number and updates the local cache.
2. The invalidation goes into a durable queue first, so a region outage delays it but never drops it.
3. A fan-out worker streams it to every other region over gRPC and retries until each one acknowledges.
4. Each region records the newest version it has heard of for that key.
5. On a read, a cached entry is trusted only if it's at least that new.
6. Anything older is refetched and counted, which turned staleness from an anecdote into a graph.

Stack: Go, Redis, gRPC

Zero-Downtime Sharded DB Migration Pipeline

Backend Engineer · CloudScale Systems · 2021–2022 · 450M records (Zero-downtime migration)

The problem

A single PostgreSQL primary held 2.8 TB of live transactional data and was close to its write limits. It had to be split into shards without a maintenance window.

Constraints

• 450 million records, written to continuously
• No downtime and no lost or duplicated writes
• Rollback had to be possible at every stage

What I did

• Built a Python backfill pipeline that copied data shard by shard in throttled batches, coordinated through AWS SQS
• Captured live changes during the backfill and replayed them onto the shards so both copies stayed in step
• Verified each shard with row counts and checksums, then moved reads first and writes last, behind a per-tenant switch

Outcome

All 450 million records moved with zero downtime, and the write headroom bought the team years of growth.

What I learned

Migrations are mostly verification. The copy took days; proving it was correct is what made the cut-over boring.

Pseudocode: Backfill, change capture, verify, cut over

Pseudocode distilled from the production system. Names and details are simplified, and nothing proprietary is shown.

# 1. Backfill: copy history in small, throttled batches
for batch in source.scan(order_by="id", size=5_000):
    sqs.send("backfill", batch.id_range)

on sqs.receive("backfill") as id_range:
    rows = source.read(id_range)
    shard_for(rows).upsert(rows)  # idempotent: safe to retry
    throttle(target_load=0.6)

# 2. Change capture: keep shards in step with live writes
for change in source.changes(since=backfill_start):
    shard_for(change).apply(change)

# 3. Verify before trusting anything
for shard in shards:
    assert shard.count() == source.count(shard.range)
    assert shard.checksum() == source.checksum(shard.range)

# 4. Cut over per tenant: reads first, then writes
for tenant in tenants:
    flags.set(tenant, reads="shards")
    observe(hours=24)  # rollback is a flag flip
    flags.set(tenant, writes="shards")

1. The source table is scanned in ID order and split into 5,000-row batches, each queued on SQS.
2. Workers copy each batch into the right shard. Upserts make retries safe, and throttling keeps production load under 60%.
3. Live changes are replayed onto the shards, so they keep up with writes during the backfill.
4. Every shard must match the source on row count and checksum before it's trusted.
5. Each tenant moves its reads first and is watched for a day. Rolling back is just flipping the flag.
6. Writes move last. 450 million records later, nobody noticed.

Stack: Python, PostgreSQL, AWS SQS

High-Throughput Metric Aggregation Daemon

Software Engineer · Telemetry Labs · 2020–2021 · Low-overhead ingestion (Concurrent metric aggregation)

Engineered high-concurrency ingestion services and low-overhead metric aggregation daemons in Rust at Telemetry Labs.

Stack: Rust, ClickHouse, TimescaleDB

GraphQL Federation Gateway & Schema Stitching

Lead Platform Engineer · Apex Commerce · 2023 · 14 microservices (One federated gateway)

Unified 14 disparate backend microservices with GraphQL federation and schema stitching as part of the Apex Commerce platform modernization.

Stack: TypeScript, GraphQL, Apollo Federation

Real-Time Collaborative State Sync Broker

Software Engineer · Telemetry Labs · 2019–2021 · Collaborative state (CRDT-based synchronization)

Built CRDT-based collaborative synchronization brokers at Telemetry Labs, supporting real-time shared state.

Stack: CRDTs, Distributed systems

Apex Commerce — Lead Platform Engineer

2023–2024

Spearheaded core platform modernization, event-driven inventory reconciliation, and GraphQL federation for an e-commerce platform.

• Cut multi-warehouse inventory sync latency by 82% at 12,000 transactions per second
• Unified 14 backend microservices behind one federated GraphQL gateway

CloudScale Systems — Senior Backend Engineer

2021–2023

Architected multi-region cache invalidation protocols and executed zero-downtime sharded PostgreSQL database migration pipelines.

• Eliminated stale cache reads across three global regions
• Migrated 450 million records (2.8 TB) with zero downtime

Telemetry Labs — Software Engineer

2019–2021

Pioneered low-overhead metric aggregation daemons in Rust and built CRDT-based collaborative synchronization brokers.

• Built high-concurrency metric ingestion and aggregation in Rust
• Shipped CRDT-based brokers for real-time shared state