Case studies

The hardest problems I've worked on

LLM agents, GPU inference, agent hosting and data sync. For each one: what broke, why it was hard, what I changed, and the numbers before and after.

Tech lead of the LUUK AI platform · MSc in AI (in progress) · Based in Bali (UTC+8) · Open to relocation

01LLM evaluation · Agent designLUUK AI · Sep–Oct 2026

Testing an LLM agent that never answers the same way twice

A 310-test answer-quality suite on frozen production data, then a rewrite of the agent that the suite could prove.

9–13% → 0.2–0.5%
empty answers
Top: production database, frozen snapshot, real agent turn run 3 times, code checks, then an LLM grader with miss, uncertain and pass bands. Bottom: before, an answer plus up to three LLM rewrite passes; after, a tool loop, one write, guards in code and a streamed answer.Open the diagram full size

The problem

Our sales assistants answer shoppers and brokers by calling tools over a live catalogue and knowledge base. After the first answer, up to three more LLM passes rewrote it (reflection, reconcile, translate), and each could change facts. Nobody could say whether a change made answers better or worse: the old tests checked helper functions on an old data dump and stayed green while the agent was broken.

Why it was hard

  • The same question gets a different answer on every run, and models differ in how they use tools.
  • Code can't judge meaning, and an LLM judge is noisy too.
  • The test must run the agent exactly as deployed, on data that doesn't drift, while production data drifts and every re-sync changes ids.
  • A full run is about 900 real conversations, so it costs real tokens, and network errors must not count as failures.

What I did

  1. 1Froze production data: read-only exports with a sha256 and provenance for every file. A refresh maps pinned ids to their new ids.
  2. 2The harness runs the real turn on the real model with the tenant's own settings, and I checked that it matched production. At first it didn't: a different token budget, search threshold and dictionary.
  3. 3Code checks come first (ids, tools and numbers precomputed from the snapshot); an LLM grader judges meaning only, with an "uncertain" band. A test passes only if all 3 runs pass, wording is never matched, and network errors are retried, never failed. About 20 minutes on 8 shards.
  4. 4Then rewrote the turn: a tool loop and one write, guards in code (it can only recommend ids the tools returned), the round budget stated in the prompt, and no fallback that hides an empty answer. An A/B against production proved it before the merge.

Results

310
real-model tests across 56 client scenarios, 3 of 3 runs to pass
9–13% → 0.2–0.5%
turns ending in an empty answer
−34%
model calls vs production in an A/B, 6 questions × 3 runs (embedding calls −82%)
12.9 → 7.0 s
p90 latency in the same A/B

Who did what: I designed the suite, its rules and the rewrite. A colleague mapped the client's scenarios, AI coding agents wrote many of the tests under my rules, and the catalogue search is a colleague's.

PYTHONFASTAPILANCEDBSQLITEGITHUB ACTIONSSSE
02GPU inference · Load balancingLUUK · Sep 2026

Getting a GPU vision API ready for a partner launch

From a service that went down at one evening's peak to four GPU servers and 393 analyses a minute, measured through the public URL.

37% → 95%
users with a full result
Clients, then Caddy routing to the least busy process, then four GPU servers running 31 analysis processes, some CPU-only, over a shared S3 mount with a watchdog. Inset: admission, lock and thread pool, ONNX Runtime at 3.6 GB. Bars: 55–68 analyses per minute on one server, 393 on four.Open the diagram full size

The problem

A face-photo skin-analysis API (the team's ONNX models behind FastAPI) was about to be embedded by a large partner. It had already failed at one evening peak: the proxy marked both backends down for 43 minutes, and only about one upload in five got a result. The servers weren't dead. Each analysis ran for 2.5–5 s on the event loop of the only process, which then couldn't answer health checks.

Why it was hard

  • Per-photo state lived in shared module objects, so the analysis couldn't simply run in parallel inside one process.
  • cuDNN's default algorithm search took about 11 GB of GPU memory on the first photo; two processes starting together ran out of memory.
  • Results lived on a shared S3 mount that cached "not found", and two servers appending to one log object hung a whole server for 13 minutes.
  • A mixed fleet, from a T4 to two A5000s, and a proxy that reused connections uvicorn was closing: 2–3% of uploads got a 502.

What I did

  1. 1Rebuilt production on an idle backup GPU server (same code, config and proxy settings) and replayed the same arrival pattern before and after every fix.
  2. 2Moved the analysis into a thread pool behind a per-process limit, with a bounded queue that answers 503 with Retry-After at once, so health checks always get through.
  3. 3Tuned the CUDA provider (heuristic cuDNN search, no max workspace, memory that grows only on request) and sized processes per GPU from measured memory. In-process parallelism gave +10–15% for +2.4 GB, so I dropped it.
  4. 4One process per port, routed to the least busy one, with keep-alive below uvicorn's timeout and result reads that try servers in order. Per-server logs and a watchdog that remounts S3 in under 2 minutes.

Results

37% → 95%
users with a full result, same load replayed on a copy of production (13% → 87% at the peak rate)
55–68 → 393
analyses per minute through the production URL, 1 → 4 GPU servers
11 → 3.6 GB
GPU memory per process; first analysis 38 s → 2 s
1,999 / 2,000
requests OK at 100 concurrent clients

Who did what: The models are the team's. I led the diagnosis, the scale-out and the load tests, working with an AI coding agent.

PYTHONFASTAPIONNX RUNTIMECUDACADDYSYSTEMD
03Platform · Process supervisionLUUK AI · Mar–Apr 2026

Replacing 25 agent services with one process manager

One manager per environment runs every partner's agent on its own Unix socket. I found and removed the 3.5 s it first added, then cut production over live.

25 → 1
services per environment
Before and after: 25 agent services, each with its own proxy route, replaced by one manager per environment that provisions agents from the admin UI and routes requests to them over Unix socketsOpen the diagram full size

The problem

Every partner's agent was its own FastAPI app, systemd unit, TCP port and proxy snippet copied by hand to an edge server: five manual steps on two machines per agent, and 25 services on the main server alone. Each agent still had to keep its own process, data and settings.

Why it was hard

  • Start, stop or reload one tenant without touching the others; big agents take up to 2 minutes to start.
  • A Python proxy now sits in front of every LLM call. The first version added about 3.5 s per request.
  • One process owns every agent, so a manager bug hits every tenant. One child writing a very long log line could kill the log pipe and restart agents.
  • The cutover had to happen on production with a way back, and hidden coupling surfaced during it: secrets from a symlinked file, config drift, an install killed by the OOM killer, a port still held by an old service.

What I did

  1. 1Wrote the manager: each agent runs on its own Unix socket, crashed agents restart, and a config change reloads only the agents it touches. New agents go live from the admin UI.
  2. 2Removed the 3.5 s: streaming instead of buffering, agents call the supervisor's socket directly (4 proxy passes per request → 2), and uvloop.
  3. 3Made the manager hard to kill: a log reader that caps lines without a newline and never raises, restarts after 3 s, and a stop that terminates, waits 10 s, then kills.
  4. 4Ran the production cutover with a rollback runbook. It surfaced 13 agents disabled in the database but still running; syncing the agents file from the admin UI fixed that for good.

Results

25 → 1
agent services → one manager per environment, now on 3 servers
5 → 1
manual steps to launch an agent (now one admin form)
~3.5 s → noise
proxy overhead per request, found and removed
2.9 → 9.9 GiB
RAM available after the 13 forgotten agents were stopped

Who did what: The manager is mine (14 of 14 commits). A teammate's small routing prototype came first.

PYTHONASYNCIOUVLOOPHTTPXUNIX SOCKETSSYSTEMD
04Backend · ConsistencyReal-estate brokerage platform (pre-launch) · 2026

Keeping a CRM I don't control in sync, without lost updates

A broker form, a webhook and a poll all write to the same cards; one locked write path keeps them consistent.

4 → 1
concurrent submits → CRM leads
Submit lane: broker form, pair lock, CRM lead, local card. Sync lane: webhook and poll, card lock, fresh read, error classification, save of changed columns, letter. Bottom: card statuses mirrored from the CRM.Open the diagram full size

The problem

A broker registers a client's phone in the app. The backend must create a lead in the client's Bitrix24 CRM, create a local card, and keep the card's status (pending, unique, meeting set, meeting held, sold or archived) in line with whatever the CRM's managers do, with no duplicate leads, lost writes or repeated letters to brokers.

Why it was hard

  • Half the lifecycle has no events: the webhook sends deal changes only, so leads and expiry need polling, and the poll and the webhook race on the same card.
  • Writes into the client's CRM are irreversible. It rate-limits, one update took 24 s, and the integration user loses sight of deals that lose their observers.
  • The framework's record API saves every column it loaded, so a broker's comment edit could undo a status change.
  • The client's process changed through seven versions in 17 days.

What I did

  1. 1A lock per broker and phone, with the duplicate check re-run under it: in a test, 4 concurrent submits create exactly one lead.
  2. 2Every sync takes the card's lock before it reads the CRM, and a batch prefetch is used only if no newer read of that card has finished. The webhook checks the local database first, so other agencies' deals no longer fail.
  3. 3Saves write only changed columns, plus a guard on the record API. A control test reproduces the lost update without the guard.
  4. 4Transient CRM errors write nothing; throttled calls retry after 1, 2 and 4 s; letters are planned from the before and after state and sent only after a successful save.

Results

42%
of webhook calls failed on staging before the fix (808 of 1,934); now only a failed local save returns 500
1,001+ → 21
CRM requests to poll 500 cards (fake CRM)
0
duplicate letters after a restart and 4 replayed webhooks, on the live CRM
276
tests in the sync package, run under the race detector

Who did what: I'm the sole backend developer and built this with AI pair-programming: I set the plan and the release gates and ran the race-detector suite myself. The app is another developer's; QA was done by the client's staff.

GOPOCKETBASESQLITEBITRIX24 REST
05LLM applications · Personal projectAgent Coach · May 2026

Long-term memory an LLM can't make up

The model proposes memory edits; code checks every one against the conversation before it's saved.

~180k tokens
of free-text memory, replaced
An exchange goes to a writer LLM, then a validator with six checks, then a memory file with canonical facts and expiring notes; every operation goes to an audit log. Before: free-text append, about 180k tokens.Open the diagram full size

The problem

My Telegram coaching bot keeps long-term memory about its user. The first design let the model append free text after every reply. After two months of daily use the memory, loaded into every turn, had grown to about 180k tokens of stale and duplicated facts, and a small extractor model had invented some.

Why it was hard

  • The writer is an LLM too, so its output changes from run to run.
  • It must tell what the user said from what the assistant suggested.
  • Same-day plans must not become permanent facts, and mistakes compound silently over months.

What I did

  1. 1The model proposes typed JSON operations with an evidence quote. Code accepts one only if the quote appears in the message it cites, facts come from the user, expiry is 1, 2, 7, 14 or 30 days, and there are at most 5 per exchange.
  2. 2Code owns the file: canonical facts plus recent notes that expire into an archive. Superseding never deletes.
  3. 3Every accepted or rejected operation goes to an audit log. I tested the design offline on 12 real exchanges first, then rebuilt the memory by replaying the chat history.

Results

~180k
tokens of free text before the rewrite
8 / 5
operation types; at most 5 per exchange
56
tests, with the LLM stubbed and the clock pinned

Who did what: All mine (66 of 66 commits), built with AI coding agents.

PYTHONTELEGRAM BOT API

War stories

Shorter ones: the symptom, the root cause and the fix.

LLM sync

Five nights of false "sync failed"

Root cause
The nightly knowledge sync went through a proxy with a 600 s timeout, while the ingest finished on the other side after about 22 minutes. I confirmed it from three logs: the proxy config, the proxy log at the run times, and the agent's log.
Fix
Bypass the proxy for that host, and store the agent's own "upserted / deleted" summary on every run. That also exposed "successful" runs that had left the index unchanged for two weeks.
LLM sync

Every save re-ingested everything for 40 minutes

Root cause
Any edit in the knowledge base triggered a full re-ingest (40–45 minutes per agent, PDFs parsed again), edits during a run queued more, and the ingest blocked the agent's only event loop, so chat waited up to 8 minutes.
Fix
Ingest only when a persisted fingerprint of the corpus changes, keep at most one follow-up run, skip PDFs whose bytes are unchanged, and run the ingest off the event loop. An unchanged re-sync now takes 0 s, and feed and index match exactly.
LLM ops

The fastest model was failing

Root cause
In a 10-query benchmark one model looked four times faster than the rest. Its answers were empty: the API rejected a temperature parameter, and the timer was measuring errors.
Fix
Check outputs, not just timers. A cached capability probe per model replaced a hard-coded model list. The real result: median latency 38.9 s → 9.0 s with the model we moved to.
Launch day

The fix only my phone could catch

Root cause
On a partner's launch day about 1 scan in 5 returned a server error. Two bugs were stacked: a crash on an empty face-detection result, and a "retake your photo" reply built with 8 values where the code had read 9 since a model update.
Fix
Reproduced it on a throwaway copy and rolled the fix out one process at a time across 31 processes. Testing on my own phone showed it still failed: the widget recognises one exact error text. A second fix matched it, and 95% of scans succeeded over the next three days.
Release · AI agent

A git pull for test agents broke production for 33 minutes

Root cause
My AI coding agent pulled new code for the test agents into a checkout that the pre-launch production agents also read. One module parses its own source file on every request, so the file on disk stopped matching the code in memory, and every chat got a 500.
Fix
I caught it and reset the checkout, wrote the postmortem the same day, and gave each environment its own code clone with symlink rollback. The version endpoint now reports the code actually loaded.
Go networking

An import that broke only through the proxy

Root cause
Cloning Go's default HTTP transport kept "h2" in the TLS handshake, so the client spoke HTTP/1.1 to a server expecting HTTP/2. On top of that, the image CDN sent GOAWAY around stream 2,000, and a nightly import stopped halfway.
Fix
Pin HTTP/1.1 for that client, retry, and skip a single bad image instead of failing the run. The catalogue import runs in one transaction, so a failure leaves nothing half-applied.

Research

Publication · 2024

Quantum annealing in machine learning: Qboost on D-Wave quantum annealer

Scholar impact

15citations on Google Scholar

Peer-reviewed paper on quantum annealing and machine learning that extends my BSc thesis (I'm the second author, after Hadi Salloum). A QBoost-style ensemble is mapped to a quadratic unconstrained binary optimization (QUBO) formulation and run on a D-Wave quantum annealer. It explores when quantum hardware can complement classical ML pipelines, how experiments are set up on real hardware, and how to read results within device constraints.

  • Connects quantum annealing with ML: QBoost framing executed on D-Wave hardware with a clear experimental narrative.
  • Sits at the intersection of QC and ML—relevant as both fields converge in research and applied work.
  • Growing citation footprint, including a 2026 citation in npj Computational Materials (Nature Portfolio).

Venue: Procedia Computer Science (Elsevier), 2024

MSc research · 2026

Disproving our own result on grokking

48–49% → 100%
held-out accuracy, changing only the learning rate

With my co-author Yazan Kbaili, I studied whether four models trained on disjoint halves of the data, with an agreement loss on unlabeled inputs, can generalise on modular addition at 50% training data, where a single model stalls at about 12%. Our milestone result didn't reproduce: my own refactor had given the distilled model a different optimizer and schedule than its baselines. I traced it, then ran single-variable controls.

  • Pilots 1–4 seemed to hit a 48–49% ceiling. Changing only the learning rate reaches 100% at about 70k steps.
  • My co-author's InfoNCE variant reached 92.2% but changed six settings at once. Under its recipe, a KL-softmax loss alone reaches 100% at about 18.5k steps.
  • Every run on a single Kaggle T4 GPU, in sweeps of 4 to 9 hours.

Also built

Go · PocketBase · 2024

Push-notification campaigns

Campaigns to a topic, a condition or a user list: a delivery record per device, a unique guard per campaign and device, fan-out and status change in one transaction, a single sender, and dry-run sends for the first two weeks.

Python · 2024

Service monitor with Raft leader election

My own implementation of Raft's leader election (terms, randomised timeouts, majority votes, heartbeats), so only the leader checks the services and sends Telegram alerts; the check state rides on the heartbeats.

React → Svelte · 2025–26

Embeddable assistant widget

One script tag on partner sites: a closed shadow root for the launcher and an iframe for the chat, so the host's CSS can't break them; Hebrew right-to-left; typed messages between five windows.

Live widget demo
SSE · 2026

Streaming answers

Server-sent events read over fetch, with a 45-second stall watchdog. A draft streams in and the final answer replaces it in place; a broken stream never sends the question twice.

How I work

Measure first

Evals before the rewrite, load tests before the scale-out, a control test that reproduces a race before the fix.

The model proposes, code decides

LLM output passes checks in code (ids the tools returned, evidence quotes, hard caps) instead of a second model cleaning up after the first.

Fix the class of bug

Root cause from the logs, a postmortem the same day, and a change that stops the whole class: separate release clones, persisted fingerprints, regression tests.

AI agents, with guardrails

I build with Claude Code and Cursor every day. I write the plan and the rules, review every change and run production deploys myself. An agent-caused incident ends in a standing rule.

Hiring for backend or AI-platform work?

I'm open to engineering roles, relocation included, and to remote contracts. Write to me and I'll send my CV.