AI/ML Engineer · Evaluation, Retrieval & Reliability

Most model failures are silent.

I work on the parts of a machine-learning system that decide whether you can trust its output: evaluation design, retrieval grounding, calibration, and the tests that catch a model degrading while every dashboard still reads green.

Interview My Work ✦ See the work Get in touch

Evidence ledger pinned to commit · re-checked in CI every week

38 Verified claims

Each checked against code, a committed data file, or a test, not against a résumé line.

7 Repositories

Every claim traced to a specific file at a specific commit, so it cannot drift after you read it.

2026-09-25 Last verified

A build fails if a number here stops matching its source, or a source link stops resolving.

Ask it anything about the work, or paste a job description and see each requirement mapped to the evidence behind it. Interrogate it →

Selected work

Six systems, and what each one got wrong

Provenance is stated on every entry. The surgical-video capstone is marked as a fork, where the upstream commits are my teammates'. Fundproof is a two-person build that Claude Code implemented from specs we wrote, and its numbers come from SYNTHETIC data. One is still being built and is not public yet, so you cannot check it. I'd rather you hear all of that here than notice it on GitHub.

Two-person build, fork 9 verified claims Oct 2026 - Present

Fundproof

LangGraph, MCP Python SDK, Monte Carlo, Tesseract OCR, FastAPI, Next.js

A two-sided lending copilot for small-business loans. The lender side turns a document packet into a cited, stress-tested recommendation that a person decides on; the borrower side simulates cash and debt service coverage under the lender's stress test. It is a two-person project: I co-wrote the specs, build prompts and evaluation protocol and review the work, and Claude Code implemented it.

The finding: a narrow gain. On 2,400 SYNTHETIC worlds drawn from its own model, the Monte Carlo feasibility simulator (1,000 paths over 36 months) scored a Brier of 0.1405 against 0.1567 for a constant base rate. The worlds come from a generator the same team wrote, so this checks internal consistency, not accuracy on real businesses.

A LangGraph agent stopped at a human checkpoint on 65 of 65 SYNTHETIC applications with 0 decisions recorded without a person, but its end-to-end routing matched the designed labels on only 49 of 65. An MCP server exposes 4 read-only tools on the same allow-list and passed 17 of 17 smoke checks over stdio and streamable HTTP. The document extractor reached field F1 0.9159 on scanned pages of 60 SYNTHETIC packets, and 36 SYNTHETIC founder packages read back 30,681 numbers with 0 untraced, which the template guarantees by design.

Where the checker is blind: it caught 627 of 627 planted number errors with 0 of 290 false flags, but only 88 of 108 comparator flips and 6 of 55 wrong-row values. It was debugged on the same sentences, so this is not a held-out test.

Caveats: every number above is measured on SYNTHETIC data. It is not deployed and has no users, no hosted LLM was called, and memos come from a template. The web app passed 5 of 5 Playwright flows on a local container only. Most results files were written from code with uncommitted changes, and several predate later code changes and were not re-run.

Repository Feasibility results Checker results Ask about it

Sole author In progress · not yet public Sept 2026 –

HoldLine

FastAPI · Anthropic API · SQLite · pytest · 13,138 lines · 520 tests

A voice agent that places real, time-limited holds on bakery inventory over the phone. The problem it exists to solve is that an assistant which sounds confident about stock it has not checked is worse than no assistant, because someone drives across town for a cake that was never there.

The design decision: every response is validated after generation against the tool results from that same turn, and violations are tiered by consequence. A wrong quantity gets one retry. An allergen claim is never retried and never softened, it hard-blocks to a human. The two are not comparable mistakes, so they do not get the same handling.

Testing it against the live model found four defects that 444 passing offline tests had not, and three of those were structurally invisible offline. "And no allergies" escalated as an allergy. "Twenty minutes" was read as containing a nut allergy. That is the argument for a live pass at the end of every phase rather than trusting a green suite.

Still being built. Four of nine phases are done; the phone layer and the LangGraph port are not. The repository is private, so nothing here is open for you to check. I verified these figures against the working tree rather than copying them off a résumé, but you cannot, and I would rather say that than imply otherwise.

Ask about it →

Sole author 11 verified claims Aug 2026

Silent Failure Detection

Python · scikit-learn · SciPy · import-linter · pytest · 8,209 lines · 261 tests

A credit-default model, frozen after training on 2013 H1, deployed across 35 monthly windows of Lending Club originations covering 1,054,948 loans, monitored with unsupervised signals only, then graded against labels that took two years to arrive.

The finding: AUC held flat (0.6719 → 0.6661) while Brier rose 17% and the calibration gap widened 27-fold. A team defining degradation as an AUC drop would have concluded nothing happened. Meanwhile the model was under-stating portfolio default risk by about 36% at its worst, and pricing consumes the probability, not the rank.

The headline number the project was built to produce was early-warning lead time. The answer was zero. Three of six detectors fired on the first window and every window after; one never fired at all. A detector that alerts on 100% of windows has maximum nominal lead time and no information, so I wrote pre_onset_alert_rate specifically to make that unquotable as a success.

It also retracts its own headline. An earlier write-up reported the estimators as anti-correlated with truth (r = −0.38, p = 0.023). Both series are autocorrelated; after correction the effective sample size falls from 35 to 13 and p becomes 0.195. The claim did not survive, so the repository says so, in the commit that removed it.

Repository The import contract Findings log X-Ray this →

NSF NRT Research-A-Thon · 4th place 7 verified claims 1 needs verification Apr–May 2026

ClinIQ

Claude · pgvector · LangGraph · FHIR R4 · UMAP · HDBSCAN · Docker

Substance-use risk detection over 52,184 public patient-written drug reviews, built for the NSF NRT Research-A-Thon. Three retrieval tiers were run against the same labelled set: rules, embedding cosine similarity, and Claude over pgvector.

The finding: no tier won. Rules scored 0.86 precision / 0.85 recall in 0.05 s. Embeddings reached perfect recall at 0.50 precision, with 295 false positives. Claude+RAG reached 0.94 precision but 0.40 recall, and took 29 minutes. The cheapest tier was the most balanced, and the expensive one bought precision by missing more than half the cases.

Flagged reviews were clustered into 37 behavioural cohorts with UMAP and HDBSCAN over 384-dimensional embeddings, with the share of low-rated reviews tracked across the 2008–2017 opioid-crisis decade. Every embedding stores its source_url, so any answer can be traced to the CMS or ICD-10 document it came from.

Since extended: FHIR R4 ingestion (118,868 synthetic-patient resources in 3.8 s, zero PHI leaks, hash-chained audit log); a LangGraph prior-authorization agent where the engine decides and only a clinician can deny (89/89 missing-evidence cases paused); and an active-learning labelling workbench that matched 95% of full-data precision with 1,300 labels instead of 10,000.

A number I reported was wrong, and here is what replaced it: the SUD proxy label matched "meth" as a substring, so methylphenidate, sulfamethoxazole and dextromethorphan all counted as substance-use related. 803 of 3,316 flagged reviews, 24%, qualified only because of that. A human labelling pilot surfaced it by disagreeing with the proxy more than anyone expected, and someone read the disagreements. Corrected, the trend is 2.0% to 24.8% over 2,513 reviews, not 1.7% to 30.5% over 3,316. I had quoted the old pair.

Caveats: the 37 cohorts predate the label fix and were not re-run. Prior-auth policies and patients are synthetic. The 4th place is self-reported.

Repository Method comparison (CSV) Schema X-Ray this →

Team course project 2 verified claims Mar–May 2025

Explainable Churn Prediction

BERT · T5 · XGBoost · SHAP · scikit-learn

A fine-tuned BERT classifier reaching 0.793 accuracy and macro F1 0.73 on 1,409 held-out customers, chained to T5 so a structured customer row becomes a plain-language risk narrative a retention team can act on without reading a model score. SHAP over XGBoost ranked the drivers: contract type, tenure, monthly charges.

Worth saying: macro F1 of 0.73 against 0.79 accuracy is the gap between the two classes. The model is noticeably weaker on churners, which is the class that matters. The notebook also contains a weaker 0.77 run, and the reported figure is the better of two.

Repository Generated summaries X-Ray this →

Team capstone · fork 3 verified claims 1 needs verification Jan–May 2026

Kinematics-Conditioned Surgical Video

Stable Diffusion 1.5 · Diffusers · LoRA (peft) · JIGSAWS · OpenCV

Stable Diffusion 1.5's CLIP text encoder replaced with a custom KinematicEncoder that maps 76-dimensional robot kinematics into cross-attention, fine-tuned with LoRA rank 4 on the JIGSAWS suturing dataset.

The question that makes it interesting: is the model tracking its conditioning signal, or just reproducing the scene average? The evaluation scores each generated frame against its true frame and against a different real frame. If both score alike, the conditioning is doing nothing. That cross-pair baseline is the part I'd defend, though it was only run over four pairs, which is too few to claim much.

Why there are no numbers here: the PSNR and SSIM figures for this model live in a team report rather than in anything committed to the repository, so I don't state them as results. The evaluation code that produces them is in the repo and is linked below.

Repository Cross-pair evaluation X-Ray this →

Experience

Where the work happened

Employer work is internal and has no public artifact, correctly so: at Hippocratic AI the underlying material is production healthcare conversation data, and my current work is confidential. It is marked self-reported throughout.

Oct 2026 - Present
Remote

Handshake AI

Software Engineer Fellow

Author complex programming and system-design challenges that benchmark the reasoning and code generation of leading AI models.
Evaluate AI-generated code for correctness, computational efficiency and reliability, diagnosing defects, flawed reasoning and architectural weaknesses, and judge production readiness from backend and data engineering experience (Python, FastAPI, REST APIs, SQL, PySpark, BigQuery, Airflow).
Develop visual question answering (VQA) datasets for multimodal vision-language models: image-grounded question-answer pairs with verified ground truth.
Run rubric-based, human-in-the-loop evaluation of model outputs across code, audio, image and text, with structured feedback on accuracy, hallucinations and failure modes that informs model training.

Self-reported. Started in October 2026. The work is confidential, so there is no public artifact, and there are no figures to report yet.

Jun 2025 – May 2026
Remote, USA

Hippocratic AI

AI/ML Engineer Intern

Developed and improved production healthcare voice AI agents built on Hippocratic AI's Polaris constellation architecture, supporting post-discharge patient conversations across a multi-hospital health system.
Ran root-cause analysis on 300+ flagged production conversations and built 240+ LLM evaluation scenarios and regression datasets from the failure patterns, with clinical, product and engineering teams.
Improved RAG retrieval (chunking, embeddings, reranking, grounding checks), tool-call argument validation, escalation and fallback logic, and the real-time ASR → LLM → TTS voice pipeline, alongside LLM supervisor models and clinical safety guardrails.
Reported outcomes: 8 regressions caught pre-release, repeat failures in the largest category down 40%, grounded-response accuracy up 12%, invalid tool calls down 70%, missed escalations down 45%, p95 latency down 25%.

Self-reported. These six figures are employer-internal and cannot be independently verified from any public source. The capability areas are real and worth interviewing on; the numbers are my account of them. The comparable public evidence for the same skills is Silent Failure Detection and ClinIQ's grounding layer.

Jan 2023 – Jun 2024
Hyderabad, India

Yrall Media Solutions

Data Engineer

12+ production ETL/ELT pipelines in Python and SQL processing 1.5M+ records daily from 8+ sources, on Google Cloud Platform with BigQuery and Cloud Storage, orchestrated as Airflow DAGs with scheduling, retries and incremental loading.
Optimised SQL and transformation logic for roughly 28% faster execution, and automated 10+ recurring validation and reporting tasks.
Record-level lineage tracing each reported figure to its source row, plus CI data-quality gates, reducing recurring issues by roughly 25% across 5M+ historical records powering Power BI for 200+ campaigns.

Self-reported. No public repository. Worth noting that the same instinct, make every output traceable to its source and fail the build when it drifts, shows up independently in code you can read: ClinIQ stores a source_url per embedding, and Silent Failure Detection enforces its module boundaries in CI.

Nov 2021 – Dec 2022
India

Movate

Data Analyst Intern

Extracted and integrated CRM and internal system data with SQL into unified reporting datasets with quality checks, and delivered Tableau KPI dashboards.

Self-reported. No public artifact, and this role is not yet on my LinkedIn, so the two sources do not currently agree.

Aug 2025 – May 2026
Kansas City, MO

UMKC School of Science and Engineering

Graduate Student Ambassador

Segmented a ~300-student international cohort in Excel, surfacing enrollment trends that fed recruitment strategy; ran a student interview series.

Method

How I work

Compare against a baseline

A number means nothing without something to beat. ClinIQ ran rules before it ran an LLM; SS-SD scores generated frames against the wrong real frame to see whether the conditioning matters at all.

Make the check structural

If an evaluation can be invalidated by a careless import, a policy in a README will not hold it. The estimator in Silent Failure Detection is blocked from reaching its own labels at the import graph, and the boundary is a test.

Ask what the processing removed

One bug appeared three times in one codebase: a safety check computed downstream of a transform that strips the evidence it looks for. Clipping the outliers before measuring outliers makes the guardrail decorative.

Report the negative result

The lead-time number I set out to produce was zero, and that is what the repository says. A finding that does not survive its own significance test gets retracted in a commit, not quietly reworded.

Toolkit

What's actually demonstrated

Chips reflect the strongest public evidence for each item, not self-assessment. There are no percentage bars here because I cannot evidence a percentage.

Retrieval & LLM

v RAG pipelines with cited sources
v pgvector, 384-dim embeddings
v Anthropic Claude API
v Baseline ladders (rules → embeddings → LLM)
? Agent orchestration, tool calling

Evaluation & reliability

v Calibration, Brier, drift detection
v Regression suites (261 tests)
v Serial-correlation-robust statistics
v Negative-result reporting
? LLM eval datasets at scale

Modelling

v PyTorch, fine-tuning BERT & T5
v scikit-learn, XGBoost, SHAP
v UMAP, HDBSCAN
v Diffusion + LoRA (peft)

Data & platform

v PostgreSQL, SQL
v Docker, docker-compose
v GitHub Actions CI, pytest
? Airflow, dbt, GCP BigQuery (employer work)
? AWS

Have a specific role in mind?

Paste the job description. You'll get each requirement mapped to direct evidence, related evidence, or nothing, with links. No match percentage, because I can't evidence one.

Interview My Work runs in the browser. With JavaScript disabled it won't open. The evidence itself is still readable as claims.json and CONFLICTS.md, which is where every answer comes from.

Evaluate my experience →