Work

Eval infrastructure for agentic automation

Zapier's agent platform lets people describe an automation in plain language and have an agent build and run it across nine thousand integrations, for more than 1.5 million users. My job is to make that measurable: to know, before a model or prompt change ships, whether agents actually build the right thing and whether what they build keeps working.

I built the deterministic evaluation environment the team releases against — a state engine with replayable tool APIs that reruns multi-step agent workflows reproducibly, so we can probe agent loops, context overflows and tool-call hallucinations without touching production. On top of it sit end-to-end scenarios grounded in a simulated world with real evidence capture, so a passing eval means the automation did the right thing to the world, not just that the transcript looked plausible. I also designed a progressive-disclosure architecture for LLM tool use — lazy field fetching and step-scoped bindings — that improved task success while cutting execution time and per-run tokens on heavy payloads, and a skill-discovery framework that took execution coverage from twenty integrations to more than six hundred.

What I learned. End-to-end evals find platform bugs at least as often as model bugs. Most "the agent is dumb" reports turned out to be contract gaps between systems, and the eval suite became the fastest way to find them.

Scoping and pricing a deal on the call

Taltrics began as a resource-management platform for operations teams in biotech, manufacturing and professional services: forecasting demand, scheduling people and deliveries, and running scenario plans before committing to one. It was the Genentech problem as a product — models that let an operation plan before reality arrived — and we sold it to a handful of enterprises, including a large biotech.

After two years we noticed where our customers actually lost time. The forecast was rarely the bottleneck. The work around it was: reading the request, scoping it, finding the people, pricing it, and getting a proposal out while the opportunity was still warm. Language models had just become good enough to do that work, so we rebuilt the company around it.

The second product sat with pre-sales teams at professional-services firms. An RFP arrives; a lead gets on a call with the prospect. While they talk, a set of agents reads the RFP, drafts the scoping document, works out resourcing and a schedule against the firm's actual bench, and returns a priced proposal — each agent owning one stage, handing structured state to the next. Work that had taken days of back-and-forth happened in minutes, during the conversation it was meant to inform.

Reliability was the product. A wrong number in a proposal is not a demo glitch; it is a price the firm has just said out loud to a customer. So we collected preference data from real usage, trained reward models, and ran RLHF and GRPO post-training to fit each firm's behavioral policy — which rates exist, what a scope exclusion means, when to ask instead of guess — and measured the drop in hallucinated tool calls and off-policy output. Every model update had to pass a golden-set regression suite that replayed multi-step trajectories against mocked tools and scored the tool-call traces, the entity mutations, and the final proposal.

What I learned. Post-training is only as good as the eval that gates it. The regression suite we built to protect customers became the asset I most wanted to build again, properly — which is what I do now.

Simulating a clinical-trial portfolio

Deciding where in the world to run a clinical trial — which countries, which sites, how many, when — determines how fast a treatment reaches patients. At Genentech I was embedded with global drug-development teams across more than three hundred active trials, seventy countries and a thousand sites, and built the models they planned with.

The core was a discrete-event simulation engine: Poisson–Gamma enrollment dynamics with hierarchical Bayesian rate priors, run at portfolio scale to forecast enrollment curves with uncertainty, rank country and site plans, and size teams by function and phase. A multi-criteria country-selection framework replaced manual deliberation with auditable recommendations and was adopted across therapeutic areas. During COVID the same engine became the crisis-response tool for re-planning trials in real time. Planning costs fell by about a quarter and decision cycles by roughly two-thirds; the dashboards became the primary decision-support surface for development-operations leadership, at around three thousand views a week.

What I learned. A simulation is only trusted if the people using it can see its assumptions. The work that mattered most was making the model's uncertainty legible to non-modelers — the same problem, it turns out, as making an agent eval legible to a product team.

The SDG Data Catalog

Research on the UN Sustainable Development Goals is slowed by data that is fragmented, uneven, and hard to find. With the AI for Good Foundation I led the SDG Data Catalog: an NLP pipeline (BiLSTM-CRF, then fine-tuned BERT) that reads millions of open-access papers and extracts the datasets, organizations and research networks behind them, with provenance and licensing, reaching over 92% recall on dataset discovery.

The work secured more than $300K in Microsoft funding, was presented at SIGKDD 2019 in the Social Impact Showcase, and led to a first-author Springer book chapter and an ACM COMPASS paper. We also built a public tracker of workforce diversity across 1.3M US organizations.

What I learned. Infrastructure that lowers search and coordination costs compounds. It is also almost invisible when it works, which is a lesson for eval tooling too.