RESEARCH · MOUNTAIN VIEW
Applied AI Engineer
Apply for this job
About Sully.ai
Sully.ai is building the AI operating system for healthcare. Our mission is to make doctors superhuman through AI, so every patient can get faster, better, and more accessible care.
Healthcare is one of the most important systems in the world, yet clinicians still lose hours every day to documentation, scheduling, intake, coding, and administrative work. We are changing that with AI medical employees: Scribes, Receptionists, Coders, Consultants and other agents that operate inside real clinical workflows.
We are backed by Amity Ventures, YC, Baidu Ventures amongst others, have raised $40M+, and are already deployed across hundreds of healthcare organizations. Our products process tens of thousands of clinical encounters every day, automate millions of minutes of administrative work, and help clinicians spend more time with patients.
The role
As an Applied AI Engineer on the Research team, you will work directly with our research and engineering org on the AI medical employees at the core of Sully — Scribe, Receptionist, Consultant, Coder, and what comes next. Your job is to turn the frontier of agent research into agents and eval systems that run in production for real clinicians within days.
This is a builder-researcher role. You read the latest work in agent, harness and context engineering, evals, and reinforcement learning, then ship from it. We do not separate "research" and "engineering" — you are expected to do both, and the work is measured by what lands in front of clinicians, not by a benchmark in isolation. Our agents are architecture- and configuration-driven: most of the behavior comes from decomposition, prompts, routing logic, context assembly, and evals rather than one-off code. You will own the systems that decide what the model sees, measure whether it worked, and make it better on its own over time.
You will own three things: evals, agents, and self-improving loops. Just as importantly, you will help define how Sully does applied research at all — the harnesses, eval infrastructure, and feedback loops that make every future agent faster to build and safer to ship. The environment is high-stakes by design: in clinical AI, accuracy has zero margin for error, and that constraint is the most interesting part of the job.
What you’ll do
Build and scale evaluation systems
Design and scale automated evaluation pipelines (LLM-as-judge + clinician review) with clinical-grade benchmarks for accuracy, factuality, attribution, safety, and instruction-following.
Build the tooling and dashboards researchers use to run evals, and the CI gates that block regressions before they reach patients.
Treat measurement as a science: understand reliability, variance, and scalability of your eval methodology, not just the headline number.
Turn fuzzy notions of "is this good enough?" into clear, defensible metrics that researchers, engineers, and clinical stakeholders can act on.
Build and ship medical AI agents
Design, implement, and ship agents (Scribe, Receptionist, Consultant, Coder, and new ones) on Sully's stack — decomposition, parallel specialists, dynamic output contracts, and per-agent model routing.
Engineer the harness: memory, context compression, retrieval, session/state, and communication patterns between agents.
Own prompts as infrastructure — structured, versioned, composable per specialty and per organization — because without a correction loop as a safety net, prompt quality is load-bearing.
Take a research result to production end-to-end, fast.
Build self-improving and auto-evolving loops
Build closed-loop systems where clinician feedback and production traces drive targeted, evidence-grounded improvements to the specific components that need them — prompt quality that improves weekly, not quarterly.
Build self-improvement loops that automate model understanding and surface where each agent is failing and why.
Stand up continuous evaluation so production traffic is the experiment and the observability layer is t