Senior Data Scientist (AI Evaluation & Improvement)

免费注册后解锁"你的匹配分 vs 高分画像"与个人技能缺口。

Your Future Evolves Here Evolent partners with health plans and providers to achieve better outcomes for people with most complex and costly health conditions. Working across specialties and primary care, we seek to connect the pieces of fragmented health care system and ensure people get the same level of care and compassion we would want for our loved ones. Evolent employees enjoy work/life balance, the flexibility to suit their work to their lives, and autonomy they need to get things done. We believe that people do their best work when they're supported to live their best lives, and when they feel welcome to bring their whole selves to work. That's one reason why diversity and inclusion are core to our business. Join Evolent for the mission. Stay for the culture. What You’ll Be Doing: Core Responsibilities Own the diagnostic loop for LLM-based clinical services: take failure modes surfaced by clinical reviewers and product managers, form root-cause hypotheses (prompt design, context assembly, retrieval, guideline encoding, model behavior, upstream data), and design experiments that isolate the cause. Test candidate fixes — prompt and configuration variants, context changes, model alternatives — and verify improvements with structured evaluations, not anecdotes; confirm fixes don't regress other behavior. Own the evaluation roadmap and quality bar for the team's AI services: which metrics gate deployment, how golden sets and regression suites grow, and what "good enough to ship" means in evidence. Use and extend the team's evaluation platform: build golden sets and regression suites, define metrics (accuracy, guideline adherence, grounding/faithfulness), run and interpret eval batteries. Fluency in *using* modern eval tooling matters; the platform exists — extending it thoughtfully is in scope, rebuilding it is not. Partner with clinical reviewers (medical directors) to turn review findings into labeled evidence and executable evaluation criteria; partner with product managers to prioritize which failure modes matter most. Mentor others in evaluation methods and grow the evals function as it scales, including readiness to take direct reports as the team expands. Support the annual clinical-guidelines update cycle with regression evaluation as guidelines, prompts, and models change; ramp with the team's senior data scientists in Q4 2026. Document the diagnostic playbook: failure taxonomies, experiment templates, variant history — a method others can run, not a private intuition. This is a practicing role — the diagnostic loop is the job, at every level of seniority. Handle clinical data (including PHI) according to organizational security, privacy, and compliance requirements. Minimum Requirements Bachelor's degree in Data Science, Computer Science, Statistics, or a related quantitative field — or equivalent experience. 5+ years of data science or applied machine learning experience, including shipping and maintaining models or AI systems in production. 2+ years of recent, hands-on experience evaluating and improving LLM-based systems: structured error analysis, prompt/configuration iteration, experiment design, metrics interpretation — as a practitioner, not only as a reviewer of others' work. Demonstrated experience with LLM evaluation methods and tools — golden/regression sets, LLM-as-judge with validation, tracing and observability tooling. Strong Python and solid data-analysis skills (SQL a plus); comfort computing and reasoning about metrics such as sensitivity, specificity, and PPV. Hypothesis-driven working style: the instinct to isolate variables and prove a fix, rather than tweak and hope. Strong written communication — findings and go/no-go evidence must be legible to engineers, clinicians, and leadership. Preferred Requirements Healthcare experience: utilization management, prior authorization, clinical documentation, or clinical/claims data; comfort reading clinical guideline and medical-policy content. Experien

你的未来在这里发展。Evolent与健康计划和医疗服务提供者合作，为患有最复杂、最昂贵健康状况的人群实现更好的结果。我们跨专科和初级保健开展工作，力求连接碎片化医疗体系的各个环节，确保人们获得我们希望自己亲人所能得到的同等照护与关怀。Evolent员工享有工作/生活平衡、让工作适应生活的灵活性，以及完成工作所需的自主权。我们相信，当人们被支持去过最好的生活，并被欢迎以完整的自我投入工作时，他们才能做出最好的工作。这也是多元与包容成为我们业务核心的原因之一。加入Evolent，为了使命。留下，为了文化。你将做什么：核心职责 负责基于LLM的临床服务的诊断闭环：将临床审查人员和产品经理发现的失败模式，形成根因假设（提示设计、上下文组装、检索、指南编码、模型行为、上游数据），并设计实验以隔离原因。测试候选修复方案——提示和配置变体、上下文变更、模型替代方案——并通过结构化评估验证改进，而非轶事；确认修复不会导致其他行为回归。负责团队AI服务的评估路线图和质量标准：哪些指标决定部署门槛、黄金集和回归套件如何增长，以及在证据层面“足以发布”意味着什么。使用并扩展团队的评估平台：构建黄金集和回归套件，定义指标（准确性、指南依从性、 grounding/faithfulness），运行并解读评估电池。熟练*使用*现代评估工具很重要；平台已经存在——深思熟虑地扩展它属于范围，重建它则不属于。与临床审查人员（医疗总监）合作，将审查发现转化为带标签的证据和可执行的评估标准；与产品经理合作，确定哪些失败模式最重要。在评估方法上指导他人，并随着评估职能扩展而发展它，包括在团队扩张时准备好承担直接下属。支持年度临床指南更新周期，在指南、提示和模型变化时进行回归评估；在2026年第四季度与团队的高级数据科学家一起 ramp。记录诊断手册：失败分类法、实验模板、变体历史——一种他人可以运行的方法，而非私人直觉。这是一个实践性角色——诊断闭环就是工作本身，适用于所有资历级别。根据组织安全、隐私和合规要求处理临床数据（包括PHI）。最低要求 数据科学、计算机科学、统计学或相关定量领域的学士学位——或同等经验。5年以上数据科学或应用机器学习经验，包括在生产中发布和维护模型或AI系统。2年以上近期、亲身实践经验，评估和改进基于LLM的系统：结构化错误分析、提示/配置迭代、实验设计、指标解读——作为实践者，而不仅仅是审查他人工作。展示出在LLM评估方法和工具方面的经验——黄金/回归集、带验证的LLM-as-judge、追踪和可观测性工具。较强的Python和扎实的数据分析技能（SQL加分）；能够计算和推理敏感性、特异性和PPV等指标。假设驱动的工作风格：本能地隔离变量并证明修复有效，而不是调整并希望奏效。较强的书面沟通能力——发现和go/no-go证据必须让工程师、临床医生和领导层都能理解。优先要求 医疗保健经验：利用管理、事前授权、临床文档或临床/索赔数据；能够舒适地阅读临床指南和医疗政策内容。经验

⚑ 岗位信息收录自公开招聘页（Himalayas），以雇主官网为准