AI Data Researcher

Own the quality of the LLM and Retrieval-Augmented Generation (RAG) evaluation system Run structured LLM and RAG evaluations and perform root-cause analysis Distinguish between retrieval, ranking, source, prompt, model, and citation failures Maintain evaluation datasets, including quality reviews, versioning, coverage, and new production cases Refine prompts, judge rubrics, and LLM-as-a-judge workflows Design controlled experiments with clear baselines, metrics, and success criteria Investigate practical solutions to recurring RAG challenges Automate repetitive analysis (e.g., Python, SQL, notebooks, APIs) Communicate findings and recommended actions to Product, Engineering, and leadership

4+ years of data research, applied analytics, or evaluation experience Experience with LLMs, prompting, RAG, and retrieval concepts Proficiency with Python Proficiency with SQL Experience in maintaining high-quality datasets or evaluation programs Ability to navigate ambiguous problems independently Strong communication skills in both spoken and written English

Startup experience Experience with LLM-as-a-judge evaluation Familiarity with (LangSmith, Ragas, DeepEval, Braintrust, Evidently, etc.) Knowledge of embeddings, search, reranking, or hybrid retrieval Experience creating evaluation rubrics, dashboards, or lightweight automation Bachelor's Degree in Computer Engineering, Computer Science, or equivalent