LLM Researcher
PointFiveAI Efficiency company
Tel Aviv, IsraelLead
Accel
Salesforce Ventures
Entrée Capital
Data & AI
About the role
TL;DR
Design evaluation methodologies for API-based coding agents.
- •PointFive is seeking a Research Scientist to develop next-generation evaluation methodologies for API-based coding agents.
- •This role involves systems research, benchmarking, and applied machine learning to enhance the accuracy, efficiency, and cost-effectiveness of AI coding systems without requiring model weights.
- •Key Responsibilities Design rigorous evaluation methodologies for API-based coding agents.
- •Build statistically sound benchmarks for cost, latency, accuracy, and reliability.
- •Develop novel techniques for optimizing agent context, retrieval, and tool interactions.
- •Design and analyze large-scale experimental campaigns using production APIs.
- •Build reusable research infrastructure, datasets, and evaluation frameworks.
- •Requirements PhD or equivalent research experience in a quantitative field.
- •Strong background in experimental design and statistical analysis.
- •Excellent programming skills in Python and/or Go.
- •Deep understanding of modern LLMs, retrieval systems, or AI agents.
Required skills
PythonGoLLMs
Nice-to-have skills
OpenAI APIAnthropic APIGemini APILangChainComputer VisionNLPKubernetesDockerAWS
Domain expertise
developer-toolsaiproductivity
Tech stack
PythonGoPyTorchOpenAI APIAnthropic APIGemini APIKubernetesDockerAWSGitHub ActionsPostgreSQL