Freelance Senior Software Engineer
TolokaAI Training company
RemoteSenior
Bezos Expeditions
Nebius Group
Mikhail Parakhin
Software Engineering
About the role
TL;DR
Create tasks and criteria to evaluate AI coding agents.
- •We're building a dataset to evaluate AI coding agents
- •how well a model handles real-world developer tasks.
- •Essentially, we're building an olympiad for AI models: the better the tasks, the more precisely we see where models are weak, and the faster they improve.
- •Key Responsibilities Create tasks and the criteria to evaluate them.
- •Build an isolated environment
- •an emulation of a developer's workstation.
- •a prompt, just like in real work.
- •Develop tests and evaluation criteria to verify the agent solved the task correctly.
- •Requirements Experience in AI/ML and software engineering.
- •Strong problem-solving skills and attention to detail.
- •Ability to work independently and manage time effectively.
- •Excellent communication skills.
- •Familiarity with Linux and development tools.
Required skills
PythonJavaScriptNode.jsLinuxDockerKubernetesCI/CD
Nice-to-have skills
ReactGit
Domain expertise
ai
Tech stack
LinuxPythonJavaScriptReactNode.jsCI/CDDockerKubernetesGit