Senior Software Engineer - AI Evaluation / Coding Agents ($100-150/hr)

About this opportunity

Turing is looking for experienced hands-on software engineers to evaluate and improve AI coding models. The role involves working with coding agents across real-world repositories, reviewing generated code and agent behavior, determining whether solutions are technically correct, identifying failure modes, and creating evaluation signals and feedback used to improve model performance.

What the listing describes

• ✓Design, refine, and iterate on rubrics and evaluation criteria for preference data
• ✓Review and label data while maintaining a high quality bar
• ✓Identify subtle issues in AI-generated code and agent behavior
• ✓Build and maintain pipelines and infrastructure supporting data generation, collection, and evaluation workflows
• ✓Synthesize findings from data work into clear write-ups, updates, and recommendations
• ✓Collaborate closely with researchers and engineers to translate qualitative judgment into scalable processes

Requirements shown on the source

• ✓5+ years of hands-on software engineering experience
• ✓Strong proficiency in Python, TypeScript/JavaScript, Go, or another major production language
• ✓Experience working in substantial real-world codebases
• ✓Strong code-review skills and technical judgment
• ✓Ability to clearly explain why an implementation is correct, incorrect, or could be improved
• ✓Strong written communication
• ✓Experience using modern LLMs or AI coding tools

Relevant backgrounds mentioned

• ✓Experience with LLM evaluation
• ✓Experience with coding agents
• ✓Experience with RLHF
• ✓Experience with preference data
• ✓Experience with rubric design
• ✓Experience with post-training