Senior Software Engineer – Python (LLM Evaluation & Repository Validation)

About this opportunity

Turing is seeking experienced software engineers at a tech-lead level who are familiar with high-quality public GitHub repositories to contribute to LLM evaluation and training datasets. The project builds verifiable software engineering tasks based on public repository histories using a synthetic approach with human-in-the-loop. The role combines hands-on software engineering with AI research and evaluation.

What the listing describes

• ✓Analyze and triage GitHub issues across trending open-source libraries
• ✓Set up and configure code repositories, including Dockerization and environment setup
• ✓Evaluate unit test coverage and quality
• ✓Modify and run codebases locally to assess LLM performance in bug-fixing scenarios
• ✓Collaborate with researchers to design and identify repositories and issues that are challenging for LLMs
• ✓Contribute to LLM evaluation and training datasets
• ✓Lead junior engineers on projects when required

Requirements shown on the source

• ✓Minimum 3+ years of overall software engineering experience
• ✓Strong experience with Python
• ✓Proficiency with Git and Docker
• ✓Basic software pipeline setup experience
• ✓Ability to understand and navigate complex codebases
• ✓Comfortable running, modifying, and testing real-world projects locally

Relevant backgrounds mentioned

• ✓Experience contributing to or evaluating open-source projects
• ✓Previous participation in LLM research or evaluation projects
• ✓Experience building or testing developer tools
• ✓Experience with automation agents