Senior Software Engineer – Ruby (LLM Evaluation & Repository Validation)

About this opportunity

Turing is seeking experienced software engineers familiar with high-quality public GitHub repositories to contribute to LLM evaluation and training datasets. The role involves hands-on software engineering work including development environment automation, issue triaging, repository setup, Dockerization, evaluating unit test coverage and quality, and modifying and running real-world codebases to assess LLM performance in bug-fixing scenarios.

What the listing describes

• ✓Analyze and triage GitHub issues across trending open-source libraries
• ✓Set up and configure code repositories, including Dockerization and development environments
• ✓Evaluate unit test coverage and quality
• ✓Modify and run codebases locally to assess LLM performance in bug-fixing scenarios
• ✓Collaborate with researchers to design and identify repositories and issues that are challenging for LLMs
• ✓Contribute to LLM evaluation and training datasets based on realistic software engineering problems
• ✓Lead junior engineers on projects when required

Requirements shown on the source

• ✓Minimum 3+ years of overall software engineering experience
• ✓Strong experience with Ruby
• ✓Proficiency with Git and Docker
• ✓Basic software pipeline setup experience
• ✓Ability to understand and navigate complex codebases
• ✓Comfortable running, modifying, and testing real-world projects locally

Relevant backgrounds mentioned

• ✓Experience contributing to or evaluating open-source projects
• ✓Previous participation in LLM research or evaluation projects
• ✓Experience building or testing developer tools
• ✓Experience with automation agents
• ✓Experience leading junior software engineers