Senior Software Engineer – C# (LLM Evaluation & Repository Validation)

About this opportunity

Turing is seeking experienced software engineers at a tech-lead level who are familiar with high-quality public GitHub repositories to contribute to an LLM evaluation and training project. The work focuses on creating verifiable software engineering tasks from public repository histories and expanding dataset coverage across programming languages and difficulty levels. Engineers will analyze and triage GitHub issues, configure repositories and development environments, evaluate unit test coverage and quality, modify and execute real codebases locally, and collaborate with researchers to identify challenging software engineering problems for LLMs.

What the listing describes

• ✓Analyze and triage GitHub issues across trending open-source libraries
• ✓Set up and configure software repositories and development environments
• ✓Dockerize code repositories and configure required environments
• ✓Evaluate unit test coverage and assess test quality
• ✓Modify real-world codebases locally to reproduce and investigate software issues
• ✓Run and test codebases locally to assess LLM performance in bug-fixing scenarios
• ✓Analyze realistic software engineering problems represented in public repositories
• ✓Collaborate with researchers to identify repositories and issues that are challenging for large language models
• ✓Contribute to the development of verifiable software engineering tasks for LLM training and evaluation
• ✓Help expand evaluation datasets across different task types and difficulty levels
• ✓Evaluate repository histories and software changes for use in benchmark tasks
• ✓Collaborate with other engineers on technical evaluation projects
• ✓Potentially lead a team of junior engineers working on project tasks

Requirements shown on the source

• ✓Minimum 3 years of overall software engineering experience
• ✓Strong professional experience with C#
• ✓Proficiency with Git
• ✓Proficiency with Docker
• ✓Basic understanding of software development pipeline setup
• ✓Ability to understand and navigate complex codebases
• ✓Comfortable running real-world software projects locally
• ✓Ability to modify and test existing codebases
• ✓Ability to analyze software bugs and GitHub issues
• ✓Strong software engineering and debugging skills
• ✓Ability to evaluate unit test coverage and quality
• ✓Ability to work with public GitHub repositories and open-source codebases
• ✓Strong analytical and problem-solving skills
• ✓Ability to collaborate with AI researchers and engineering teams
• ✓Ability to work independently in a remote environment
• ✓Availability for at least 4 hours per day and a minimum of 20 hours per week
• ✓Ability to provide 4 hours of overlap with PST working hours
• ✓Must be located in India, Pakistan, Nigeria, Kenya, Egypt, Ghana, Bangladesh, Turkey, or Mexico

Relevant backgrounds mentioned

• ✓Experience contributing to open-source software projects
• ✓Experience evaluating open-source projects
• ✓Previous participation in LLM research projects
• ✓Previous experience with LLM evaluation
• ✓Experience building developer tools
• ✓Experience testing developer tools
• ✓Experience developing automation agents
• ✓Experience evaluating AI coding agents
• ✓Experience with software engineering benchmarks
• ✓Experience leading or mentoring junior software engineers