Senior Python Developer

About this opportunity

This position is part of a project with a foundational LLM company. The role involves creating high-quality proprietary data for model fine-tuning and evaluation, writing Python solutions for coding tasks, evaluating outputs from different LLM versions, and providing detailed feedback. The role does not involve building or fine-tuning LLMs directly.

What the listing describes

• ✓Design, develop, and maintain efficient, high-quality Python code for AI model training and optimization
• ✓Conduct evaluations to benchmark model performance and analyze results
• ✓Evaluate and rank AI model responses across diverse domains
• ✓Develop comprehensive explanations and rationales for evaluations
• ✓Lead Supervised Fine-Tuning (SFT) efforts, including creating and maintaining task-specific datasets
• ✓Collaborate with researchers and annotators on Reinforcement Learning with Human Feedback (RLHF)
• ✓Design evaluation strategies and processes to improve model alignment
• ✓Create and refine optimal responses to improve AI performance
• ✓Conduct peer reviews of code and documentation
• ✓Collaborate with cross-functional teams to improve model performance and product capabilities
• ✓Explore and integrate new tools, techniques, and methodologies for AI training processes

Requirements shown on the source

• ✓3+ years of strong experience with Python
• ✓Industry experience and knowledge of code quality, formatting, and software development best practices
• ✓Experience with Python testing ecosystems, including unit, integration, and property-based testing
• ✓Knowledge of multi-threading and asynchronous programming in Python
• ✓Ability to work with architectural patterns and refactor code without introducing regressions
• ✓Strong debugging skills, including fixing memory and concurrency issues
• ✓Fluent conversational and written English communication skills

Relevant backgrounds mentioned

• ✓Experience working on AI model evaluation
• ✓Experience with Supervised Fine-Tuning (SFT)
• ✓Experience with Reinforcement Learning with Human Feedback (RLHF)
• ✓Experience creating datasets for AI model training and evaluation
• ✓Experience evaluating and comparing outputs from multiple LLMs