Overview

About LH2 AI Labs Built by second-time founders who have built and sold companies before, LH2 AI Labs is building the post-training infrastructure for frontier AI models.

Responsibilities

• This is a strong fit for a research-oriented software engineer with practical
• experience in coding benchmarks such as SWE-bench, SWE-Bench Pro, DeepSWE, Terminal-Bench, or equivalent repository-based evaluation frameworks.
• Design realistic coding tasks covering debugging, feature development, refactoring, code review, repository navigation, and test-based problem solving.
• Source and structure tasks from repositories, issues, pull requests, commits, diffs, test suites, and customer-provided codebases.
• Build reproducible evaluation environments using repositories, Docker containers, dependencies, test harnesses, sandboxes, and code-execution workflows.
• Define task instructions, acceptance criteria, reference solutions, grading rubrics, tests, hidden tests, and programmatic verifiers.
• Run and adapt coding benchmarks such as SWE-bench, SWE-Bench Pro, DeepSWE, Terminal-Bench, or similar internal benchmarks.
• Evaluate outputs and trajectories generated by LLMs, humans, and coding agents, and determine whether tasks have been correctly solved.
• Identify ambiguous, broken, trivial, contaminated, non-reproducible, or incorrectly verified tasks before they reach customers or training pipelines.
• Analyse model and agent failure modes across repository understanding, planning, implementation, tool usage, debugging, and test execution.
• Build datasets for supervised fine-tuning, preference optimisation, RLHF, RLVR, and other LLM post-training workflows.
• Convert successful and unsuccessful agent trajectories into demonstrations, corrections, preference pairs, critiques, and outcome-labelled training examples.
• Build and improve pipelines for collecting, validating, deduplicating, analysing, and packaging evaluation and post-training datasets.
• Work with customers and internal teams to convert high-level evaluation or training goals into clear benchmark workflows, technical environments, and quality standards.

Qualifications

• 3-7+ years of professional experience in software engineering, AI research engineering, machine-learning engineering, or developer tooling.
• Strong programming ability in Python and at least one additional language such as JavaScript/ TypeScript, Java, Go, C++, or Rust.
• Strong ability to read, understand, modify, and debug unfamiliar production codebases.
• Ability to create and debug reproducible environments using Docker, Linux, shell scripts, package managers, and cloud or local execution setups.
• Hands-on experience building, running, adapting, or analysing coding-model or coding-agent benchmarks.
• Practical knowledge of benchmarks such as SWE-bench, SWE-bench Verified, SWE-Bench Pro, DeepSWE, Terminal-Bench, or equivalent internal frameworks.
• Understanding of benchmark contamination, solution leakage, test overfitting, flaky environments, and reproducibility challenges.
• Ability to distinguish between model failures, agent failures, task-design problems, verifier problems, and environment failures.
• Strong engineering judgment around correctness, reproducibility, code quality, task difficulty, and evaluation reliability.
• Strong written and verbal communication skills for documenting task design, methodology, results, and failure analysis.
• Familiarity with additional benchmarks such as LiveCodeBench, HumanEval+, MBPP+, RepoBench, or Multi-SWE-bench.
• Prior experience working with AI labs, evaluation teams, data companies, or applied AI organisations.
• Coding tasks are realistic, reproducible, well-scoped, challenging, and objectively verifiable.
• Public and private repositories can be converted into reliable benchmark environments.
• Evaluation results accurately reflect model capabilities rather than problems in the task, verifier, or infrastructure.
• Broken, ambiguous, contaminated, or low-quality tasks are identified early.
• Model failures are converted into useful benchmark improvements and high-quality post-training examples.
• Customers receive clean datasets, clear methodologies, meaningful analysis, and trustworthy evaluation results.