Software Engineer - Benchmark Auditor

San Francisco, CA

$70 to $90 a hour

About the job

About the job Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark , General Catalyst , Peter Thiel , Adam D'Angelo , Larry Summers , and Jack Dorsey . Position: SWE-Bench Task Auditor Type: Contract Compensation: $70–$90/hour Location: Remote Role Responsibilities Evaluate the quality, correctness, and reproducibility of software-engineering benchmark tasks . Assess repository-level tasks, reference patches, test harnesses, and grading integrity. Provide clear, rubric-based written feedback to improve AI model training . Audit reference patches, test runners, and Docker isolation to detect answer...