Senior Software Engineer — AI Evaluation & Benchmarks
Build the benchmarks that test the world's most advanced AI models. $80–$100/hr, fully remote, Python experts wanted.
- Organization
- Alignerr
- Type
- Hourly Contract
- Pay
- $80–$100 / hour
- Location
- Remote
- Contract Length
- 3 Months
- Commitment
- Full-time availability preferred
About the Role
What if the code you write could determine how smart the next generation of AI truly is? We’re looking for experienced Software Engineers to design and build the coding benchmarks and data pipelines used to evaluate frontier AI models — the systems that decide whether an AI can actually reason, debug, and write production-quality software.
This is high-impact, technically demanding work at the intersection of software engineering and AI research. You’ll work with large codebases, multiple programming languages, and scalable infrastructure to create evaluation systems that push the boundaries of what AI can do.
This is a fully remote contract role. If you thrive in fast-paced engineering environments and want your work to directly shape the trajectory of AI — this is the role.
- Organization: Alignerr
- Type: Hourly Contract
- Pay: $80–$100 / hour
- Location: Remote
- Contract Length: 3 Months
- Commitment: Full-time availability preferred
What You’ll Do
- Design and implement coding benchmarks used to evaluate frontier AI models across real-world programming tasks
- Build and maintain scalable data pipelines for AI evaluation workflows
- Analyze AI-generated code for correctness, reliability, and edge-case failures
- Create structured evaluation scenarios that rigorously test reasoning, debugging, and code quality
- Work with large code repositories and multi-language environments
- Collaborate on systems that improve how AI models understand and generate software
- Provide detailed technical feedback on model performance and failure patterns
- Contribute to the design of evaluation frameworks that set industry standards
Who You Are
- 4+ years of professional software engineering experience — this is non-negotiable
- Experience working at a high-growth tech company or top-tier software organization
- Expert proficiency in Python — you write clean, performant, well-tested Python code
- Hands-on experience with code repositories and working in large, complex codebases
- Proven experience designing and implementing LLM coding benchmarks and data pipelines
- Track record of working in high-performance engineering environments with large-scale products or platforms
- Strong command of version control systems (Git) and modern development workflows
- Bilingual or native English speaker with strong written communication skills
- Self-directed, technically rigorous, and comfortable operating with autonomy
What Makes a Perfect Match
Candidates with these additional qualifications have the highest chance of success:
- Senior or Lead-level engineering profiles with a history of technical ownership
- Bachelor’s or Master’s degree in Computer Science, Machine Learning, or a related field — or equivalent professional experience
- Proficiency in one or more additional languages: JavaScript, Go, C++, or other relevant languages
- Experience with CI/CD pipelines and writing robust unit tests (pytest, Mocha, JUnit)
- Background in security engineering or significant open-source contributions
- Familiarity with AI/ML evaluation methodologies or model benchmarking
Why Join
- Work on cutting-edge AI evaluation projects alongside world-class research teams
- Fully remote — work from anywhere with a reliable internet connection
- Your benchmarks directly influence how the most advanced AI systems in the world are measured and improved
- Freelance autonomy with meaningful, high-stakes engineering work
- Collaborate with a global community of elite engineers and researchers
- Potential for contract extension and ongoing engagement as new evaluation challenges emerge
Preparing for Alignerr’s AI-Led Interview
Alignerr uses AI to conduct its interviews, so after you apply you will likely talk to an AI interviewer rather than a recruiter. That rewards clear, specific, well-paced answers. Expect to talk in detail about benchmarks or data pipelines you have built, how you test Python code, and how you would decide whether a piece of AI-generated code is actually correct. Read the invitation instructions carefully and follow Alignerr’s rules for the interview itself.
The best preparation is rehearsing out loud beforehand. With InterviewAce you can run mock interviews, practise answering follow-up questions about your own projects, and get feedback before the real one. These guides walk you through it:
- Interviewed by an AI? What to expect from an AI interviewerHow AI-led screening interviews work and how to come across clearly.
- Technical interview prep for software engineersA structured plan for coding, system design and behavioral rounds.
- How to use AI tools to practice job interviewsRehearse out loud with an AI interviewer that asks follow-ups.
- Remote interview tips for video callsSet-up, lighting, audio and pacing for interviews you take from home.
- Top 5 mistakes in technical interviewsThe habits that sink strong engineers, and how to avoid them.
- Behavioral questions with the STAR methodTurn your projects into tight, specific stories.
- How large language models workA refresher on the systems you would be evaluating.
Free Tools for Your Application
Before you hit apply, make sure your resume tells the right story. Our free career tools need no sign-up:
- Free Resume BuilderTighten your resume around Python, benchmarks and AI work before you apply.
- Free Cover Letter GeneratorDraft a focused note about the systems you have built.
- Free Salary CalculatorCompare an hourly contract rate with a salaried offer.
Ready to Apply?
Applications are handled by Alignerr. The button below opens the application for Senior Software Engineer — AI Evaluation & Benchmarks on Alignerr’s site.
Apply for This Job →InterviewAce lists this role as a partner of Alignerr and may receive a referral benefit if you are hired. It costs you nothing and does not affect your application. Alignerr makes all hiring decisions.
Also hiring: Senior Software Engineer — Agentic Coding. See all open jobs, or find out where InterviewAce has been featured.