This job is no longer available
This job expired on 09/08/2026. It no longer accepts applications.
ML Evaluation Designer - Expert
Mercor
Job description
About the role
Mercor is seeking an experienced Machine Learning Engineer to design and refine evaluation tasks for large language models. This remote, contract position focuses on creating robust rubrics, metrics, and grading processes that directly improve model performance.
Key responsibilities
- Design ML/LLM evaluation tasks, rubrics, and metrics.
- Grade model and agent outputs, iterating to enhance evaluation quality.
- Apply training‑side judgment in SFT, RLHF, and reward modeling to shape evaluation design.
- Collaborate with AI research teams to refine evaluation signals and improve model outputs.
- Work independently and asynchronously, meeting deadlines for a 30+ hour weekly commitment.
Required profile
- 5+ years of experience as a Machine Learning Engineer with hands‑on training and evaluation work.
- Strong written communication skills.
Required skills
- Proficiency with PyTorch.
- Proficiency with JAX.
- Experience using Hugging Face libraries.
What we offer
- Contract rate ranging from $45 to $140 per hour.
- Fully remote work environment.
- Opportunity to influence cutting‑edge AI research.
Questions fréquentes
Why are you reporting this job?
Explore further
Salaries, guides and searches in Ireland.
A question about this job?
Ask it here: you will get the full job summary by e-mail, right away.
Boost your chances
Upload your CV — we will match you with relevant openings.
Analyzing your CV...
Mercor