AI Evaluation Specialist – Remote Contract

About the role

We are looking for a meticulous AI Evaluation Specialist to design, execute, and document evaluation tasks that benchmark AI performance in real‑world workflows. This remote contract position offers flexible hours (10‑40 hrs/week) and competitive pay.

Key responsibilities

• Create self‑contained evaluation tasks, including prompts, supporting files, and detailed grading rubrics.
• Define clear, unambiguous success criteria for diverse administrative and workflow scenarios.
• Observe AI agent behavior, produce concise summaries, and generate high‑quality English reports.
• Iterate and refine tasks and rubrics based on feedback and team collaboration.
• Collaborate with the client’s team to share insights and drive continuous improvement in evaluation techniques.

Required profile

• Minimum experience in roles that demand precise written communication (e.g., paralegal, executive assistant, junior analyst, librarian, research assistant, technical writer, QA analyst).
• Native or fluent English writing ability with a track record of producing succinct, specific, and unambiguous observations.
• Demonstrated skill in designing or applying rubric‑based evaluation or structured scoring frameworks.
• High attention to detail and ability to notice subtle patterns or inconsistencies.
• Strong written and verbal communication skills for documenting nuanced feedback.

Required skills

What we offer

• Hourly compensation ranging from $20 to $35.
• Fully remote work with flexible scheduling.
• Opportunity to influence AI evaluation methodology for a growing client base.