AI Evaluation Specialist – Remote Contract

About the role

We are seeking a detail‑oriented AI Evaluation Specialist to design, execute, and refine evaluation tasks that benchmark AI performance in real‑world computer‑based workflows. This contract position is fully remote and offers flexible hours ranging from 10 to 40 per week.

Key responsibilities

• Design self‑contained evaluation tasks, including prompts, supporting files, and grading rubrics.
• Define clear, unambiguous criteria for successful and unsuccessful task outcomes across diverse administrative scenarios.
• Observe AI agent behavior, document observations, and produce concise, high‑quality English reports.
• Iterate on tasks and rubrics based on feedback and team collaboration.
• Collaborate with the client’s team to share insights and drive continuous improvement in evaluation techniques.

Required profile

• Minimum experience in roles that demand precise written communication and structured thinking (e.g., paralegal, executive assistant, analyst, librarian, technical writer, QA analyst).
• Native or fluent English writing ability with a track record of succinct, specific, and unambiguous documentation.
• Proven experience designing or applying rubric‑based evaluation or structured scoring frameworks.
• High attention to detail and ability to spot subtle patterns or inconsistencies.
• Strong written and verbal communication skills for documenting nuanced observations.

Required skills

What we offer

• Competitive hourly rate of $20‑$35.
• Remote work with flexible scheduling.
• Opportunity to influence AI evaluation methodology for a leading client.