Human Eval

Key Responsibilities
Oversee day to day operations of the human evaluation team.
Lead calibration sessions ensure consistent application of rubrics.
Review flagged responses and resolve ambiguous or edge case evaluations.
Document any rubric or tooling issues and escalate systemic concerns.
Drive continuous improvement in evaluation quality and team performance.
Support onboarding and training for new evaluators.
Tools & Systems
All tools described above for the evaluator role
Performance tracking dashboards( post 6-8 months into the project)
Client reporting and documentation templates
QA automation scripts or tooling (as needed)
Domain & Skill Requirements
Proficient in geospatial analysis: Strong understanding of spatial data, map structures, and geographic context.
Expertise in GIS tools, especially Google Earth Pro (placemarks, polygons, historical imagery).
Familiarity with remote sensing data: Analysing historical/current satellite images and interpreting vector data (points, lines, polygons).
Experience in data annotation and labelling: Assigning names, descriptions, and styles to geographic features.
Geospatial visualization: Creating and interpreting maps in Google Earth Pro.
Familiarity with Google Maps, spatial boundaries, and map-based QA workflows.
Comfortable with rubric-based assessments and rating scales (e.g., Likert, Pass/Fail).
Excellent observation and analytical skills.
Stakeholder management and reporting expertise.
Experience auditing and maintaining quality standards across a team.
Ability to lead team calibration sessions and ensure evaluation consistency across raters.
Proven experience in mentoring, coaching, and performance management of evaluation teams.
Bonus: Experience with LLM feedback cycles or AI evaluations.