Generalist
Evaluate Content:
- Assess responses for accuracy, relevance, helpfulness, language usage, and alignment with user intent.
Review and Compare Content:
- Conduct research & fact-checking by comparing response with other generally available sources (ex: Google search, social media, 3rd party materials) and identify key differences and trends,
- Conduct side-by-side comparisons and identify key differences between responses.
Rate Content:
- Rate and stack rank responses
- Provide clear reasoning while ensuring adherence to guidelines
- Ensure consistency in rating decisions
Undergraduate / Masters or comparable experience
- Knowledge of established guidelines and best practices for content rating.
- Experience in evaluating and comparing content from different sources.
- Awareness of privacy policies, copyright guidelines, and industry regulations.
- Work experience that is inclusive of journalism, literature, educators, writing, Linguistic, instructional designing or other comparable experience
- Research skills to evaluate content accuracy and identify trustworthy sources.
- Analytical skills to assess content factuality, relevance, and alignment with user intent.
- Critical thinking skills to identify biases, inconsistencies, and limitations in AI-generated responses.
- Attention to detail to review content for grammar, style, and adherence to ethical/legal standards.
- Communication skills to provide clear and constructive feedback.
Assess Quality:
- Identify one or more user intent from the provided prompt and review if the responses are aligned with the user's primary intent
- Identify potential biases, inconsistencies or contradiction within responses
Improve the Model:
- Identify limitations
- Create high quality training data
- Provide constructive feedback to the team on how to improve Quality
- Provide guidance and clarification to other raters, conduct training sessions
- Participate in training and rating calibration exercises
- Collaborate with project teams to refine quality assurance model
- Undergraduate / Masters or comparable experience