GenAI QA Analyst

Location: PCS CA~TORONTO

Years of Experience: 8 Years

Job Summary: The GenAI QA Analyst is responsible for ensuring the quality, reliability, and compliance of generative AI solutions deployed across enterprise environments. This role is critical in validating AI-driven outputs, monitoring system performance, and ensuring that solutions meet both technical and business expectations. Positioned at the intersection of technology and business, the analyst collaborates closely with delivery leads, developers, and product owners to safeguard the accuracy, scalability, and ethical use of AI applications.

Responsibilities:

• Collaborate with delivery leads and product owners to define QA strategies for generative AI projects.
• Design and execute test plans that validate AI outputs for accuracy, consistency, and compliance.
• Conduct both automated and manual testing to evaluate AI models across diverse business scenarios.
• Monitor AI system performance using enterprise tools, ensuring proactive detection and resolution of issues.
• Validate integration of AI platforms into enterprise workflows, ensuring seamless adoption and reliability.
• Ensure GitHub Copilot–driven development aligns with QA standards and supports reproducible outcomes.
• Document QA findings in clear, business-readable formats for stakeholders.
• Advocate for continuous improvement in AI testing practices and promote a quality-first culture across teams.

Mandatory Skills:

• GitHub Copilot (Expert)
• Generative AI Testing Frameworks (Advanced)
• Automated QA Tools & Scripting (Advanced)
• AI Model Validation Techniques (Advanced)
• Performance Monitoring (Dynatrace Davis) (Intermediate)

Preferred Skills:

• AI Foundry
• Microsoft Copilot Studio
• Atlassian Rovo
• Claude Code
• Documentation & Reporting Tools

Qualifications: A minimum of 5-7 years of experience in Quality Assurance, with at least 2 years focused on AI/GenAI projects. Strong analytical skills and a commitment to quality are essential.