Senior AI Quality Assurance Engineer

About the role

SoftServe is delivering an AI Student Tutor education platform integrated with Canvas. The Senior Quality Assurance Engineer will own quality across the engagement, working with AI/ML, application, product and data engineering teams and coordinating directly with the client’s product, SME, IT and security stakeholders.

Key responsibilities

• Create and maintain the Acceptance Testing Plan, defining criteria, test types, data, evidence expectations and sign‑off requirements.
• Design and execute functional, integration, regression, load/performance, accessibility, safety and pedagogical‑quality testing.
• Test AI agent components such as routing logic, prompt governance, persona‑adaptive behavior, content validation and resistance to prompt‑injection.
• Run safety scenario testing across distress‑handling cases and measure detection accuracy.
• Drive UAT by building test scripts, executing them with client stakeholders and managing defect triage and re‑testing.
• Serve as the primary client‑facing point of contact for quality matters, communicating status, defect trends and pilot‑readiness risk.
• Verify and reproduce reported bugs with the client, translate them into clear defect reports and hand them off to the Europe‑based delivery team.

Required profile

• Proven experience leading QA/QC for complex, multi‑workstream software programs and authoring test strategies from scratch.
• Experience testing AI/LLM‑based conversational agents, including prompt‑driven behavior, RAG/grounding validation and safety guardrails.
• Strong written documentation skills for test plans, traceability matrices, defect reports and readiness packages.
• Excellent client‑facing communication skills and ability to run bug‑verification calls directly with stakeholders.
• Availability during the US‑Europe daily overlap window to support live client bug verification.

Required skills

• AI/LLM testing and prompt governance.
• Canvas/LTI LMS integration testing (SSO, roster sync).