--- title: 'AI Evaluation Engineer at Zafin' canonical: 'https://feeny.ai/job/ai-evaluation-engineer-zafin-toronto-30ah6kgqw02b' type: 'job' last_seen: '2026-09-11' --- # AI Evaluation Engineer at Zafin - **Company:** Zafin - **Location:** Toronto, Canada - **Posted:** 2026-07-23 - **Last confirmed live:** 2026-09-11 - **Apply:** https://job-boards.greenhouse.io/zafinlabsamericasinc/jobs/5188972007 ## Job description Zafin is an AI platform company helping regulated institutions modernize how critical work is designed, governed, and delivered. Our technology enables organizations to move faster while maintaining the governance, accountability, and control required in highly regulated environments. Our portfolio includes Zafin AIOS, an agent orchestration platform for governed AI work; the Zafin Banking Platform, which helps banks modernize product, pricing, offers, billing, loyalty, and relationship management; and Zafin IO, an integration platform that connects data, systems, and workflows across complex enterprise environments. Headquartered in Toronto, Canada, Zafin partners with leading financial institutions across North America, Europe, the Middle East, Africa, and Asia-Pacific. As AI transforms the future of financial services, we're building the platforms that help regulated organizations adopt AI responsibly and at scale. What’s the Opportunity? The AI Evaluation Engineer ensures AI agent solutions are accurate, reliable, safe, and production-ready within regulated banking environments. The role provides objective, evidence-based evaluation of AI agent behaviour against defined business, quality, risk, performance, and regulatory criteria, helping ensure AI solutions deliver consistent, trusted outcomes in production. Working within the Reliability Testing phase of the AIOS (Zafin's AI Operating System) delivery lifecycle, the role designs, executes, and leads evaluation activities that validate AI agent behaviour across the full development lifecycle. This includes developing meaningful evaluation scenarios, identifying defects and failure modes, monitoring quality across releases, and providing actionable feedback that continuously improves AI agent reliability. The role helps operationalize AIOS's principle of reliability first, velocity second through disciplined evaluation and objective production-readiness decisions. The AI Evaluation Engineer works closely with Agent Engineers, Industry Consultants, Agent Architects, AI Knowledge & Governance, Product, and Delivery teams to ensure evaluation reflects intended business logic, regulatory requirements, technical standards, and real-world operating conditions. Depending on experience and level, the role may also lead evaluation activities, coach other evaluation engineers, improve evaluation practices, and drive quality improvements across multiple AI agent initiatives. Ultimately, the role helps ensure AI agent capabilities earn and maintain customer trust by delivering consistent, reliable, and explainable outcomes in production. What Will You Do? - Design and execute structured evaluation scenarios that validate AI agent accuracy, reliability, safety, compliance, and business outcomes. - Validate AI agent behaviour against approved business rules, policies, technical requirements, source knowledge, and expected outcomes. - Conduct regression evaluation across releases and monitor behavioural drift, performance degradation, and newly introduced failure modes. - Identify, document, prioritize, and track quality issues, defects, and production risks. - Support production-readiness decisions through objective evaluation evidence and recommendations. - Analyze evaluation results to identify root causes, recurring quality trends, and opportunities to improve prompts, workflows, knowledge, integrations, and engineering practices. - Maintain reusable evaluation scenarios, benchmark datasets, expected outcomes, regression suites, and supporting evidence. - Contribute to continuous improvement of evaluation methodologies, automation, tooling, and engineering feedback loops. - Support investigation of production issues and validate corrective actions. - Partner with Agent Engineering, Industry Consultants, Agent Architects, AI Knowledge & Governance, Product, and Delivery teams to ensure evaluation reflects business requirements and production expectations. - Participate in production-readiness reviews, release planning, and AI agent optimization activities. - Communicate evaluation findings, quality risks, and recommendations clearly to technical and business stakeholders. - Depending on experience and level, you may also: - Lead evaluation activities across one or more AI agent initiatives or squads. - Review evaluation approaches, production-readiness recommendations, and quality evidence produced by other Evaluation Engineers. - Coach and mentor less experienced Evaluation Engineers, supporting technical growth and consistent evaluation practices. - Coordinate evaluation priorities across multiple concurrent initiatives and support delivery planning. - Drive improvements in evaluation tooling, automation, benchmark management, CI/CD quality integration, and operational effectiveness. - Analyze systemic quality trends and lead continuous improvement initiatives across multiple AI agent capabilities. - Partner with Delivery and Engineering leadership to improve quality outcomes, operational consistency, and AI agent reliability. What Do You Need to Succeed? Must Haves - Typically 3–10+ years of relevant experience in software quality engineering, AI evaluation, AI quality engineering, machine learning evaluation, software testing, or related disciplines. - Level and scope of responsibility will be determined based on demonstrated technical capability, evaluation expertise, leadership experience, independence, and ability to influence quality outcomes. - Experience leading evaluation activities, mentoring technical professionals, or coordinating quality initiatives is advantageous for more senior levels. - Degree in Computer Science, Software Engineering, Data Science, Artificial Intelligence, or related discipline, or equivalent practical experience. - Strong understanding of AI evaluation, large language model behaviour, reasoning quality, hallucination detection, safety, instruction adherence, factual accuracy, and business correctness. - Experience with structured software testing, regression evaluation, production-readiness assessment, and quality engineering. - Working knowledge of SDLC, CI/CD, automated evaluation, AI observability, and engineering delivery practices. - Familiarity with benchmark management, evaluation tooling, quality automation, and AI engineering workflows. - Understanding of privacy, security, governance, and regulatory considerations relevant to enterprise AI. - Proficiency with Python, SQL, or similar tools supporting evaluation and analysis. ## Nice to Have - Experience evaluating LLMs, RAG systems, AI agents, or agentic AI platforms. - Experience with AI evaluation platforms such as LangSmith, OpenAI Evals, or comparable tools. - Experience integrating automated evaluation into CI/CD or MLOps workflows. - Experience with model observability, behavioural-drift detection, or AI production monitoring. - Banking, financial services, or other regulated industry experience. - Experience leading technical teams, quality initiatives, or engineering improvement programs. Additional Job Details - Expected Salary Range: $80,000 - $180,000; we hire into multiple career levels for this role based on a candidate's experience, skills and demonstrated capabilities. - Vacancy Status: Open Position to be filled - Mode of Work: Hybrid - Use of AI: Zafin may use Artificial Intelligence (AI) and/or other forms of automated technology to screen and/or assess applicants for this position. Zafin will not utilize AI for conducting interviews and/or making hiring decisions. What’s in it for you Joining our team means being part of a culture that values diversity, teamwork, and high-quality work. We offer competitive salaries, annual bonus potential, generous paid time off, paid volunteering days, wellness benefits, and robust opportunities for professional growth and career advancement. Want to learn more about what you can look forward to during your career with us? Visit our careers site and our openings: [zafin.com/careers](https://zafin.com/careers/) Zafin welcomes and encourages applications from people with disabilities. Accommodations are available on request for candidates taking part in all aspects of the selection process. Zafin is committed to protecting the privacy and security of the personal information collected from all applicants throughout the recruitment process. The methods by which Zafin contains uses, stores, handles, retains, or discloses applicant information can be accessed by reviewing Zafin’s privacy policy at [https://zafin.com/privacy-notice/.](https://zafin.com/privacy-notice/) By submitting a job application, you confirm that you agree to the processing of your personal data by Zafin described in the [candidate privacy notice.](https://zafin.com/candidate-privacy-notice/) ## About Zafin ## Company Overview - **One-liner**: Zafin provides an AI platform for regulated institutions, enabling governed agentic work, banking execution, and enterprise integration. - **Entity Type**: Private (acquired by Nordic Capital in 2024) - **Headquarters**: Toronto, Canada - **Founded**: 2002 - **Founders**: Not publicly available ## Core Business - Primary industry/industries: Financial services technology (banking & regulated institutions) - Target customers: B2B – large banks, credit unions, and other regulated financial institutions - Mission or purpose statement: “Transform the future of finance” by helping institutions modernize critical work, systems and decisions with greater speed, control and trust. ## Products & Services - **Zafin AIOS**: End-to-end agent orchestration platform and control plane for governed agentic work – governs how agentic work is requested, grounded, routed, executed, reviewed, evidenced, cost-managed and improved. - **Zafin Banking Platform**: AI-powered banking execution platform that externalizes product, pricing, offers, billing, loyalty, relationship and deal logic so banks can launch propositions faster and personalize value. - **Zafin IO**: Enterprise integration platform that connects data, events, services, workflows and applications across legacy and modern systems for progressive modernization. ## Market Standing - **Valuation/Market Cap**: Not disclosed (private company; majority stake acquired by Nordic Capital) - **Key Metric**: 500M+ accounts processed every day; ~25% of the top 15 banks worldwide use Zafin; 800+ employees; 25% of annual revenue invested in R&D - **Notable Investors/Partners**: Nordic Capital (majority owner), Microsoft (Global Partner of the Year, ISV Innovation – Americas Partner of the Year), Curinos, OpenAI - **Growth Signals**: Launched Zafin AIOS in 2025/2026; won CNBC World’s Top Fintech award; expanded into LATAM and APAC; won BIAN Best-in-Class Partner and Microsoft Global Partner of the Year awards. ## Competitive Advantages - Over 20 years of regulated banking technology experience with deep domain expertise in governance, auditability and accountability. - Purpose-built AI governance framework for agentic work in highly regulated environments, differentiating from generic AI platforms. - Strong platform ecosystem partnerships (Microsoft, OpenAI, Curinos) and an existing base of top-tier banking clients. ## Strategic Focus - Extending regulated technology discipline to govern agentic AI work across agents, models, tools and workflows. - Enabling progressive modernization without disrupting critical operations through integration and platform capabilities. - Expanding global footprint, especially in LATAM and APAC, and deepening cloud-based SaaS delivery. ## Why Work Here - **Culture**: 93% employee satisfaction; recognized as a “Great Place To Work” with a focus on care, innovation, transparency, and diversity/inclusion. - **Work-Life Flexibility**: Emphasizes a balanced approach with paid volunteer days, comprehensive health benefits from Day 1, gym discounts, and employee assistance programs. - **Perks & Growth**: Annual performance-based incentives, employer-matched retirement savings, paid learning hours, mentorship, and a career development framework. - **Engineering Culture**: 25% of revenue reinvested into R&D; teams work across North America, Europe, and APAC (major hubs in Toronto and Trivandrum, India). Hybrid/remote flexibility is likely but not explicitly stated. ## Sources 1. [zafin.com](https://zafin.com/) 2. [zafin.com/our-story](https://zafin.com/our-story/) 3. [zafin.com/careers](https://zafin.com/careers/) 4. [job-boards.greenhouse.io/zafinlabsamericasinc](http://job-boards.greenhouse.io/zafinlabsamericasinc) 5. [linkedin.com/company/zafin](https://www.linkedin.com/company/zafin) ## Other roles at Zafin - [QA Engineer II](https://feeny.ai/job/qa-engineer-ii-zafin-trivandrum-3rs2b9zk3cy9) — Trivandrum, India - [Product and Pricing Consultant II](https://feeny.ai/job/product-and-pricing-consultant-ii-zafin-trivandrum-584j4mp3sbk9) — Trivandrum, India - [Project Manager](https://feeny.ai/job/project-manager-zafin-trivandrum-1dpr3ge1b3w6) — Trivandrum, India - [Product & Pricing Consultant II/III](https://feeny.ai/job/product-pricing-consultant-ii-iii-zafin-toronto-z9t2sdwa4kt1) — Toronto, Canada - [Technical Consultant II/III](https://feeny.ai/job/technical-consultant-ii-iii-zafin-toronto-sgkx9ejb900t) — Toronto, Canada - [QA Engineer III](https://feeny.ai/job/qa-engineer-iii-zafin-toronto-40nnhenzpk35) — Toronto, Canada - [Solution Architect](https://feeny.ai/job/solution-architect-zafin-toronto-bzdgt89efyzb) — Toronto, Canada - [Video Content Designer](https://feeny.ai/job/video-content-designer-zafin-chennai-j4grvcdcygax) — Chennai, India - [Video Producer](https://feeny.ai/job/video-producer-zafin-chennai-d42n9y3wat9q) — Chennai, India - [Senior Analyst, PMO](https://feeny.ai/job/senior-analyst-pmo-zafin-trivandrum-vw94c7qengww) — Trivandrum, India