See all roles

AI Automation Engineer

Work from home Full-time role Hiring

About the Company:

Netomi is the leading agentic AI platform for enterprise customer experience. We work with the largest global brands like Delta Airlines, MetLife, MGM, United, and others to enable agentic automation at scale across the entire customer journey. Our no-code platform delivers the fastest time to market, lowest total cost of ownership, and simple, scalable management of AI agents for any CX use case. Backed by WndrCo, Y Combinator, and Index Ventures, we help enterprises drive efficiency, lower costs, and deliver higher quality customer experiences.

Want to be part of the AI revolution and transform how the world’s largest global brands do business? Join us!

About the Role

An AI Automation Engineer focuses on building automated systems to evaluate, test, and enhance the performance of conversational AI models (LLMs, chatbots, voice assistants). They sit at the intersection of machine learning, software engineering, and QA automation, ensuring that AI systems produce high-quality, safe, and reliable responses at scale.

Responsibilities

  • Automation Pipelines: Build Python-based pipelines for automated quality testing of AI responses.
  • Integrate LLMs into automated evaluation frameworks (e.g., using GPT-based evaluators, embeddings, or custom scoring).
  • Automate regression and stress testing for conversational AI flows.
  • Quality & Evaluation: Define evaluation metrics (relevance, factuality, coherence, safety, empathy).
  • Implement both rule-based and AI-driven quality checks.
  • Monitor model drift, bias, and hallucinations using automated workflows.
  • Integration & Deployment: Work with APIs, SDKs, and CI/CD pipelines to embed automated AI evaluation in production.
  • Develop monitoring dashboards to visualize conversation quality.
  • Collaborate with ML engineers, product managers, and QA teams to close the feedback loop.
  • Optimization & Innovation: Experiment with prompt engineering and automated prompt-testing frameworks.
  • Explore reinforcement learning, self-critique models, or human-in-the-loop automation for continuous improvement.
  • Automate compliance and policy adherence checks for enterprise AI systems.
  • Requirements

  • Strong 2-5 Years of experience in Python development (automation, scripting, data handling).
  • Experience with LLMs/NLP frameworks.
  • Understanding of MLOps / AI deployment pipelines.
  • Additional Information

    Netomi is an equal opportunity employer committed to diversity in the workplace. We evaluate qualified applicants without regard to race, color, religion, sex, sexual orientation, disability, veteran status, and other protected characteristics.

    Apply to this Job

    You might like

    Prompt Engineer

    Work from home Full-time role

    Senior Salesforce Engineer

    Work from home Full-time role

    Store Assistant, Full Time – Amazon Store

    Work from home Full-time role

    Senior Customer Service Representative – Amazon Store

    Work from home Full-time role

    Drinking Water Technical Manager (California)

    Work from home Full-time role

    Manager, ERP

    Work from home Full-time role

    Senior Software Engineer - AI/Deep Learning

    Work from home Full-time role

    Senior Salesforce Developer

    Work from home Full-time role

    Business Development Manager for Sora Union

    Work from home Full-time role

    Vertriebsmitarbeiter im Außendienst (m/w/d)

    Work from home Full-time role

    Experienced Data Entry Clerk – Flexible Part-Time Remote Work Opportunity with Arenaflex

    Work from home Full-time role

    [PART_TIME Remote] Freelance Data Entry Typist

    Work from home Full-time role

    Intern - Business Development & Account Mgmt Sales (Summer 2025)

    Work from home Full-time role

    Customer Service Manager

    Work from home Full-time role

    Experienced Seafood Customer Service Representative – Providing Exceptional Service and Ensuring Freshness in a Fast-Paced blithequark Store Environment

    Work from home Full-time role

    Experienced Customer Service Account Associate - Delivering Exceptional Support and Building Lasting Relationships with arenaflex

    Work from home Full-time role

    Traveling Scheduling Manager - MSG - Biotech/Pharma

    Work from home Full-time role

    Experienced Remote Customer Service Specialist - Technology Support Professional at arenaflex ($16-$35/hour)

    Work from home Full-time role

    First-Line Client Support Agent - Remote, Entry Level

    Work from home Full-time role

    Remote Virtual Customer Service Agent – Travel & Tourism Industry

    Work from home Full-time role