Senior Machine Learning/AI Engineer

Senior AI Engineer — Search, Retrieval & Evaluation

The Opportunity

We are seeking a Senior AI Engineer to help build sophisticated, production-grade AI products and workflows.

This role sits at the intersection of generative AI, information retrieval, evaluation systems and distributed software systems. You will help determine how complex data is indexed and retrieved, how models use that information, and how the organization measures whether its AI systems are consistently producing accurate and valuable results.

This is an opportunity to move beyond proof-of-concept AI applications and build reliable systems that solve complex, real-world problems.

What You’ll Do

• Design and build production AI applications and multi-step workflows using Python, LangChain and LangGraph.
• Develop indexing, search and retrieval strategies across structured and unstructured data.
• Determine what information should be indexed, how it should be organized and how relevant evidence should be retrieved.
• Build hybrid search solutions using techniques such as keyword search, semantic search, metadata filtering, ranking and reranking.
• Design evaluation frameworks for generative AI applications, retrieval systems and agentic workflows.
• Create representative test datasets, expected outputs, scoring criteria and automated regression tests.
• Evaluate models and workflows for accuracy, relevance, groundedness, consistency, latency and cost.
• Diagnose inconsistent model behavior and determine whether failures originate in the data, retrieval process, prompt, model or broader workflow.
• Compare models and technical approaches to determine whether the quality improvement justifies the additional cost and complexity.
• Partner with engineering, product and business stakeholders to establish AI development and evaluation standards.
• Help the organization make informed decisions about how AI capabilities should be designed, tested and deployed.
• Contribute to system architecture, production reliability and distributed systems decisions.

What We’re Looking For

• Significant professional software engineering or applied machine learning experience.
• Strong Python development skills.
• Experience building and operating production AI, machine learning or LLM-based applications.
• Deep understanding of search, indexing or information retrieval.
• Experience with semantic search, embeddings, vector databases, keyword search or hybrid retrieval.
• Experience designing evaluation systems for LLMs, RAG applications, search systems or other nondeterministic AI products.
• Ability to separate and evaluate retrieval quality from final-answer quality.
• Experience diagnosing variable or inconsistent AI outputs.
• Strong understanding of model selection, system performance, latency and cost tradeoffs.
• Experience designing scalable, reliable distributed systems.
• Ability to communicate technical tradeoffs and establish best practices across an engineering organization.

Preferred Qualifications

• Master’s degree in computer science, machine learning, information retrieval or a related field.
• Experience with LangChain or LangGraph.
• Experience building agentic or multi-step AI workflows.
• Experience with Elasticsearch, OpenSearch, Solr or vector-search technologies.
• Familiarity with retrieval metrics such as precision, recall, ranking quality or NDCG.
• Experience with human-in-the-loop evaluation, golden datasets, model benchmarking or AI observability.
• Experience with Azure or Azure OpenAI.

Azure experience is helpful but not required.