Develop a deep understanding of Tekion’s AI agents, ML models, and the quality dimensions that matter for each business domain.

Design, enhance, and own Tekion’s AI evaluation infrastructure as a shared capability used across ML teams.

Create, curate, and maintain evaluation datasets (evals) and golden/ground-truth sets across use cases and domains.

Define quality metrics for AI outputs — accuracy, relevance, faithfulness/groundedness, consistency, safety, and task success.

Build automated scoring pipelines, including LLM-as-judge, rubric-based, and referencebased evaluation methods.

Validate user intents and measure response accuracy and consistency for AI-powered capabilities such as the Analytics Agent.

Identify hallucinations, unsafe or biased outputs, and edge cases; design targeted eval suites to catch them.

Build both offline evaluation (pre-release benchmarking) and online evaluation (production quality monitoring, A/B, drift detection).

Establish evaluation gates in CI/CD so model, prompt, or data changes are quality-checkedbefore release.

Develop dashboards and reporting that make AI quality visible and actionable for ML and product teams.

Use AI/LLMs to scale evaluation — automated judges, synthetic data generation, and eval \tooling

Champion evaluation and responsible-AI quality best practices across the organization.

8+ years in SDET, quality engineering, ML engineering, or data science, with hands-on experience building evaluation or measurement systems — or a strong SDET background with deep LLM/ML fluency.

Strong programming skills in Python, with the ability to build robust, reusable evaluation pipelines and tooling.

Deep understanding of ML/LLM evaluation, benchmark design, and the pitfalls of evaluating non-deterministic systems.

Hands-on experience with LLM-as-judge, rubric-based scoring, or human-in-the-loop evaluation.

Solid grasp of LLM/agent concepts — prompting, RAG, embeddings, tool use — and generative failure modes (hallucination, drift, prompt sensitivity, bias).

Experience designing and curating datasets, including labeling/annotation strategy and data quality.

Strong statistical intuition for interpreting eval results and significance.

Excellent communication skills to translate quality signals into decisions for ML and product teams.

Develop a deep understanding of Tekion’s AI agents, ML models, and the quality dimensions that matter for each business domain.

Design, enhance, and own Tekion’s AI evaluation infrastructure as a shared capability used across ML teams.

Create, curate, and maintain evaluation datasets (evals) and golden/ground-truth sets across use cases and domains.

Define quality metrics for AI outputs — accuracy, relevance, faithfulness/groundedness, consistency, safety, and task success.

Build automated scoring pipelines, including LLM-as-judge, rubric-based, and referencebased evaluation methods.

Validate user intents and measure response accuracy and consistency for AI-powered capabilities such as the Analytics Agent.

Identify hallucinations, unsafe or biased outputs, and edge cases; design targeted eval suites to catch them.

Build both offline evaluation (pre-release benchmarking) and online evaluation (production quality monitoring, A/B, drift detection).

Establish evaluation gates in CI/CD so model, prompt, or data changes are quality-checkedbefore release.

Develop dashboards and reporting that make AI quality visible and actionable for ML and product teams.

Use AI/LLMs to scale evaluation — automated judges, synthetic data generation, and eval \tooling

Champion evaluation and responsible-AI quality best practices across the organization.

8+ years in SDET, quality engineering, ML engineering, or data science, with hands-on experience building evaluation or measurement systems — or a strong SDET background with deep LLM/ML fluency.

Strong programming skills in Python, with the ability to build robust, reusable evaluation pipelines and tooling.

Deep understanding of ML/LLM evaluation, benchmark design, and the pitfalls of evaluating non-deterministic systems.

Hands-on experience with LLM-as-judge, rubric-based scoring, or human-in-the-loop evaluation.

Solid grasp of LLM/agent concepts — prompting, RAG, embeddings, tool use — and generative failure modes (hallucination, drift, prompt sensitivity, bias).

Experience designing and curating datasets, including labeling/annotation strategy and data quality.

Strong statistical intuition for interpreting eval results and significance.

Excellent communication skills to translate quality signals into decisions for ML and product teams.