LLM evaluation framework for Fielddesk
Automated evaluation of prompts and models so every change is measured before release. A case study by Arjun Patel, AI Engineer — Chatbots & Document AI.
Challenge
Fielddesk, a field service software company, changed prompts based on a few manual tests. Quality regressions reached customers twice in one month.
Solution
I built an evaluation framework with curated test sets, model-graded and rule-based metrics, cost and latency tracking, and a CI job that blocks releases that reduce quality.