AI Agents – Designing and Evaluating AI for Job-Listing Quality

Overview

A national online employment service, where employers publish job listings and job seekers search for work, wanted to help employers write clearer, more complete listings. To do this, a set of AI agents was developed to review listings and suggest improvements.

I worked as the AI and UX designer in the team, responsible for the research behind the agents, their instructions and the evaluation that showed whether they worked.

Challenge

AI models can sound convincing even when they are wrong or inconsistent. For a public service, the agents had to give feedback that employers could trust, and the same listing had to receive the same assessment each time.

The challenge was to:

• Define what a good job listing actually is, in testable terms.

• Make each agent assess listings consistently and accurately.

• Decide, based on evidence, when an agent was good enough to use.

Approach

I started with the listings themselves: what separates a clear posting from a weak one. I turned these findings into concrete quality criteria that defined what each agent should check.

I wrote and refined the agents' instructions and set up a systematic way to evaluate their output. I used AI as a sparring partner to look for gaps the instructions did not yet cover, and then checked every suggestion against real-world examples.

I worked alongside developers in a standard development workflow and documented the findings for both technical and non-technical colleagues.

Outcome

• AI agents that give employers consistent, useful feedback on their listings.

• A clear, evidence-based way to judge AI quality before release.

• A documented basis for using AI-driven quality checks in a public service.

Impact & Reflection

This project changed how I think about AI in design. The most important design work happened before and after the prompts: understanding what quality means for the people involved, and building ways to prove that the AI actually delivers it. It taught me to treat AI output as something to verify, never something to assume.