AI Agents for IT Service Desk Triage: What Broke and What Works

Most AI triage projects fail before the model matters. The category list is a mess, the ticket text has no context, and nobody feeds corrections back.

Fix those three things and an agent can route tickets well. Skip them and it will route tickets confidently and wrongly.

This guide covers AI agents for the IT service desk: why first attempts break, what to clean up, how to measure routing accuracy, which tickets to leave alone, and a one-week pilot.

Why AI agents for the IT service desk fail at triage

Triage looks like an easy job for a language model. Read the ticket, pick a category, pick a team. In practice it fails in five predictable ways.

1. The categories are bad. Over the years, service desks collect categories nobody trims. Duplicates, near-duplicates, “Other,” and groups named after people who left. A model trained on that data learns the mess. ServiceNow’s own tuning guidance warns that assignment groups with the same meaning and different spellings, or empty groups, lower model precision and should be cleaned or excluded from training.

2. The ticket text is thin or noisy. Users write “it’s broken” or paste a screenshot. Researchers who studied about 26,000 real industrial IT tickets describe noisy text, subjective writing, jargon and ambiguous labels, and report that hierarchical classification worked poorly on inconsistently labeled data.

3. The agent never sees context. Who is the user? What device? Is there an open incident on the same service? Without that, the agent guesses. The same words from a developer and from a new hire in finance can mean different things.

4. The agent must always answer. If the system forces a category on every ticket, it creates false coverage. A triage agent needs a “I don’t know” path that goes to a person.

5. No feedback loop. Every time an analyst moves a ticket, that is a label. If nobody captures it, the agent never improves, and trust drops.

Skip any of these and the project drifts. A triage project with no accuracy number is exactly the kind that gets canceled.

Clean the category list first

Do this before you touch a model. It is cheap, and it helps your human analysts too.

1. Export 90 days of tickets with category, assignment group, resolution group and reassignment history.
2. Count tickets per category. List the bottom 30 percent. Most can be merged.
3. Find the catch-alls. If “Other” or “General IT” holds a large share, the taxonomy has failed. Split it using real ticket text.
4. Define each category by who resolves it. A good category answers “which team does the work?” and not “what topic is this?”
5. Write one sentence per category saying what belongs and what does not. Add two example tickets.
6. Cap the list. Fewer than 25 routing targets is a good starting point. I’d rather start with 12 clean ones than 80 messy ones.
7. Add an “unclear” bucket on purpose. Tickets land here when the agent is not sure.

The TickIt research team at ByteDance did something similar. They predefined ticket categories based on the responsibilities of support analysts and used an “Others” class for tickets that did not need escalation. Categories tied to who does the work gave the model something it could learn.

One more warning from the ServiceNow guidance: if 90 percent of tickets already land in one group, you do not need a model to predict that. Look at the spread before you build.

Routing before resolving

Start with routing. Do not start with resolution.

Routing means: read the ticket, add context, pick the team and priority, and leave the work to a human. It is low risk because a wrong route costs minutes. A wrong resolution can lock out a user, grant access, or change a system.

Here is the order I recommend.

Stage	What the agent does	Risk	Human role
1. Suggest	Proposes category and team in a side field	Very low	Analyst accepts or changes
2. Route	Assigns low-risk categories automatically	Low	Analyst reviews exceptions
3. Enrich	Adds user, device, related incidents, known fix links	Low	Analyst uses it
4. Resolve	Handles narrow, reversible requests	Medium	Spot checks and audit

Only move to the next stage when the numbers from the stage before hold up. This is the same gate logic in our guides on human in the loop for AI agents and AI pilot success criteria.

Priority is part of routing. Most service desks set priority from impact and urgency, and Atlassian’s documentation defines priority as a result of those two. Have the agent suggest impact and urgency separately, then compute priority with your existing matrix. That keeps your rules in charge and lets you grade each part.

Measuring routing accuracy

“Accuracy” alone tells you little. Measure these instead.

• First-route accuracy. The share of tickets whose first team assignment was correct, judged by the team that closed it. This is the main number.
• Per-category accuracy. Overall accuracy hides weak categories. A category with 5 percent of volume can be wrong half the time and barely move the average.
• Precision and coverage. Precision is how often the agent is right when it answers. Coverage is the share of tickets it answers at all. Raising one usually lowers the other. ServiceNow’s guidance says models with precision above 70% and coverage above 80% are typically more effective than agents sorting thousands of incidents a month. Treat that as a vendor reference point, not a law.
• Reassignment rate. How often a ticket moves after the first route.
• Time to first useful touch. Minutes until the right person sees it.
• Catch-all share. The share of tickets in “unclear” or “other.”

Accuracy depends heavily on what you ask the model to predict. In the 26,000-ticket study above, the best model reached about 98% accuracy on impact but about 77% on urgency. Urgency is more subjective. Expect your numbers to vary by label, and report them that way.

Escalation decisions are also hard. TickIt’s best method scored an F1 of 86.2% on deciding which cloud-support tickets to escalate. Their paper notes that exaggerated wording can trigger needless escalations and bland wording can hide serious problems. Those results come from one large cloud provider’s support tickets, not an internal IT desk, so use them as a sign of what is possible, not a promise.

How to build your test set

Take 200 recently closed tickets. Have the resolving teams confirm the correct route. Hide those labels from the agent, run it, and compare. Use tickets from different months so seasonal spikes are covered. Do not tune the agent on this set and then report the same set as your result.

Tickets to leave alone

Some tickets should stay with people, at least at first. My rule: leave it alone if a wrong action is hard to undo or carries security weight.

Ticket type	Why to leave it	What the agent can still do
Access grants to sensitive systems	A wrong grant is a security event	Route to approver, attach request context
Account recovery for privileged users	High abuse risk	Flag and route
Suspected security incidents	Needs trained judgment	Route to security fast
Executive or VIP issues	High visibility, custom handling	Route and notify
Outage reports tied to an open major incident	Context changes by the minute	Link to the incident
Hardware swaps, physical work	Needs a person on site	Route and book
Ambiguous, angry or multi-issue tickets	Misread risk is high	Send to “unclear”
Anything the agent scores below its confidence threshold	By design	Human queue

Password reset and unlock requests are the classic automation target. They are high volume and repetitive. They are also the request attackers impersonate. If you automate them, use your identity provider’s verified self-service flow rather than letting an agent judge a chat message. I could not find a reliable public figure for what share of tickets are password resets, so measure your own mix before you build a business case.

OWASP’s guidance on excessive agency names three root causes: too much functionality, too many permissions and too much autonomy. Its mitigations are to minimize permissions and require approval for high-impact actions. Keep a triage agent’s rights to a minimum, as set out in our post on least-privilege permissions for AI agents.

Feedback loop from analysts

The feedback loop is what separates a demo from a system that improves.

Capture three signals:

1. Re-routes. When an analyst changes the team, store the original guess, the new team, and the ticket text.
2. Rejected suggestions. In suggest mode, log what the analyst saw and what they chose.
3. A one-click reason. Offer short reasons: “wrong team,” “needs more info,” “agent missed context,” “category does not exist.”

Review the log weekly. Most weeks you will find one of three fixes: a category to split, a context field to add, or a rule to hard-code. Hard rules are fine. If “VPN” plus “new laptop” always goes to one team, write the rule and let the model handle the rest.

Keep humans in charge of the taxonomy. Categories drift as services change. One person should own the list and approve changes.

The step from “triage works” to “agent handles whole requests” is covered in AI agent escalation design, which shows how to hand a case to a person without losing context.

A one-week pilot scope

Our one-week pilot method fits triage well because the scope is narrow and the output is easy to grade. Here is a plan you can run in five working days.

Day 1: Data and categories. Export 90 days of tickets. Clean the category list to 12 to 25 targets. Name an owner for each.

Day 2: Test set and baseline. Build the 200-ticket test set. Measure how the current process does on first-route accuracy and reassignment rate. This baseline matters more than the agent’s score.

Day 3: Build suggest mode. The agent reads ticket text plus user and device context, and writes a suggested team, impact, urgency and confidence into side fields. It changes nothing.

Day 4: Shadow run. Run it on live tickets for a full day. Analysts do their normal work. Compare its suggestions with what they did.

Day 5: Review and decide. Review per-category accuracy, coverage, and the list of misses. Decide per category: keep in suggest mode, allow auto-route, or drop.

Pass bar (my suggestion, adjust it): auto-route only categories where the agent beats the human baseline on first-route accuracy, with no serious misses in the test set. Everything else stays in suggest mode.

Write the pass bar down before day 1. For the money side, see AI agents for back-office operations, and for the same ideas applied to external customers, AI agents for customer service.

For a rough value check, HDI defines cost per ticket as monthly operating expense divided by monthly ticket volume, and says handle time and agent utilization drive it most. Triage helps by cutting reassignments and wait time, not by removing tickets. Say that plainly in your business case.

Bottom Line

Fix the categories first. They decide whether any triage agent can work.

Route before you resolve. Measure first-route accuracy per category against your human baseline.

Leave access, security and anything hard to undo with people. Capture every analyst correction.

Then run the one-week pilot in suggest mode and let the numbers decide what gets automated.

FAQ

Can AI resolve IT tickets? Some, yes, but start with routing. Narrow, reversible requests such as known how-to questions can be resolved with a clear audit trail. Anything touching access or security should keep a person in the loop. Prove routing accuracy first, then widen scope one ticket type at a time.

How accurate is triage routing? It depends on the label and the data. In published research on about 26,000 IT tickets, one model reached about 98% accuracy on impact but about 77% on urgency. Your result will depend on category quality, so build a 200-ticket test set and measure it yourself.

Which tickets should stay with humans? Sensitive access grants, privileged account recovery, suspected security incidents, VIP issues, tickets tied to an open major incident, and anything ambiguous or below the agent’s confidence threshold. The agent can still route and enrich these.

How do I fix bad categories? Export 90 days of tickets, merge near-duplicates, split the catch-alls, and define each category by which team does the work. Keep the list short, add an “unclear” bucket, and give one person ownership of changes.

What does a first-week pilot cover? Clean categories on day 1, a 200-ticket test set and baseline on day 2, a suggest-only agent on day 3, a live shadow run on day 4, and a per-category go or no-go review on day 5. The agent changes nothing until the numbers justify it.

Three Ways to Work With TMA

Need an agent built? We deploy production AI agents in your infrastructure. Working pilot. Real data. Measurable ROI. → Schedule Demo

Want to co-build a product? We’re not a dev agency. We’re co-builders. Shared cost. Shared upside. → Partner with Us

Want to join the Guild? Ship pilots, earn bounties, share profit. Community + equity + path to exit. → Become an AI Architect