Debugging + monitoring — call logs + rejection-rate dashboard

Authoring an agent and operating it are different seats. This lesson designs the monitoring surface that surfaces whether the agent is operating correctly at a glance.

Agent call logs — the primary debugging entry point

D.Hub agents are designed so that every call writes one row to a dataset. The om_event_history dataset on map_control is that pattern.

Columns that belong on one call-log row:

• event_id — Unique call identifier
• agent_name — Which agent's call
• user_id — Who called
• input_text — The user's natural-language input
• intent_classification — Classification result (intent + confidence)
• tools_called — Tools invoked, with input + output summaries
• llm_recommendation — LLM's recommendation + grounding
• human_decision — Reviewer's decision (HITL case)
• actors_fired — Actors that fired
• created_at — UTC timestamp

This one row carries the full trace of one call and becomes the primary entry point for post-hoc debugging.

Operational dashboard — four signals

Build a dashboard with four operational widgets on top of the call log.

1. Intent distribution (donut)

Distribution of intent_classification. Ratio of allowed categories (e.g. UI_CONTROL) vs rejected categories (DATA_QUERY, SYSTEM_CONFIG, UNKNOWN).

Operational signal — when rejection rate exceeds 30% persistently, check two things:

• Is the classification LLM's allowed-category definition too narrow (over-conservative)?
• Are users routing to the wrong agent (separate-tool guidance needed)?

2. HITL agreement rate (statistic)

Agreement rate between llm_recommendation and human_decision. The rate at which the reviewer accepts the recommendation as-is.

Operational signal — when agreement rate exceeds 95% persistently, HITL's value is effectively zero (the warning from Lesson 3). Two possible causes:

• The LLM recommendation is accurate enough for reviewers to trust (good — automation candidate)
• Reviewers lack the context to judge and accept by default (incident-prone)

To distinguish the latter, add an override reason column to the reviewer UI so they can leave one line on why they decided differently from the recommendation.

3. Per-tool call frequency (bar)

Unnest tools_called and aggregate per-tool call counts. Which tools get used how often.

Operational signals:

• Tools with 0 or very low call counts — either unnecessary tools or tools the LLM can't figure out when to use. Check the tool description (Lesson 2).
• Tools called abnormally often — the LLM calls the same tool multiple times due to missing context. Check whether the output schema carries everything needed for one call.

4. RAG search similarity distribution (histogram)

Distribution of similarity scores from search_* tool outputs.

Operational signal — when many calls land below 0.5, two possible causes:

• Insufficient knowledge content (no paragraph that answers the question). Augment the policy body.
• The user's question patterns are misaligned with the knowledge body's § structure. Redesign the body's § split.

Post-hoc debugging — why did that recommendation come out?

When a complaint of the recommendation differed from intent lands in production, pull the call-log row and check four points.

1. Intent classification result — Did the intent latch onto an allowed category? Confidence sufficient?
2. Tool output — Did the tools return accurate context? Is the dataset row fresh?
3. RAG retrieval results — Were the retrieved paragraphs actually relevant to the question? Similarity score?
4. LLM reasoning — Given that context, did a reasonable recommendation come out? Traces of hallucination?

Most incidents trace to (2) or (3) — input context quality. LLM reasoning being wrong on its own is unexpectedly rare.

Wrap-up — the agent-builder cycle

If you made it this far, you've handled D.Hub's agent-builder cycle end-to-end — three axes (Tool · Actor · Intent) → tool definition → HITL → RAG → intent gating → monitoring.

Three next directions:

• The three workshops — refund_approval · docai · map_control — walk the full flow end-to-end so you can see how each pattern fits in a real scenario. This Path was the textbook; the workshops are the integrated exercises.
• Admin Path — When an agent rolls out organization-wide, permission · policy · audit surfaces stack on top.
• Ontology Modeler Path — Modeling for when the agent's knowledge grounding extends to a graph base.

Great job.