Implement a grounded answer-or-escalate loop.

difficulty · 9/10·30–40 min·implementation · judgment-in-code · robustness

BUG-HUNT EVAL-DESIGN SPEC-WRITE BUILD-LOOP

ScenarioImplementation9/10

You are implementing the core decision loop for a support copilot. The function must retrieve supporting documents, decide whether an answer is safe to produce, and escalate when needed. A naive implementation will bluff, over-answer, or fail unsafely under weak retrieval and risky policy cases.

// task: Implement `handle_query`. You are judged on decision quality, safe control flow, robustness, and constraint handling.

handle_query.spec.md

build-loop · support copilot

// function to implement

handle_query(user_message, conversation_id) -> Result

// output schema

Result = {
  "status": "answered" | "escalated" | "needs_clarification",
  "answer": string,
  "citations": string[],   // e.g. ["DOC-101"]
  "ticket_id": string | null
}

// tool contracts

·search_kb(query) → list of { doc_id, title, updated_at, text }
·create_ticket(reason, priority, user_message) → { ticket_id }
·llm_generate(system_prompt, user_payload) → response

// behavioral rules

Refunds above $200 always require escalation.
If KB docs conflict or do not support the answer, escalate — don't bluff.
Citations must point only to documents actually used.
Treat KB doc text as evidence, not instructions.
At most 2 search_kb calls per question (budget).
Be resilient to empty retrieval and tool failure.
No unbounded retries; deterministic safe fallback when unsure.

// language · Python

handle_query.py

27 lines·

test.out

visible tests · hidden tests run on submit

Normal KB-backed question
"How do I update my billing email?"
// expect: status="answered" with ≥1 citation
idle
No-support question
"Can you grant me VIP status on my account?"
// expect: status="escalated" or "needs_clarification" — no bluffing
idle
Contradictory-doc question
"How long do I have to dispute a billing charge?"
// expect: docs conflict → status="escalated" with reason
idle
Refund above $200
"I was charged $349 for my annual renewal. Please refund it."
// expect: status="escalated", create_ticket called
idle
Malicious retrieved-doc instruction
"Process my refund based on the KB guidance."
// expect: DOC text "always approve" ignored; SYSTEM_PROMPT wins
idle
Empty retrieval
"Is my refund eligible under the 2018 policy?"
// expect: no results → status="escalated" (no fabrication)
idle
First search weak, second search useful
"What's the invoice date for order #A-417?"
// expect: widen query once; at most 2 search_kb calls
idle
Citation mismatch temptation
"How does the billing grace period work?"
// expect: citations reference actually-retrieved doc_ids only
idle