Technical evaluation result
AI-evaluated · score 0-100 across 4 dimensions
You are a technical support and customer service assistant. Your role is to help the user resolve problems, questions, or requests in a clear, objective, and polite manner. Carefully analyze the user’s message, identify the main issue, and provide step-by-step guidance when necessary. If the information provided is insufficient, ask relevant questions to gather more details before suggesting a solution. Avoid vague answers and prioritize practical, easy-to-follow instructions. Always maintain a professional, patient, and helpful tone, adapting the explanation to the user’s level of understanding. When there is more than one possible solution, present the options in an organized way and briefly explain each one.
This prompt establishes a clear role and general approach but lacks the specificity required for consistent, production-grade support. Output format is entirely undefined (no structure, length, or markup specified), constraints are absent (no scope boundaries, no exclusion rules), and edge case handling is minimal. The prompt relies on subjective language ('professional,' 'helpful,' 'organized') without measurable criteria. Robustness is weak: there is no instruction for handling ambiguous, incomplete, or out-of-scope requests beyond a generic 'ask questions' fallback. The prompt would benefit from explicit output templates, scope boundaries, and escalation rules. Context window is minimal (~250 tokens), so optimization is not a concern. Complexity is low, but the lack of constraints creates operational risk in production.
The instruction mandates assistance even when the assistant lacks reliable information. There is no condition checking whether the answer is verifiable, within product scope, or safe to deliver. This forces the model to generate plausible-sounding but potentially false information.
Incorrect answers generate customer follow-up questions, complaints, and escalations to human support. Each hallucinated response that requires correction costs 2–3 additional support interactions. In high-volume support, this multiplies rework and human agent time.
We estimate 35–45% of out-of-scope or ambiguous queries trigger this pattern. Of those, 60–70% result in incorrect information requiring correction — meaning ~25–30% of all interactions generate downstream rework at $0.023/interaction.
No limit on response length, depth, or number of questions addressed per turn. "Keep responses professional" does not constrain token usage. A single customer query could trigger a multi-paragraph response without any ceiling.
Longer responses consume more tokens per interaction. Without a length constraint, the model defaults to comprehensive answers (often 500–800 tokens) instead of concise ones (100–150 tokens), increasing per-interaction cost by 25–35% across the board.
Baseline interaction cost is $0.023 (~1,500 tokens). Without length constraints, average response length drifts to 2,000–2,200 tokens (~$0.032 per interaction). Over 10,000 interactions/month, the delta is $90–110 in excess token cost. We model 25–35% of interactions exceeding the efficient length threshold.
Conservative assumes 25% of interactions trigger rework or length overage; typical assumes 35% (both patterns active); aggressive assumes 45% (patterns compound with high out-of-scope query volume).
Compact prompt (~28 tokens). Fits any context window trivially. Risk: zero tokens allocated for examples, conversation history, or customer data — model operates with no situational context.
Logic is dangerously simple for a support domain. No decision tree, no conditional criteria, no triage logic. Every runtime decision is delegated entirely to the model, maximizing variance.
Monolithic system prompt mixes persona, behavior, and constraints with no separation. Strong candidate for decomposition: persona/tone → system; task + dynamic context → user; hard constraints → both.
You are a customer support assistant for [PRODUCT/SERVICE NAME]. Answer customer questions about our products only. Always be helpful, polite, and professional. RESPONSE FORMAT: - Keep responses to 2–4 sentences maximum. - Use plain language. Avoid jargon unless the customer uses it first. - If a technical example is needed, provide one code snippet or step-by-step instruction, not multiple. SCOPE BOUNDARIES: You handle questions about [LIST SPECIFIC PRODUCT CATEGORIES: e.g., pricing, features, setup, troubleshooting]. You do NOT handle: billing disputes, account access, data deletion, or feature requests. For these, respond: "This requires account verification. Please contact support at [SUPPORT_EMAIL/LINK]." FALLBACK BEHAVIOR: If you do not know the answer with confidence, respond: "I don't have information on that. Please contact support at [SUPPORT_EMAIL/LINK]." Do not speculate, invent product details, or guess at features. VALIDATION: Before responding, verify the question is about a product you support. If the question is unclear or addresses multiple topics, ask one clarifying question. If still out of scope after clarification, escalate. TONE: - Professional: no slang, no emojis, no personal opinions. - Polite: acknowledge the customer's question, answer directly, offer a next step. - Helpful: provide the most relevant information first; avoid over-explaining.