Test & optimize AI
How to test AI assistant
AI test zone helps simulate customer interactions before going live, ensuring your AI responds appropriately to customer inquiries.
Before AI assistant test
- Add your data (FAQs, product info, etc.)
- Add custom instruction for AI response (tone, voice, length of answer)
- Set up other settings (chatbot name, avatar, etc.)
Can I test AI assistant without enabling it?
You can still test AI assistant with data sources that you added without enabling AI assistant.
Go to test zone
Go to AI agent → Go to Test AI
You can also click Test AI in the top right of any other AI agent page, such as Settings. It opens the same test chat in a side panel, so you can try a change without leaving the page you are editing.
Chat with your AI
In chatbox, ask and interact with AI assistant as a customer. Click the reset icon in the header to clear the test conversation and start over.
Tips for better testing:
- Test one topic at a time
- Ask the same question in different ways
- Make sure responses aren't too long or too short
Test order questions with sample orders
You can test order questions before your store has real orders. Test AI has five sample orders that belong to one sample customer.
- Go to AI agent → Skills and make sure Order tracking is turned on.
- Go back to AI agent and click Test AI.
- In Type your message, send an order number with the sample email. Example:
Where is my order TEST-1002? My email is sample.customer@chatty-test.io - To test a phone lookup, send:
Has order TEST-1003 arrived? My phone is +1 555 0100
| Order | What it is |
|---|---|
| TEST-1001 | Paid, not shipped yet |
| TEST-1002 | In transit with DHL |
| TEST-1003 | Delivered with UPS |
| TEST-1004 | Refunded |
| TEST-1005 | Cancelled |
Expected result: AI first says the order is sample data, then answers with the order status. Shipped orders include a tracking link. Dates count back from today, so the answers always look recent.
Why does AI say it can't find the sample order?
The order number must come with the sample email sample.customer@chatty-test.io or the phone +1 555 0100. If the email or phone is different, AI treats the order as not found and asks you to check your details.
Can my customers see these sample orders?
No. Sample orders work only in Test AI. Real customer chats never see them.
Check the responses and make improvements
What to check in AI response:
- Is the information correct?
- Is the tone friendly but professional?
- Are product details accurate?
- Are prices and availability correct?
You can also review what data sources your AI uses to answer customer questions in test zone.
How to optimize AI responses
A. Unresolved questions
Unresolved questions are customer inquiries that your AI Assistant couldn't answer effectively.
The page counts them for the past 30 days and splits them by reason, so you can see what is really going wrong:
| Reason | What it means |
|---|---|
| Missing knowledge | Your training data has no answer for the question |
| Customer request human | The customer asked to speak to a person |
| Negative sentiment | The customer was frustrated or angry |
| Configured handover | One of your own handover rules sent the chat to your team |
We only show unresolved questions in the past 30 days
How to optimize unresolved questions
Go to Answer gaps
Go to AI agent → click the AI unresolved questions card. It opens Training data → Knowledge base on the Answer gaps tab. The tabs above the list filter questions by reason.
Check unresolved questions
What you should do with unresolved questions:
- Look for patterns in unresolved questions to identify gaps in your training data
- Group similar unresolved questions to create comprehensive answers that address various phrasings of the same inquiry
- Use unresolved questions to identify new FAQs to add to your help center
Add answers to unresolved questions
Click a question in the list, or the edit icon on its row. The Add answer window opens. Enter your answer and click Save.
The answer is saved as a new entry in your knowledge base.
Test again
Go to AI agent → Test AI and ask the question to ensure that AI assistant can handle that question now.
With duplicate or spam questions, click the delete icon on the row to remove the question from the list. You can also click Dismiss in the Add answer window to close the question without adding an answer.
B. AI responses review
Each AI response now includes a "Review sources" button to see what information was used
This helps you understand why the AI responded the way it did.
Use this feature to:
- Add missing product information the AI should have included
- Correct inaccurate policy details
- Expand on partial answers that need more detail
- Add variations of common questions that weren't properly understood
What you should do
- Identify and correct inaccurate responses
- When you notice incorrect or incomplete answers, go to Test AI and click Review sources under the response
- Click Add Q&A to add the right answer, or Add other sources to open your training data in a new tab
- Watch for patterns in corrections
- If you're frequently adding the same type of information, consider creating comprehensive entries about that topic
- Group similar corrections into comprehensive answers to improve overall AI performance
Chatty Help Center