
An AI chatbot on the website can answer questions, find information and direct visitors to the right person. It can also be confidently wrong. The latter is a trait the technology shares with certain project meetings, but that does not make it any less important to manage.
The specific question, therefore, is not whether the company can install an AI chatbot. In most cases, it can. The question is whether the chatbot solves a clearly defined task better than good content, a search field, a form or a phone number. A bouncing speech bubble is not automatically customer service. Sometimes it is simply the website’s answer to a salesperson jumping out from behind a shelf.
Start with the problem, not the chatbot
An AI chatbot makes the most sense when visitors ask many variations of the same questions, and the answers are found in a defined and maintained knowledge base. Typical examples include questions about products, delivery, opening hours, documentation, ordering processes or how to use a service.
It is less suitable when every answer requires individual assessment, legal advice, access to sensitive information or up-to-date information from several systems. In such cases, the chatbot can still help collect the necessary information and explain the next step, but it should not pretend to have decision-making authority.
Three questions you should be able to answer
- What specific task should the chatbot solve? For example, reducing a particular type of inquiry or helping users find the right product documentation.
- What content should the answers be based on? It must be clear who owns the content, where it is stored and how it is updated.
- What should happen when the chatbot does not know? It must be able to say so, suggest relevant alternatives and transfer the user to a person or another channel.
If the answer to the first question is “increase the pace of innovation”, you currently have a buzzword, not a use case. Ask for an example from an actual customer journey: What does the user ask, what should the chatbot answer and what is the desired outcome?
Understand what kind of chatbot you are actually buying
A rule-based chatbot follows predefined choices and responses. It is predictable, but handles language variation less effectively. A generative AI chatbot can interpret free text and formulate responses, but requires stricter guardrails and quality control.
Many solutions use a method in which the language model retrieves relevant information from the company’s approved content before answering. This is often referred to as RAG, or retrieval-augmented generation. Think of the solution as a librarian allowed to use one controlled bookshelf: It can find and summarize the content quickly, but it should not fill in missing pages with its own imagination.

Vendors may describe this as simply uploading your documents and letting the “magic” do the rest. The magic often turns out to be a search field wearing a wizard’s cloak. The documents still need to be accurate, well organized, understandable and free of conflicting information.
The knowledge base determines quality
The chatbot should only use content you have approved for the purpose. Outdated product sheets, internal notes and three nearly identical versions of the return policy do not create a rich knowledge base. They give a machine more opportunities to choose incorrectly with impressive fluency.
- Remove outdated and duplicate content.
- Label content with an owner, scope of validity and date for the next review.
- Distinguish between public information and internal material.
- Use clear terminology consistently across websites and documents.
- Clarify which topics the chatbot should not answer questions about.
A good knowledge base also improves the website and customer service in general. If you discover that no one can explain the current process without opening six PDFs and calling a former employee, the chatbot is not the main problem.
Clarify privacy, security and responsibility
Visitors may enter names, contact details, customer numbers, health information or other personal data even if they are not asked to do so. You therefore need to know what data the solution collects, where it is processed, how long it is stored, who has access and whether the data is used to train models further.
This must be assessed based on the specific vendor, agreement, data types and the organization’s use. Marketing phrases such as “secure AI” are not a technical specification. Ask for precise answers about data processing, access control, logging, deletion, subcontractors and the geographical location of processing. Where processing involves higher risk, involve privacy and security officers early.
The chatbot must also be protected against instructions attempting to make it ignore its rules, disclose internal information or use unauthorized content. Such attempts cannot be handled with one solemn system prompt and optimism. Limit data access, filter content, test misuse scenarios and ensure that the model cannot perform actions it does not need to perform.
Assign responsibility to named roles
- Content owner approves what the chatbot can base its answers on.
- Service owner follows up on goals, costs and user experience.
- Technical owner handles integrations, access and errors.
- Customer service or subject-matter owner reviews conversations and improves the responses.
- Privacy and security officer assesses data processing and the necessary measures.
It should be easy to stop or restrict the chatbot if quality declines, content becomes outdated or the vendor changes the solution. An emergency brake is less glamorous than an AI strategy document, but considerably more useful when something actually happens.
Test with real questions before full launch
Start with a limited area and a representative set of questions from search logs, customer service and customer-facing employees. Include simple questions, unclear wording, typos, conflicting premises and questions the chatbot should reject.
Assess more than whether the answer sounds good. Check whether it is correct, relevant, understandable and based on approved content. Also test whether the chatbot acknowledges uncertainty, refers the user onward at the right time and retains the necessary context throughout the conversation.

- Choose one clearly defined use case with low to moderate risk.
- Build a controlled knowledge base.
- Create a test bank with realistic and challenging questions.
- Have subject-matter experts assess the answers against set criteria.
- Run a limited pilot and review conversation logs regularly.
- Correct the content and rules before expanding to more topics.
Measure whether users get their task completed, whether the chatbot provides correct answers, how often it needs to refer users onward, and which questions it does not cover. The number of conversations alone says little. A fire alarm is not successful simply because it makes a lot of noise.
At the same time, create a simple path to human help. The chatbot should be able to offer a contact form, phone, email or another relevant channel without forcing users through a digital obstacle course. A process involving nine clarification meetings to approve the word “customer service” is admittedly thorough, but hardly seamless.
When should you say yes, wait or walk away?
Say yes to a pilot when you have a clearly defined use case, a lot of repetitive information, a well-organized knowledge base and resources for ongoing follow-up. The chatbot should have clear boundaries and a good transition to human help.
Wait when the content is outdated, responsibility is unclear or no one has the capacity to review conversations after launch. Use the time to clean up the website and documentation instead. That work will be useful regardless of which technology you choose later.
Walk away if the solution is primarily intended to spruce up the website, or if incorrect answers could have serious consequences without you being able to establish adequate controls. Good search, clear navigation and accessible subject-matter experts are not old-fashioned. They are simply less suited to product demos with dramatic background music.
A useful AI chatbot is therefore not one that can talk about everything. It is one that solves a clearly defined task, uses approved information, knows its own limitations and refers the user onward when appropriate. That sounds almost ordinary. It is usually a good sign.




