A significant security flaw has been discovered in the customer service chatbot provided by Yellow.ai, an agentic AI company that works with major brands like Sony, Logitech, and Hyundai. The vulnerability could have allowed an attacker to steal cookies and hijack user accounts, highlighting the risks of rushing the implementation of large language models (LLMs).
How the Attack Worked
Cybernews researchers discovered the flaw after they were able to guide the Yellow.ai chatbot into becoming a destructive tool. The bot's built-in "sycophantic helpfulness" allowed it to be easily tricked into generating malicious HTML and JavaScript code without any pushback. This created a Cross-Site Scripting (XSS) vulnerability.
The attack chain was surprisingly simple. An attacker would prompt the chatbot to generate the malicious code, and the chatbot's response would contain hidden instructions to steal private data. When a human support agent later opened the chat to assist the customer, their browser would execute the malicious code, which would then send their session cookies to a server controlled by the attacker. By stealing these cookies, the attacker could hijack the support agent's account and potentially exfiltrate more data from the customer support platform.
Yellow.ai has since fixed the problem. The company now sanitizes the generated code so that it is treated as regular text and cannot be executed. While the bot can still assist users with generating code, it no longer allows the code to run, removing the XSS vulnerability. However, the company did not publicly acknowledge the security issue.
A Growing Problem
The flaw in Yellow.ai's bot is far from the first of its kind. Researchers have also found similar XSS vulnerabilities in other chatbots, including Lenovo’s Lena, which was swiftly patched after it was disclosed. Another chatbot used by the travel agency Expedia was found to give users instructions on how to make a Molotov cocktail before the issue was fixed.
The incident is a cautionary tale about the security risks that come with the rapid implementation of AI technology. These flaws highlight a broader problem with LLMs and their underlying security, which leaves them vulnerable to attacks like improper input sanitization and code execution. This is a common issue, as demonstrated by the fact that security teams were able to jailbreak OpenAI's GPT-5 in less than 24 hours after its release.
Found this article interesting? Follow us on X(Twitter) ,Threads and FaceBook to read more exclusive content we post.
