Photo credit: www.darkreading.com
As developers increasingly utilize AI programming assistants to assist with coding, recent research emphasizes the necessity of thoroughly evaluating code suggestions prior to their integration into existing projects to prevent the introduction of potential security vulnerabilities.
A research team from three universities unveiled insights last week into methods of corrupting training datasets, leading to scenarios in which large language models (LLMs) could be coerced into generating insecure code. This new approach, termed CodeBreaker, enables the crafting of code samples that evade detection by static analysis tools, yet can still guide AI-driven code-completion assistants to propose flaws and exploitable code to developers. The technique enhances previous poisoning strategies, demonstrating a greater ability to conceal harmful code, thereby effectively embedding backdoors during the coding process.
Shenao Yan, a doctoral candidate specializing in trustworthy machine learning at the University of Connecticut and a contributor to the paper presented at the USENIX Security Conference, stresses that developers must rigorously examine any code provided by LLMs rather than blindly copying and pasting snippets. He states, “It is crucial to train the developers to foster a critical attitude toward accepting code suggestions, ensuring they review not only functionality but also the security of their code.”
Despite the continuing evolution of code-generating tools, the inclusion of insecure code within developer resources isn’t a novel problem. Vulnerabilities within tutorials and code suggestions on platforms like StackOverflow have previously been documented, with findings showing that out of 2,560 C++ snippets, 69 contained flaws that were replicated across more than 2,800 public projects.
This recent study falls within a broader context illustrating how AI models are vulnerable to manipulation through the incorporation of malicious examples within their training data, according to Gary McGraw, co-founder of the Berryville Institute of Machine Learning. He remarks, “LLMs become their data, and if the data are poisoned, they happily eat the poison.”
Bad Code and Poison Pills
The findings from the CodeBreaker project build upon earlier initiatives, such as COVERT and TrojanPuzzle. The basic strategy of data poisoning involves introducing vulnerable code samples into LLM training datasets, leading to compromised code suggestions. The COVERT technique circumvents static detection by embedding insecure suggestions within comments or documentation of a program. Improving on this approach, TrojanPuzzle utilizes a diverse array of samples to instruct an AI model in generating insecure code.
CodeBreaker advances these concepts through code transformations that produce vulnerable code, which functions correctly but eludes detection by mainstream static analysis security tests. David Evans, a computer science professor at the University of Virginia and co-author of the TrojanPuzzle paper, notes that the development illustrates the potential for executing realistic attacks through methods that do not overtly manifest malicious code.
While the techniques for poisoning LLMs are concerning, many code-generating models already contain a significant amount of vulnerable code acquired from online sources. This reality elevates the overarching risk of developers accepting outputs from code-recommendation systems without sufficient scrutiny, as emphasized by Neal Swaelens, who leads LLM Security products at Protect AI. He suggests that while initial usage might involve careful examination of generated code, complacency may set in over time as developers begin to place undue trust in the system. This phenomenon is akin to “dialogue fatigue,” where developers inadvertently approve generated code without critical evaluation.
According to Swaelens, organizations planning to connect AI systems to automated actions should prioritize the rectification of LLM inaccuracies before relying on such technology.
Better Data Selection
The developers of code assistant tools must ensure they are thoroughly vetting their training datasets, avoiding reliance on substandard security metrics that may overlook obfuscated yet malicious code, Yan advises. Popularity metrics for open-source projects, for instance, are inadequate as indicators of security, since external services can artificially enhance these ratings.
Yan elaborates, “To enhance the likelihood of inclusion in fine-tuning datasets, attackers might inflate their repository’s rating.” He illustrates that repositories often selected for fine-tuning are based on metrics like GitHub’s star ratings, where merely 600 stars can qualify a project as a top contender within the repository index.
Developers, too, should maintain a discerning perspective on code suggestions, whether sourced from AI or other online platforms. Cultivating skills in prompt crafting can further help in generating more robust code.
Moreover, there is a call for tools that empower developers to identify potentially malicious code effectively, as emphasized by Evans from the University of Virginia. He points out the critical role of code reviews in mature software development practices, which typically involve both human assessments and automated analysis tools. “This is the best hope for catching vulnerabilities, whether they are introduced by human error, inserted with malicious intent, or stem from suggestions made by tainted AI assistants,” he concludes.
Source
www.darkreading.com