AI Struggles to Audit Smart Contracts

Recent testing indicates that while generative artificial intelligence (AI) is capable of performing a vast array of tasks, OpenAI’s ChatGPT-4 is unable to audit smart contracts as effectively as human auditors.

To determine whether AI tools could replace human auditors, Mariko Wakabayashi and Felix Wegener of blockchain security firm OpenZeppelin pitted ChatGPT-4 against the firm’s Ethernaut security challenge.

Although the AI model passed most levels, it struggled with those introduced after its training data cutoff date of September 2021, as the plugin enabling web connectivity was not included in the test.

Ethernaut is a wargame played on the Ethereum Virtual Machine with 28 hackable smart contracts or levels. In other words, levels are completed upon discovering the appropriate exploit.

According to testing conducted by OpenZeppelin’s AI team, ChatGPT-4 could identify the exploit and pass 20 of the 28 levels but required additional prompting to solve some stories after the initial question: “Does the following smart contract contain a vulnerability?”

Wegener explained that OpenZeppelin expects its auditors to be able to complete all Ethernaut levels, as should all competent authors.

While Wakabayashi and Wegener concluded that ChatGPT-4 could not currently replace human auditors, they noted that it could be used to improve the efficiency of smart contract auditors and identify security vulnerabilities.

Wakabayashi stated in a May 31 Twitter thread that large language models (LLMs) such as ChatGPT are not yet ready for smart contract security auditing because the task requires a high level of precision and LLMs are optimized to generate text and have human-like conversations.

Wakabayashi suggested, however, that an AI model trained using customized data and output goals could provide more dependable solutions than chatbots currently available to the public that have been prepared using massive amounts of data.