GPT-4’s Role in Smart Contract Security

GPT-4 is deemed ineffective for security assessments due to low recall rates and accuracy, with a maximum accuracy of 33%.

A team of researchers from Salus Security—a blockchain security firm with locations in the US, EU, and Asia—recently released research demonstrating GPT-4’s abilities to parse and audit smart contracts.

Despite AI’s apparent proficiency in code generation and parsing, it is not a smart choice for a security assessment.

As per the article:

“GPT-4 can be a useful tool in assisting with smart contract auditing, especially in code parsing and providing vulnerability hints. However, given its limitations in vulnerability detection, it cannot fully replace professional auditing tools and experienced auditors at this time.”

The researchers from Salus evaluated the AI’s capacity to identify possible security flaws across seven prevalent categories of vulnerabilities using a dataset consisting of 35 smart contracts (referred to as the SolidiFI-benchmark vulnerability library), which had a grand total of 732 vulnerabilities.

Their research shows that ChatGPT is effective at identifying real vulnerabilities, or “true positives,” that warrant further investigation even when run in a controlled setting. In testing, it achieved an accuracy of over 80%.

It seems, nonetheless, that it struggles with producing false negatives. An indicator of this is the “recall rate,” which dropped to 11% in the trials conducted by the Salus team (a higher number is preferable).

The results show “that GPT-4’s vulnerability detection capabilities are lacking, with the highest accuracy being only 33%,” according to the researchers. Therefore, until AI systems like GPT-4 are fully functional, the researchers suggest auditing smart contracts using specialized tools and good ol’ fashioned human expertise.

“In summary, GPT-4 can be a useful tool in assisting with smart contract auditing, especially in code parsing and providing vulnerability hints. … When using GPT-4, it should be combined with other auditing methods and tools to enhance the overall accuracy and efficiency of the audit.”