Friends or Foes? Combining Static Analysis Tools and LLMs for Vulnerability Detection

Industrial products and devices (e.g., vehicles, drones) are broadly accessed and managed using web applications. These applications have been a major concern for enterprises, as they are the preferred targets of attackers due to persistent vulnerabilities in their code. To address vulnerabilities, Static analysis tools (SASTs) have been widely used for detection, alongside the growing trend of employing prompt-engineered large language models (LLMs). Although they have proven useful for detection, both techniques tend to generate false positives (FPs), thereby unnecessarily increasing manual effort in the search for non-existent vulnerabilities; moreover, SASTs tend to miss vulnerabilities. In contrast, fine-tuned LLMs have proven effective at reasoning and classification tasks, but often require expensive training with balanced corpora. In this paper, we study SASTs, both types of LLMs, and their combination to improve overall vulnerability detection in web applications. We tested two modern SAST tools and two LLM models, across seven datasets for SQL injection (SQLi) vulnerability detection. Our f indings reveal that combining the results of multiple solutions can improve vulnerability detection. The best combination integrates both LLMs and a SAST, where i) the fine-tuned LLM, together with the SAST, reduces FPs, mainly produced by the prompt-engineering LLM, and ii) both LLMs overcome SAST’s limitation of missing vulnerabilities. On average, the F1-Score increases by 17-60% when SASTS and LLMs are combined. In particular, it can improve from 6% (with a standalone solution) to ≈100% when LLMs are combined with SASTs