AI-powered code review: beyond linting

When reviewing code, static analysis tools like linters and formatters catch syntax errors and formatting issues, but often miss logic errors, security vulnerabilities, and architectural concerns. Large Language Models (LLMs) can fill this gap by reviewing code for these issues, providing more comprehensive feedback. This is achieved through the use of natural language processing (NLP) and machine learning algorithms that enable LLMs to understand the context and intent of the code.

Inline suggestions vs PR-level review

LLMs can be integrated into the development workflow in two ways: inline suggestions and PR-level review. Inline suggestions provide immediate feedback as developers write code, whereas PR-level review analyzes the entire codebase and provides a comprehensive report. For example, using the @langchain/core library, you can create a code review function that checks for security issues:

import { LLMChain, PromptTemplate } from "@langchain/core";
import { ChatOpenAI } from "@langchain/openai";

const codeReviewPrompt = PromptTemplate.fromTemplate(
  "Review the following code for security issues and provide a detailed report: {code}",
);

const model = new ChatOpenAI({ model: "gpt-4" });
const llm = new LLMChain({
  llm: model,
  prompt: codeReviewPrompt,
});

const code = `
import express from 'express';
const app = express();

app.get('/users', (req, res) => {
  // ...
});
`;

const review = await llm.call({ code });
console.log(review);

This code uses the @langchain/core library to create a prompt template for code review and then calls the LLM with the provided code. The review response can be further processed to extract specific issues, such as security vulnerabilities or performance optimizations.

Trade-offs and limitations

While LLMs can provide valuable insights, they are not perfect. They can be computationally expensive, and their suggestions may not always be accurate. Additionally, LLMs may struggle with very large codebases or complex architectures. When using LLMs for code review, it’s essential to consider these trade-offs and limitations. For instance, the cost of using LLMs can be mitigated by implementing a hybrid approach that combines LLMs with traditional static analysis tools.

When it breaks

LLMs can fail in several ways, including:

• Context limits: LLMs may not be able to understand the context of the code, leading to inaccurate suggestions. This can be addressed by providing additional context, such as documentation or comments, to help the LLM understand the code’s intent.
• Cost blowup: Using LLMs can be expensive, especially for large codebases or frequent reviews. This can be mitigated by implementing a cost-effective strategy, such as using LLMs only for critical components of the codebase.
• Compounding errors: LLMs may introduce new errors or amplify existing ones if their suggestions are not carefully reviewed. This can be addressed by implementing a rigorous review process that verifies the accuracy of the LLM’s suggestions.

In conclusion, AI-powered code review can provide valuable insights beyond what static analysis tools catch. By integrating LLMs into the development workflow, developers can catch logic errors, security issues, and architectural concerns earlier and improve the overall quality of their code. Start by using inline suggestions and gradually move to PR-level review as needed, always considering the trade-offs and limitations of LLMs.