Skip to content
AI Tools in Practice

Is CodeRabbit Worth It? What It Flagged in 4 Real Open-Source Pull Requests

By AI Tools in Practice ·

Short answer: CodeRabbit is good at catching the boring-but-costly mistakes a tired reviewer misses: missing indexes, unsafe defaults, files committed by accident, broken links after a refactor. It also produces a lot of comments that developers reasonably decline. In the four pull requests we read, the most useful comments were the ones CodeRabbit itself rated Major or Critical; the ones developers pushed back on were mostly Trivial refactors or design choices the bot did not understand. It is free for public repositories. For private code, the free plan only writes pull-request summaries.

How we did this

CodeRabbit posts its reviews as normal GitHub comments, so on public repositories anyone can read what it said and how the humans responded. We picked four recent pull requests from active projects, pulled every inline comment CodeRabbit left, and checked each one for one of three outcomes:

  • Led to a change: the author replied with a fix, or CodeRabbit marked the thread "Addressed in commit…".
  • Declined: the author explained why no change was needed.
  • No visible response (yet).

Four pull requests are not a statistical sample. They show what kind of feedback you get, not how often it is right. For the "how often", see the study further down.

Case 1: AutoGPT, 14 comments on a cost-tracking feature

AutoGPT #12651 added platform cost tracking: 51 files, about 6,400 lines. CodeRabbit left 14 inline comments (9 Major, 5 Minor), and the author answered every one.

OutcomeCountExamples
Led to a change9skip cost logging on dry runs; keep cost logs when a user is deleted (onDelete: SetNull); add a tiebreaker to paginated queries; remove transcript files linked to production that had been committed by accident
Declined, with a reason3normalize datetimes to UTC ("admin-only page… No change needed"); a suggestion to widen a type, declined because the page "follows existing admin page patterns"; a field left out of the API on purpose
Stale1"missing total_users": the field was already there; the comment referred to an older commit
Moot1A metadata mismatch in a file that was deleted anyway

The accidental transcript commit is the kind of thing that matters most: a human skimming 51 files can easily miss a fixture folder. Two of the nine accepted comments were the same suggestion (index createdAt) raised in two files. The pull request was eventually closed without being merged, so we cannot say how the final code looked.

Case 2: Asymmetric, 27 comments on a UI library migration

Asymmetric #309 migrated a frontend from Radix UI to Base UI: 261 files, merged in July 2026. CodeRabbit left 27 comments: 1 Critical, 5 Major, 8 Minor and 13 Trivial.

  • 18 led to a fix, including the only Critical one: "Collapsed nav rail links no longer navigate", a real regression introduced by the migration.
  • 9 were declined or deferred, and all 9 were comments CodeRabbit itself had labeled Trivial: extract a shared helper, hoist a repeated class string, switch Tailwind syntax. The maintainer's replies followed one line, for example: "Valid refactor to hoist the repeated transition utility, but out of scope for this migration PR."

That pattern is useful: in this pull request, the severity label was a good filter. Read the Critical and Major comments first.

Case 3: LangWatch, 336 comments on a single pull request

LangWatch #6900 was huge: 904 files and more than 120,000 added lines, merged in September 2026. CodeRabbit left 336 inline comments (103 Major, 176 Minor, 57 Trivial). The maintainer replied to 305 of them. By our rough keyword count, about 267 replies describe a fix and about 32 decline or defer.

The declines are worth reading because they show where the bot lacks context. In one thread the maintainer rejected a quoting fix after testing it: "verified empirically against ClickHouse… a digit-leading ALIAS needs no quoting". Several nitpicks got the same reply: "the current form follows the surrounding file's conventions and changing it buys no behavior."

The lesson is about cost, not accuracy: someone had to read and answer 336 comments. On very large pull requests, CodeRabbit can create as much review work as it saves, unless you tune it (see below).

Case 4: CARE, 67 comments still waiting

CARE #16618, from the Open Healthcare Network, rebuilds a questionnaire builder (701 files). It was still open when we read it. Of CodeRabbit's 67 comments, 22 are marked as addressed and 45 have no visible response yet.

About 17 of the 67 are accessibility issues: missing accessible names, radio buttons without a group, controls unreachable by keyboard. Only one of those was marked as addressed. CodeRabbit also flagged a cache key that ignored the organization ID, which could mix data across organizations; that one was fixed. A security comment suggesting an allowlist instead of a denylist was still open.

If your team does not already review for accessibility, expect CodeRabbit to surface a lot of it. Whether that is noise or value depends on whether you plan to act on it.

What the research says

A 2026 preprint from researchers at the University of Melbourne and Monash University (posted on arXiv, so not yet peer-reviewed as far as we can tell), "Is Agentic Code Review Helpful?", analyzed 31,073 CodeRabbit comments and the developer responses across 10,191 pull requests in 239 open-source repositories (September 2023 to September 2025).

Developer responseShare
Accepted (changed the code)36.4%
Led to discussion7.3%
Rejected56.3%

Most rejections were invalid suggestions (false positives, out of scope or redundant); the rest clashed with an intentional design trade-off or the developer's preference. The authors labeled most comments with GPT-5.1, checked against a hand-labeled sample. So, across many projects, a bit more than one in three CodeRabbit comments leads to a code change. Three of the four pull requests above had higher acceptance (CARE, at about one in three so far, is in line with the study), but those maintainers clearly engaged with the bot. Do not expect that rate by default.

What other developers report

  • Useful, but noisy. On Hacker News (January 2026), a user running it on GitLab said it "regularly finds problems, including subtle but important problems that human reviewers struggle to find" and, in the same comment, that it "regularly complains about things that are possible in theory but impossible in practice." Another wrote that it had "made PRs unreadable."
  • Mostly useful, occasionally very wrong. Davide Muttoni, who used it on a Python team, writes that "our ratio of useful comments is about 10:1". He also describes a hallucination: "it once left 25 comments on a PR insisting that Pydantic v2 wasn't enabled when it actually was." His conclusion: "This DOES NOT replace humans."
  • Low false positives, many review cycles. A three-week comparison of four AI reviewers on 146 pull requests found CodeRabbit had the lowest false-positive rate among the tools that made mistakes (2.3%), but most of its fixes were mechanical and every new commit triggered new findings: "Plan on five-to-six review cycles per non-trivial PR." Note: the author works at Sentry, whose own reviewer was in the comparison.

Is CodeRabbit free?

It depends on where your code lives. From CodeRabbit's plans documentation, read on November 18, 2026:

SituationWhat you getLimits
Public (open-source) repositoryTeam-plan features at no costRate limits vary with the repo's star count; under 10 stars, reviews must be triggered manually (@coderabbitai review)
Private repository, Free planPull-request summaries only, no line-by-line review1 PR per developer per hour
IDE extension and CLI, Free planReviews of local changes3 reviews per hour each
New organization14-day trial of the Advanced plan10 reviews per developer per hour

Paid plans, per developer per month (pricing page): Essentials $24 billed annually or $30 monthly, Team $48 or $60, Advanced $72 or $90. Some third-party articles describe the free tier differently, so check the plans page before you decide; this changes often.

How to make it less noisy

Most complaints above are about volume. Two settings in .coderabbit.yaml, both from the configuration reference, address that directly:

reviews:
  profile: quiet          # default is "chill"; "quiet" posts only the most important comments inline
  path_filters:
    - "!dist/**"          # skip build output
    - "!**/*.lock"        # skip lockfiles

The quiet profile "posts only the most important comments inline and groups the other review comments in the review summary". Given that the declined comments in our cases were mostly low-severity, that is the setting we would try first on a busy repository. A developer write-up on TabNews (in Portuguese) also recommends adding per-folder path_instructions so the bot knows your conventions instead of suggesting generic "best practices". The config also has a language option if your team prefers reviews in another language.

What happens to your code

CodeRabbit's FAQ says "your code is shared with OpenAI and/or Anthropic for reviewing purposes only" and that "CodeRabbit never uses customer code for training." It caches encrypted code and keeps longer-lived review context ("Learnings", which you can opt out of); caching can be turned off with reviews.disable_cache. The FAQ does not state retention periods. If your code is under an NDA, read the full policy first.

FAQ

Is CodeRabbit free?

For public repositories, yes: open-source projects get Team-plan features at no cost, with rate limits that depend on the repository's star count. For private repositories, the Free plan only writes pull-request summaries; line-by-line reviews need a paid plan, the IDE extension, the CLI or the 14-day trial (plans page, read November 18, 2026).

How much does CodeRabbit cost?

Paid plans are priced per developer per month: Essentials $24 billed annually or $30 monthly, Team $48 or $60, and Advanced $72 or $90 (pricing page, read November 18, 2026).

How often do developers accept CodeRabbit's suggestions?

A 2026 arXiv preprint that analyzed 31,073 CodeRabbit comments found 36.4% were accepted, 7.3% led to discussion and 56.3% were rejected, mostly as false positives, out of scope or redundant.

Does CodeRabbit train on my code?

CodeRabbit's FAQ says code is shared with OpenAI and/or Anthropic for reviewing purposes only and that CodeRabbit never uses customer code for training. It does not state retention periods.

How do I make CodeRabbit less noisy?

Set reviews.profile to quiet in .coderabbit.yaml, which posts only the most important comments inline, and use reviews.path_filters to skip generated files such as build output and lockfiles.

Verdict

  • Worth trying if you maintain a public repository (it costs nothing) or a small team where pull requests regularly wait for a reviewer. Treat it as a first pass, read Critical and Major comments first, and start with the quiet profile.
  • Think twice if your pull requests are routinely huge: expect hundreds of comments and several review rounds, as in the LangWatch case.
  • Not a replacement for a human reviewer. Even in well-run projects, a large share of its comments are declined for good reasons.

If you maintain a public repository, the cheapest test is to install CodeRabbit on it and open one small pull request.