I spent way too long fighting with PR review automation before figuring out how to make MergeWatch and GitHub Actions actually work together
Code Reviews are sometimes a mess. Reviewers would leave generic comments like "LGTM" on diffs that clearly hadn't been fully read. I started looking for something that could bring more structure to the feedback loop, ideally something that could actually inspect what changed in a PR and produce a useful, automated first-pass review. I'd heard of a few tools in this space, but one brought me into my attention and it is MergeWatch AI . Let me walk through how I got it working alongside GitHub Actions, what the integration actually looks like in practice, and a few gotchas I ran into, including the part that got genuinely interesting: wiring up AI agents to do the heavy lifting on diff analysis. What MergeWatch AI Actually Does MergeWatch AI watches pull requests and runs configurable inspection rules against the diff. Think of it less like a linter and more like a programmable reviewer. You define what you care about: file patterns, change thresholds, forbidden strings, structural conditions, whatever your team needs to enforce. It then posts review comments (or a summary review) back to the PR through the GitHub API. That said, the static rule engine is just the beginning. MergeWatch AI also ships with an agents configuration block that lets you plug in AI-powered reviewers alongside your deterministic rules. That's where things get interesting. The High-Level Setup The core idea is simple: GitHub Actions triggers on pull_request events, checks out the repo, runs MergeWatch AI against the diff, and MergeWatch AI posts its output back to the PR as a review. Three moving parts: The GitHub Actions workflow file The MergeWatch AI config ( .mergewatch.yml ), including any agent definitions A GitHub token with the right permissions to post reviews No external webhook server, no extra infrastructure. Everything runs inside the Actions runner. Wiring Up the GitHub Actions Workflow Here's the workflow I landed on after a couple of iterations. The first version I wrote was triggering on push instead of pull_request , which meant it ran on commits to main too. Not ideal. name: PR Review with MergeWatch on: pull_request: types: [opened, synchronize, reopened] permissions: pull-requests: write contents: read jobs: mergewatch-review: runs-on: ubuntu-latest steps: - name: Checkout code uses: actions/checkout@v4 with: fetch-depth: 0 - name: Set up MergeWatch run: | curl -sSL https://github.com/mergewatch/mergewatch/releases/download/v0.9.2/mergewatch-linux-amd64 \ -o /usr/local/bin/mergewatch chmod +x /usr/local/bin/mergewatch - name: Run MergeWatch env: GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }} PR_NUMBER: ${{ github.event.pull_request.number }} REPO: ${{ github.repository }} BASE_SHA: ${{ github.event.pull_request.base.sha }} HEAD_SHA: ${{ github.event.pull_request.head.sha }} run: | mergewatch review \ --config .mergewatch.yml \ --repo "$REPO" \ --pr "$PR_NUMBER" \ --base "$BASE_SHA" \ --head "$HEAD_SHA" \ --token "$GITHUB_TOKEN" A few things worth pointing out. The fetch-depth: 0 on the checkout step is non-negotiable. Without it, you get a shallow clone and MergeWatch AI can't compute the diff between the base and head commits. I spent about 45 minutes debugging a cryptic "ref not found" error before I figured that one out. Don't skip that line. The permissions block at the top is also required if your org has restricted default token permissions (which, honestly, most orgs should). Without pull-requests: write , the tool can't post reviews back to the PR and will fail silently in some configurations. That's its own special kind of frustrating. I've also added OPENAI_API_KEY to the environment here because the agent configuration I'll show below needs it. If you're using a different model provider, swap it out accordingly. The MergeWatch Config: Static Rules The .mergewatch.yml file lives at the root of the repo and defines both the deterministic rules and the agent configuration. Static rules form the baseline, so let's start there. Here's a simplified version of the config we use on the portal project, adapted for a Java Spring Boot codebase: version: 1 rules: - name: "No hardcoded secrets" match: pattern: "(password|secret|apiKey)\\s*=\\s*\"[^\"]+\"" files: "**/*.java" severity: error message: | Looks like a hardcoded credential. Please use environment variables or the secrets manager instead. - name: "Liquibase changelog needs a corresponding entity change" match: files: "src/main/resources/db/changelog/**" require: files: "src/main/java/**/entity/**" severity: warning message: | You've added a Liquibase changelog but I don't see a change to the entity classes. This might be intentional, but worth double-checking. - name: "Large PRs" threshold: lines_changed: 500 severity: info message: | This PR changes more than 500 lines. Consider splitting it up if the changes are logically separable. - name: "Test coverage for new services" match: files: "src/main/java/**/service/**/*.java" change_type: added require: files: "src/test/java/**/service/**" severity: warning message: "New service class detected. Don't forget unit tests." The require key is my favorite feature in the static rule set. It caught a real problem on the portal project: a developer had added a Liquibase changelog to create a new user_tier column but hadn't touched the JPA entity class. The entity was essentially out of sync with the database schema. MergeWatch flagged it, the reviewer caught it, and nobody had to spend an afternoon debugging a MappingException in staging. Well, okay, one person did have to reproduce it in staging first to believe it. But that's close enough. Adding AI Agents to the Mix This is the part I didn't expect to work as well as it does. MergeWatch AI supports an agents block in the config where you can define AI-powered reviewers that run against the diff alongside your static rules. Each agent gets a role, a model, and a prompt. The diff content is injected automatically. Here's what our base agent config looks like: agents: - name: "general-reviewer" model: openai/gpt-4o role: | You are a senior Java engineer reviewing a pull request diff on a Spring Boot application. Focus on logic errors, edge cases, and anything that looks like it could cause a bug in production. Do not comment on formatting or style issues; those are handled by Checkstyle and SpotBugs. severity_map: critical: error suggestion: info max_tokens: 1024 temperature: 0.2 The temperature: 0.2 is intentional. I tried higher values during testing and the comments got... creative, not in a good way. Lower temperature keeps the agent focused and less prone to inventing problems that don't exist. When this runs, MergeWatch AI feeds the agent the full unified diff and the agent responds with structured feedback. MergeWatch AI then maps that feedback to GitHub review comments, with severity determined by the severity_map you define. Scoping Agents to Specific Files One thing I learned after the first week of running the general reviewer: it's noisy if you let it look at everything. Infrastructure files, auto-generated code, Liquibase changelogs, documentation updates. None of those need GPT-4o's opinion. MergeWatch lets you scope agents to specific file patterns with a files filter: agents: - name: "general-reviewer" model: openai/gpt-4o files: include: - "src/main/java/**/*.java" exclude: - "src/main/java/**/generated/**" - "src/m...