Skip to content
Holits
AI Infrastructure3 min readBy Holits

Dependency bots tell you what changed. They don't tell you if it matters.

Why we built an LLM-scored, bundle-aware upgrade pipeline instead of accepting the one-PR-per-package model Renovate and Dependabot default to.

Renovate and Dependabot solve a real problem well: detection. A new version of a package exists, here is a PR. What they don't solve is judgment — whether that version bump is a security fix, a breaking API removal, or a no-op internal refactor looks identical from the outside, because all three produce the same artifact: a version number that went up.

The second failure is worse in practice than the first. Dependency updates rarely arrive alone. A TypeScript bump, a @types/react bump and a React bump are one upgrade wearing three separate pull requests, and merging them in the wrong order is how a team ends up with three green CI runs and a broken build on main.

Scoring urgency from signals, not from the size of the version jump

The obvious first attempt at "how urgent is this" is a heuristic off the version diff: a major bump is probably more disruptive than a patch. That's true on average and wrong often enough to matter. We initially estimated a release's age the same way — major ≈ 180 days old, minor ≈ 90, patch ≈ 30 — which is a guess wearing the clothes of a measurement. It's now replaced with the component's actual registry release date wherever that lookup succeeds, falling back to the heuristic only when it doesn't.

Urgency also isn't just "how old is this" — it's "does the changelog describe a breaking change, and does your codebase actually call the API it breaks." An LLM reads the changelog and flags what it believes will break; a separate codebase-impact scan then pattern-matches that claim against the connected repository's real source. A changelog claiming a function was removed is a useful signal. A changelog claiming a function was removed that your code calls thirty times is a different priority entirely, and only the second scan tells you which one you're looking at.

Bundling: the part a bot can't see

Ecosystem-aware upgrades matter because dependency graphs aren't flat. We hardcoded recognition for over a dozen ecosystem groups — React and its types, Vue, Angular, Prisma, Next.js, the major test runners, ESLint, Babel, Webpack, Vite, GraphQL, Tailwind, Storybook, NestJS — and when a new release belongs to one of them, the recommender bundles it with the packages that have to move alongside it and topologically sorts the bundle so a types package upgrades before the package whose types it supplies.

Bundle execution runs one job at a time, deliberately. It would be faster to run bundles in parallel, and it would also mean two bundles could mutate the same repository simultaneously — a race we chose not to have, at the cost of throughput we can afford to give up.

Confidence instead of a coin flip

When a bundle's CI run finishes, "tests passed" isn't the whole answer we give. The confidence score blends test pass rate, code coverage, how closely the tests that actually failed correlate with the breaking changes that were predicted, and the historical success rate of upgrading that specific component before. The reason for the blend is that a binary gate misses in both directions: tests can pass by accident on an upgrade that broke something untested, and tests can fail on something completely unrelated to the upgrade itself. A single number that accounts for both is more honest than a green checkmark.

None of this replaces a bot that opens a PR. It replaces the assumption that opening the PR was the hard part.