Skip to content
Holits
Menu

AI Infrastructure

Upgrade Intelligence

An LLM-driven dependency-update platform that turns raw changelogs into scored, bundled upgrade plans — and checks whether your codebase actually uses the APIs that are about to break.

Discuss a similar problem
By Holits

The problem

Renovate and Dependabot solve detection: a new version exists, here is a PR. They do not solve judgment. A changelog might describe a security fix, a no-op internal refactor, or a breaking API removal, and from the bot's perspective these look identical — a version bump. Nor do they know whether the deprecated function the changelog warns about is one your code actually calls.

The second problem is worse: dependency updates rarely arrive alone. A TypeScript bump, a @types/react bump and a React bump are one upgrade wearing three PRs, and merging them out of order is how a green CI run still leaves the build broken.

The approach

Upgrade Intelligence tracks components (npm packages, GitHub repos, GitLab projects) a team depends on, and runs each new release through a pipeline: fetch the changelog, send it to an LLM for structured analysis, score urgency from real signals rather than a version-diff guess, scan the connected codebase for actual usage of anything the changelog flags as breaking, and — when the component is part of a known ecosystem — bundle it with the other packages that have to move together.

Heavy work happens off the request path. API handlers enqueue a job to Redis-backed BullMQ and return immediately; seven workers process changelog analysis, periodic checks, dependency-graph builds, CI correlation, impact scanning, bundle execution and transitive dependency audits.

Ecosystem-aware bundling

A single-package view of "what needs upgrading" misses the fact that React, @types/react and TypeScript are one upgrade. The bundle recommender recognises 15+ hardcoded ecosystem groups — React, Vue, Angular, Prisma, Next.js, TypeScript-and-its-types, testing libraries, ESLint, Babel, Webpack, Vite, GraphQL, Tailwind, Storybook, NestJS — and topologically orders the bundle with Kahn's algorithm, so a type-level dependency upgrades before the package whose types it supplies.

Bundle execution runs sequentially, not in parallel, by explicit design: worker concurrency for the bundle-execution queue is capped at 1 to avoid one job mutating a repository another job is still working in.

Checking the codebase, not just the changelog

An LLM reading a changelog can tell you an API was removed. It cannot tell you, on its own, whether your code calls that API. The impact scanner closes that gap: once a changelog analysis flags a breaking change, a separate pass fetches the connected repository's source and pattern/AST-matches the deprecated surface against real usage, so a team only gets urgency escalated for changes that would actually touch their code.

Confidence, not just a pass/fail gate

When an upgrade PR's CI run finishes, the platform doesn't just report red or green. It builds a 0–100 confidence score from four independent signals: test pass rate, code coverage, a similarity correlation between which tests failed and which breaking changes were predicted, and the historical success rate of upgrading that component before. The point is to catch the case a binary gate misses in both directions — tests that pass by accident, and tests that fail on something unrelated to the upgrade.

Design decisions worth noting

  • Two Redis instances, not one. Queue coordination (BullMQ) and LLM output/changelog caching are deliberately split onto separate Redis instances with independent configuration, so cache pressure and queue throughput never compete with each other.
  • Multi-provider LLM adapter with a circuit breaker. Analysis can run against OpenAI, OpenRouter, Ollama or a local LM Studio endpoint, with a configurable fallback order and a circuit breaker so one provider's outage degrades the pipeline instead of stalling it. Local reasoning models proved the harder case in practice — several return their answer in a separate reasoning_content field with an empty message.content, which was silently tripping the circuit breaker until the adapter learned to normalise both response shapes, strip <think> blocks, and retry once with a larger token budget when a reasoning model runs out of budget mid-thought.
  • Version age comes from the registry, not a guess. Urgency scoring originally estimated how "old" a version was from the size of the version jump alone (major ≈ 180 days, minor ≈ 90, patch ≈ 30). It now fetches the component's real release date and only falls back to the heuristic when that lookup fails.
  • A stub LLM provider exists for CI. The full integration suite runs in GitLab CI against a deterministic stub, so the pipeline is tested without external API keys ever touching CI configuration.

Case study

See this in production: Queue-based infrastructure for scoring, bundling and verifying dependency upgrades.

Availability

Private. Built and operated in-house; not distributed under an open licence.