superagent_

// use cases

Validate dependency updates end to end with Devin

[ view markdown ]

Combine Dependency Validation exploit verification with Devin application regression testing on native dependency update PRs.

Use Dependency Validation to investigate known vulnerabilities on Dependabot or Renovate security PRs, and Devin to run application regression and integration checks on those same PRs.

Configure the repository

  1. Enable native Dependabot security updates or your Renovate security update workflow.
  2. In Superagent, have an organization owner open Repository → Dependency scans and enable Verify vulnerable dependency updates.
  3. Configure application setup instructions and isolated test credentials in the repository's Environment tab.
  4. Configure the repository's setup and integration commands in Devin.

Superagent attempts to exploit the vulnerable application at the PR base commit and verifies the proposed fix at the head commit. The PR comment contains evidence, priority, and limitations. Eligible security updates use Dependency Validation in place of the regular PR security and supply chain scans. Other pull requests retain their configured checks.

Run regression checks with Devin

Use your existing GitHub PR automation to start Devin for dependency update PRs. Provide the exact PR head commit and instruct it to install the committed lockfile, start the application, exercise the changed dependency's relevant features, and run the repository's integration checks. Ask it to report failures and missing configuration without claiming unexecuted tests passed.

For example:

Check out this dependency update PR at its current head. Follow repository setup instructions and run its integration checks. Exercise functionality affected by the dependency change, then report commands, outcomes, and blockers on the PR. Leave the merge decision to repository reviewers.

Re-run Dependency Validation when the PR head changes. Review both the Dependency Validation result and regression evidence before merging. A failed exploit attempt does not prove non-applicability, and a verified security fix does not establish that all application behavior remains compatible.

The retired Superagent updater's dependency_update.published webhook is no longer the trigger for this workflow; use the native GitHub PR lifecycle instead.