A synk integration with the github clone and/or Dependabot alerts might work

For PHP we have two systems of security scanning:

• php-composer-security job in Jenkins, which notifies the security team whenever a new vulnerability is reported and repeats every day
• LibUp, which just posts to its own webpage https://libraryupgrader2.wmcloud.org/vulns/composer?branch=main - mostly intended for dev deps

I do want to add Python/Poetry support to LibUp, but for critical security alerts, LibUp's system is unnecessarily complicated and does break occasionally. So I think we want a simple, does-only-one-thing job in CI that actively notifies someone when a vulnerability is found.

https://github.com/pypa/advisory-db is the new freely licensed database from PyPA, there's no tool for it yet but we should be able to write a small Python script that reads the poetry lockfile, and hits the API for every dependency.

In T288535#7273267, @Legoktm wrote:

For PHP we have two systems of security scanning:

• php-composer-security job in Jenkins, which notifies the security team whenever a new vulnerability is reported and repeats every day
• LibUp, which just posts to its own webpage https://libraryupgrader2.wmcloud.org/vulns/composer?branch=main - mostly intended for dev deps

I do want to add Python/Poetry support to LibUp, but for critical security alerts, LibUp's system is unnecessarily complicated and does break occasionally. So I think we want a simple, does-only-one-thing job in CI that actively notifies someone when a vulnerability is found.

https://github.com/pypa/advisory-db is the new freely licensed database from PyPA, there's no tool for it yet but we should be able to write a small Python script that reads the poetry lockfile, and hits the API for every dependency.

Hello @Legoktm won't adding python/poetry support be a useful feature despite the issues mentioned above? our team is looking to start working on a patch for libup to support python because of how important this is.

In T288535#7613998, @Raymond_Ndibe wrote:

Hello @Legoktm won't adding python/poetry support be a useful feature despite the issues mentioned above? our team is looking to start working on a patch for libup to support python because of how important this is.

I know @Legoktm can respond to this, but they did already note some reasons above for not wanting to add such functionality to LibUp:

In T288535#7273267, @Legoktm wrote:

I do want to add Python/Poetry support to LibUp, but for critical security alerts, LibUp's system is unnecessarily complicated and does break occasionally. So I think we want a simple, does-only-one-thing job in CI that actively notifies someone when a vulnerability is found.

Setting up a job similar to the existing php-composer-security job (although I'd recommend having the recipients be the code owners, instead of or in addition to the security team's admin feed) or having a test or script in various projects' repos that runs safety check or poetry export -f requirements.txt | safety check --stdin should be simple, perhaps interim, solutions here. I know the latter is not technically using the "free" version of the db, so if that's a dealbreaker, some doc could be provided on manually installing and running it during development, until a new tool/cli can be written and then used for automation.

In T288535#7614314, @sbassett wrote:

In T288535#7613998, @Raymond_Ndibe wrote:

Hello @Legoktm won't adding python/poetry support be a useful feature despite the issues mentioned above? our team is looking to start working on a patch for libup to support python because of how important this is.

I know @Legoktm can respond to this, but they did already note some reasons above for not wanting to add such functionality to LibUp:

In T288535#7273267, @Legoktm wrote:

I do want to add Python/Poetry support to LibUp, but for critical security alerts, LibUp's system is unnecessarily complicated and does break occasionally. So I think we want a simple, does-only-one-thing job in CI that actively notifies someone when a vulnerability is found.

Setting up a job similar to the existing php-composer-security job (although I'd recommend having the recipients be the code owners, instead of or in addition to the security team's admin feed) or having a test or script in various projects' repos that runs safety check or poetry export -f requirements.txt | safety check --stdin should be simple, perhaps interim, solutions here. I know the latter is not technically using the "free" version of the db, so if that's a dealbreaker, some doc could be provided on manually installing and running it during development, until a new tool/cli can be written and then used for automation.

fair enough

LibUp vulnerability checking shines when there are a significant amount of repos that are likely to have very similar vulnerabilities, because it can aggregate them, like how most MediaWiki extensions and skins depend on roughly the same set of npm/composer dependencies. If one is vulnerable, it's likely that 30+ repositories are. For something like Toolhub I think having a dedicated Python job that just checks those dependencies and reports on them is the ideal first step. Then it would be reasonable to add support to LibUp to also detect those outdated/vulnerable deps and attempt to automatically upgrade them.

Stumbled upon this today: https://www.whitesourcesoftware.com/free-developer-tools/renovate/

WhiteSource Renovate automates open source dependency updates, reducing risk by mitigating security vulnerabilities and saving developers’ time. The Renovate App has enabled a diverse user base across github.com and gitlab.com to keep dependencies up-to-date since 2018, and has generated millions of pull requests in the process. The inclusion of WhiteSource Renovate as a positive indicator in the OpenSSF and Google Secure Open Source Rewards program further solidifies its position as the leading open source dependency management solution for developers.

WhiteSource is built on the most comprehensive vulnerability database in the industry, providing the widest coverage for threats and attack vectors. Our solution helps enterprises like Microsoft, IBM, Comcast, Philips, and many more reduce security risk and increase the productivity of their security and development teams.

WhiteSource renovate is open source, free and supports a variety of programming languages, including python and nodejs. On the minus side, its implementation is based on the PR workflow, so has no native support for Gerrit the way it does for GitHub and GitLab.

In T288535#7670107, @Slst2020 wrote:

Stumbled upon this today: https://www.whitesourcesoftware.com/free-developer-tools/renovate/

Looks interesting. We'd want to investigate the licensing though, to make sure there aren't any hidden caveats for "large organizations", "running on-prem", etc. In talking to some reps from snyk (an obvious competitor) a while back, they basically admitted they didn't currently have a compatible license for a non-profit org/movement like Wikimedia, even though their product was also, technically, "free for open-source projects". They did say they were working on developing a license for large, non-profit organizations though.

WhiteSource renovate is open source, free and supports a variety of programming languages, including python and nodejs. On the minus side, its implementation is based on the PR workflow, so has no native support for Gerrit the way it does for GitHub and GitLab.

If renovate offers a fairly standard cli (it seems to), it should be able to run just about anywhere, and not just within the context of MRs/PRs. This also might be a bit moot as I believe we're still on pace to migrate much of our code from gerrit to gitlab this year(?) and thus begin making heavy use of MRs for code review.

Just FYI - this details what the Security-Team has settled on, for now, for our appsec ci pipeline templates within gitlab: T301830#7782646.