Importer Overview

Importers are responsible to scrape vulnerability data from various data sources without creating a complete relational model between vulnerabilites, their fixes and store them in a structured fashion.

All importer implementation related code is defined in vulnerabilites/importer.py.

Whereas, the framework related code for actually invoking and processing the importers are situated in vulnerabilites/import_runner.py.

The importers, after scraping, provide with AdvisoryData objects. These objects are then processed and inserted into the Advisory model.

While implementing an importer, it is important to make sure that the importer does not alter the upstream data at all. Its only job is to convert the data from a data source into structured - yet non relational - data. The importers must not be smart or performing trickeries under the hood. This ensures that we always have a true copy of an advisory without any speculations or improvements.

As importers do not speculate and given that a lot of advisories publish version ranges of affected packages, it is necessary to store those ranges in a structured manner. Vers was designed to solve this problem. It has been implemented in the univers library whose development goes hand in hand with VulnerableCode.

The data imported by importers is not useful by itself, it must be processed into a relational model. The version ranges are required to be dissolved into concrete ranges. These are achieved by Improvers. For more, see: Improver Overview

As of now, the following importers have been implemented in VulnerableCode

Importer Name

Data Source

Ecosystems Covered

rust

https://github.com/RustSec/advisory-db

rust crates

alpine

https://secdb.alpinelinux.org/

alpine packages

archlinux

https://security.archlinux.org/json

arch packages

debian

https://security-tracker.debian.org/tracker/data/json

debian packages

npm

https://github.com/nodejs/security-wg.git

npm packages

ruby

https://github.com/rubysec/ruby-advisory-db.git

ruby gems

ubuntu

https://people.canonical.com/~ubuntu-security/oval/

ubuntu packages

retiredotnet

https://github.com/RetireNet/Packages.git

.NET packages

suse_backports

http://ftp.suse.com/pub/projects/security/yaml/

SUSE packages

debian_oval

https://www.debian.org/security/oval/

debian packages

redhat

https://access.redhat.com/hydra/rest/securitydata/cve.json

rpm packages

nvd

https://nvd.nist.gov/vuln/data-feeds#JSON_FEED

none

gentoo

https://anongit.gentoo.org/git/data/glsa.git

gentoo packages

openssl

https://www.openssl.org/news/vulnerabilities.xml

openssl

ubuntu_usn

https://usn.ubuntu.com/usn-db/database-all.json.bz2

ubuntu packages

github

https://api.github.com/graphql

maven, .NET, php-composer, pypi packages. ruby gems

msr2019

https://raw.githubusercontent.com/SAP/project-kb/master/MSR2019/dataset/vulas_db_msr2019_release.csv

maven packages

apache_httpd

https://httpd.apache.org/security/vulnerabilities-httpd.xml

apache-httpd

kaybee

https://github.com/SAP/project-kb.git

maven packages

nginx

http://nginx.org/en/security_advisories.html

nginx

postgresql

https://www.postgresql.org/support/security/

postgresql

elixir_security

https://github.com/dependabot/elixir-security-advisories

hex packages

suse_scores

https://ftp.suse.com/pub/projects/security/yaml/suse-cvss-scores.yaml

vulnerability severity scores by SUSE

mozilla

https://github.com/mozilla/foundation-security-advisories

mozilla

mattermost

https://mattermost.com/security-updates/

mattermost server, desktop and mobile apps