Importer Overview

Importers are responsible to scrape vulnerability data from various data sources without creating a complete relational model between vulnerabilites, their fixes and store them in a structured fashion.

All importer implementation related code is defined in vulnerabilites/importer.py.

Whereas, the framework related code for actually invoking and processing the importers are situated in vulnerabilites/import_runner.py.

The importers, after scraping, provide with AdvisoryData objects. These objects are then processed and inserted into the Advisory model.

While implementing an importer, it is important to make sure that the importer does not alter the upstream data at all. Its only job is to convert the data from a data source into structured - yet non relational - data. The importers must not be smart or performing trickeries under the hood. This ensures that we always have a true copy of an advisory without any speculations or improvements.

As importers do not speculate and given that a lot of advisories publish version ranges of affected packages, it is necessary to store those ranges in a structured manner. Vers was designed to solve this problem. It has been implemented in the univers library whose development goes hand in hand with VulnerableCode.

The data imported by importers is not useful by itself, it must be processed into a relational model. The version ranges are required to be dissolved into concrete ranges. These are achieved by Improvers. For more, see: Improver Overview

As of now, the following importers have been implemented in VulnerableCode

Importer Name	Data Source	Ecosystems Covered
rust	https://github.com/RustSec/advisory-db	rust crates
alpine	https://secdb.alpinelinux.org/	alpine packages
archlinux	https://security.archlinux.org/json	arch packages
debian	https://security-tracker.debian.org/tracker/data/json	debian packages
npm	https://github.com/nodejs/security-wg.git	npm packages
ruby	https://github.com/rubysec/ruby-advisory-db.git	ruby gems
ubuntu	https://people.canonical.com/~ubuntu-security/oval/	ubuntu packages
retiredotnet	https://github.com/RetireNet/Packages.git	.NET packages
suse_backports	http://ftp.suse.com/pub/projects/security/yaml/	SUSE packages
debian_oval	https://www.debian.org/security/oval/	debian packages
redhat	https://access.redhat.com/hydra/rest/securitydata/cve.json	rpm packages
nvd	https://nvd.nist.gov/vuln/data-feeds#JSON_FEED	none
gentoo	https://anongit.gentoo.org/git/data/glsa.git	gentoo packages
openssl	https://www.openssl.org/news/vulnerabilities.xml	openssl
ubuntu_usn	https://usn.ubuntu.com/usn-db/database-all.json.bz2	ubuntu packages
github	https://api.github.com/graphql	maven, .NET, php-composer, pypi packages. ruby gems
msr2019	https://raw.githubusercontent.com/SAP/project-kb/master/MSR2019/dataset/vulas_db_msr2019_release.csv	maven packages
apache_httpd	https://httpd.apache.org/security/vulnerabilities-httpd.xml	apache-httpd
kaybee	https://github.com/SAP/project-kb.git	maven packages
nginx	http://nginx.org/en/security_advisories.html	nginx
postgresql	https://www.postgresql.org/support/security/	postgresql
elixir_security	https://github.com/dependabot/elixir-security-advisories	hex packages
suse_scores	https://ftp.suse.com/pub/projects/security/yaml/suse-cvss-scores.yaml	vulnerability severity scores by SUSE
mozilla	https://github.com/mozilla/foundation-security-advisories	mozilla
mattermost	https://mattermost.com/security-updates/	mattermost server, desktop and mobile apps