Importer Overview
Importers are responsible to scrape vulnerability data from various data sources without creating a complete relational model between vulnerabilites, their fixes and store them in a structured fashion.
All importer implementation related code is defined in vulnerabilites/importer.py
.
Whereas, the framework related code for actually invoking and processing the importers are
situated in vulnerabilites/import_runner.py
.
The importers, after scraping, provide with AdvisoryData
objects. These objects are then
processed and inserted into the Advisory
model.
While implementing an importer, it is important to make sure that the importer does not alter the upstream data at all. Its only job is to convert the data from a data source into structured - yet non relational - data. The importers must not be smart or performing trickeries under the hood. This ensures that we always have a true copy of an advisory without any speculations or improvements.
As importers do not speculate and given that a lot of advisories publish version ranges of affected packages, it is necessary to store those ranges in a structured manner. Vers was designed to solve this problem. It has been implemented in the univers library whose development goes hand in hand with VulnerableCode.
The data imported by importers is not useful by itself, it must be processed into a relational
model. The version ranges are required to be dissolved into concrete ranges. These are achieved by
Improvers
. For more, see: Improver Overview
As of now, the following importers have been implemented in VulnerableCode
Importer Name |
Data Source |
Ecosystems Covered |
---|---|---|
rust |
rust crates |
|
alpine |
alpine packages |
|
archlinux |
arch packages |
|
debian |
debian packages |
|
npm |
npm packages |
|
ruby |
ruby gems |
|
ubuntu |
ubuntu packages |
|
retiredotnet |
.NET packages |
|
suse_backports |
SUSE packages |
|
debian_oval |
debian packages |
|
redhat |
rpm packages |
|
nvd |
none |
|
gentoo |
gentoo packages |
|
openssl |
openssl |
|
ubuntu_usn |
ubuntu packages |
|
github |
maven, .NET, php-composer, pypi packages. ruby gems |
|
msr2019 |
https://raw.githubusercontent.com/SAP/project-kb/master/MSR2019/dataset/vulas_db_msr2019_release.csv |
maven packages |
apache_httpd |
apache-httpd |
|
kaybee |
maven packages |
|
nginx |
nginx |
|
postgresql |
postgresql |
|
elixir_security |
hex packages |
|
suse_scores |
https://ftp.suse.com/pub/projects/security/yaml/suse-cvss-scores.yaml |
vulnerability severity scores by SUSE |
mozilla |
mozilla |
|
mattermost |
mattermost server, desktop and mobile apps |