Importer Overview

Importers are responsible for scraping advisories data such as advisories and their fixes and for storing the scraped information in a structured fashion. The structured data created by the importer then provides input to an improver (see Improver Overview), which is responsible for creating a relational model for advisories, affected packages and fixed packages.

All importer implementation-related code is defined in vulnerabilites/importer.py.

In addition, the framework-related code for actually invoking and processing the importers is located in vulnerabilites/import_runner.py.

The importers, after scraping, provide AdvisoryData objects. These objects are then processed and inserted into the Advisory model.

While implementing an importer, it is important to make sure that the importer does not alter the upstream data at all. Its only job is to convert the data from a data source into structured – yet non-relational – data. This ensures that we always have a true copy of an advisory without any modifications.

Given that a lot of advisories publish version ranges of affected packages, it is necessary to store those ranges in a structured manner. Vers was designed to solve this problem. It has been implemented in the univers library whose development goes hand in hand with VulnerableCode.

The data imported by importers is not useful by itself: it must be processed into a relational model. The version ranges are required to be resolved into concrete ranges. These are achieved by Improvers (see Improver Overview for details).

As of now, the following importers have been implemented in VulnerableCode:

Importer Name

Data Source

Ecosystems Covered

archlinux

https://security.archlinux.org/json

arch packages

apache_kafka

https://kafka.apache.org/community/cve-list

apache-kafka

nvd

https://nvd.nist.gov/vuln/data-feeds#JSON_FEED

none

elixir_security

https://github.com/dependabot/elixir-security-advisories

hex packages

npm

https://github.com/nodejs/security-wg.git

npm packages

vulnrichment

https://github.com/cisagov/vulnrichment.git

all

apache_httpd

https://httpd.apache.org/security/json

apache-httpd

pypa

https://github.com/pypa/advisory-database.git

python packages

gitlab

https://gitlab.com/gitlab-org/advisories-community.git

all

pysec

https://osv-vulnerabilities.storage.googleapis.com/PyPI/all.zip

python packages

xen

https://xenbits.xen.org/xsa/xsa.json

xen packages

curl

https://curl.se/docs/vuln.json

curl packages

oss_fuzz

https://github.com/google/oss-fuzz-vulns.git

OSS-Fuzz target packages

istio

https://github.com/istio/istio.io/tree/master/content/en/news/security

istio packages

postgresql

https://www.postgresql.org/support/security/

postgresql

mozilla

https://github.com/mozilla/foundation-security-advisories

mozilla

github_osv

https://github.com/github/advisory-database.git

all

redhat

https://access.redhat.com/hydra/rest/securitydata/cve.json

rpm packages

aosp_dataset

https://github.com/pypa/advisory-database.git

android packages

project_kb_statements

https://github.com/SAP/project-kb.git

maven, python packages

project_kb_msr2019

https://github.com/SAP/project-kb/blob/main/MSR2019/dataset/vulas_db_msr2019_release.csv

maven, python packages

ruby

https://github.com/rubysec/ruby-advisory-db.git

ruby gems

epss

https://epss.cyentia.com/epss_scores-current.csv.gz

exploit prediction scoring system

gentoo

https://anongit.gentoo.org/git/data/glsa.git

gentoo packages

nginx

http://nginx.org/en/security_advisories.html

nginx

debian

https://security-tracker.debian.org/tracker/data/json

debian packages

mattermost

https://mattermost.com/security-updates/

mattermost server, desktop and mobile apps

glibc

https://sourceware.org/git/glibc.git

GNU C packages

apache_tomcat

https://tomcat.apache.org/security

apache_tomcat packages

suse_scores

http://ftp.suse.com/pub/projects/security/yaml/

vulnerability severity scores by SUSE

retiredotnet

https://github.com/RetireNet/Packages.git

.NET packages

ubuntu_osv

https://github.com/canonical/ubuntu-security-notices.git

ubuntu packages

alpine_linux

https://github.com/aboutcode-org/aboutcode-mirror-alpine-secdb

alpine packages

linux_kernel

https://github.com/nluedtke/linux_kernel_cves

linux kernel packages

openssl

https://www.openssl.org/news/vulnerabilities.xml

openssl

fireeye

https://github.com/mandiant/Vulnerability-Disclosures

none

fix_commits