For large scale deployments with more than a few thousand applications, contact ThreadFix Support for involvement in the deployment and configuration process.

new deployment, optimal configuration

High

Prior to bringing the ThreadFix deployment down or restarting, allow scans in the ingestion pipeline to flush and finish processing without interruption.

restarting, upgrades, applying config changes, maintenance routines

Critical

Monitor CPU/RAM average/peak utilization on the database and application servers during peak scan ingestion activity. Under-allocated resources can hinder performance and stability of the ingestion pipeline.

If the database server resources aren’t enough, and increasing them isn’t option, consider scaling down scan ingestion services. Especially the data (writer) service.

It’s recommended to start with slightly over-allocated resources. Allocation can be optimized after monitoring resource utilization in the environment under the typical/average activity and load.

resource allocation, utilization, and monitoring

Critical

Scan/Application/Team delete jobs acquire global locks blocking other jobs while processing, so it’s recommended to run these jobs during scheduled maintenance hours.

scan ingestion throughput hindrance, maintenance routines

High

Contains 2 consumers that process the following:

Raw Scan File Consumer
Parses raw scan files, saves the parsed data to the staging storage, and the scan is placed on the pipeline for further processing and ingestion.

Pending Scan Statues:

• Queued for Import → Importing

• Queued for Processing

Remote Provider Import Request Consumer
Handles import requests initiated by a user or scheduled job to import for a single app or a bulk import for all mapped apps for a remote provider connection.

Remote Provider Import Request Statuses:

For a bulk import, this request imports new scans for all mapped Remote Provider Apps sequentially.

• Queued → Processing → Finished/Failed

Remote Provider Application Import Attempt Statuses

Pending Scan Statues:

• Queued for Processing: a PendingScan is created with this start status after the scan is received and imported from the external Remote Provider.

Essentially both of these consumers produce a parsed and normalized scan which is stored to the staging storage (Minio) and some metadata to the database.

Consumes a parsed and normalized scan

• Deduplicates findings within the scan file

• Deduplicates against
  application channel history and carries over updates

• Merges scan findings with other findings and vulnerabilities application-wide. Also identifies and creates new vulnerabilities.

  Findings reporting the same risk cluster and merge into ThreadFix Vulnerabilities.

• Processed results are produced to Kafka to be picked up by the Data Writer Service where they get written to the database in efficient batches.

Pending Scan Statues:

• Queued for Processing → Processing

• Queueing for Ingestion

• Queued for Ingestion

• Handles the majority of database writes for scan ingestion

• Includes process to reconcile application vulnerability data and statistics

• Rolls back and cleans up failed pending scans.

Pending Scan Statues:

• Queued for Ingestion → Queued for Reconcile

• Reconciling

• Completed

~2

~1

** Does not include additional threads that can be utilized by Kafka's asynchronous producers used to produce Processed Finding and Vulnerability results to Kafka for Database ingestion.

~20

TF Default: 2 hours

Consider increasing this to allow Remote Provider Bulk Imports Requests to run for longer periods.

Allow importing scans for all configured and mapped Remote Provider Applications.

If any connection configurations listed below apply:
a) Have more than 1000 Remote Provider Apps with small scan data. A typical scan will have around 350 findings.
b) Takes longer than 2 hours.

TF Default: 6 minutes

Consider increasing this if processing very large scans frequently and the “Processing” stage needs more time to successfully complete.

Kafka Default: 5 minutes

TF Default: 2 hours

APPSEC_IMPORTER_MAX_PROCESSING_TIME_MS=7200000

APPSEC_VIP_MAX_PROCESSING_TIME_MS=360000

Override the Kafka max.poll.interval.ms for this service if only truly necessary.

APPSEC_VIP_MIN_PARTITION_COUNT=10

APPSEC_DATA_MIN_PARTITION_COUNT=10

A. Remote Provider Imports
Imports for different Remote Provider Connection Configurations can be imported concurrently if the Importer Service is scaled to match the desired concurrency.

Ideally the number of Importer Services and Remote Provider Import concurrency can match the number of Remote Provider connection configurations. Scaling beyond this number will likely result in idle services/consumers.

A bulk import for a single Remote Provider connection configuration is picked up and processed by a single Importer Service. The Bulk Import job will sequentially import scans for each mapped app and drop them on the ingestion pipeline.

B. Scan File Imports
If reducing time for the following Pending Scan stages is desired:

• Queued for Import (Scan File)

If reducing time for the following Pending Scan stages is desired:

• Queued for Processing

If reducing time for the following Pending Scan stages is desired:

• Database Ingestion Time

• Queued for Reconcile (conditional)

Warnings
Monitor CPU/RAM average/peak utilization on the database server during peak scan ingestion activity. An under-resourced database server can hinder performance and stability of the ingestion pipeline.

If the database server resources aren’t enough, and increasing them isn’t an option, consider scaling down the number of data writer services.

Limited to 1

docker-compose scale appsec-importer=2

docker-compose scale appsec-vip=4

docker-compose scale appsec-data=4