Story #8088
Updated by pulpbot over 2 years ago
**Ticket moved to GitHub**: "pulp/pulpcore/1957":https://github.com/pulp/pulpcore/issues/1957 ---- ## Motivation The business policy here is that all downloaded files should be scanned for viruses. The on_demand policy is being used with a PyPI remote because syncing all of PyPI to just use a few files is not practical. ## User Experience 1. The user sets up a virus scanner like `clamav` in daemon mode. 2. Administrator configures the setting `SECURITY_SCAN_SHELL`. 3. User restarts Pulp 4. All content already in saved in Pulp is *not* scanned. Any new content brought into Pulp through any policy type, e.g. `immediate|on_demand|streamed` is first scanned before being saved or handed to a client. ## Implementation ### on_demand and streamed policies The [`pulpcore_content app`](https://github.com/pulp/pulpcore/blob/master/pulpcore/content/handler.py) handles the `policy=on_demand` and `policy=streamed` modes. Currently those modes "stream" bits to clients meaning they begin serving data prior to receiving all of it. When scanning is enabled those modes can no longer do this, they have to "store, scan, and forward" the data to clients because security scanners can only scan entire files not a stream of data. So the first change to prototype is having a store and forward change when this setting is in place. ### immediate policy To handle the immediate policy a new "Artifact" stage should be created. The [Artifact stages live here](https://github.com/pulp/pulpcore/blob/master/pulpcore/plugin/stages/artifact_stages.py). The pipeline is a series of stages, and this stage should only be used when the `SECURITY_SCAN_SHELL` setting is set. The default stage construction [happens here](https://github.com/pulp/pulpcore/blob/master/pulpcore/plugin/stages/declarative_version.py#L107-L133). ## Submitting data for scanning In all policy types data should be submitted for scanning by calling the `SECURITY_SCAN_SHELL` with the path to the temporary file added into that command somehow. The response code 0 from the AV scanner shell subprocess call will tell Pulp to accept the file and that's its safe. Any other response code will tell Pulp to throw away the file and send the client an error code of some kind. ## Notes * The on_demand and streamed policies have to submit files to the scanner one-by-one. This is unavoidable. * Having the `clamav` scanner run in daemon mode should resolve the "startup time" problem. * The calls to the shell need to be validly callable by the Pulp user on wherever the `pulpcore_content` process is running *and* any `pulpcore reserved worker`. ## TBD * What HTTP error code to submit to clients when the AV scanner says the file is not safe? * The file saved may have a filename like `/path/to/tmp/dirs/e0e9fe76-a80a-47c7-8255-1ec4c8f8c7da`. This doesn't have a filetype. Does it need a filetype? * How will the subprocess call know where in the command to put the path to the file to scan?
.