Pulp

 
Description of problem: 

Customer had at first problem that workers died on failed docker image sync, for that we applied https://github.com/pulp/pulp_docker/pull/275/commits/b09801b1de94ea7a542b565e6c9caa456aae8619 after discussion with pulp dev. That fixed the issue and at least gracefully fail if needed. After commit was applied we restarted services which flushed cache. When we verified that commit works we applied the same for all Capsules.

Right now when syncing Capsule, pulp task will finish successfully but do not clean its /var/cache/pulp, because of that "No space left out" appear for different task and sync will never finish.

Customer have multiple capsules and this is happening only on one of them.

Version-Release number of selected component (if applicable):

satellite-capsule-6.4.2-2.el7sat.noarch
pulp-admin-client-2.16.4.1-1.el7sat.noarch
pulp-docker-plugins-3.1.4.1-1.el7sat.noarch
pulp-katello-1.0.2-5.el7sat.noarch
pulp-ostree-plugins-1.3.0-1.el7sat.noarch
pulp-puppet-plugins-2.16.4-3.el7sat.noarch
pulp-rpm-admin-extensions-2.16.4.1-5.el7sat.noarch
pulp-rpm-plugins-2.16.4.1-5.el7sat.noarch
pulp-selinux-2.16.4.1-1.el7sat.noarch
pulp-server-2.16.4.1-1.el7sat.noarch
python-isodate-0.5.0-5.pulp.el7sat.noarch
python-pulp-bindings-2.16.4.1-1.el7sat.noarch
python-pulp-client-lib-2.16.4.1-1.el7sat.noarch
python-pulp-common-2.16.4.1-1.el7sat.noarch
python-pulp-docker-common-3.1.4.1-1.el7sat.noarch
python-pulp-integrity-2.16.4.1-5.el7sat.noarch
python-pulp-oid_validation-2.16.4.1-1.el7sat.noarch
python-pulp-ostree-common-1.3.0-1.el7sat.noarch
python-pulp-puppet-common-2.16.4-3.el7sat.noarch
python-pulp-repoauth-2.16.4.1-1.el7sat.noarch
python-pulp-rpm-common-2.16.4.1-5.el7sat.noarch
python-pulp-streamer-2.16.4.1-1.el7sat.noarch
rubygem-smart_proxy_pulp-1.3.0-1.git.0.b5c2768.el7sat.noarch

How reproducible: 
Not easily reproducible as it is happening only on one Capsule from many.

Actual results:
Capsule sync task will never finish as different task will fail with "No Space Left on Device"

Expected results: 
Cache of completed pulp task will clean and Capsule sync finish.

Additional info:

du -sh /var/cache/pulp/*
0    /var/cache/pulp/reserved_resource_worker-0@capsule.example.com
20G    /var/cache/pulp/reserved_resource_worker-1@capsule.example.com
0    /var/cache/pulp/reserved_resource_worker-2@capsule.example.com
21G    /var/cache/pulp/reserved_resource_worker-3@capsule.example.com
20G    /var/cache/pulp/reserved_resource_worker-4@capsule.example.com
0    /var/cache/pulp/reserved_resource_worker-5@capsule.example.com
0    /var/cache/pulp/reserved_resource_worker-6@capsule.example.com
0    /var/cache/pulp/reserved_resource_worker-7@capsule.example.com
0    /var/cache/pulp/resource_manager@capsule.example.com

du -sh /var/cache/pulp/reserved_resource_worker-3@capsule.example.com/*
21G    /var/cache/pulp/reserved_resource_worker-3@capsule.example.com/dc57ee7e-3226-45aa-977a-2c3a55487cea

One of tasks detail with not cleaned cache

+----------------------------------------------------------------------+
                              Task Details
+----------------------------------------------------------------------+

Operations:       sync
Resources:        1-rhel7-production-7bd5ddcc-6cfa-4fd2-a607-a60c36cad74b
                  (repository)
State:            Successful
Start Time:       2019-04-25T06:30:13Z
Finish Time:      2019-04-25T08:40:13Z
Result:           
  Added Count:      4341
  Completed:        2019-04-25T08:40:13Z
  Details:          
    Description:   Downloading manifests
    Details:       
    Error Details: 
    Items Total:   201
    Num Failures:  0
    Num Processed: 402
    Num Success:   402
    State:         FINISHED
    Step Id:       fca85786-e6f8-45b8-8207-21a8586fb52c
    Step Type:     sync_step_metadata
    Description:   Copying units already in pulp
    Details:       
    Error Details: 
    Items Total:   1
    Num Failures:  0
    Num Processed: 1
    Num Success:   1
    State:         FINISHED
    Step Id:       227a13f8-5d63-40b8-a6d0-7c12c5dceedf
    Step Type:     get_local
    Description:   Copying units already in pulp
    Details:       
    Error Details: 
    Items Total:   1
    Num Failures:  0
    Num Processed: 1
    Num Success:   1
    State:         FINISHED
    Step Id:       0d76f77f-46e3-4e73-8c94-b9489ddd41c7
    Step Type:     get_local
    Description:   Downloading remote files
    Details:       
    Error Details: 
    Items Total:   3551
    Num Failures:  0
    Num Processed: 3551
    Num Success:   3551
    State:         FINISHED
    Step Id:       9efa7a84-4aff-4535-969f-44a4a5cd25a4
    Step Type:     sync_step_download
    Description:   Saving Manifests and Blobs
    Details:       
    Error Details: 
    Items Total:   1
    Num Failures:  0
    Num Processed: 1
    Num Success:   1
    State:         FINISHED
    Step Id:       a11ef1b7-788c-40e3-a490-b3dd1a53c134
    Step Type:     sync_step_save
    Description:   Saving Tags
    Details:       
    Error Details: 
    Items Total:   402
    Num Failures:  0
    Num Processed: 402
    Num Success:   402
    State:         FINISHED
    Step Id:       90f85de7-4c91-45bb-bd48-69ab29066ebb
    Step Type:     sync_step_save
  Error Message:    None
  Exception:        None
  Id:               5cc1726d508b576efb86fc28
  Importer Id:      docker_importer
  Importer Type Id: docker_importer
  Removed Count:    0
  Repo Id:          1-rhel7-production-7bd5ddcc-6cfa-4fd2-a607-a60c36cad74b
  Result:           success
  Started:          2019-04-25T06:30:13Z
  Summary:          
    Get Local:          FINISHED
    Sync Step Download: FINISHED
    Sync Step Metadata: FINISHED
    Sync Step Save:     FINISHED
  Traceback:        None
  Updated Count:    0
Task Id:          dc57ee7e-3226-45aa-977a-2c3a55487cea
Worker Name:      reserved_resource_worker-3@
  Docker Importer: 
    Description:   Downloading manifests
    Details:       
    Error Details: 
    Items Total:   201
    Num Failures:  0
    Num Processed: 402
    Num Success:   402
    State:         FINISHED
    Step Id:       fca85786-e6f8-45b8-8207-21a8586fb52c
    Step Type:     sync_step_metadata
    Description:   Copying units already in pulp
    Details:       
    Error Details: 
    Items Total:   1
    Num Failures:  0
    Num Processed: 1
    Num Success:   1
    State:         FINISHED
    Step Id:       227a13f8-5d63-40b8-a6d0-7c12c5dceedf
    Step Type:     get_local
    Description:   Copying units already in pulp
    Details:       
    Error Details: 
    Items Total:   1
    Num Failures:  0
    Num Processed: 1
    Num Success:   1
    State:         FINISHED
    Step Id:       0d76f77f-46e3-4e73-8c94-b9489ddd41c7
    Step Type:     get_local
    Description:   Downloading remote files
    Details:       
    Error Details: 
    Items Total:   3551
    Num Failures:  0
    Num Processed: 3551
    Num Success:   3551
    State:         FINISHED
    Step Id:       9efa7a84-4aff-4535-969f-44a4a5cd25a4
    Step Type:     sync_step_download
    Description:   Saving Manifests and Blobs
    Details:       
    Error Details: 
    Items Total:   1
    Num Failures:  0
    Num Processed: 1
    Num Success:   1
    State:         FINISHED
    Step Id:       a11ef1b7-788c-40e3-a490-b3dd1a53c134
    Step Type:     sync_step_save
    Description:   Saving Tags
    Details:       
    Error Details: 
    Items Total:   402
    Num Failures:  0
    Num Processed: 402
    Num Success:   402
    State:         FINISHED
    Step Id:       90f85de7-4c91-45bb-bd48-69ab29066ebb
    Step Type:     sync_step_save