Container Support

Getting filesystem layer verification failed for digest sha256 while pulling image from pulp.

(SuSE15SP2 - EDT time)
--------------------------------------------------
mtlb-cp-ctrl:~ # docker pull ci-gl-mgmt-pulp-vm1/tensorflow/tensorflow:latest-gpu-py3-jupyter
latest-gpu-py3-jupyter: Pulling from tensorflow/tensorflow
7ddbc47eeb70: Pulling fs layer
c1bbdc448b72: Pulling fs layer
8c3b70e39044: Pulling fs layer
45d437916d57: Pulling fs layer
d8f1569ddae6: Pulling fs layer
85386706b020: Waiting
ee9b457b77d0: Waiting
bebfcc1316f7: Waiting
ee9b457b77d0: Pull complete
d6c0f989e873: Download complete
7a8e64f26211: Download complete
c33b03e4dd22: Download complete
bca93af797c1: Download complete
47f6c197be35: Verifying Checksum
e5da48aa9554: Download complete
ca68d98a90c4: Download complete
2059de27f7c8: Download complete
55d02aea1458: Download complete
32162ecb0c59: Download complete
47520dc72e8e: Download complete
3dafed94e1f2: Download complete
dc228e76e4f0: Download complete
2c6922dc5a5f: Download complete
a960e6d108fd: Download complete
6818a780ae00: Download complete
06dfbeeed7ba: Download complete
5890e026a0a0: Download complete
eeddfe30f3d2: Download complete
187170305445: Download complete
2e20a8960c42: Download complete
9f1bf726c909: Download complete
filesystem layer verification failed for digest sha256:47f6c197be35b90bd9791c459                                                                                                                                                          eda7c735cd6abefe6a95bb863be04b98afc847c


/var/log/messages
2021-09-07T06:20:08.192518-04:00 mtlb-cp-ctrl dockerd[4030]: time="2021-09-07T06:20:08.192401967-04:00" level=error msg="filesystem layer verification failed for digest sha256:bebfcc1316f7a3e5e6b5e78a78c2739c292d6d712a97fe7fac4f346e66f6812d"
2021-09-07T06:20:08.192788-04:00 mtlb-cp-ctrl dockerd[4030]: time="2021-09-07T06:20:08.192440461-04:00" level=error msg="Download failed: filesystem layer verification failed for digest sha256:bebfcc1316f7a3e5e6b5e78a78c2739c292d6d712a97fe7fac4f346e66f6812d"
2021-09-07T06:20:08.729142-04:00 mtlb-cp-ctrl dockerd[4030]: time="2021-09-07T06:20:08.728994866-04:00" level=error msg="filesystem layer verification failed for digest sha256:644140fd95a90c8f2831acb22741a672e2107db148d2033e62cb253bc5646c60"
2021-09-07T06:20:08.729404-04:00 mtlb-cp-ctrl dockerd[4030]: time="2021-09-07T06:20:08.729072081-04:00" level=error msg="Download failed: filesystem layer verification failed for digest sha256:644140fd95a90c8f2831acb22741a672e2107db148d2033e62cb253bc5646c60"
2021-09-07T06:20:09.315464-04:00 mtlb-cp-ctrl dockerd[4030]: time="2021-09-07T06:20:09.315322013-04:00" level=error msg="Error running exec d0c8f1c78fdfa35fac418154a0f373a9258e9e894ac9e0dfc48f8b6b198586ae in container: exec attach failed: error attaching stdout stream: write unix /var/run/docker.sock->@: write: broken pipe"
2021-09-07T06:20:12.969350-04:00 mtlb-cp-ctrl dockerd[4030]: time="2021-09-07T06:20:12.969230614-04:00" level=error msg="filesystem layer verification failed for digest sha256:47f6c197be35b90bd9791c459eda7c735cd6abefe6a95bb863be04b98afc847c"
2021-09-07T06:20:12.969802-04:00 mtlb-cp-ctrl dockerd[4030]: time="2021-09-07T06:20:12.969757194-04:00" level=error msg="Download failed: filesystem layer verification failed for digest sha256:47f6c197be35b90bd9791c459eda7c735cd6abefe6a95bb863be04b98afc847c"
2021-09-07T06:20:13.317583-04:00 mtlb-cp-ctrl dockerd[4030]: time="2021-09-07T06:20:13.317439193-04:00" level=info msg="Attempting next endpoint for pull after error: filesystem layer verification failed for digest sha256:47f6c197be35b90bd9791c459eda7c735cd6abefe6a95bb863be04b98afc847c"
2021-09-07T06:20:13.318708-04:00 mtlb-cp-ctrl dockerd[4030]: time="2021-09-07T06:20:13.318635666-04:00" level=info msg="Layer sha256:15107c79f38ac74631fa18cd30286516a3547400484cc7f4f9f16795d80e8192 cleaned up"
2021-09-07T06:20:13.338263-04:00 mtlb-cp-ctrl dockerd[4030]: time="2021-09-07T06:20:13.338133450-04:00" level=info msg="Layer sha256:11923d9012649f7d9848ae82a805a7f021db44c9e398e02a341a467f57d660ca cleaned up"
2021-09-07T06:20:13.338471-04:00 mtlb-cp-ctrl dockerd[4030]: time="2021-09-07T06:20:13.338169632-04:00" level=info msg="Layer sha256:2736b61577fa5951cb4bc962e5018a6b8b5c8899d2f0c8e14532f7f9a01fe807 cleaned up"
2021-09-07T06:20:13.338893-04:00 mtlb-cp-ctrl dockerd[4030]: time="2021-09-07T06:20:13.338830162-04:00" level=info msg="Layer sha256:4fc26b0b0c6903db3b4fe96856034a1bd9411ed963a96c1bc8f03f18ee92ac2a cleaned up"
2021-09-07T06:20:13.339783-04:00 mtlb-cp-ctrl dockerd[4030]: time="2021-09-07T06:20:13.339734277-04:00" level=info msg="Layer sha256:b53837dafdd21f67e607ae642ce49d326b0c30b39734b6710c682a50a9f932bf cleaned up"
2021-09-07T06:20:13.340544-04:00 mtlb-cp-ctrl dockerd[4030]: time="2021-09-07T06:20:13.340502582-04:00" level=info msg="Layer sha256:565879c6effe6a013e0b2e492f182b40049f1c083fc582ef61e49a98dca23f7e cleaned up"
2021-09-07T06:20:13.394846-04:00 mtlb-cp-ctrl dockerd[4030]: time="2021-09-07T06:20:13.394716018-04:00" level=info msg="Layer sha256:cc967c529ced563b7746b663d98248bc571afdb3c012019d7f54d6c092793b8b cleaned up"

Pulp VM (CentOS) - UTC time
-------------------------------------------
pulp component versions used
{
    "versions": [
        {
            "component": "pulpcore",
            "version": "3.9.0"
        },
        {
            "component": "pulp_rpm",
            "version": "3.8.0"
        },
        {
            "component": "pulp_file",
            "version": "1.5.0"
        },
        {
            "component": "pulp_container",
            "version": "2.2.0"
        }
    ],
journalctl -u pulpcore-api.service  -f
 
"GET /v2/tensorflow/tensorflow/blobs/sha256:ce8f7398433c18dd7a76057256042f902ff6bcdca1ad808dba6e1575c2fae1c7 HTTP/1.0" 302 0 "-" "docker/19.03.5 go/go1.12.12 git-commit/633a0ea838f1 kernel/5.3.18-24.49-default os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \(linux\))"
Sep 07 10:19:34 glhc-mgmt-pulp-vm1 gunicorn[2627970]: pulp [None]: django_guid:INFO: Header `Correlation-ID` was not found in the incoming request. Generated new GUID: 1d7a6865124a4b5888053dabf2b03481
Sep 07 10:19:34 glhc-mgmt-pulp-vm1 gunicorn[2627970]: 172.29.57.132 - - [07/Sep/2021:10:19:34 +0000] "GET /v2/tensorflow/tensorflow/blobs/sha256:7ddbc47eeb70dc7f08e410a6667948b87ff3883024eb41478b44ef9a81bf400c HTTP/1.0" 302 0 "-" "docker/19.03.5 go/go1.12.12 git-commit/633a0ea838f1 kernel/5.3.18-24.49-default os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \(linux\))"
Sep 07 10:19:34 glhc-mgmt-pulp-vm1 gunicorn[2627970]: pulp [None]: django_guid:INFO: Header `Correlation-ID` was not found in the incoming request. Generated new GUID: 2e7592a14d1c4a09bc217eaef2eeef2d
Sep 07 10:19:34 glhc-mgmt-pulp-vm1 gunicorn[2627970]: 172.29.57.132 - - [07/Sep/2021:10:19:34 +0000] "GET /v2/tensorflow/tensorflow/blobs/sha256:c1bbdc448b7263673926b8fe2e88491e5083a8b4b06ddfabf311f2fc5f27e2ff HTTP/1.0" 302 0 "-" "docker/19.03.5 go/go1.12.12 git-commit/633a0ea838f1 kernel/5.3.18-24.49-default os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \(linux\))"
Sep 07 10:19:34 glhc-mgmt-pulp-vm1 gunicorn[2627970]: pulp [None]: django_guid:INFO: Header `Correlation-ID` was not found in the incoming request. Generated new GUID: 06da53e39bde4d3cbebb90a8adbdda1c
Sep 07 10:19:34 glhc-mgmt-pulp-vm1 gunicorn[2627970]: 172.29.57.132 - - [07/Sep/2021:10:19:34 +0000] "GET /v2/tensorflow/tensorflow/blobs/sha256:8c3b70e3904492c753652606df4726430426f42ea56e06ea924d6fea7ae162a1 HTTP/1.0" 302 0 "-" "docker/19.03.5 go/go1.12.12 git-commit/633a0ea838f1 kernel/5.3.18-24.49-default os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \(linux\))"
Sep 07 10:19:34 glhc-mgmt-pulp-vm1 gunicorn[2627970]: pulp [None]: django_guid:INFO: Header `Correlation-ID` was not found in the incoming request. Generated new GUID: 9ac3c2f616e54dba820414f2b75a5065
Sep 07 10:19:34 glhc-mgmt-pulp-vm1 gunicorn[2627970]: 172.29.57.132 - - [07/Sep/2021:10:19:34 +0000] "GET /v2/tensorflow/tensorflow/blobs/sha256:45d437916d5781043432f2d72608049dcf74ddbd27daa01a25fa63c8f1b9adc4 HTTP/1.0" 302 0 "-" "docker/19.03.5 go/go1.12.12 git-commit/633a0ea838f1 kernel/5.3.18-24.49-default os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \(linux\))"
Sep 07 10:19:35 glhc-mgmt-pulp-vm1 gunicorn[2627970]: pulp [None]: django_guid:INFO: Header `Correlation-ID` was not found in the incoming request. Generated new GUID: eb85689d1d18472481012d36655cc91c
Sep 07 10:19:35 glhc-mgmt-pulp-vm1 gunicorn[2627970]: 172.29.57.132 - - [07/Sep/2021:10:19:35 +0000] "GET /v2/tensorflow/tensorflow/blobs/sha256:d8f1569ddae616589c5a2dabf668fadd250ee9d89253ef16f0cb0c8a9459b322 HTTP/1.0" 302 0 "-" "docker/19.03.5 go/go1.12.12 git-commit/633a0ea838f1 kernel/5.3.18-24.49-default os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \(linux\))"
Sep 07 10:19:35 glhc-mgmt-pulp-vm1 gunicorn[2627970]: pulp [None]: django_guid:INFO: Header `Correlation-ID` was not found in the incoming request. Generated new GUID: 010c00e6d05542d8831f07c0b1dbe2c5
Sep 07 10:19:35 glhc-mgmt-pulp-vm1 gunicorn[2627970]: 172.29.57.132 - - [07/Sep/2021:10:19:35 +0000] "GET /v2/tensorflow/tensorflow/blobs/sha256:85386706b02069c58ffaea9de66c360f9d59890e56f58485d05c1a532ca30db1 HTTP/1.0" 302 0 "-" "docker/19.03.5 go/go1.12.12 git-commit/633a0ea838f1 kernel/5.3.18-24.49-default os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \(linux\))"
Sep 07 10:19:37 glhc-mgmt-pulp-vm1 gunicorn[2627970]: pulp [None]: django_guid:INFO: Header `Correlation-ID` was not found in the incoming request. Generated new GUID: aec7513d8ca447e7b7f321ff28c6ed3a
Sep 07 10:19:37 glhc-mgmt-pulp-vm1 gunicorn[2627970]: 172.29.57.132 - - [07/Sep/2021:10:19:37 +0000] "GET /v2/tensorflow/tensorflow/blobs/sha256:ee9b457b77d047ff322858e2de025e266ff5908aec569560e77e2e4451fc23f4 HTTP/1.0" 302 0 "-" "docker/19.03.5 go/go1.12.12 git-commit/633a0ea838f1 kernel/5.3.18-24.49-default os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \(linux\))"
Sep 07 10:19:37 glhc-mgmt-pulp-vm1 gunicorn[2627970]: pulp [None]: django_guid:INFO: Header `Correlation-ID` was not found in the incoming request. Generated new GUID: 761d7a8e674d4034af6ca1548d46e41f
Sep 07 10:19:37 glhc-mgmt-pulp-vm1 gunicorn[2627970]: 172.29.57.132 - - [07/Sep/2021:10:19:37 +0000] "GET /v2/tensorflow/tensorflow/blobs/sha256:bebfcc1316f7a3e5e6b5e78a78c2739c292d6d712a97fe7fac4f346e66f6812d HTTP/1.0" 302 0 "-" "docker/19.03.5 go/go1.12.12 git-commit/633a0ea838f1 kernel/5.3.18-24.49-default os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \(linux\))"
Sep 07 10:19:37 glhc-mgmt-pulp-vm1 gunicorn[2627970]: pulp [None]: django_guid:INFO: Header `Correlation-ID` was not found in the incoming request. Generated new GUID: 3f4b8850e9ef4165a81d69a49b038091
Sep 07 10:19:37 glhc-mgmt-pulp-vm1 gunicorn[2627970]: 172.29.57.132 - - [07/Sep/2021:10:19:37 +0000] "GET /v2/tensorflow/tensorflow/blobs/sha256:644140fd95a90c8f2831acb22741a672e2107db148d2033e62cb253bc5646c60 HTTP/1.0" 302 0 "-" "docker/19.03.5 go/go1.12.12 git-commit/633a0ea838f1 kernel/5.3.18-24.49-default os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \(linux\))"
Sep 07 10:19:37 glhc-mgmt-pulp-vm1 gunicorn[2627970]: pulp [None]: django_guid:INFO: Header `Correlation-ID` was not found in the incoming request. Generated new GUID: 7a7b2db6154c4ea28c1791c3e50c325b
Sep 07 10:19:37 glhc-mgmt-pulp-vm1 gunicorn[2627970]: 172.29.57.132 - - [07/Sep/2021:10:19:37 +0000] "GET /v2/tensorflow/tensorflow/blobs/sha256:d6c0f989e8738fa2bdfebf4d112e0da5930e7f5a14aec68e03943e1b7af0c773 HTTP/1.0" 302 0 "-" "docker/19.03.5 go/go1.12.12 git-commit/633a0ea838f1 kernel/5.3.18-24.49-default os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \(linux\))"
Sep 07 10:19:37 glhc-mgmt-pulp-vm1 gunicorn[2627970]: pulp [None]: django_guid:INFO: Header `Correlation-ID` was not found in the incoming request. Generated new GUID: 08d842f0c1f6431aaf88005f3daa6065
Sep 07 10:19:37 glhc-mgmt-pulp-vm1 gunicorn[2627970]: 172.29.57.132 - - [07/Sep/2021:10:19:37 +0000] "GET /v2/tensorflow/tensorflow/blobs/sha256:7a8e64f262115c5f82c42c49f720928f37da6b0028e1f1e55db1e2dd20ab91c0 HTTP/1.0" 302 0 "-" "docker/19.03.5 go/go1.12.12 git-commit/633a0ea838f1 kernel/5.3.18-24.49-default os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \(linux\))"
Sep 07 10:19:38 glhc-mgmt-pulp-vm1 gunicorn[2627970]: pulp [None]: django_guid:INFO: Header `Correlation-ID` was not found in the incoming request. Generated new GUID: 6a14a90df52749df9b46a92d08712c03
Sep 07 10:19:38 glhc-mgmt-pulp-vm1 gunicorn[2627970]: 172.29.57.132 - - [07/Sep/2021:10:19:38 +0000] "GET /v2/tensorflow/tensorflow/blobs/sha256:c33b03e4dd226568c166378a84a18cd5b10e1508fd61706074be15c3a8fe746e HTTP/1.0" 302 0 "-" "docker/19.03.5 go/go1.12.12 git-commit/633a0ea838f1 kernel/5.3.18-24.49-default os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \(linux\))"
Sep 07 10:19:38 glhc-mgmt-pulp-vm1 gunicorn[2627970]: pulp [None]: django_guid:INFO: Header `Correlation-ID` was not found in the incoming request. Generated new GUID: 95391301ef5b4e01b8fc5738ea7acabe
Sep 07 10:19:38 glhc-mgmt-pulp-vm1 gunicorn[2627970]: 172.29.57.132 - - [07/Sep/2021:10:19:38 +0000] "GET /v2/tensorflow/tensorflow/blobs/sha256:bca93af797c124524e827e572bd160706d6c49a9133baf22cf1152d65a966dda HTTP/1.0" 302 0 "-" "docker/19.03.5 go/go1.12.12 git-commit/633a0ea838f1 kernel/5.3.18-24.49-default os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \(linux\))"
Sep 07 10:19:38 glhc-mgmt-pulp-vm1 gunicorn[2627970]: pulp [None]: django_guid:INFO: Header `Correlation-ID` was not found in the incoming request. Generated new GUID: 7d7975f8319643bda48eeb55fd593f9e
Sep 07 10:19:38 glhc-mgmt-pulp-vm1 gunicorn[2627970]: 172.29.57.132 - - [07/Sep/2021:10:19:38 +0000] "GET /v2/tensorflow/tensorflow/blobs/sha256:47f6c197be35b90bd9791c459eda7c735cd6abefe6a95bb863be04b98afc847c HTTP/1.0" 302 0 "-" "docker/19.03.5 go/go1.12.12 git-commit/633a0ea838f1 kernel/5.3.18-24.49-default os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \(linux\))"
Sep 07 10:19:38 glhc-mgmt-pulp-vm1 gunicorn[2627970]: pulp [None]: django_guid:INFO: Header `Correlation-ID` was not found in the incoming request. Generated new GUID: 5634fddc54334602ad9bfedb3727028a
Sep 07 10:19:39 glhc-mgmt-pulp-vm1 gunicorn[2627970]: 172.29.57.132 - - [07/Sep/2021:10:19:39 +0000] "GET /v2/tensorflow/tensorflow/blobs/sha256:e5da48aa9554efec444c2b6395a7d45d3129d66693cf4c446719736a77af7df5 HTTP/1.0" 302 0 "-" "docker/19.03.5 go/go1.12.12 git-commit/633a0ea838f1 kernel/5.3.18-24.49-default os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \(linux\))"
Sep 07 10:19:40 glhc-mgmt-pulp-vm1 gunicorn[2627970]: pulp [None]: django_guid:INFO: Header `Correlation-ID` was not found in the incoming request. Generated new GUID: 85c95eddb7c9499fb8f54ab09f2439c0
Sep 07 10:19:40 glhc-mgmt-pulp-vm1 gunicorn[2627970]: 172.29.57.132 - - [07/Sep/2021:10:19:40 +0000] "GET /v2/tensorflow/tensorflow/blobs/sha256:ca68d98a90c42199152014e30fe667ab71885a6a00a13ad7207b11a8b51209b0 HTTP/1.0" 302 0 "-" "docker/19.03.5 go/go1.12.12 git-commit/633a0ea838f1 kernel/5.3.18-24.49-default os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \(linux\))"
Sep 07 10:19:40 glhc-mgmt-pulp-vm1 gunicorn[2627970]: pulp [None]: django_guid:INFO: Header `Correlation-ID` was not found in the incoming request. Generated new GUID: 1d06e452e0434d05a871ab48288feea0
Sep 07 10:19:40 glhc-mgmt-pulp-vm1 gunicorn[2627970]: 172.29.57.132 - - [07/Sep/2021:10:19:40 +0000] "GET /v2/tensorflow/tensorflow/blobs/sha256:2059de27f7c8152660f7cbbf3cfffcdeddc2bedceee09cba578d1c55763b8e02 HTTP/1.0" 302 0 "-" "docker/19.03.5 go/go1.12.12 git-commit/633a0ea838f1 kernel/5.3.18-24.49-default os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \(linux\))"
Sep 07 10:19:40 glhc-mgmt-pulp-vm1 gunicorn[2627970]: pulp [None]: django_guid:INFO: Header `Correlation-ID` was not found in the incoming request. Generated new GUID: f0d9e73799e74f93b0401a74dde27c55
Sep 07 10:19:40 glhc-mgmt-pulp-vm1 gunicorn[2627970]: 172.29.57.132 - - [07/Sep/2021:10:19:40 +0000] "GET /v2/tensorflow/tensorflow/blobs/sha256:55d02aea14585dc12e92beaaec0dd0e53141d8e710cf2d75b185bd0877e81ff3 HTTP/1.0" 302 0 "-" "docker/19.03.5 go/go1.12.12 git-commit/633a0ea838f1 kernel/5.3.18-24.49-default os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \(linux\))"
Sep 07 10:19:40 glhc-mgmt-pulp-vm1 gunicorn[2627970]: pulp [None]: django_guid:INFO: Header `Correlation-ID` was not found in the incoming request. Generated new GUID: 040f7e9c187f463d9abb979aab9bfbda
Sep 07 10:19:40 glhc-mgmt-pulp-vm1 gunicorn[2627970]: 172.29.57.132 - - [07/Sep/2021:10:19:40 +0000] "GET /v2/tensorflow/tensorflow/blobs/sha256:32162ecb0c5944e4791343f01541173a00dcaf2d686afc143aae5fb1bcca58f4 HTTP/1.0" 302 0 "-" "docker/19.03.5 go/go1.12.12 git-commit/633a0ea838f1 kernel/5.3.18-24.49-default os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \(linux\))"
Sep 07 10:19:41 glhc-mgmt-pulp-vm1 gunicorn[2627970]: pulp [None]: django_guid:INFO: Header `Correlation-ID` was not found in the incoming request. Generated new GUID: bcf1b56bb92d470e8dfd5c85dbcedeb3
Sep 07 10:19:41 glhc-mgmt-pulp-vm1 gunicorn[2627970]: 172.29.57.132 - - [07/Sep/2021:10:19:41 +0000] "GET /v2/tensorflow/tensorflow/blobs/sha256:47520dc72e8eda76f2fefa1cc4143e9c798fb9b04101dec1024223f257bcea98 HTTP/1.0" 302 0 "-" "docker/19.03.5 go/go1.12.12 git-commit/633a0ea838f1 kernel/5.3.18-24.49-default os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \(linux\))"
Sep 07 10:19:41 glhc-mgmt-pulp-vm1 gunicorn[2627970]: pulp [None]: django_guid:INFO: Header `Correlation-ID` was not found in the incoming request. Generated new GUID: d1b696b7895d495b90273398251eb371
Sep 07 10:19:41 glhc-mgmt-pulp-vm1 gunicorn[2627970]: 172.29.57.132 - - [07/Sep/2021:10:19:41 +0000] "GET /v2/tensorflow/tensorflow/blobs/sha256:3dafed94e1f29798964e8a4da21dcd4566b77b1734609f09854690c1406403ad HTTP/1.0" 302 0 "-" "docker/19.03.5 go/go1.12.12 git-commit/633a0ea838f1 kernel/5.3.18-24.49-default os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \(linux\))"
Sep 07 10:19:41 glhc-mgmt-pulp-vm1 gunicorn[2627970]: pulp [None]: django_guid:INFO: Header `Correlation-ID` was not found in the incoming request. Generated new GUID: a6071eb23108457890e157f23a18c28b
Sep 07 10:19:41 glhc-mgmt-pulp-vm1 gunicorn[2627970]: 172.29.57.132 - - [07/Sep/2021:10:19:41 +0000] "GET /v2/tensorflow/tensorflow/blobs/sha256:dc228e76e4f0f1d0c907e9b99a22c0081a03af10588874fb01113f48fdf03566 HTTP/1.0" 302 0 "-" "docker/19.03.5 go/go1.12.12 git-commit/633a0ea838f1 kernel/5.3.18-24.49-default os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \(linux\))"
Sep 07 10:19:42 glhc-mgmt-pulp-vm1 gunicorn[2627970]: pulp [None]: django_guid:INFO: Header `Correlation-ID` was not found in the incoming request. Generated new GUID: fd002ec85e5648c59331c487e005309e
Sep 07 10:19:42 glhc-mgmt-pulp-vm1 gunicorn[2627970]: 172.29.57.132 - - [07/Sep/2021:10:19:42 +0000] "GET /v2/tensorflow/tensorflow/blobs/sha256:2c6922dc5a5f5c2b5181a99142aa313b91371622de496ed60bfb15eb8967a74c HTTP/1.0" 302 0 "-" "docker/19.03.5 go/go1.12.12 git-commit/633a0ea838f1 kernel/5.3.18-24.49-default os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \(linux\))"
Sep 07 10:19:42 glhc-mgmt-pulp-vm1 gunicorn[2627970]: pulp [None]: django_guid:INFO: Header `Correlation-ID` was not found in the incoming request. Generated new GUID: 20333960e10e4793b698ebdbe65a621e
Sep 07 10:19:42 glhc-mgmt-pulp-vm1 gunicorn[2627970]: 172.29.57.132 - - [07/Sep/2021:10:19:42 +0000] "GET /v2/tensorflow/tensorflow/blobs/sha256:a960e6d108fd4decf7192c0780615e667b52e2680263ab586b9171b8825b262e HTTP/1.0" 302 0 "-" "docker/19.03.5 go/go1.12.12 git-commit/633a0ea838f1 kernel/5.3.18-24.49-default os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \(linux\))"
Sep 07 10:19:42 glhc-mgmt-pulp-vm1 gunicorn[2627970]: pulp [None]: django_guid:INFO: Header `Correlation-ID` was not found in the incoming request. Generated new GUID: 28500880c73f4003b9a2cfe03a059e12
Sep 07 10:19:42 glhc-mgmt-pulp-vm1 gunicorn[2627970]: 172.29.57.132 - - [07/Sep/2021:10:19:42 +0000] "GET /v2/tensorflow/tensorflow/blobs/sha256:6818a780ae00b7fe320cb22fa51acdaacef1a1fec51d6e238601d6b35e483d1f HTTP/1.0" 302 0 "-" "docker/19.03.5 go/go1.12.12 git-commit/633a0ea838f1 kernel/5.3.18-24.49-default os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \(linux\))"
Sep 07 10:19:43 glhc-mgmt-pulp-vm1 gunicorn[2627970]: pulp [None]: django_guid:INFO: Header `Correlation-ID` was not found in the incoming request. Generated new GUID: d8f08f634e3f4470871a56fc31116d9b
Sep 07 10:19:43 glhc-mgmt-pulp-vm1 gunicorn[2627970]: 172.29.57.132 - - [07/Sep/2021:10:19:43 +0000] "GET /v2/tensorflow/tensorflow/blobs/sha256:06dfbeeed7bae8b395ad8ffc7a133ed460558f0b78030314d46cbb3d4a33b852 HTTP/1.0" 302 0 "-" "docker/19.03.5 go/go1.12.12 git-commit/633a0ea838f1 kernel/5.3.18-24.49-default os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \(linux\))"
Sep 07 10:19:43 glhc-mgmt-pulp-vm1 gunicorn[2627970]: pulp [None]: django_guid:INFO: Header `Correlation-ID` was not found in the incoming request. Generated new GUID: 79566aaabd84446482f6b0301cdfb2a7
Sep 07 10:19:43 glhc-mgmt-pulp-vm1 gunicorn[2627970]: 172.29.57.132 - - [07/Sep/2021:10:19:43 +0000] "GET /v2/tensorflow/tensorflow/blobs/sha256:5890e026a0a0d6ce4fe9494f1256e9193ab26db904ebb1ac605b289f61b6518c HTTP/1.0" 302 0 "-" "docker/19.03.5 go/go1.12.12 git-commit/633a0ea838f1 kernel/5.3.18-24.49-default os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \(linux\))"
Sep 07 10:19:43 glhc-mgmt-pulp-vm1 gunicorn[2627970]: pulp [None]: django_guid:INFO: Header `Correlation-ID` was not found in the incoming request. Generated new GUID: 404456c53646429ca7898ae5af0974ba
Sep 07 10:19:43 glhc-mgmt-pulp-vm1 gunicorn[2627970]: 172.29.57.132 - - [07/Sep/2021:10:19:43 +0000] "GET /v2/tensorflow/tensorflow/blobs/sha256:eeddfe30f3d26fa11442a01c1b43e8f7849ab07b80ef16313be0c036fa61d282 HTTP/1.0" 302 0 "-" "docker/19.03.5 go/go1.12.12 git-commit/633a0ea838f1 kernel/5.3.18-24.49-default os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \(linux\))"
Sep 07 10:19:43 glhc-mgmt-pulp-vm1 gunicorn[2627970]: pulp [None]: django_guid:INFO: Header `Correlation-ID` was not found in the incoming request. Generated new GUID: fd9c8d8d14be41c6ae18cefcaa8bfc8e
Sep 07 10:19:43 glhc-mgmt-pulp-vm1 gunicorn[2627970]: 172.29.57.132 - - [07/Sep/2021:10:19:43 +0000] "GET /v2/tensorflow/tensorflow/blobs/sha256:187170305445701a5e50c50d8d5aee1113649b5c13de671823f319640f8970e9 HTTP/1.0" 302 0 "-" "docker/19.03.5 go/go1.12.12 git-commit/633a0ea838f1 kernel/5.3.18-24.49-default os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \(linux\))"
Sep 07 10:19:43 glhc-mgmt-pulp-vm1 gunicorn[2627970]: pulp [None]: django_guid:INFO: Header `Correlation-ID` was not found in the incoming request. Generated new GUID: 4c1233e701ae4e909a35c1ce9bf8f02a
Sep 07 10:19:43 glhc-mgmt-pulp-vm1 gunicorn[2627970]: 172.29.57.132 - - [07/Sep/2021:10:19:43 +0000] "GET /v2/tensorflow/tensorflow/blobs/sha256:2e20a8960c429200c0cb2b28835b4c077da278dbb42f6afb32f6808416bb5f75 HTTP/1.0" 302 0 "-" "docker/19.03.5 go/go1.12.12 git-commit/633a0ea838f1 kernel/5.3.18-24.49-default os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \(linux\))"
Sep 07 10:19:43 glhc-mgmt-pulp-vm1 gunicorn[2627970]: pulp [None]: django_guid:INFO: Header `Correlation-ID` was not found in the incoming request. Generated new GUID: 2bdec3565cab45fd847f4f77defdbefd
Sep 07 10:19:43 glhc-mgmt-pulp-vm1 gunicorn[2627970]: 172.29.57.132 - - [07/Sep/2021:10:19:43 +0000] "GET /v2/tensorflow/tensorflow/blobs/sha256:9f1bf726c909892ac35caa545d960e68356cec5ffbbb7f03765a0b4a60d79c19 HTTP/1.0" 302 0 "-" "docker/19.03.5 go/go1.12.12 git-commit/633a0ea838f1 kernel/5.3.18-24.49-default os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \(linux\))"

journalctl -u pulpcore-api.service  -f

Sep 07 10:19:34 glhc-mgmt-pulp-vm1 gunicorn[2627992]: 172.29.57.132 [07/Sep/2021:10:19:34 +0000] "GET /pulp/container/tensorflow/tensorflow/manifests/latest-gpu-py3-jupyter?validate_token=457ff25db607ce8f2ab13b70cef216a5dca068bc798a86ae3015adacfa9e025b:b8d249a8549ccdf79fba45f5d2d6762d280d9d15bc4f220e585986a2016bdf57 HTTP/1.0" 200 7304 "https://ci-gl-mgmt-pulp-vm1.glhc-hpe.local/v2/tensorflow/tensorflow/manifests/latest-gpu-py3-jupyter" "docker/19.03.5 go/go1.12.12 git-commit/633a0ea838f1 kernel/5.3.18-24.49-default os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \(linux\))"
Sep 07 10:19:34 glhc-mgmt-pulp-vm1 gunicorn[2627992]: 172.29.57.132 [07/Sep/2021:10:19:34 +0000] "GET /pulp/container/tensorflow/tensorflow/blobs/sha256:c1bbdc448b7263673926b8fe2e88491e5083a8b4b06ddfabf311f2fc5f27e2ff?validate_token=d89e515a23e76a5e3df388af754df1a50168467ae8eafce092f6494cd207118e:1d16a5ebbd944189bc72e1deb3a3cc426f03f330be996e1f9758d9ffda2e59a4 HTTP/1.0" 200 35863 "https://ci-gl-mgmt-pulp-vm1.glhc-hpe.local/v2/tensorflow/tensorflow/blobs/sha256:c1bbdc448b7263673926b8fe2e88491e5083a8b4b06ddfabf311f2fc5f27e2ff" "docker/19.03.5 go/go1.12.12 git-commit/633a0ea838f1 kernel/5.3.18-24.49-default os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \(linux\))"
Sep 07 10:19:34 glhc-mgmt-pulp-vm1 gunicorn[2627992]: 172.29.57.132 [07/Sep/2021:10:19:34 +0000] "GET /pulp/container/tensorflow/tensorflow/blobs/sha256:7ddbc47eeb70dc7f08e410a6667948b87ff3883024eb41478b44ef9a81bf400c?validate_token=a6a55d4e3870bf0ec718fa3d16086ddaef2aa23876d11340f501bca41f5f4d42:728ffc3cf74847a50f4f54519fa03f5d420ac114a0d2c19e0eb4a4fa01b4f97f HTTP/1.0" 200 26689351 "https://ci-gl-mgmt-pulp-vm1.glhc-hpe.local/v2/tensorflow/tensorflow/blobs/sha256:7ddbc47eeb70dc7f08e410a6667948b87ff3883024eb41478b44ef9a81bf400c" "docker/19.03.5 go/go1.12.12 git-commit/633a0ea838f1 kernel/5.3.18-24.49-default os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \(linux\))"
Sep 07 10:19:34 glhc-mgmt-pulp-vm1 gunicorn[2627992]: 172.29.57.132 [07/Sep/2021:10:19:34 +0000] "GET /pulp/container/tensorflow/tensorflow/blobs/sha256:8c3b70e3904492c753652606df4726430426f42ea56e06ea924d6fea7ae162a1?validate_token=176e847923c2fcc89225575939ed203b7ad524114441e9389f59a6ea0a1bffe5:ac29f521ad7c6c3574267775f84e976c948f3fef627a5e320636a1f7883290e2 HTTP/1.0" 200 1344 "https://ci-gl-mgmt-pulp-vm1.glhc-hpe.local/v2/tensorflow/tensorflow/blobs/sha256:8c3b70e3904492c753652606df4726430426f42ea56e06ea924d6fea7ae162a1" "docker/19.03.5 go/go1.12.12 git-commit/633a0ea838f1 kernel/5.3.18-24.49-default os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \(linux\))"
Sep 07 10:19:35 glhc-mgmt-pulp-vm1 gunicorn[2627992]: 172.29.57.132 [07/Sep/2021:10:19:35 +0000] "GET /pulp/container/tensorflow/tensorflow/blobs/sha256:45d437916d5781043432f2d72608049dcf74ddbd27daa01a25fa63c8f1b9adc4?validate_token=016ee94c366a5d746f70e2c0e97ed5d6637b95ef5e43dfbe855abee81e5e65cb:7775aaaf3923dceaccf82dee1c4149324293a607824355418d2556b43de17012 HTTP/1.0" 200 661 "https://ci-gl-mgmt-pulp-vm1.glhc-hpe.local/v2/tensorflow/tensorflow/blobs/sha256:45d437916d5781043432f2d72608049dcf74ddbd27daa01a25fa63c8f1b9adc4" "docker/19.03.5 go/go1.12.12 git-commit/633a0ea838f1 kernel/5.3.18-24.49-default os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \(linux\))"
Sep 07 10:19:35 glhc-mgmt-pulp-vm1 gunicorn[2627992]: 172.29.57.132 [07/Sep/2021:10:19:35 +0000] "GET /pulp/container/tensorflow/tensorflow/blobs/sha256:85386706b02069c58ffaea9de66c360f9d59890e56f58485d05c1a532ca30db1?validate_token=b3496e4569614870a383b556dcf8a7ad84cb49dbc5988d66ace91015c67bd80b:6cd519a626640cdc984ab9db9b14c33434fa82b34f9fe54ca3bcaf049e5a03c9 HTTP/1.0" 200 8446478 "https://ci-gl-mgmt-pulp-vm1.glhc-hpe.local/v2/tensorflow/tensorflow/blobs/sha256:85386706b02069c58ffaea9de66c360f9d59890e56f58485d05c1a532ca30db1" "docker/19.03.5 go/go1.12.12 git-commit/633a0ea838f1 kernel/5.3.18-24.49-default os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \(linux\))"
Sep 07 10:19:37 glhc-mgmt-pulp-vm1 gunicorn[2627992]: 172.29.57.132 [07/Sep/2021:10:19:37 +0000] "GET /pulp/container/tensorflow/tensorflow/blobs/sha256:bebfcc1316f7a3e5e6b5e78a78c2739c292d6d712a97fe7fac4f346e66f6812d?validate_token=41c4d62e52761c1fa3aabd0c35bf02db612a05b69dd1990c080840593a7a0d07:468a363fbbf9246dfbeced1bbed761654d4db47981b8a856b5b1e9700a4b62ed HTTP/1.0" 200 74023416 "https://ci-gl-mgmt-pulp-vm1.glhc-hpe.local/v2/tensorflow/tensorflow/blobs/sha256:bebfcc1316f7a3e5e6b5e78a78c2739c292d6d712a97fe7fac4f346e66f6812d" "docker/19.03.5 go/go1.12.12 git-commit/633a0ea838f1 kernel/5.3.18-24.49-default os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \(linux\))"
Sep 07 10:19:37 glhc-mgmt-pulp-vm1 gunicorn[2627992]: 172.29.57.132 [07/Sep/2021:10:19:37 +0000] "GET /pulp/container/tensorflow/tensorflow/blobs/sha256:644140fd95a90c8f2831acb22741a672e2107db148d2033e62cb253bc5646c60?validate_token=d802b6aa256b18671318dd2847000cf75914659ddda62f189e5b3a60d2164f29:7a3b22d166e361a9024ce83e1a458b465d1d4acce1a3b38add6cdfb4ae45b692 HTTP/1.0" 200 65307128 "https://ci-gl-mgmt-pulp-vm1.glhc-hpe.local/v2/tensorflow/tensorflow/blobs/sha256:644140fd95a90c8f2831acb22741a672e2107db148d2033e62cb253bc5646c60" "docker/19.03.5 go/go1.12.12 git-commit/633a0ea838f1 kernel/5.3.18-24.49-default os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \(linux\))"
Sep 07 10:19:38 glhc-mgmt-pulp-vm1 gunicorn[2627992]: 172.29.57.132 [07/Sep/2021:10:19:37 +0000] "GET /pulp/container/tensorflow/tensorflow/blobs/sha256:7a8e64f262115c5f82c42c49f720928f37da6b0028e1f1e55db1e2dd20ab91c0?validate_token=d736316c29ecad6c5ab92c0bdf32d472439c87f4dce486157dd415c89f47df82:bbd12b0a66fbb1ca874d5b092bc26b03106eab5171dff93322fa4e7010f82fcc HTTP/1.0" 200 79656174 "https://ci-gl-mgmt-pulp-vm1.glhc-hpe.local/v2/tensorflow/tensorflow/blobs/sha256:7a8e64f262115c5f82c42c49f720928f37da6b0028e1f1e55db1e2dd20ab91c0" "docker/19.03.5 go/go1.12.12 git-commit/633a0ea838f1 kernel/5.3.18-24.49-default os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \(linux\))"
Sep 07 10:19:38 glhc-mgmt-pulp-vm1 gunicorn[2627992]: 172.29.57.132 [07/Sep/2021:10:19:38 +0000] "GET /pulp/container/tensorflow/tensorflow/blobs/sha256:c33b03e4dd226568c166378a84a18cd5b10e1508fd61706074be15c3a8fe746e?validate_token=80f4c54125acfd226cf605fe00006654bf01fe71daa2d1bfbeadd9e3370527bf:8fd4be3bc0cfcdf7c3ebc182a0b1bfe109bac1500fad7c52b228615b15c75a50 HTTP/1.0" 200 3482804 "https://ci-gl-mgmt-pulp-vm1.glhc-hpe.local/v2/tensorflow/tensorflow/blobs/sha256:c33b03e4dd226568c166378a84a18cd5b10e1508fd61706074be15c3a8fe746e" "docker/19.03.5 go/go1.12.12 git-commit/633a0ea838f1 kernel/5.3.18-24.49-default os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \(linux\))"
Sep 07 10:19:38 glhc-mgmt-pulp-vm1 gunicorn[2627992]: 172.29.57.132 [07/Sep/2021:10:19:38 +0000] "GET /pulp/container/tensorflow/tensorflow/blobs/sha256:bca93af797c124524e827e572bd160706d6c49a9133baf22cf1152d65a966dda?validate_token=8b56feee87da398a15d442f2e367f5da309235728181a85567cd5924be73d044:db0a55be486abc7422c3390988b2f80ca35a6ef58cb86eae3d7a2d39d0ea7579 HTTP/1.0" 200 674 "https://ci-gl-mgmt-pulp-vm1.glhc-hpe.local/v2/tensorflow/tensorflow/blobs/sha256:bca93af797c124524e827e572bd160706d6c49a9133baf22cf1152d65a966dda" "docker/19.03.5 go/go1.12.12 git-commit/633a0ea838f1 kernel/5.3.18-24.49-default os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \(linux\))"
Sep 07 10:19:39 glhc-mgmt-pulp-vm1 gunicorn[2627992]: 172.29.57.132 [07/Sep/2021:10:19:38 +0000] "GET /pulp/container/tensorflow/tensorflow/blobs/sha256:47f6c197be35b90bd9791c459eda7c735cd6abefe6a95bb863be04b98afc847c?validate_token=94c92aa16609f780879837f61bded129158bfced6fb2a8ec2e847d4157d00141:6750b4d3889fd3c073d9aeb70dbb0571cab870b5fc420b17dc7193ec7475f157 HTTP/1.0" 200 263483897 "https://ci-gl-mgmt-pulp-vm1.glhc-hpe.local/v2/tensorflow/tensorflow/blobs/sha256:47f6c197be35b90bd9791c459eda7c735cd6abefe6a95bb863be04b98afc847c" "docker/19.03.5 go/go1.12.12 git-commit/633a0ea838f1 kernel/5.3.18-24.49-default os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \(linux\))"
Sep 07 10:19:40 glhc-mgmt-pulp-vm1 gunicorn[2627992]: 172.29.57.132 [07/Sep/2021:10:19:40 +0000] "GET /pulp/container/tensorflow/tensorflow/blobs/sha256:ca68d98a90c42199152014e30fe667ab71885a6a00a13ad7207b11a8b51209b0?validate_token=0d760c2d0101112b94bb75fdb4195a8f7a9708b902431f5a16f21ac3835e0367:627e6f05c1f6890872e0e9162769c4d7fb229bd6b8cc22ddc518f928331f62a5 HTTP/1.0" 200 1541 "https://ci-gl-mgmt-pulp-vm1.glhc-hpe.local/v2/tensorflow/tensorflow/blobs/sha256:ca68d98a90c42199152014e30fe667ab71885a6a00a13ad7207b11a8b51209b0" "docker/19.03.5 go/go1.12.12 git-commit/633a0ea838f1 kernel/5.3.18-24.49-default os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \(linux\))"
Sep 07 10:19:40 glhc-mgmt-pulp-vm1 gunicorn[2627992]: 172.29.57.132 [07/Sep/2021:10:19:40 +0000] "GET /pulp/container/tensorflow/tensorflow/blobs/sha256:2059de27f7c8152660f7cbbf3cfffcdeddc2bedceee09cba578d1c55763b8e02?validate_token=9120ec689c2c8dd4eff60f5e3f752fffabe4ca622251ff55433057f1ad259f34:e1ae2e779ab8c6162f038ff5107adc80d87113bb8e289555b0d2aa43c2af0fd0 HTTP/1.0" 200 69997130 "https://ci-gl-mgmt-pulp-vm1.glhc-hpe.local/v2/tensorflow/tensorflow/blobs/sha256:2059de27f7c8152660f7cbbf3cfffcdeddc2bedceee09cba578d1c55763b8e02" "docker/19.03.5 go/go1.12.12 git-commit/633a0ea838f1 kernel/5.3.18-24.49-default os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \(linux\))"
Sep 07 10:19:40 glhc-mgmt-pulp-vm1 gunicorn[2627992]: 172.29.57.132 [07/Sep/2021:10:19:40 +0000] "GET /pulp/container/tensorflow/tensorflow/blobs/sha256:55d02aea14585dc12e92beaaec0dd0e53141d8e710cf2d75b185bd0877e81ff3?validate_token=55a722b9ce58dbff6e960cf3b8034050ac51a1fb9c7447df065db582ae529f05:e2716b973e15d55fc6543a528633563ebf8f1a5b8fe8c0c2ad1db73de7a55b94 HTTP/1.0" 200 92499 "https://ci-gl-mgmt-pulp-vm1.glhc-hpe.local/v2/tensorflow/tensorflow/blobs/sha256:55d02aea14585dc12e92beaaec0dd0e53141d8e710cf2d75b185bd0877e81ff3" "docker/19.03.5 go/go1.12.12 git-commit/633a0ea838f1 kernel/5.3.18-24.49-default os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \(linux\))"
Sep 07 10:19:40 glhc-mgmt-pulp-vm1 gunicorn[2627992]: 172.29.57.132 [07/Sep/2021:10:19:40 +0000] "GET /pulp/container/tensorflow/tensorflow/blobs/sha256:32162ecb0c5944e4791343f01541173a00dcaf2d686afc143aae5fb1bcca58f4?validate_token=adf801a702e74404c9556453212a28b8c12b8afbdeeb9ddcda08bf0b1d4586ab:05b024d1f7bc1ab8ce4bd3ce5e7f542d4a0f945ea425365d591ef2976c088928 HTTP/1.0" 200 760 "https://ci-gl-mgmt-pulp-vm1.glhc-hpe.local/v2/tensorflow/tensorflow/blobs/sha256:32162ecb0c5944e4791343f01541173a00dcaf2d686afc143aae5fb1bcca58f4" "docker/19.03.5 go/go1.12.12 git-commit/633a0ea838f1 kernel/5.3.18-24.49-default os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \(linux\))"
Sep 07 10:19:41 glhc-mgmt-pulp-vm1 gunicorn[2627992]: 172.29.57.132 [07/Sep/2021:10:19:41 +0000] "GET /pulp/container/tensorflow/tensorflow/blobs/sha256:3dafed94e1f29798964e8a4da21dcd4566b77b1734609f09854690c1406403ad?validate_token=2fb652d9136acfff1223a82c49be03b25a47002763b6c9f113341fa4f0b7437c:a418877bddd28192dc75ef387adda23dee9624c7032c0cda529e63f41c8d76a5 HTTP/1.0" 200 628 "https://ci-gl-mgmt-pulp-vm1.glhc-hpe.local/v2/tensorflow/tensorflow/blobs/sha256:3dafed94e1f29798964e8a4da21dcd4566b77b1734609f09854690c1406403ad" "docker/19.03.5 go/go1.12.12 git-commit/633a0ea838f1 kernel/5.3.18-24.49-default os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \(linux\))"
Sep 07 10:19:42 glhc-mgmt-pulp-vm1 gunicorn[2627992]: 172.29.57.132 [07/Sep/2021:10:19:42 +0000] "GET /pulp/container/tensorflow/tensorflow/blobs/sha256:dc228e76e4f0f1d0c907e9b99a22c0081a03af10588874fb01113f48fdf03566?validate_token=3f8ebb2cf40d63bd571ee755f1280a5c0cfbd3ce7046d47a4b1cdc2d18244826:df977989dad1b70d34ab41edf6af9e52385fc713c3b2760d2aaf2aa643064fc8 HTTP/1.0" 200 634942 "https://ci-gl-mgmt-pulp-vm1.glhc-hpe.local/v2/tensorflow/tensorflow/blobs/sha256:dc228e76e4f0f1d0c907e9b99a22c0081a03af10588874fb01113f48fdf03566" "docker/19.03.5 go/go1.12.12 git-commit/633a0ea838f1 kernel/5.3.18-24.49-default os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \(linux\))"
Sep 07 10:19:43 glhc-mgmt-pulp-vm1 gunicorn[2627992]: 172.29.57.132 [07/Sep/2021:10:19:43 +0000] "GET /pulp/container/tensorflow/tensorflow/blobs/sha256:2e20a8960c429200c0cb2b28835b4c077da278dbb42f6afb32f6808416bb5f75?validate_token=c75213b8213bccada83b1afff29cbc97162060f1aaa18fca0f8d1af64b43fb36:da6e91f7f74f5f0e2118ef75dbe15d7fc0a15455aa849a5732cbeac9dc2195a0 HTTP/1.0" 200 330524 "https://ci-gl-mgmt-pulp-vm1.glhc-hpe.local/v2/tensorflow/tensorflow/blobs/sha256:2e20a8960c429200c0cb2b28835b4c077da278dbb42f6afb32f6808416bb5f75" "docker/19.03.5 go/go1.12.12 git-commit/633a0ea838f1 kernel/5.3.18-24.49-default os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \(linux\))"
Sep 07 10:19:43 glhc-mgmt-pulp-vm1 gunicorn[2627992]: 172.29.57.132 [07/Sep/2021:10:19:43 +0000] "GET /pulp/container/tensorflow/tensorflow/blobs/sha256:9f1bf726c909892ac35caa545d960e68356cec5ffbbb7f03765a0b4a60d79c19?validate_token=c60e60a6f6fb01b00e9ffa3959126adeb2991332f02e9a3d865e10bf1e78a749:438be1ff8bc4c2d8534c6a7a5cd654bb5fd9f985ba675b35d457468e91cb31f0 HTTP/1.0" 200 4341 "https://ci-gl-mgmt-pulp-vm1.glhc-hpe.local/v2/tensorflow/tensorflow/blobs/sha256:9f1bf726c909892ac35caa545d960e68356cec5ffbbb7f03765a0b4a60d79c19" "docker/19.03.5 go/go1.12.12 git-commit/633a0ea838f1 kernel/5.3.18-24.49-default os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \(linux\))"

1. On docker node: Cleaning and restarting of docker daemon didn't solve
2. On pulp node: Deleting of repositories , remotes and distributions in pulp and recreating for specific repo (tensorflow) didn't solve
3. Any help to solve this is appreciated