Container Support

During regular synchronization, the exception "Cannot connect to host production.cloudflare.docker.com:443 ssl:default [Connect call failed ('104.18.121.25', 443)]" is raised from time to time. It looks like there may be present some connectivity issues during the sync process. I am not experiencing this problem when I am directly connected to Red Hat's network, disregarding the VPN. This behavior could be, therefore, caused by the obscure network configuration provided by the local Internet provider (UPC) too.

I stumbled upon this issue a few weeks ago while trying to sync the content from home. The issue can be still relevant to the old pulp_docker plugin.

I strove to resolve the issue by my own. I noticed that when I had changed the download_concurrency, for example, to 2 (https://pulp-docker.readthedocs.io/en/latest/restapi.html#operation/remotes_docker_docker_create, the default value is 20), everything worked fine. On the other hand, by lowering a number of concurrent connections, the overall performance rapidly decreased. But this was expected.

Please, find below the steps to reproduce the behavior. Note the attached script which was used to sync the content.

1. Clean up the database and run the sync.

$ pclean && pstart
$ bash dockersync.bash

HTTP/1.1 201 Created
Allow: GET, POST, HEAD, OPTIONS
Connection: close
Content-Length: 314
Content-Type: application/json
Date: Sun, 17 Nov 2019 12:15:58 GMT
Location: /pulp/api/v3/repositories/container/container/b1e6372c-ad1d-40b1-b05a-a8c68fe5589e/
Server: gunicorn/20.0.0
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN

{
    "description": null,
    "latest_version_href": null,
    "name": "foo",
    "pulp_created": "2019-11-17T12:15:57.994754Z",
    "pulp_href": "/pulp/api/v3/repositories/container/container/b1e6372c-ad1d-40b1-b05a-a8c68fe5589e/",
    "versions_href": "/pulp/api/v3/repositories/container/container/b1e6372c-ad1d-40b1-b05a-a8c68fe5589e/versions/"
}

HTTP/1.1 201 Created
Allow: GET, POST, HEAD, OPTIONS
Connection: close
Content-Length: 437
Content-Type: application/json
Date: Sun, 17 Nov 2019 12:15:58 GMT
Location: /pulp/api/v3/remotes/container/container/4213f1b8-1078-4829-a5dc-9a774baf8548/
Server: gunicorn/20.0.0
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN

{
    "ca_cert": null,
    "client_cert": null,
    "client_key": null,
    "download_concurrency": 20,
    "name": "library/busybox",
    "policy": "immediate",
    "proxy_url": null,
    "pulp_created": "2019-11-17T12:15:58.735655Z",
    "pulp_href": "/pulp/api/v3/remotes/container/container/4213f1b8-1078-4829-a5dc-9a774baf8548/",
    "pulp_last_updated": "2019-11-17T12:15:58.735668Z",
    "tls_validation": true,
    "upstream_name": "busybox",
    "url": "https://registry-1.docker.io",
    "whitelist_tags": null
}

HTTP/1.1 202 Accepted
Allow: POST, OPTIONS
Connection: close
Content-Length: 67
Content-Type: application/json
Date: Sun, 17 Nov 2019 12:15:59 GMT
Server: gunicorn/20.0.0
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN

{
    "task": "/pulp/api/v3/tasks/4e93f3bf-d7fd-426d-8a72-0ad63cd49843/"
}

2. Check the status of the task.

$ http :24817/pulp/api/v3/tasks/4e93f3bf-d7fd-426d-8a72-0ad63cd49843/

HTTP/1.1 200 OK
Allow: GET, PATCH, DELETE, HEAD, OPTIONS
Connection: close
Content-Length: 4320
Content-Type: application/json
Date: Sun, 17 Nov 2019 12:17:25 GMT
Server: gunicorn/20.0.0
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN

{
    "created_resources": [],
    "error": {
        "description": "Cannot connect to host production.cloudflare.docker.com:443 ssl:default [Connect call failed ('104.18.121.25', 443)]",
        "traceback": "  File \"/usr/local/lib/pulp/lib64/python3.7/site-packages/rq/worker.py\", line 822, in perform_job\n    rv = job.perform()\n  File \"/usr/local/lib/pulp/lib64/python3.7/site-packages/rq/job.py\", line 605, in perform\n    self._result = self._execute()\n  File \"/usr/local/lib/pulp/lib64/python3.7/site-packages/rq/job.py\", line 611, in _execute\n    return self.func(*self.args, **self.kwargs)\n  File \"/home/vagrant/devel/pulp_container/pulp_container/app/tasks/synchronize.py\", line 44, in synchronize\n    dv.create()\n  File \"/home/vagrant/devel/pulpcore/pulpcore/plugin/stages/declarative_version.py\", line 149, in create\n    loop.run_until_complete(pipeline)\n  File \"/usr/lib64/python3.7/asyncio/base_events.py\", line 584, in run_until_complete\n    return future.result()\n  File \"/home/vagrant/devel/pulpcore/pulpcore/plugin/stages/api.py\", line 209, in create_pipeline\n    await asyncio.gather(*futures)\n  File \"/home/vagrant/devel/pulpcore/pulpcore/plugin/stages/api.py\", line 43, in __call__\n    await self.run()\n  File \"/home/vagrant/devel/pulpcore/pulpcore/plugin/stages/artifact_stages.py\", line 132, in run\n    pb.done += task.result()  # download_count\n  File \"/home/vagrant/devel/pulpcore/pulpcore/plugin/stages/artifact_stages.py\", line 157, in _handle_content_unit\n    await asyncio.gather(*downloaders_for_content)\n  File \"/home/vagrant/devel/pulpcore/pulpcore/plugin/stages/models.py\", line 81, in download\n    download_result = await downloader.run(extra_data=self.extra_data)\n  File \"/home/vagrant/devel/pulpcore/pulpcore/download/base.py\", line 221, in run\n    return await self._run(extra_data=extra_data)\n  File \"/usr/local/lib/pulp/lib64/python3.7/site-packages/backoff/_async.py\", line 133, in retry\n    ret = await target(*args, **kwargs)\n  File \"/home/vagrant/devel/pulp_container/pulp_container/app/downloaders.py\", line 61, in _run\n    async with self.session.get(self.url, headers=headers, proxy=self.proxy) as response:\n  File \"/usr/local/lib/pulp/lib64/python3.7/site-packages/aiohttp/client.py\", line 1012, in __aenter__\n    self._resp = await self._coro\n  File \"/usr/local/lib/pulp/lib64/python3.7/site-packages/aiohttp/client.py\", line 483, in _request\n    timeout=real_timeout\n  File \"/usr/local/lib/pulp/lib64/python3.7/site-packages/aiohttp/connector.py\", line 523, in connect\n    proto = await self._create_connection(req, traces, timeout)\n  File \"/usr/local/lib/pulp/lib64/python3.7/site-packages/aiohttp/connector.py\", line 859, in _create_connection\n    req, traces, timeout)\n  File \"/usr/local/lib/pulp/lib64/python3.7/site-packages/aiohttp/connector.py\", line 1004, in _create_direct_connection\n    raise last_exc\n  File \"/usr/local/lib/pulp/lib64/python3.7/site-packages/aiohttp/connector.py\", line 986, in _create_direct_connection\n    req=req, client_error=client_error)\n  File \"/usr/local/lib/pulp/lib64/python3.7/site-packages/aiohttp/connector.py\", line 943, in _wrap_create_connection\n    raise client_error(req.connection_key, exc) from exc\n"
    },
    "finished_at": "2019-11-17T12:16:43.184042Z",
    "name": "pulp_container.app.tasks.synchronize.synchronize",
    "parent": null,
    "progress_reports": [
        {
            "code": "downloading.tag_list",
            "done": 1,
            "message": "Downloading tag list",
            "state": "completed",
            "suffix": null,
            "total": 1
        },
        {
            "code": "downloading.artifacts",
            "done": 644,
            "message": "Downloading Artifacts",
            "state": "failed",
            "suffix": null,
            "total": null
        },
        {
            "code": "associating.content",
            "done": 834,
            "message": "Associating Content",
            "state": "canceled",
            "suffix": null,
            "total": null
        },
        {
            "code": "processing.tag",
            "done": 139,
            "message": "Processing Tags",
            "state": "completed",
            "suffix": null,
            "total": 139
        }
    ],
    "pulp_created": "2019-11-17T12:15:59.455435Z",
    "pulp_href": "/pulp/api/v3/tasks/4e93f3bf-d7fd-426d-8a72-0ad63cd49843/",
    "reserved_resources_record": [
        "/pulp/api/v3/remotes/container/container/4213f1b8-1078-4829-a5dc-9a774baf8548/",
        "/pulp/api/v3/repositories/container/container/b1e6372c-ad1d-40b1-b05a-a8c68fe5589e/"
    ],
    "spawned_tasks": [],
    "started_at": "2019-11-17T12:15:59.541635Z",
    "state": "failed",
    "worker": "/pulp/api/v3/workers/ebc6615a-2149-4618-8aba-61050edb48ec/"
}

3. Wait at least 5 minutes to let the server restore its cache. Otherwise, it immediately crashes on the same error after a start. Clean up the database and run the sync again with updated download_concurrency.

$ pclean && pstart
$ bash dockersync.bash

HTTP/1.1 201 Created
Allow: GET, POST, HEAD, OPTIONS
Connection: close
Content-Length: 314
Content-Type: application/json
Date: Sun, 17 Nov 2019 12:36:44 GMT
Location: /pulp/api/v3/repositories/container/container/d5f0f584-1e8e-4b11-843d-98116ce962ee/
Server: gunicorn/20.0.0
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN

{
    "description": null,
    "latest_version_href": null,
    "name": "foo",
    "pulp_created": "2019-11-17T12:36:44.516158Z",
    "pulp_href": "/pulp/api/v3/repositories/container/container/d5f0f584-1e8e-4b11-843d-98116ce962ee/",
    "versions_href": "/pulp/api/v3/repositories/container/container/d5f0f584-1e8e-4b11-843d-98116ce962ee/versions/"
}

HTTP/1.1 201 Created
Allow: GET, POST, HEAD, OPTIONS
Connection: close
Content-Length: 436
Content-Type: application/json
Date: Sun, 17 Nov 2019 12:36:45 GMT
Location: /pulp/api/v3/remotes/container/container/603ef7c0-f961-4a78-a8ee-d690d7fa287f/
Server: gunicorn/20.0.0
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN

{
    "ca_cert": null,
    "client_cert": null,
    "client_key": null,
    "download_concurrency": 2,
    "name": "library/busybox",
    "policy": "immediate",
    "proxy_url": null,
    "pulp_created": "2019-11-17T12:36:45.355781Z",
    "pulp_href": "/pulp/api/v3/remotes/container/container/603ef7c0-f961-4a78-a8ee-d690d7fa287f/",
    "pulp_last_updated": "2019-11-17T12:36:45.355795Z",
    "tls_validation": true,
    "upstream_name": "busybox",
    "url": "https://registry-1.docker.io",
    "whitelist_tags": null
}

HTTP/1.1 202 Accepted
Allow: POST, OPTIONS
Connection: close
Content-Length: 67
Content-Type: application/json
Date: Sun, 17 Nov 2019 12:36:46 GMT
Server: gunicorn/20.0.0
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN

{
    "task": "/pulp/api/v3/tasks/1e3b4439-4abb-4c7c-980c-071460ce1f6e/"
}

4. Check the status of the task.

$ http :24817/pulp/api/v3/tasks/1e3b4439-4abb-4c7c-980c-071460ce1f6e/

HTTP/1.1 200 OK
Allow: GET, PATCH, DELETE, HEAD, OPTIONS
Connection: close
Content-Length: 1212
Content-Type: application/json
Date: Sun, 17 Nov 2019 12:50:05 GMT
Server: gunicorn/20.0.0
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN

{
    "created_resources": [
        "/pulp/api/v3/repositories/container/container/d5f0f584-1e8e-4b11-843d-98116ce962ee/versions/1/"
    ],
    "error": null,
    "finished_at": "2019-11-17T12:49:22.160145Z",
    "name": "pulp_container.app.tasks.synchronize.synchronize",
    "parent": null,
    "progress_reports": [
        {
            "code": "downloading.tag_list",
            "done": 1,
            "message": "Downloading tag list",
            "state": "completed",
            "suffix": null,
            "total": 1
        },
        {
            "code": "processing.tag",
            "done": 139,
            "message": "Processing Tags",
            "state": "completed",
            "suffix": null,
            "total": 139
        },
        {
            "code": "downloading.artifacts",
            "done": 1490,
            "message": "Downloading Artifacts",
            "state": "completed",
            "suffix": null,
            "total": null
        },
        {
            "code": "associating.content",
            "done": 1785,
            "message": "Associating Content",
            "state": "completed",
            "suffix": null,
            "total": null
        }
    ],
    "pulp_created": "2019-11-17T12:36:46.112538Z",
    "pulp_href": "/pulp/api/v3/tasks/1e3b4439-4abb-4c7c-980c-071460ce1f6e/",
    "reserved_resources_record": [
        "/pulp/api/v3/remotes/container/container/603ef7c0-f961-4a78-a8ee-d690d7fa287f/",
        "/pulp/api/v3/repositories/container/container/d5f0f584-1e8e-4b11-843d-98116ce962ee/"
    ],
    "spawned_tasks": [],
    "started_at": "2019-11-17T12:36:46.199769Z",
    "state": "completed",
    "worker": "/pulp/api/v3/workers/eccc84fa-2436-44d7-922b-0512545fff74/"
}

Do you think this issue can be resolved in pulp_container, or even in pulpcore?