Ticket moved to GitHub: "pulp/pulp_rpm/2308":https://github.com/pulp/pulp_rpm/issues/2308

Hi All

Pulp 3.16.1 on docker.

The Sync from remote never completes. If you could please assist.

We are seeing Ansible error:

An exception occurred during task execution. To see the full traceback, use -vvv. The error was: Exception: Task failed to complete. (failed; A file located at the url https://rmt.abcgroup.net/repo/SUSE/Updates/SLE-SERVER/12-SP5/x86_64/update/src/kernel-source-4.12.14-122.98.1.src.rpm failed validation due to checksum.)
failed: [localhost] (item={'name': 'SLES12-SP5-Updates', 'backend': 'https://rmt.abcgroup.net/repo/SUSE/Updates/SLE-SERVER/12-SP5/x86_64/update/', 'base': 'SLE/SLES12/SP5/updates'}) => {"ansible_index_var": "s_idx", "ansible_loop_var": "item", "changed": false, "item": {"backend": "https://rmt.abcgroup.net/repo/SUSE/Updates/SLE-SERVER/12-SP5/x86_64/update/", "base": "SLE/SLES12/SP5/updates", "name": "SLES12-SP5-Updates"}, "msg": "Task failed to complete. (failed; A file located at the url https://rmt.abcgroup.net/repo/SUSE/Updates/SLE-SERVER/12-SP5/x86_64/update/src/kernel-source-4.12.14-122.98.1.src.rpm failed validation due to checksum.)", "s_idx": 3}

On remote: rmt.abcgroup.net

sysctl -a|grep aio¶

fs.aio-max-nr = 65536 fs.aio-nr = 2080

On pulp server

sysctl -a|grep aio¶

fs.aio-max-nr = 65536 fs.aio-nr = 2

================ Attached is the Docker pulp log. docker logs pulp

aio reported errors for various syncs(tries) of the same repo.

The sha256 for the RPMs are correct on the remote server's repodata.

Backing off download_wrapper(...) for 0.7s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed)
pulp [6efc07543b0c4821922029cf5cda3e90]: backoff:INFO: Backing off download_wrapper(...) for 0.7s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed)

Backing off download_wrapper(...) for 3.5s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed)
pulp [6efc07543b0c4821922029cf5cda3e90]: backoff:INFO: Backing off download_wrapper(...) for 3.5s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed)
Backing off download_wrapper(...) for 0.4s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed)
pulp [896ae2851414447db9e177f69c677185]: backoff:INFO: Backing off download_wrapper(...) for 0.4s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed)
Backing off download_wrapper(...) for 0.3s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed)
pulp [896ae2851414447db9e177f69c677185]: backoff:INFO: Backing off download_wrapper(...) for 0.3s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed)
Backing off download_wrapper(...) for 1.8s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed)
pulp [896ae2851414447db9e177f69c677185]: backoff:INFO: Backing off download_wrapper(...) for 1.8s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed)
Backing off download_wrapper(...) for 0.9s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed)
pulp [896ae2851414447db9e177f69c677185]: backoff:INFO: Backing off download_wrapper(...) for 0.9s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed)
Backing off download_wrapper(...) for 3.1s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed)
pulp [896ae2851414447db9e177f69c677185]: backoff:INFO: Backing off download_wrapper(...) for 3.1s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed)
Backing off download_wrapper(...) for 1.1s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed)
pulp [896ae2851414447db9e177f69c677185]: backoff:INFO: Backing off download_wrapper(...) for 1.1s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed)
Backing off download_wrapper(...) for 1.6s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed)
pulp [896ae2851414447db9e177f69c677185]: backoff:INFO: Backing off download_wrapper(...) for 1.6s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed)
Backing off download_wrapper(...) for 2.5s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed)
pulp [896ae2851414447db9e177f69c677185]: backoff:INFO: Backing off download_wrapper(...) for 2.5s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed)
Backing off download_wrapper(...) for 0.8s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed)
pulp [5f8f6f6814aa494da5817dc65d65dca2]: backoff:INFO: Backing off download_wrapper(...) for 0.8s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed)
Backing off download_wrapper(...) for 0.8s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed)
pulp [5f8f6f6814aa494da5817dc65d65dca2]: backoff:INFO: Backing off download_wrapper(...) for 0.8s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed)
Backing off download_wrapper(...) for 0.9s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed)
pulp [5f8f6f6814aa494da5817dc65d65dca2]: backoff:INFO: Backing off download_wrapper(...) for 0.9s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed)
Backing off download_wrapper(...) for 0.4s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed)
pulp [5f8f6f6814aa494da5817dc65d65dca2]: backoff:INFO: Backing off download_wrapper(...) for 0.4s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed)
Backing off download_wrapper(...) for 3.8s (aiohttp.client_exceptions.ServerDisconnectedError: Server disconnected)
pulp [5f8f6f6814aa494da5817dc65d65dca2]: backoff:INFO: Backing off download_wrapper(...) for 3.8s (aiohttp.client_exceptions.ServerDisconnectedError: Server disconnected)
Backing off download_wrapper(...) for 1.2s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed)
pulp [5f8f6f6814aa494da5817dc65d65dca2]: backoff:INFO: Backing off download_wrapper(...) for 1.2s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed)
Backing off download_wrapper(...) for 6.6s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed)
pulp [5f8f6f6814aa494da5817dc65d65dca2]: backoff:INFO: Backing off download_wrapper(...) for 6.6s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed)
Backing off download_wrapper(...) for 6.3s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed)
pulp [5f8f6f6814aa494da5817dc65d65dca2]: backoff:INFO: Backing off download_wrapper(...) for 6.3s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed)
Giving up download_wrapper(...) after 5 tries (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed)
pulp [5f8f6f6814aa494da5817dc65d65dca2]: backoff:ERROR: Giving up download_wrapper(...) after 5 tries (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed)

The failures reported soon after the reported aio errors: (see attached log file)

pulp [2fa84b10f57c41f39635eddffc89ee0a]: pulpcore.tasking.pulpcore_worker:INFO: Task 086ebba6-1f81-471c-9f60-47eda47969a6 failed ()
pulp [2fa84b10f57c41f39635eddffc89ee0a]: pulpcore.tasking.pulpcore_worker:INFO:   File "/usr/local/lib/python3.8/site-packages/pulpcore/tasking/pulpcore_worker.py", line 370, in _perform_task
--
pulp [6efc07543b0c4821922029cf5cda3e90]: pulpcore.tasking.pulpcore_worker:INFO: Task c375f81f-7b64-4356-983b-b8729048094f failed (A file located at the url https://rmt.abcgroup.net/repo/SUSE/Updates/SLE-SERVER/12-SP5/x86_64/update/noarch/kernel-source-azure-4.12.14-16.76.2.noarch.rpm failed validation due to checksum.)
pulp [6efc07543b0c4821922029cf5cda3e90]: pulpcore.tasking.pulpcore_worker:INFO:   File "/usr/local/lib/python3.8/site-packages/pulpcore/tasking/pulpcore_worker.py", line 370, in _perform_task

--¶

Installation information:

pulp rpm remote show --name "remote_SLES12-SP5-Updates"
"download_concurrency": 2,  (tried 1, 4, 10 - same result)

pulp --version pulp3 command line interface, version 0.12.0

pip freeze appdirs==1.4.3 asn1crypto==0.24.0 Babel==2.8.0 beautifulsoup4==4.10.0 bs4==0.0.1 certifi==2021.5.30 cffi==1.13.2 chardet==4.0.0 click==8.0.1 colorama==0.4.4 commonmark==0.9.1 contextlib2==21.6.0 cryptography==2.8 dataclasses==0.8 httpie==1.0.3 idna==2.10 importlib-metadata==4.6.1 iotop==0.6 isc==2.0 Jinja2==2.10.1 MarkupSafe==1.0 packaging==20.3 pip-search==0.0.7 ply==3.10 pulp-cli==0.12.0 py==1.8.1 pyasn1==0.4.2 pycparser==2.17 Pygments==2.6.1 pyOpenSSL==19.0.0 pyparsing==2.4.7 pytz==2020.5 PyYAML==5.4.1 requests==2.25.1 rich==10.15.2 schema==0.7.4 six==1.14.0 soupsieve==2.3.1 toml==0.10.2 typing-extensions==3.10.0.0 urllib3==1.26.6 zipp==3.5.0

pulp status "versions": [ { "component": "core", "version": "3.16.1" }, { "component": "rpm", "version": "3.16.1" }, { "component": "python", "version": "3.5.2" }, { "component": "file", "version": "1.10.1" }, { "component": "deb", "version": "2.16.0" }, { "component": "container", "version": "2.9.1" }, { "component": "certguard", "version": "1.5.1" }, { "component": "ansible", "version": "0.10.1" }

Docker server

cat /etc/os-release
NAME="SLES"
VERSION="15-SP3"
VERSION_ID="15.3"
PRETTY_NAME="SUSE Linux Enterprise Server 15 SP3"

Thanks so much for the support. Jan