Issue #9631
Updated by fao89 over 2 years ago
**Ticket moved to GitHub**: "pulp/pulp_rpm/2308":https://github.com/pulp/pulp_rpm/issues/2308 ---- Hi All Pulp 3.16.1 on docker. The Sync from remote never completes. If you could please assist. We are seeing Ansible error: ~~~ An exception occurred during task execution. To see the full traceback, use -vvv. The error was: Exception: Task failed to complete. (failed; A file located at the url https://rmt.abcgroup.net/repo/SUSE/Updates/SLE-SERVER/12-SP5/x86_64/update/src/kernel-source-4.12.14-122.98.1.src.rpm failed validation due to checksum.) failed: [localhost] (item={'name': 'SLES12-SP5-Updates', 'backend': 'https://rmt.abcgroup.net/repo/SUSE/Updates/SLE-SERVER/12-SP5/x86_64/update/', 'base': 'SLE/SLES12/SP5/updates'}) => {"ansible_index_var": "s_idx", "ansible_loop_var": "item", "changed": false, "item": {"backend": "https://rmt.abcgroup.net/repo/SUSE/Updates/SLE-SERVER/12-SP5/x86_64/update/", "base": "SLE/SLES12/SP5/updates", "name": "SLES12-SP5-Updates"}, "msg": "Task failed to complete. (failed; A file located at the url https://rmt.abcgroup.net/repo/SUSE/Updates/SLE-SERVER/12-SP5/x86_64/update/src/kernel-source-4.12.14-122.98.1.src.rpm failed validation due to checksum.)", "s_idx": 3} ~~~ On remote: rmt.abcgroup.net # sysctl -a|grep aio fs.aio-max-nr = 65536 fs.aio-nr = 2080 On pulp server # sysctl -a|grep aio fs.aio-max-nr = 65536 fs.aio-nr = 2 ================ Attached is the Docker pulp log. docker logs pulp aio reported errors for various syncs(tries) of the same repo. The sha256 for the RPMs are correct on the remote server's repodata. ~~~ Backing off download_wrapper(...) for 0.7s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed) pulp [6efc07543b0c4821922029cf5cda3e90]: backoff:INFO: Backing off download_wrapper(...) for 0.7s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed) Backing off download_wrapper(...) for 3.5s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed) pulp [6efc07543b0c4821922029cf5cda3e90]: backoff:INFO: Backing off download_wrapper(...) for 3.5s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed) Backing off download_wrapper(...) for 0.4s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed) pulp [896ae2851414447db9e177f69c677185]: backoff:INFO: Backing off download_wrapper(...) for 0.4s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed) Backing off download_wrapper(...) for 0.3s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed) pulp [896ae2851414447db9e177f69c677185]: backoff:INFO: Backing off download_wrapper(...) for 0.3s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed) Backing off download_wrapper(...) for 1.8s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed) pulp [896ae2851414447db9e177f69c677185]: backoff:INFO: Backing off download_wrapper(...) for 1.8s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed) Backing off download_wrapper(...) for 0.9s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed) pulp [896ae2851414447db9e177f69c677185]: backoff:INFO: Backing off download_wrapper(...) for 0.9s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed) Backing off download_wrapper(...) for 3.1s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed) pulp [896ae2851414447db9e177f69c677185]: backoff:INFO: Backing off download_wrapper(...) for 3.1s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed) Backing off download_wrapper(...) for 1.1s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed) pulp [896ae2851414447db9e177f69c677185]: backoff:INFO: Backing off download_wrapper(...) for 1.1s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed) Backing off download_wrapper(...) for 1.6s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed) pulp [896ae2851414447db9e177f69c677185]: backoff:INFO: Backing off download_wrapper(...) for 1.6s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed) Backing off download_wrapper(...) for 2.5s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed) pulp [896ae2851414447db9e177f69c677185]: backoff:INFO: Backing off download_wrapper(...) for 2.5s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed) Backing off download_wrapper(...) for 0.8s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed) pulp [5f8f6f6814aa494da5817dc65d65dca2]: backoff:INFO: Backing off download_wrapper(...) for 0.8s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed) Backing off download_wrapper(...) for 0.8s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed) pulp [5f8f6f6814aa494da5817dc65d65dca2]: backoff:INFO: Backing off download_wrapper(...) for 0.8s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed) Backing off download_wrapper(...) for 0.9s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed) pulp [5f8f6f6814aa494da5817dc65d65dca2]: backoff:INFO: Backing off download_wrapper(...) for 0.9s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed) Backing off download_wrapper(...) for 0.4s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed) pulp [5f8f6f6814aa494da5817dc65d65dca2]: backoff:INFO: Backing off download_wrapper(...) for 0.4s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed) Backing off download_wrapper(...) for 3.8s (aiohttp.client_exceptions.ServerDisconnectedError: Server disconnected) pulp [5f8f6f6814aa494da5817dc65d65dca2]: backoff:INFO: Backing off download_wrapper(...) for 3.8s (aiohttp.client_exceptions.ServerDisconnectedError: Server disconnected) Backing off download_wrapper(...) for 1.2s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed) pulp [5f8f6f6814aa494da5817dc65d65dca2]: backoff:INFO: Backing off download_wrapper(...) for 1.2s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed) Backing off download_wrapper(...) for 6.6s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed) pulp [5f8f6f6814aa494da5817dc65d65dca2]: backoff:INFO: Backing off download_wrapper(...) for 6.6s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed) Backing off download_wrapper(...) for 6.3s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed) pulp [5f8f6f6814aa494da5817dc65d65dca2]: backoff:INFO: Backing off download_wrapper(...) for 6.3s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed) Giving up download_wrapper(...) after 5 tries (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed) pulp [5f8f6f6814aa494da5817dc65d65dca2]: backoff:ERROR: Giving up download_wrapper(...) after 5 tries (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed) ~~~ -------- The failures reported soon after the reported aio errors: (see attached log file) ~~~ pulp [2fa84b10f57c41f39635eddffc89ee0a]: pulpcore.tasking.pulpcore_worker:INFO: Task 086ebba6-1f81-471c-9f60-47eda47969a6 failed () pulp [2fa84b10f57c41f39635eddffc89ee0a]: pulpcore.tasking.pulpcore_worker:INFO: File "/usr/local/lib/python3.8/site-packages/pulpcore/tasking/pulpcore_worker.py", line 370, in _perform_task -- pulp [6efc07543b0c4821922029cf5cda3e90]: pulpcore.tasking.pulpcore_worker:INFO: Task c375f81f-7b64-4356-983b-b8729048094f failed (A file located at the url https://rmt.abcgroup.net/repo/SUSE/Updates/SLE-SERVER/12-SP5/x86_64/update/noarch/kernel-source-azure-4.12.14-16.76.2.noarch.rpm failed validation due to checksum.) pulp [6efc07543b0c4821922029cf5cda3e90]: pulpcore.tasking.pulpcore_worker:INFO: File "/usr/local/lib/python3.8/site-packages/pulpcore/tasking/pulpcore_worker.py", line 370, in _perform_task ~~~ -- ------------ Installation information: ~~~ pulp rpm remote show --name "remote_SLES12-SP5-Updates" "download_concurrency": 2, (tried 1, 4, 10 - same result) ~~~ pulp --version pulp3 command line interface, version 0.12.0 pip freeze appdirs==1.4.3 asn1crypto==0.24.0 Babel==2.8.0 beautifulsoup4==4.10.0 bs4==0.0.1 certifi==2021.5.30 cffi==1.13.2 chardet==4.0.0 click==8.0.1 colorama==0.4.4 commonmark==0.9.1 contextlib2==21.6.0 cryptography==2.8 dataclasses==0.8 httpie==1.0.3 idna==2.10 importlib-metadata==4.6.1 iotop==0.6 isc==2.0 Jinja2==2.10.1 MarkupSafe==1.0 packaging==20.3 pip-search==0.0.7 ply==3.10 pulp-cli==0.12.0 py==1.8.1 pyasn1==0.4.2 pycparser==2.17 Pygments==2.6.1 pyOpenSSL==19.0.0 pyparsing==2.4.7 pytz==2020.5 PyYAML==5.4.1 requests==2.25.1 rich==10.15.2 schema==0.7.4 six==1.14.0 soupsieve==2.3.1 toml==0.10.2 typing-extensions==3.10.0.0 urllib3==1.26.6 zipp==3.5.0 pulp status "versions": [ { "component": "core", "version": "3.16.1" }, { "component": "rpm", "version": "3.16.1" }, { "component": "python", "version": "3.5.2" }, { "component": "file", "version": "1.10.1" }, { "component": "deb", "version": "2.16.0" }, { "component": "container", "version": "2.9.1" }, { "component": "certguard", "version": "1.5.1" }, { "component": "ansible", "version": "0.10.1" } Docker server ~~~ cat /etc/os-release NAME="SLES" VERSION="15-SP3" VERSION_ID="15.3" PRETTY_NAME="SUSE Linux Enterprise Server 15 SP3" ~~~ Thanks so much for the support. Jan
.