Issue #6471
closedUnable to sync large openshift docker repos
Description
We are trying to sync the following docker images from satellite server and we got the error while doing sync 1.openshift3/mysql-apb 2.openshift3/mediawiki-apb 3.openshift3/mariadb-apb 4.openshift3/metrics-hawkular-metrics
Getting the below error, when syncing the repository from http://registry.redhat.io/https://registry.redhat.io
Error:
DKR1020: Image download(s) from http://registry.redhat.io/v2/openshift3/mysql-apb/blobs/sha256:1e3f598e03f841ddc4933fafb143d44393d4ce0517604c5cb2367024975e871c failed. Sync task has failed to prevent a corrupted repository.
DKR1020: Image download(s) from https://registry.redhat.io/v2/openshift3/mysql-apb/blobs/sha256:2a81c83bf7ad14e424acbfc7eecaf84aec2b273c25124807d820c4da607644e1 failed. Sync task has failed to prevent a corrupted repository.
Stacktrace:
Apr 1 09:45:10 satellite pulp: nectar.downloaders.threaded:ERROR: (22861-60960) (104, 'Connection reset by peer')
Apr 1 09:45:10 satellite pulp: nectar.downloaders.threaded:ERROR: (22861-60960) Traceback (most recent call last):
Apr 1 09:45:10 satellite pulp: nectar.downloaders.threaded:ERROR: (22861-60960) File "/usr/lib/python2.7/site-packages/nectar/downloaders/threaded.py", line 292, in _fetch
Apr 1 09:45:10 satellite pulp: nectar.downloaders.threaded:ERROR: (22861-60960) for chunk in chunks:
Apr 1 09:45:10 satellite pulp: nectar.downloaders.threaded:ERROR: (22861-60960) File "/usr/lib/python2.7/site-packages/requests/models.py", line 656, in generate
Apr 1 09:45:10 satellite pulp: nectar.downloaders.threaded:ERROR: (22861-60960) for chunk in self.raw.stream(chunk_size, decode_content=True):
Apr 1 09:45:10 satellite pulp: nectar.downloaders.threaded:ERROR: (22861-60960) File "/usr/lib/python2.7/site-packages/requests/packages/urllib3/response.py", line 273, in stream
Apr 1 09:45:10 satellite pulp: nectar.downloaders.threaded:ERROR: (22861-60960) data = self.read(amt=amt, decode_content=decode_content)
Apr 1 09:45:10 satellite pulp: nectar.downloaders.threaded:ERROR: (22861-60960) File "/usr/lib/python2.7/site-packages/requests/packages/urllib3/response.py", line 203, in read
Apr 1 09:45:10 satellite pulp: nectar.downloaders.threaded:ERROR: (22861-60960) data = self._fp.read(amt)
Apr 1 09:45:10 satellite pulp: nectar.downloaders.threaded:ERROR: (22861-60960) File "/usr/lib64/python2.7/httplib.py", line 611, in read
Apr 1 09:45:10 satellite pulp: nectar.downloaders.threaded:ERROR: (22861-60960) s = self.fp.read(amt)
Apr 1 09:45:10 satellite pulp: nectar.downloaders.threaded:ERROR: (22861-60960) File "/usr/lib64/python2.7/socket.py", line 380, in read
Apr 1 09:45:10 satellite pulp: nectar.downloaders.threaded:ERROR: (22861-60960) data = self._sock.recv(left)
Apr 1 09:45:10 satellite pulp: nectar.downloaders.threaded:ERROR: (22861-60960) File "/usr/lib/python2.7/site-packages/requests/packages/urllib3/contrib/pyopenssl.py", line 188, in recv
Apr 1 09:45:10 satellite pulp: nectar.downloaders.threaded:ERROR: (22861-60960) data = self.connection.recv(*args, **kwargs)
Apr 1 09:45:10 satellite pulp: nectar.downloaders.threaded:ERROR: (22861-60960) SysCallError: (104, 'Connection reset by peer')
The workaround is to limit the tags to sync but it is not feasible for some users because they will have to manually update the tags to sync when the upstream repo is updated
Related issues
Updated by hyu over 4 years ago
I made a patch to let the docker plugin to retry docker blob downloads when hitting the connection drop issue. After applying this patch, I can sync the large docker repositories successfully.
Updated by ipanova@redhat.com over 4 years ago
- Status changed from NEW to POST
- Triaged changed from No to Yes
Updated by ipanova@redhat.com over 4 years ago
- Status changed from POST to CLOSED - NOTABUG
Pulp should not keep hitting on the server if the server dropped the connection. There is a ticket opened with IT to solve issue on the server side. https://projects.engineering.redhat.com/browse/CLOUDDST-285
Updated by dkliban@redhat.com about 4 years ago
- Status changed from CLOSED - NOTABUG to MODIFIED
We merged some retry logic into nectar.
Updated by ggainey about 4 years ago
- Status changed from MODIFIED to CLOSED - CURRENTRELEASE
Updated by ipanova@redhat.com almost 4 years ago
- Copied to Issue #7911: Unable to sync large openshift docker repos added