Project

Profile

Help

Issue #1808

exporting a sufficiently large repo with 'on_demand' policy results in BSON error

Added by cduryee over 4 years ago. Updated over 1 year ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
2.8.3
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Sprint 1
Quarter:

Description

If you attempt to export a repo with many RPMs, you'll get a BSON document too large error.

To repro:

  • create a large repo with "on_demand" policy, such as Fedora 23 everything
  • sync repo
  • create repo group with said repo
  • attempt to publish repo group

actual result:

BSON document too large error

expected result:

error that either a.) not all units are on-disk and repo/repo group cannot be exported, or b.) 'on_demand' repos cannot be exported

Associated revisions

Revision dd5f8c7e View on GitHub
Added by mhrivnak over 4 years ago

copytree no longer accumulates an unbounded number of errors

https://pulp.plan.io/issues/1808

fixes #1808

Revision dd5f8c7e View on GitHub
Added by mhrivnak over 4 years ago

copytree no longer accumulates an unbounded number of errors

https://pulp.plan.io/issues/1808

fixes #1808

History

#1 Updated by mhrivnak over 4 years ago

A patch such as this will prevent the copytree() function from accumulating an unbounded number of errors before giving up. That unbounded collection of error messages is what caused the mongodb document size limit to be exceeded.

diff --git a/server/pulp/server/util.py b/server/pulp/server/util.py
index 5411e87..98fede3 100644
--- a/server/pulp/server/util.py
+++ b/server/pulp/server/util.py
@@ -207,6 +207,8 @@ def copytree(src, dst, symlinks=False, ignore=None):
     os.makedirs(dst)
     errors = []
     for name in names:
+        if len(errors) >= 100:
+            raise Error(errors)
         if name in ignored_names:
             continue
         srcname = os.path.join(src, name)

#2 Updated by mhrivnak over 4 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to mhrivnak
  • Sprint/Milestone set to 19

#3 Updated by mhrivnak over 4 years ago

  • Status changed from ASSIGNED to POST

#4 Updated by mhrivnak over 4 years ago

  • Platform Release set to 2.8.3

#6 Updated by mhrivnak over 4 years ago

  • Status changed from POST to MODIFIED
  • % Done changed from 0 to 100

#7 Updated by mhrivnak over 4 years ago

  • Triaged changed from No to Yes

#8 Updated by pthomas@redhat.com over 4 years ago

Verified
pre>
[root@mgmt5 ~]# pulp-admin rpm repo export run --repo-id rhel67+----------------------------------------------------------------------+
Publishing Repository [rhel67]
--------------------------------------------------------------------

This command may be exited via ctrl+c without affecting the request.

Task Failed

The repository cannot be exported because some units are not downloaded.

[root@mgmt5 ~]# pulp-admin rpm repo group export run --group-id grp1
--------------------------------------------------------------------
Exporting Repository Group [grp1]
--------------------------------------------------------------------

This command may be exited via ctrl+c without affecting the request.

Task Failed

The repository group cannot be exported because these repos have units that are
not downloaded: zoo, rhel67

[root@mgmt5 ~]#

#9 Updated by semyers over 4 years ago

  • Status changed from MODIFIED to 5

#10 Updated by semyers over 4 years ago

  • Status changed from 5 to CLOSED - CURRENTRELEASE

#11 Updated by bmbouter over 2 years ago

  • Sprint set to Sprint 1

#12 Updated by bmbouter over 2 years ago

  • Sprint/Milestone deleted (19)

#13 Updated by bmbouter over 1 year ago

  • Tags Pulp 2 added

Please register to edit this issue

Also available in: Atom PDF