Project

Profile

Help

Issue #1808

closed

exporting a sufficiently large repo with 'on_demand' policy results in BSON error

Added by cduryee almost 7 years ago. Updated almost 4 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
2.8.3
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Sprint 1
Quarter:

Description

If you attempt to export a repo with many RPMs, you'll get a BSON document too large error.

To repro:

  • create a large repo with "on_demand" policy, such as Fedora 23 everything
  • sync repo
  • create repo group with said repo
  • attempt to publish repo group

actual result:

BSON document too large error

expected result:

error that either a.) not all units are on-disk and repo/repo group cannot be exported, or b.) 'on_demand' repos cannot be exported

Actions #1

Updated by mhrivnak almost 7 years ago

A patch such as this will prevent the copytree() function from accumulating an unbounded number of errors before giving up. That unbounded collection of error messages is what caused the mongodb document size limit to be exceeded.

diff --git a/server/pulp/server/util.py b/server/pulp/server/util.py
index 5411e87..98fede3 100644
--- a/server/pulp/server/util.py
+++ b/server/pulp/server/util.py
@@ -207,6 +207,8 @@ def copytree(src, dst, symlinks=False, ignore=None):
     os.makedirs(dst)
     errors = []
     for name in names:
+        if len(errors) >= 100:
+            raise Error(errors)
         if name in ignored_names:
             continue
         srcname = os.path.join(src, name)
Actions #2

Updated by mhrivnak almost 7 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to mhrivnak
  • Sprint/Milestone set to 19

Added by mhrivnak almost 7 years ago

Revision dd5f8c7e

copytree no longer accumulates an unbounded number of errors

https://pulp.plan.io/issues/1808

fixes #1808

Added by mhrivnak almost 7 years ago

Revision dd5f8c7e

copytree no longer accumulates an unbounded number of errors

https://pulp.plan.io/issues/1808

fixes #1808

Actions #3

Updated by mhrivnak almost 7 years ago

  • Status changed from ASSIGNED to POST
Actions #4

Updated by mhrivnak almost 7 years ago

  • Platform Release set to 2.8.3
Actions #6

Updated by mhrivnak almost 7 years ago

  • Status changed from POST to MODIFIED
  • % Done changed from 0 to 100
Actions #7

Updated by mhrivnak almost 7 years ago

  • Triaged changed from No to Yes
Actions #8

Updated by pthomas@redhat.com almost 7 years ago

Verified
pre>
[root@mgmt5 ~]# pulp-admin rpm repo export run --repo-id rhel67+----------------------------------------------------------------------+
Publishing Repository [rhel67]
--------------------------------------------------------------------

This command may be exited via ctrl+c without affecting the request.

Task Failed

The repository cannot be exported because some units are not downloaded.

[root@mgmt5 ~]# pulp-admin rpm repo group export run --group-id grp1
--------------------------------------------------------------------
Exporting Repository Group [grp1]
--------------------------------------------------------------------

This command may be exited via ctrl+c without affecting the request.

Task Failed

The repository group cannot be exported because these repos have units that are
not downloaded: zoo, rhel67

[root@mgmt5 ~]#

Actions #9

Updated by semyers almost 7 years ago

  • Status changed from MODIFIED to 5
Actions #10

Updated by semyers almost 7 years ago

  • Status changed from 5 to CLOSED - CURRENTRELEASE
Actions #11

Updated by bmbouter about 5 years ago

  • Sprint set to Sprint 1
Actions #12

Updated by bmbouter about 5 years ago

  • Sprint/Milestone deleted (19)
Actions #13

Updated by bmbouter almost 4 years ago

  • Tags Pulp 2 added

Also available in: Atom PDF