Pulp

I had been following issue #2605 and was hopeful that when the issue was closed at the 2.13.1 release, that syncs would work for the sles12 updates repo, however, I have found that (in term of stable releases) they haven't worked since 2.10.3. Here is more information that was noticed as I moved through to find the last stable release that would work for the sync on this repo.

name: SUSE_Linux_Enterprise_Server_12_x86_64:SLES12-Updates.repo
baseurl: https://updates.suse.com/SUSE/Updates/SLE-SERVER/12/x86_64/update

(not providing token to access), but that was entered into pulp via REST api and from pulp, I was able to access repo url.

error received on sync (stable versions from 2.11.1 and forward):

--------------------------------------------------------------------
Synchronizing Repository [sles-12-updates-x86_64]
--------------------------------------------------------------------

This command may be exited via ctrl+c without affecting the request.

Downloading metadata...
[|]
... completed

Downloading repository content...
[-]

Task Failed

Malformed repository: metadata is missing for some packages in filelists.xml and
in other.xml

contents in the baseurl on updates.suse.com for this repo:

289fcab0500fd4c25ff9891986b9e94f19e55d3025a659aca24452450cd91001-filelists.xml.gz 20-Jan-2017 03:59 10M
[ ] 924bc93b54e975cf73e7981691ffa8e5252188481719cf838952509faa549edf-other.xml.gz 20-Jan-2017 03:59 3.7M
[ ] 963e7cd7a693a08591c36de6f5f6c570ffd67c7cd0143f035756eac1faa86680-appdata.xml.gz 20-Jan-2017 03:59 22K
[ ] 3716b64ae0bc83476f1eb60cf0492c149668d68feeb84f588ca9b8d092ececde-primary.xml.gz 20-Jan-2017 03:59 8.0M
[ ] 7550cd2032cfdbfa03de56105981904213199269986c57686acb45db36b9cd39-susedata.xml.gz 20-Jan-2017 04:00 140K
[ ] 8893d6a534cf081df9cf62ddf806fc9e58ad2c1c157e4cc75f8033660f5307ea-updateinfo.xml.gz 20-Jan-2017 03:59 421K
[ ] d4e5931ae5de09e4ec59db630200226a46ff33c0ae4ca417a08c51e506e89cfb-app-icons.tar.gz 20-Jan-2017 03:59 2.3K
[ ] d66a570bf21cbbc3f6b1cd7a4233d1d9b5846452b150f7d9d0db5112ecfedd28-deltainfo.xml.gz 20-Jan-2017 04:00 108K
[TXT] repomd.xml 20-Jan-2017 04:00 3.6K
[TXT] repomd.xml.asc 20-Jan-2017 04:00 481
[TXT] repomd.xml.key 20-Jan-2017 04:00 972

observations:

I notice that there are some files listed (and called out as SUSE specific) in the pulp-rpm-plugins rpm under pulp_rpm/plugins/importers/yum/repomd/metadata.py that do not have names of the "units" listed above, (e.g. appdata, app-icons)

1. These are metadata file types listed in "repomd" that we do not want to store as units
   .
2. deltainfo and susedata are SUSE specific files that we need to ignore for now, as ther
   e's no
3. handling yet and copying them creates broken SUSE repos.
   KNOWN_TYPES = set(['group', 'group_gz',
   'filelists', 'filelists_db',
   'other', 'other_db',
   'primary', 'primary_db',
   'deltainfo', 'susedata',
   'prestodelta',
   'updateinfo', 'updateinfo_db'])

although this is probably a separate issue, thought I would mention it.

the crux of the error occurs when 2 numbers are compared and found to be not equal from the pulp_rpm/plugins/importers/yum/sync.py under function _decide_rpms_to_download

if primary_rpm_count != metadata_files.rpm_count:

---> primary_rpm_count: 3927 metadata_files_rpm_count: 3924

due to needing to move forward with an upgrade of the system using a version (even our current if necessary) that works, I didn't spend the time to find out which 3 rpms are the difference. If needed, I could provide as an attachment the list of output for serialized_pkgs that I put together as a pulp-admin sync that failed was running.

this is the only sles12 repo that has failed sync with the error posted above for me.