Issue #6630: Missing revision-tag in repomd.xml is causing errors - RPM Support - Pulp

Actions

Send by e-mail Copy link

Issue #6630

closed

Missing revision-tag in repomd.xml is causing errors

Added by andreas.dijkman almost 4 years ago. Updated almost 4 years ago.

Status:

CLOSED - CURRENTRELEASE

Priority:

Normal

Assignee:

Sprint/Milestone:

Pulp RPM 3.3.1

Start date:

Due date:

Estimated time:

Severity:

2. Medium

Version:

Master

Platform Release:

OS:

Triaged:

Yes

Groomed:

Sprint Candidate:

Tags:

Sprint:

Quarter:

Description

If a repomd.xml-file is missing the revision-tag, repository can't be synced and throws errors. I think this is introduced in commit https://github.com/pulp/pulp_rpm/commit/444b7acc9eb9db81444ff853cccba50a3834544a, which is related to #6367.

Example of syncing repositories without revision-tag, almost all Oracle Linux yum-repositories. For example: http://yum.oracle.com/repo/OracleLinux/OL8/UEKR6/x86_64/repodata/repomd.xml

The errors I get are

pulp: rq.worker:ERROR: django.db.utils.IntegrityError: null value in column "last_sync_revision_number" violates not-null constraint
DETAIL:  Failing row contains (9f9b589b-beb7-4b6a-ba6d-03eeb26d6d62, f, null, null, 0, null, {"other": "SHA", "primary": "SHA", "other_db": "SHA", "filelists...).
Traceback (most recent call last):
 File "/var/lib/pulp/virtualenv/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
   return self.cursor.execute(sql, params)
psycopg2.IntegrityError: null value in column "last_sync_revision_number" violates not-null constraint
DETAIL:  Failing row contains (9f9b589b-beb7-4b6a-ba6d-03eeb26d6d62, f, null, null, 0, null, {"other": "SHA", "primary": "SHA", "other_db": "SHA", "filelists...).
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
 File "/var/lib/pulp/virtualenv/lib/python3.6/site-packages/rq/worker.py", line 886, in perform_job
   rv = job.perform()
 File "/var/lib/pulp/virtualenv/lib/python3.6/site-packages/rq/job.py", line 664, in perform
   self._result = self._execute()
 File "/var/lib/pulp/virtualenv/lib/python3.6/site-packages/rq/job.py", line 670, in _execute
   return self.func(*self.args, **self.kwargs)
 File "/var/lib/pulp/virtualenv/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 167, in synchronize
   dv.create()
 File "/var/lib/pulp/virtualenv/lib/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py", line 149, in create
   loop.run_until_complete(pipeline)
 File "/var/lib/pulp/virtualenv/lib/python3.6/site-packages/pulpcore/app/models/repository.py", line 787, in __exit__
   self.repository.save()
 File "/var/lib/pulp/virtualenv/lib/python3.6/site-packages/pulpcore/app/models/repository.py", line 62, in save
   super().save(*args, **kwargs)
 File "/var/lib/pulp/virtualenv/lib/python3.6/site-packages/pulpcore/app/models/base.py", line 107, in save
   return super().save(*args, **kwargs)
 File "/var/lib/pulp/virtualenv/lib/python3.6/site-packages/django/db/models/base.py", line 741, in save
   force_update=force_update, update_fields=update_fields)
 File "/var/lib/pulp/virtualenv/lib/python3.6/site-packages/django/db/models/base.py", line 779, in save_base
   force_update, using, update_fields,
 File "/var/lib/pulp/virtualenv/lib/python3.6/site-packages/django/db/models/base.py", line 851, in _save_table
   forced_update)
 File "/var/lib/pulp/virtualenv/lib/python3.6/site-packages/django/db/models/base.py", line 900, in _do_update
   return filtered._update(values) > 0
 File "/var/lib/pulp/virtualenv/lib/python3.6/site-packages/django/db/models/query.py", line 760, in _update
   return query.get_compiler(self.db).execute_sql(CURSOR)
 File "/var/lib/pulp/virtualenv/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1469, in execute_sql
   cursor = super().execute_sql(result_type)
 File "/var/lib/pulp/virtualenv/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1140, in execute_sql
   cursor.execute(sql, params)
 File "/var/lib/pulp/virtualenv/lib/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
   return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
 File "/var/lib/pulp/virtualenv/lib/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
   return executor(sql, params, many, context)
 File "/var/lib/pulp/virtualenv/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
   return self.cursor.execute(sql, params)
 File "/var/lib/pulp/virtualenv/lib/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
   raise dj_exc_value.with_traceback(traceback) from exc_value
 File "/var/lib/pulp/virtualenv/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
   return self.cursor.execute(sql, params)
django.db.utils.IntegrityError: null value in column "last_sync_revision_number" violates not-null constraint

I beleive it is originating in app/tasks/synchronizing.py where repomd.revision is used. As it is None it can't be inserted in the table. If you remove the not null contraint from the column, the problem shifts to a comparison from a previous revision to a current. The repomd.revision can't be cast to a digit (None) and throws an error:

AttributeError: 'NoneType' object has no attribute 'isdigit'
Traceback (most recent call last):
 File "/var/lib/pulp/virtualenv/lib/python3.6/site-packages/rq/worker.py", line 886, in perform_job
   rv = job.perform()
 File "/var/lib/pulp/virtualenv/lib/python3.6/site-packages/rq/job.py", line 664, in perform
   self._result = self._execute()
 File "/var/lib/pulp/virtualenv/lib/python3.6/site-packages/rq/job.py", line 670, in _execute
   return self.func(*self.args, **self.kwargs)
 File "/var/lib/pulp/virtualenv/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 167, in synchronize
   dv.create()
 File "/var/lib/pulp/virtualenv/lib/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py", line 149, in create
   loop.run_until_complete(pipeline)
 File "/usr/lib64/python3.6/asyncio/base_events.py", line 484, in run_until_complete
   return future.result()
 File "/var/lib/pulp/virtualenv/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 228, in create_pipeline
   await asyncio.gather(*futures)
 File "/var/lib/pulp/virtualenv/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 43, in __call__
   await self.run()
 File "/var/lib/pulp/virtualenv/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 346, in run
   is_previous_version(repomd.revision, self.repository.last_sync_revision_number)
 File "/var/lib/pulp/virtualenv/lib/python3.6/site-packages/pulp_rpm/app/shared_utils.py", line 49, in is_previous_version
   if version.isdigit() and target_version.isdigit():
AttributeError: 'NoneType' object has no attribute 'isdigit'

Installation is done on Oracle Linux 8.1 x86_64 via pip-method over version 3.3.0. DB is PostgreSQL 12 and Redis 5. Python 3.6

Actions

Copy link

Updated by andreas.dijkman almost 4 years ago

The following patch fixed it for me by the way:

--- synchronizing.py-orig	2020-05-01 23:18:50.775612239 +0200
+++ synchronizing.py	2020-05-01 23:21:55.118202698 +0200
@@ -2,6 +2,7 @@
 import gzip
 import logging
 import os
+import time
 
 from collections import defaultdict
 from gettext import gettext as _  # noqa:F401
@@ -331,6 +332,10 @@
             repomd_path = result.path
             repomd = cr.Repomd(repomd_path)
 
+            # It the tag revision is missing, set it at the current Unix TimeStamp
+            if (repomd.revision is None):
+                repomd.set_revision(str(int(time.time())))
+
             # Caution: we are not storing when the remote was last updated, so the order of this
             # logic must remain in this order where we first check the version number as other
             # changes than sync could have taken place such that the date or repo version will be

I think something like this was fixed earlier in #1263

Actions

Copy link

Updated by ttereshc almost 4 years ago

Thank you for the report and for the proposed patch. I'm not sure Pulp should assume any revision number and maybe it can just keep it as null.

Other users ran into a different issue #6662 with not being able to have null as a revision number. I'm leaning towards allowing null value.

What do others think?

Actions

Copy link

Updated by ttereshc almost 4 years ago

Status changed from NEW to MODIFIED
Triaged changed from No to Yes

This issue is fixed by #6662. Null value for revision number is accepted.

For repositories with no revision number, there will be no sync optmization because it's not possible to tell whether there were any changes in the remote repository or not.

Actions

Copy link