Issue #2377
closed
0025_importer_schema_change hits issue with last_sync being None
Status:
CLOSED - CURRENTRELEASE
Description
0025_importer_schema_change causes the following error during run:
Applying pulp.server.db.migrations version 25
Applying migration pulp.server.db.migrations.0025_importer_schema_change failed.
Halting migrations due to a migration failure.
'NoneType' object has no attribute 'split'
Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/pulp/server/db/manage.py", line 194, in main
return _auto_manage_db(options)
File "/usr/lib/python2.6/site-packages/pulp/server/db/manage.py", line 257, in _auto_manage_db
migrate_database(options)
File "/usr/lib/python2.6/site-packages/pulp/server/db/manage.py", line 124, in migrate_database
update_current_version=not options.test)
File "/usr/lib/python2.6/site-packages/pulp/server/db/migrate/models.py", line 186, in apply_migration
migration.migrate()
File "/usr/lib/python2.6/site-packages/pulp/server/db/migrations/0025_importer_schema_change.py", line 19, in migrate
importer[updated_key] = dateutils.parse_iso8601_datetime(importer['last_sync'])
File "/usr/lib/python2.6/site-packages/pulp/common/dateutils.py", line 176, in parse_iso8601_datetime
return isodate.parse_datetime(datetime_str)
File "/usr/lib/python2.6/site-packages/isodate/isodatetime.py", line 48, in parse_datetime
datestring, timestring = datetimestring.split('T')
AttributeError: 'NoneType' object has no attribute 'split'
This was worked around by changing "elif 'last_sync' in importer.keys():" to "elif importer.get('last_sync') is not None:" in 0025_importer_schema_change.py. This appears to be working fine in production, though testing was extra-medium at best
vitaminmoo wrote:
This was worked around by changing "elif 'last_sync' in importer.keys():" to "elif importer.get('last_sync') is not None:" in 0025_importer_schema_change.py. This appears to be working fine in production, though testing was extra-medium at best
This is a diff, of the change described above:
diff --git a/server/db/migrations/0025_importer_schema_change.py b/server/db/migrations/0025_importer_schema_change.py
index 3de6284..6157760 100644
--- a//server/db/migrations/0025_importer_schema_change.py
+++ b/server/db/migrations/0025_importer_schema_change.py
@@ -15,7 +15,7 @@ def migrate(*args, **kwargs):
if updated_key in importer.keys():
continue
- elif 'last_sync' in importer.keys():
+ elif importer.get('last_sync') is not None:
importer[updated_key] = dateutils.parse_iso8601_datetime(importer['last_sync'])
else:
importer[updated_key] = dateutils.now_utc_datetime_with_tzinfo()
Where this file lands varies from distribution to distribution. We also need more testing before calling this "the" fix for folks running 2.10.1.
Yes, the real fix should also put the parsing in a try...except block.
This patch should be more robust. I'm not able to test it at the moment, so I could use help with that.
diff --git a/server/pulp/server/db/migrations/0025_importer_schema_change.py b/server/pulp/server/db/migrations/0025_importer_schema_change.py
index 3de6284..836b25f 100644
--- a/server/pulp/server/db/migrations/0025_importer_schema_change.py
+++ b/server/pulp/server/db/migrations/0025_importer_schema_change.py
@@ -15,9 +15,11 @@ def migrate(*args, **kwargs):
if updated_key in importer.keys():
continue
- elif 'last_sync' in importer.keys():
+
+ try:
importer[updated_key] = dateutils.parse_iso8601_datetime(importer['last_sync'])
- else:
+ # The attribute doesn't exist, or parsing failed. It's safe to set a newer timestamp.
+ except:
importer[updated_key] = dateutils.now_utc_datetime_with_tzinfo()
collection.save(importer)
- Priority changed from Normal to Urgent
- Triaged changed from No to Yes
- Status changed from NEW to ASSIGNED
- Assignee set to semyers
- Status changed from ASSIGNED to POST
- Platform Release set to 2.11.0
- OS deleted (
RHEL 6)
This issue was discovered in 2.10.1, but required a different fix for that version than 2.11. #2378 fixes 2.10.1, this issue fixes 2.11.0.
- Status changed from POST to MODIFIED
I can confirm that our install was upgraded numerous times.
The final version of the fix was very similar to the workaround you put in place, with a little bit more protection again failure.
I'm building a 2.10.2 hotfix release now, which should get anyone on 2.10.0 or 2.10.1 (broken or not) upgraded successfully.
- Status changed from MODIFIED to 5
- Related to Issue #2378: Bad merges forward on 2.10-dev added
- Status changed from 5 to MODIFIED
I'm going to be building Beta 3 tomorrow; I'll put this back ON_QA once that's available with all the fixes that we included in the 2.10.2 hotfix as well as 2.10.3.
- Status changed from MODIFIED to 5
- Status changed from 5 to 6
Verified
> use pulp_database
switched to db pulp_database
>
>
> db.repo_importers.find()
{ "_id" : ObjectId("58261de01661f00b3558afe7"), "repo_id" : "zoo", "importer_type_id" : "yum_importer", "config" : { "feed" : "https://repos.fedorapeople.org/pulp/pulp/fixtures/rpm/" }, "_ns" : "repo_importers", "scratchpad" : { "repomd_revision" : 1478850732, "previous_skip_list" : [ ] }, "last_sync" : "2016-11-11T20:44:18Z" }
>
>
> db.repo_importers.update( { "repo_id" : "zoo" }, { $set: { "last_sync": null} } )
WriteResult({ "nMatched" : 1, "nUpserted" : 0, "nModified" : 1 })
>
>
> db.repo_importers.find()
{ "_id" : ObjectId("58261de01661f00b3558afe7"), "repo_id" : "zoo", "importer_type_id" : "yum_importer", "config" : { "feed" : "https://repos.fedorapeople.org/pulp/pulp/fixtures/rpm/" }, "_ns" : "repo_importers", "scratchpad" : { "repomd_revision" : 1478850732, "previous_skip_list" : [ ] }, "last_sync" : null }
>
> exit
bye
[root@mgmt13 ~]# pulp-admin rpm repo sync run --repo-id zoo
+----------------------------------------------------------------------+
Synchronizing Repository [zoo]
+----------------------------------------------------------------------+
This command may be exited via ctrl+c without affecting the request.
Downloading metadata...
[|]
... completed
Downloading repository content...
[==================================================] 100%
RPMs: 32/32 items
Delta RPMs: 0/0 items
... completed
Downloading distribution files...
[==================================================] 100%
Distributions: 0/0 items
... completed
Importing errata...
[-]
... completed
Importing package groups/categories...
[-]
... completed
Cleaning duplicate packages...
[-]
... completed
Task Succeeded
Copying files
[-]
... completed
Initializing repo metadata
[-]
... completed
Publishing Distribution files
[-]
... completed
Publishing RPMs
[==================================================] 100%
32 of 32 items
... completed
Publishing Delta RPMs
... skipped
Publishing Errata
[==================================================] 100%
4 of 4 items
... completed
Publishing Comps file
[==================================================] 100%
4 of 4 items
... completed
Publishing Metadata.
[-]
... completed
Closing repo metadata
[-]
... completed
Generating sqlite files
... skipped
Generating HTML files
... skipped
Publishing files to web
[-]
... completed
Writing Listings File
[-]
... completed
Task Succeeded
- Status changed from 6 to CLOSED - CURRENTRELEASE
Also available in: Atom
PDF
.
Create migration 27 to replace the now-defunct migration 25
This includes the migration fix from https://pulp.plan.io/issues/2377#note-4 as well as the changes made in 2.10 to deal with https://pulp.plan.io/issues/2378
fixes #2377 https://pulp.plan.io/issues/2377