Project

Profile

Help

Actions
Send by e-mail Copy link

Issue #7239

closed

ValueError: time data '' does not match format '%Y-%m-%dT%H:%M:%SZ' for any Pulp 3 task

Added by iballou almost 3 years ago. Updated almost 3 years ago.

Status:
CLOSED - WORKSFORME
Priority:
Normal
Assignee:
-
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
OS:
Triaged:
No
Groomed:
No
Sprint Candidate:
No
Tags:
Katello
Sprint:
Quarter:

Description

When I run my Pulp 2 to 3 migration, I see the stack trace below. I've tried flushing the rq queue and resetting the pulpcore DB, but I still see the error. The result is that my migration task is stuck "waiting" forever and doesn't receive a worker.

Related versions:

pulp-2to3-migration (0.2.0b7.dev0) pulp-2to3-migration-client (0.2.0b5) pulp-rpm (3.5.0.dev0) pulpcore (3.4.0)

Stack trace:

    Jul 28 22:24:35 centos7-katello-devel-stable.example.com pulpcore-resource-manager[20338]: pulp: rq.worker:ERROR: Worker rq:worker:resource-manager: found an unhandled exception, quitting...
    Jul 28 22:24:35 centos7-katello-devel-stable.example.com pulpcore-resource-manager[20338]: Traceback (most recent call last):
    Jul 28 22:24:35 centos7-katello-devel-stable.example.com pulpcore-resource-manager[20338]: File "/usr/lib/python3.6/site-packages/rq/utils.py", line 169, in utcparse
    Jul 28 22:24:35 centos7-katello-devel-stable.example.com pulpcore-resource-manager[20338]: return datetime.datetime.strptime(string, _TIMESTAMP_FORMAT)
    Jul 28 22:24:35 centos7-katello-devel-stable.example.com pulpcore-resource-manager[20338]: File "/usr/lib64/python3.6/_strptime.py", line 565, in _strptime_datetime
    Jul 28 22:24:35 centos7-katello-devel-stable.example.com pulpcore-resource-manager[20338]: tt, fraction = _strptime(data_string, format)
    Jul 28 22:24:35 centos7-katello-devel-stable.example.com pulpcore-resource-manager[20338]: File "/usr/lib64/python3.6/_strptime.py", line 362, in _strptime
    Jul 28 22:24:35 centos7-katello-devel-stable.example.com pulpcore-resource-manager[20338]: (data_string, format))
    Jul 28 22:24:35 centos7-katello-devel-stable.example.com pulpcore-resource-manager[20338]: ValueError: time data '' does not match format '%Y-%m-%dT%H:%M:%S.%fZ'
    Jul 28 22:24:35 centos7-katello-devel-stable.example.com pulpcore-resource-manager[20338]: During handling of the above exception, another exception occurred:
    Jul 28 22:24:35 centos7-katello-devel-stable.example.com pulpcore-resource-manager[20338]: Traceback (most recent call last):
    Jul 28 22:24:35 centos7-katello-devel-stable.example.com pulpcore-resource-manager[20338]: File "/usr/lib/python3.6/site-packages/rq/worker.py", line 510, in work
    Jul 28 22:24:35 centos7-katello-devel-stable.example.com pulpcore-resource-manager[20338]: result = self.dequeue_job_and_maintain_ttl(timeout)
    Jul 28 22:24:35 centos7-katello-devel-stable.example.com pulpcore-resource-manager[20338]: File "/usr/lib/python3.6/site-packages/rq/worker.py", line 578, in dequeue_job_and_maintain_ttl
    Jul 28 22:24:35 centos7-katello-devel-stable.example.com pulpcore-resource-manager[20338]: job_class=self.job_class)
    Jul 28 22:24:35 centos7-katello-devel-stable.example.com pulpcore-resource-manager[20338]: File "/usr/lib/python3.6/site-packages/rq/queue.py", line 550, in dequeue_any
    Jul 28 22:24:35 centos7-katello-devel-stable.example.com pulpcore-resource-manager[20338]: job = job_class.fetch(job_id, connection=connection)
    Jul 28 22:24:35 centos7-katello-devel-stable.example.com pulpcore-resource-manager[20338]: File "/usr/lib/python3.6/site-packages/rq/job.py", line 303, in fetch
    Jul 28 22:24:35 centos7-katello-devel-stable.example.com pulpcore-resource-manager[20338]: job.refresh()
    Jul 28 22:24:35 centos7-katello-devel-stable.example.com pulpcore-resource-manager[20338]: File "/usr/lib/python3.6/site-packages/rq/job.py", line 515, in refresh
    Jul 28 22:24:35 centos7-katello-devel-stable.example.com pulpcore-resource-manager[20338]: self.restore(data)
    Jul 28 22:24:35 centos7-katello-devel-stable.example.com pulpcore-resource-manager[20338]: File "/usr/lib/python3.6/site-packages/rq/job.py", line 478, in restore
    Jul 28 22:24:35 centos7-katello-devel-stable.example.com pulpcore-resource-manager[20338]: self.started_at = str_to_date(obj.get('started_at'))
    Jul 28 22:24:35 centos7-katello-devel-stable.example.com pulpcore-resource-manager[20338]: File "/usr/lib/python3.6/site-packages/rq/utils.py", line 256, in str_to_date
    Jul 28 22:24:35 centos7-katello-devel-stable.example.com pulpcore-resource-manager[20338]: return utcparse(as_text(date_str))
    Jul 28 22:24:35 centos7-katello-devel-stable.example.com pulpcore-resource-manager[20338]: File "/usr/lib/python3.6/site-packages/rq/utils.py", line 172, in utcparse
    Jul 28 22:24:35 centos7-katello-devel-stable.example.com pulpcore-resource-manager[20338]: return datetime.datetime.strptime(string, '%Y-%m-%dT%H:%M:%SZ')
    Jul 28 22:24:35 centos7-katello-devel-stable.example.com pulpcore-resource-manager[20338]: File "/usr/lib64/python3.6/_strptime.py", line 565, in _strptime_datetime
    Jul 28 22:24:35 centos7-katello-devel-stable.example.com pulpcore-resource-manager[20338]: tt, fraction = _strptime(data_string, format)
    Jul 28 22:24:35 centos7-katello-devel-stable.example.com pulpcore-resource-manager[20338]: File "/usr/lib64/python3.6/_strptime.py", line 362, in _strptime
    Jul 28 22:24:35 centos7-katello-devel-stable.example.com pulpcore-resource-manager[20338]: (data_string, format))
    Jul 28 22:24:35 centos7-katello-devel-stable.example.com pulpcore-resource-manager[20338]: ValueError![](: time data '' does not match format '%Y-%m-%dT%H:%M:%SZ')
Actions
Copy link
 #1

Updated by iballou almost 3 years ago

I confirmed that this error is happening for any Pulp 3 task. I tried to sync a repo and got the same result.

Actions
Copy link
 #2

Updated by iballou almost 3 years ago

I've fixed it by upgrading to python3-rq-1.4.2-1.el7.noarch

Actions
Copy link
 #3

Updated by ttereshc almost 3 years ago

  • Project changed from Migration Plugin to Pulp
  • Subject changed from ValueError when running Pulp 2 to 3 migration for RPM content to ValueError: time data '' does not match format '%Y-%m-%dT%H:%M:%SZ' for any Pulp 3 task
  • Description updated (diff)
Actions
Copy link
 #4

Updated by iballou almost 3 years ago

We realized this could have been caused by how Pulp was installed on my system. With Katello, Pulp 3 was installed via RPMs. To upgrade the migration plugin, I did a system-wide pip3 install of the migration plugin. I saw an rq version in /usr/local/lib/python3.6/site-packages/, so the pip3 install might have included rq with the migration plugin. This could then have caused multiple versions of rq being used across different processes.

Just a theory, but something that could be tested.

Actions
Copy link
 #5

Updated by fao89 almost 3 years ago

  • Status changed from NEW to CLOSED - WORKSFORME
Actions
Send by e-mail Copy link

Also available in: Atom PDF