Project

Profile

Help

Issue #8115

Satellite issue sync is broken

Added by bmbouter 11 months ago. Updated about 2 months ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
CI/CD
Sprint:
Sprint 107
Quarter:

Description

The Redmine Bugzilla sync is failing repeatedly:

https://github.com/pulp/pulp-ci/runs/1722097819?check_suite_focus=true#step:4:536

It's failing due to timeout issues. Here are some ideas:

  1. increase the timeouts somehow
  2. batch the queries to make more, smaller queries
  3. Have the queries "retry" if a timeout occurs

I suspect option (1) or (2) are our best options.

Associated revisions

Revision 5345e87d View on GitHub
Added by daviddavis about 2 months ago

Apply retry fix from redmine loop

fixes #8115

History

#1 Updated by fao89 11 months ago

  • Triaged changed from No to Yes
  • Sprint set to Sprint 88

#2 Updated by fao89 11 months ago

  • Triaged changed from Yes to No
  • Sprint deleted (Sprint 88)

#3 Updated by ttereshc 11 months ago

  • Sprint set to Sprint 88

#4 Updated by fao89 11 months ago

  • Triaged changed from No to Yes

#5 Updated by rchan 11 months ago

  • Sprint changed from Sprint 88 to Sprint 89

#6 Updated by rchan 10 months ago

  • Sprint changed from Sprint 89 to Sprint 90

#7 Updated by rchan 10 months ago

  • Sprint changed from Sprint 90 to Sprint 91

#8 Updated by rchan 9 months ago

  • Sprint changed from Sprint 91 to Sprint 92

#9 Updated by rchan 9 months ago

  • Sprint changed from Sprint 92 to Sprint 93

#10 Updated by rchan 8 months ago

  • Sprint changed from Sprint 93 to Sprint 94

#11 Updated by rchan 8 months ago

  • Sprint changed from Sprint 94 to Sprint 95

#12 Updated by rchan 7 months ago

  • Sprint changed from Sprint 95 to Sprint 96

#13 Updated by rchan 7 months ago

  • Sprint changed from Sprint 96 to Sprint 97

#14 Updated by rchan 6 months ago

  • Sprint changed from Sprint 97 to Sprint 98

#15 Updated by rchan 6 months ago

  • Sprint changed from Sprint 98 to Sprint 99

#16 Updated by rchan 5 months ago

  • Sprint changed from Sprint 99 to Sprint 100

#17 Updated by rchan 5 months ago

  • Sprint changed from Sprint 100 to Sprint 101

#18 Updated by ipanova@redhat.com 4 months ago

  • Sprint changed from Sprint 101 to Sprint 102

#19 Updated by rchan 4 months ago

  • Sprint changed from Sprint 102 to Sprint 103

#20 Updated by rchan 3 months ago

  • Sprint changed from Sprint 103 to Sprint 104

#21 Updated by rchan 3 months ago

  • Sprint changed from Sprint 104 to Sprint 105

#22 Updated by rchan 2 months ago

  • Sprint changed from Sprint 105 to Sprint 106

#23 Updated by daviddavis about 2 months ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to daviddavis

#24 Updated by daviddavis about 2 months ago

  • Status changed from ASSIGNED to POST

#25 Updated by daviddavis about 2 months ago

  • Status changed from POST to ASSIGNED

I fixed the exception when trying to read an issue from redmine.

But now we get this exception from BZ:

Traceback (most recent call last):
  File "./ci/redmine_bugzilla.py", line 371, in <module>
    main()
  File "./ci/redmine_bugzilla.py", line 298, in main
    add_cc_list_to_bugzilla_bug(bug)
  File "./ci/redmine_bugzilla.py", line 77, in add_cc_list_to_bugzilla_bug
    bug.addcc(pulp_cc_username)
  File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/bugzilla/bug.py", line 257, in addcc
    return self.bugzilla.update_bugs(self.bug_id, vals)
  File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/bugzilla/base.py", line 1379, in update_bugs
    return self._backend.bug_update(listify(ids), tmp)
  File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/bugzilla/_backendxmlrpc.py", line 197, in bug_update
    return self._xmlrpc_proxy.Bug.update(data)
  File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/xmlrpc/client.py", line 1112, in __call__
    return self.__send(self.__name, args)
  File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/bugzilla/_backendxmlrpc.py", line 132, in _ServerProxy__request
    ret = ServerProxy._ServerProxy__request(self, methodname, (newparams,))
  File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/xmlrpc/client.py", line 1452, in __request
    verbose=self.__verbose
  File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/bugzilla/_backendxmlrpc.py", line 100, in request
    return self.__request_helper(url, request_body)
  File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/bugzilla/_backendxmlrpc.py", line 45, in __request_helper
    "POST", url, data=request_body)
  File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/bugzilla/_session.py", line 102, in request
    response = self._session.request(*args, **kwargs)
  File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/requests/adapters.py", line 529, in send
    raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='bugzilla.redhat.com', port=443): Read timed out. (read timeout=300.0)

#26 Updated by daviddavis about 2 months ago

  • Status changed from ASSIGNED to MODIFIED

#27 Updated by daviddavis about 2 months ago

  • Status changed from MODIFIED to ASSIGNED

#28 Updated by daviddavis about 2 months ago

I raised the bugzilla request timeout from 300 to 1200 and even 3600 but it still seems to timeout:

https://github.com/pulp/pulp-ci/runs/3837799211?check_suite_focus=true

requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='bugzilla.redhat.com', port=443): Read timed out. (read timeout=3600.0)

What's odd is that the timeout (3600s or one hour) is greater than the job execution time (42 min) so I am not sure how it's hitting that timeout.

#29 Updated by rchan about 2 months ago

  • Sprint changed from Sprint 106 to Sprint 107

#30 Updated by daviddavis about 2 months ago

  • Status changed from ASSIGNED to CLOSED - CURRENTRELEASE

Not sure if the BZ timeouts are totally solved but for now the script has been behaving properly. I fixed several broken links where BZs weren't linking back to redmine (or vice versa).

There are a couple issues left with bad states but ttereshc said she would handle them.

https://github.com/pulp/pulp-ci/runs/3872180194?check_suite_focus=true

Please register to edit this issue

Also available in: Atom PDF