Project

Profile

Help

Issue #8115

closed

Satellite issue sync is broken

Added by bmbouter about 3 years ago. Updated over 2 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
CI/CD
Sprint:
Sprint 107
Quarter:

Description

The Redmine Bugzilla sync is failing repeatedly:

https://github.com/pulp/pulp-ci/runs/1722097819?check_suite_focus=true#step:4:536

It's failing due to timeout issues. Here are some ideas:

  1. increase the timeouts somehow
  2. batch the queries to make more, smaller queries
  3. Have the queries "retry" if a timeout occurs

I suspect option (1) or (2) are our best options.

Actions #1

Updated by fao89 about 3 years ago

  • Triaged changed from No to Yes
  • Sprint set to Sprint 88
Actions #2

Updated by fao89 about 3 years ago

  • Triaged changed from Yes to No
  • Sprint deleted (Sprint 88)
Actions #3

Updated by ttereshc about 3 years ago

  • Sprint set to Sprint 88
Actions #4

Updated by fao89 about 3 years ago

  • Triaged changed from No to Yes
Actions #5

Updated by rchan about 3 years ago

  • Sprint changed from Sprint 88 to Sprint 89
Actions #6

Updated by rchan about 3 years ago

  • Sprint changed from Sprint 89 to Sprint 90
Actions #7

Updated by rchan about 3 years ago

  • Sprint changed from Sprint 90 to Sprint 91
Actions #8

Updated by rchan about 3 years ago

  • Sprint changed from Sprint 91 to Sprint 92
Actions #9

Updated by rchan about 3 years ago

  • Sprint changed from Sprint 92 to Sprint 93
Actions #10

Updated by rchan about 3 years ago

  • Sprint changed from Sprint 93 to Sprint 94
Actions #11

Updated by rchan about 3 years ago

  • Sprint changed from Sprint 94 to Sprint 95
Actions #12

Updated by rchan almost 3 years ago

  • Sprint changed from Sprint 95 to Sprint 96
Actions #13

Updated by rchan almost 3 years ago

  • Sprint changed from Sprint 96 to Sprint 97
Actions #14

Updated by rchan almost 3 years ago

  • Sprint changed from Sprint 97 to Sprint 98
Actions #15

Updated by rchan almost 3 years ago

  • Sprint changed from Sprint 98 to Sprint 99
Actions #16

Updated by rchan almost 3 years ago

  • Sprint changed from Sprint 99 to Sprint 100
Actions #17

Updated by rchan almost 3 years ago

  • Sprint changed from Sprint 100 to Sprint 101
Actions #18

Updated by ipanova@redhat.com over 2 years ago

  • Sprint changed from Sprint 101 to Sprint 102
Actions #19

Updated by rchan over 2 years ago

  • Sprint changed from Sprint 102 to Sprint 103
Actions #20

Updated by rchan over 2 years ago

  • Sprint changed from Sprint 103 to Sprint 104
Actions #21

Updated by rchan over 2 years ago

  • Sprint changed from Sprint 104 to Sprint 105
Actions #22

Updated by rchan over 2 years ago

  • Sprint changed from Sprint 105 to Sprint 106

Added by daviddavis over 2 years ago

Revision 5345e87d | View on GitHub

Apply retry fix from redmine loop

fixes #8115

Actions #23

Updated by daviddavis over 2 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to daviddavis
Actions #24

Updated by daviddavis over 2 years ago

  • Status changed from ASSIGNED to POST
Actions #25

Updated by daviddavis over 2 years ago

  • Status changed from POST to ASSIGNED

I fixed the exception when trying to read an issue from redmine.

But now we get this exception from BZ:

Traceback (most recent call last):
  File "./ci/redmine_bugzilla.py", line 371, in <module>
    main()
  File "./ci/redmine_bugzilla.py", line 298, in main
    add_cc_list_to_bugzilla_bug(bug)
  File "./ci/redmine_bugzilla.py", line 77, in add_cc_list_to_bugzilla_bug
    bug.addcc(pulp_cc_username)
  File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/bugzilla/bug.py", line 257, in addcc
    return self.bugzilla.update_bugs(self.bug_id, vals)
  File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/bugzilla/base.py", line 1379, in update_bugs
    return self._backend.bug_update(listify(ids), tmp)
  File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/bugzilla/_backendxmlrpc.py", line 197, in bug_update
    return self._xmlrpc_proxy.Bug.update(data)
  File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/xmlrpc/client.py", line 1112, in __call__
    return self.__send(self.__name, args)
  File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/bugzilla/_backendxmlrpc.py", line 132, in _ServerProxy__request
    ret = ServerProxy._ServerProxy__request(self, methodname, (newparams,))
  File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/xmlrpc/client.py", line 1452, in __request
    verbose=self.__verbose
  File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/bugzilla/_backendxmlrpc.py", line 100, in request
    return self.__request_helper(url, request_body)
  File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/bugzilla/_backendxmlrpc.py", line 45, in __request_helper
    "POST", url, data=request_body)
  File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/bugzilla/_session.py", line 102, in request
    response = self._session.request(*args, **kwargs)
  File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/requests/adapters.py", line 529, in send
    raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='bugzilla.redhat.com', port=443): Read timed out. (read timeout=300.0)
Actions #26

Updated by daviddavis over 2 years ago

  • Status changed from ASSIGNED to MODIFIED
Actions #27

Updated by daviddavis over 2 years ago

  • Status changed from MODIFIED to ASSIGNED
Actions #28

Updated by daviddavis over 2 years ago

I raised the bugzilla request timeout from 300 to 1200 and even 3600 but it still seems to timeout:

https://github.com/pulp/pulp-ci/runs/3837799211?check_suite_focus=true

requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='bugzilla.redhat.com', port=443): Read timed out. (read timeout=3600.0)

What's odd is that the timeout (3600s or one hour) is greater than the job execution time (42 min) so I am not sure how it's hitting that timeout.

Actions #29

Updated by rchan over 2 years ago

  • Sprint changed from Sprint 106 to Sprint 107
Actions #30

Updated by daviddavis over 2 years ago

  • Status changed from ASSIGNED to CLOSED - CURRENTRELEASE

Not sure if the BZ timeouts are totally solved but for now the script has been behaving properly. I fixed several broken links where BZs weren't linking back to redmine (or vice versa).

There are a couple issues left with bad states but ttereshc said she would handle them.

https://github.com/pulp/pulp-ci/runs/3872180194?check_suite_focus=true

Also available in: Atom PDF