Actions
Issue #5221
closed
Tests are failing in Travis: Cannot connect to host repos.fedorapeople.org:443 ssl:None
Status:
CLOSED - CURRENTRELEASE
Priority:
High
Assignee:
-
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Sprint 57
Quarter:
Description
Tests started failing about a week ago. Here's a recent example: https://travis-ci.com/pulp/pulpcore/builds/121878618
Updated by daviddavis over 4 years ago
It seems like serving the fixtures locally in Travis might be the best solution. This would mean either (A) generating the fixtures in Travis or (B) downloading the fixtures. I think (A) might be difficult given that some of the fixtures require rpm dependencies and Travis uses Ubuntu. For (B), we could maybe check the fixtures into the pulpqe/pulp-fixtures repo and have Travis clone the repo.
There's also the question of how to point the tests to fixtures locally instead of fedorapeople.org. I think we can serve them using python (python3 -m http.server) or nginx and then add a fixture url setting to pulp-smash.
Updated by bmbouter over 4 years ago
daviddavis wrote:
It seems like serving the fixtures locally in Travis might be the best solution. This would mean either (A) generating the fixtures in Travis or (B) downloading the fixtures. I think (A) might be difficult given that some of the fixtures require rpm dependencies and Travis uses Ubuntu. For (B), we could maybe check the fixtures into the pulpqe/pulp-fixtures repo and having Travis clone the repo.
There's also the question of how to point the tests to fixtures locally instead of fedorapeople.org. I think we can serve them using python (python3 -m http.server) or nginx and then add a fixture url setting to pulp-smash.
What about going with option B, but have the fixture data strive to be small and live in the repo w/ the test data itself (rather than 1 fixture repo shared by all content types. The reasoning is that the functional tests should be portable and bringing the test data with them would make them even easier to run.
In practice I think we need to use nginx to serve up these fixtures. Some of the tests test concurrency and I don't believe the Python webserver will provide the server-side concurrency we need. Otherwise we'll be limited in various, subtle ways around asserting correctness around concurrency.
We should do something super simple that maybe serves the fixutres on another port. We need to have this working in Travis and pulplift easily when it goes live.
Updated by dkliban@redhat.com over 4 years ago
We should store the fixtures in a git repository. In pulplift and on Traivs we should serve those fixtures using nginx on port 1234.
Updated by daviddavis over 4 years ago
Doing some more investigation, it looks like this isn't fedorapeople's fault and switching off fedorapeople to use local fixtures probably won't solve the problem. It looks like the cause is that Travis is failing to allocate a socket to Pulp while it is trying to open a connection to fedorapeople ("Cannot assign requested address"). We found a related issue in aiohttp although the solution it proposes, we already follow:
Updated by kersom over 4 years ago
Traceback of the one test failures:
('Task report /pulp/api/v3/tasks/7648343b-6356-4421-9cb7-7039315934a1/ '
"contains a error: {'code': None, 'description': 'Cannot connect to host "
"repos.fedorapeople.org:443 ssl:None [Cannot assign requested address]', "
"'traceback': ' File "
'"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/rq/worker.py", '
'line 822, in perform_job\\n rv = job.perform()\\n File '
'"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/rq/job.py", '
'line 605, in perform\\n self._result = self._execute()\\n File '
'"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/rq/job.py", '
'line 611, in _execute\\n return self.func(*self.args, **self.kwargs)\\n '
'File '
'"/home/travis/build/pulp/pulp_file/pulp_file/app/tasks/synchronizing.py", '
'line 45, in synchronize\\n dv.create()\\n File '
'"/home/travis/build/pulp/pulpcore-plugin/pulpcore/plugin/stages/declarative_version.py", '
'line 169, in create\\n loop.run_until_complete(pipeline)\\n File '
'"/opt/python/3.7.1/lib/python3.7/asyncio/base_events.py", line 573, in '
'run_until_complete\\n return future.result()\\n File '
'"/home/travis/build/pulp/pulpcore-plugin/pulpcore/plugin/stages/api.py", '
'line 209, in create_pipeline\\n await asyncio.gather(*futures)\\n File '
'"/home/travis/build/pulp/pulpcore-plugin/pulpcore/plugin/stages/api.py", '
'line 43, in __call__\\n await self.run()\\n File '
'"/home/travis/build/pulp/pulpcore-plugin/pulpcore/plugin/stages/artifact_stages.py", '
'line 132, in run\\n pb.done += task.result() # download_count\\n File '
'"/home/travis/build/pulp/pulpcore-plugin/pulpcore/plugin/stages/artifact_stages.py", '
'line 155, in _handle_content_unit\\n await '
'asyncio.gather(*downloaders_for_content)\\n File '
'"/home/travis/build/pulp/pulpcore-plugin/pulpcore/plugin/stages/models.py", '
'line 78, in download\\n download_result = await '
'downloader.run(extra_data=self.extra_data)\\n File '
'"/home/travis/build/pulp/pulpcore-plugin/pulpcore/plugin/download/base.py", '
'line 212, in run\\n return await self._run(extra_data=extra_data)\\n '
'File '
'"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/backoff/_async.py", '
'line 131, in retry\\n ret = await target(*args, **kwargs)\\n File '
'"/home/travis/build/pulp/pulpcore-plugin/pulpcore/plugin/download/http.py", '
'line 182, in _run\\n async with self.session.get(self.url, '
'proxy=self.proxy) as response:\\n File '
'"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/aiohttp/client.py", '
'line 1005, in __aenter__\\n self._resp = await self._coro\\n File '
'"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/aiohttp/client.py", '
'line 476, in _request\\n timeout=real_timeout\\n File '
'"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/aiohttp/connector.py", '
'line 522, in connect\\n proto = await self._create_connection(req, '
'traces, timeout)\\n File '
'"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/aiohttp/connector.py", '
'line 854, in _create_connection\\n req, traces, timeout)\\n File '
'"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/aiohttp/connector.py", '
'line 992, in _create_direct_connection\\n raise last_exc\\n File '
'"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/aiohttp/connector.py", '
'line 974, in _create_direct_connection\\n req=req, '
'client_error=client_error)\\n File '
'"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/aiohttp/connector.py", '
'line 931, in _wrap_create_connection\\n raise '
"client_error(req.connection_key, exc) from exc\\n'}\n"
"Full task report: {'_href': "
"'/pulp/api/v3/tasks/7648343b-6356-4421-9cb7-7039315934a1/', '_created': "
"'2019-08-05T13:19:30.829697Z', 'state': 'failed', 'name': "
"'pulp_file.app.tasks.synchronizing.synchronize', 'started_at': "
"'2019-08-05T13:19:30.969822Z', 'finished_at': '2019-08-05T13:21:42.290066Z', "
"'non_fatal_errors': [], 'error': {'code': None, 'description': 'Cannot "
'connect to host repos.fedorapeople.org:443 ssl:None [Cannot assign requested '
"address]', 'traceback': ' File "
'"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/rq/worker.py", '
'line 822, in perform_job\\n rv = job.perform()\\n File '
'"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/rq/job.py", '
'line 605, in perform\\n self._result = self._execute()\\n File '
'"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/rq/job.py", '
'line 611, in _execute\\n return self.func(*self.args, **self.kwargs)\\n '
'File '
'"/home/travis/build/pulp/pulp_file/pulp_file/app/tasks/synchronizing.py", '
'line 45, in synchronize\\n dv.create()\\n File '
'"/home/travis/build/pulp/pulpcore-plugin/pulpcore/plugin/stages/declarative_version.py", '
'line 169, in create\\n loop.run_until_complete(pipeline)\\n File '
'"/opt/python/3.7.1/lib/python3.7/asyncio/base_events.py", line 573, in '
'run_until_complete\\n return future.result()\\n File '
'"/home/travis/build/pulp/pulpcore-plugin/pulpcore/plugin/stages/api.py", '
'line 209, in create_pipeline\\n await asyncio.gather(*futures)\\n File '
'"/home/travis/build/pulp/pulpcore-plugin/pulpcore/plugin/stages/api.py", '
'line 43, in __call__\\n await self.run()\\n File '
'"/home/travis/build/pulp/pulpcore-plugin/pulpcore/plugin/stages/artifact_stages.py", '
'line 132, in run\\n pb.done += task.result() # download_count\\n File '
'"/home/travis/build/pulp/pulpcore-plugin/pulpcore/plugin/stages/artifact_stages.py", '
'line 155, in _handle_content_unit\\n await '
'asyncio.gather(*downloaders_for_content)\\n File '
'"/home/travis/build/pulp/pulpcore-plugin/pulpcore/plugin/stages/models.py", '
'line 78, in download\\n download_result = await '
'downloader.run(extra_data=self.extra_data)\\n File '
'"/home/travis/build/pulp/pulpcore-plugin/pulpcore/plugin/download/base.py", '
'line 212, in run\\n return await self._run(extra_data=extra_data)\\n '
'File '
'"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/backoff/_async.py", '
'line 131, in retry\\n ret = await target(*args, **kwargs)\\n File '
'"/home/travis/build/pulp/pulpcore-plugin/pulpcore/plugin/download/http.py", '
'line 182, in _run\\n async with self.session.get(self.url, '
'proxy=self.proxy) as response:\\n File '
'"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/aiohttp/client.py", '
'line 1005, in __aenter__\\n self._resp = await self._coro\\n File '
'"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/aiohttp/client.py", '
'line 476, in _request\\n timeout=real_timeout\\n File '
'"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/aiohttp/connector.py", '
'line 522, in connect\\n proto = await self._create_connection(req, '
'traces, timeout)\\n File '
'"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/aiohttp/connector.py", '
'line 854, in _create_connection\\n req, traces, timeout)\\n File '
'"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/aiohttp/connector.py", '
'line 992, in _create_direct_connection\\n raise last_exc\\n File '
'"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/aiohttp/connector.py", '
'line 974, in _create_direct_connection\\n req=req, '
'client_error=client_error)\\n File '
'"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/aiohttp/connector.py", '
'line 931, in _wrap_create_connection\\n raise '
"client_error(req.connection_key, exc) from exc\\n'}, 'worker': "
"'/pulp/api/v3/workers/2caefc6d-cda3-49b3-8d1d-aa4d9fe4161d/', 'parent': "
"None, 'spawned_tasks': [], 'progress_reports': [{'message': 'Downloading "
"Metadata', 'state': 'completed', 'total': 1, 'done': 1, 'suffix': None}, "
"{'message': 'Parsing Metadata Lines', 'state': 'completed', 'total': 250, "
"'done': 250, 'suffix': None}, {'message': 'Downloading Artifacts', 'state': "
"'failed', 'total': 237, 'done': 237, 'suffix': None}, {'message': "
"'Associating Content', 'state': 'canceled', 'total': 200, 'done': 200, "
"'suffix': None}], 'created_resources': []}",
{'_created': '2019-08-05T13:19:30.829697Z',
'_href': '/pulp/api/v3/tasks/7648343b-6356-4421-9cb7-7039315934a1/',
'created_resources': [],
'error': {'code': None,
'description': 'Cannot connect to host repos.fedorapeople.org:443 '
'ssl:None [Cannot assign requested address]',
'traceback': ' File '
'"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/rq/worker.py", '
'line 822, in perform_job\n'
' rv = job.perform()\n'
' File '
'"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/rq/job.py", '
'line 605, in perform\n'
' self._result = self._execute()\n'
' File '
'"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/rq/job.py", '
'line 611, in _execute\n'
' return self.func(*self.args, **self.kwargs)\n'
' File '
'"/home/travis/build/pulp/pulp_file/pulp_file/app/tasks/synchronizing.py", '
'line 45, in synchronize\n'
' dv.create()\n'
' File '
'"/home/travis/build/pulp/pulpcore-plugin/pulpcore/plugin/stages/declarative_version.py", '
'line 169, in create\n'
' loop.run_until_complete(pipeline)\n'
' File '
'"/opt/python/3.7.1/lib/python3.7/asyncio/base_events.py", '
'line 573, in run_until_complete\n'
' return future.result()\n'
' File '
'"/home/travis/build/pulp/pulpcore-plugin/pulpcore/plugin/stages/api.py", '
'line 209, in create_pipeline\n'
' await asyncio.gather(*futures)\n'
' File '
'"/home/travis/build/pulp/pulpcore-plugin/pulpcore/plugin/stages/api.py", '
'line 43, in __call__\n'
' await self.run()\n'
' File '
'"/home/travis/build/pulp/pulpcore-plugin/pulpcore/plugin/stages/artifact_stages.py", '
'line 132, in run\n'
' pb.done += task.result() # download_count\n'
' File '
'"/home/travis/build/pulp/pulpcore-plugin/pulpcore/plugin/stages/artifact_stages.py", '
'line 155, in _handle_content_unit\n'
' await asyncio.gather(*downloaders_for_content)\n'
' File '
'"/home/travis/build/pulp/pulpcore-plugin/pulpcore/plugin/stages/models.py", '
'line 78, in download\n'
' download_result = await '
'downloader.run(extra_data=self.extra_data)\n'
' File '
'"/home/travis/build/pulp/pulpcore-plugin/pulpcore/plugin/download/base.py", '
'line 212, in run\n'
' return await self._run(extra_data=extra_data)\n'
' File '
'"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/backoff/_async.py", '
'line 131, in retry\n'
' ret = await target(*args, **kwargs)\n'
' File '
'"/home/travis/build/pulp/pulpcore-plugin/pulpcore/plugin/download/http.py", '
'line 182, in _run\n'
' async with self.session.get(self.url, '
'proxy=self.proxy) as response:\n'
' File '
'"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/aiohttp/client.py", '
'line 1005, in __aenter__\n'
' self._resp = await self._coro\n'
' File '
'"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/aiohttp/client.py", '
'line 476, in _request\n'
' timeout=real_timeout\n'
' File '
'"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/aiohttp/connector.py", '
'line 522, in connect\n'
' proto = await self._create_connection(req, '
'traces, timeout)\n'
' File '
'"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/aiohttp/connector.py", '
'line 854, in _create_connection\n'
' req, traces, timeout)\n'
' File '
'"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/aiohttp/connector.py", '
'line 992, in _create_direct_connection\n'
' raise last_exc\n'
' File '
'"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/aiohttp/connector.py", '
'line 974, in _create_direct_connection\n'
' req=req, client_error=client_error)\n'
' File '
'"/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/aiohttp/connector.py", '
'line 931, in _wrap_create_connection\n'
' raise client_error(req.connection_key, exc) from '
'exc\n'},
'finished_at': '2019-08-05T13:21:42.290066Z',
'name': 'pulp_file.app.tasks.synchronizing.synchronize',
'non_fatal_errors': [],
'parent': None,
'progress_reports': [{'done': 1,
'message': 'Downloading Metadata',
'state': 'completed',
'suffix': None,
'total': 1},
{'done': 250,
'message': 'Parsing Metadata Lines',
'state': 'completed',
'suffix': None,
'total': 250},
{'done': 237,
'message': 'Downloading Artifacts',
'state': 'failed',
'suffix': None,
'total': 237},
{'done': 200,
'message': 'Associating Content',
'state': 'canceled',
'suffix': None,
'total': 200}],
'spawned_tasks': [],
'started_at': '2019-08-05T13:19:30.969822Z',
'state': 'failed',
'worker': '/pulp/api/v3/workers/2caefc6d-cda3-49b3-8d1d-aa4d9fe4161d/'})
Fixtures used in the test: https://repos.fedorapeople.org/pulp/pulp/fixtures/file-many/
Updated by amacdona@redhat.com over 4 years ago
- Triaged changed from No to Yes
- Sprint set to Sprint 57
Updated by daviddavis over 4 years ago
- Status changed from NEW to CLOSED - CURRENTRELEASE
Turns out this is a problem in Travis that has been resolved:
Actions
.