Project

Profile

Help

Downloading » History » Sprint/Milestone 30

jortel@redhat.com, 09/06/2017 06:44 PM

1 1 jortel@redhat.com
# Downloading
2
3 24 jortel@redhat.com
In pulp3, there are two competing technologies and designs being considered. For the purposes of the discussion we'll name them **Jupiter** and **Saturn**. The *Jupiter* solution is based on *concurrent.futures* and the Saturn solution is based on *asyncio*. In addition to the underlying technology difference, the solutions meet the requirements in different ways. The *Jupiter* solution includes more classes, provides more abstraction and supports customization through delegation and object composition. The *Saturn* solution meets the requirements with the fewest classes possible and minimum abstraction. Customization is supported though subclassing.
4 3 jortel@redhat.com
5 5 jortel@redhat.com
The three actors for our use cases is the *Importer*, *Streamer* and Plugin Writer. The *ChangeSet* shares a subset of the Streamer requirements but not included in this discussion.
6 3 jortel@redhat.com
7 21 jortel@redhat.com
## Design Goals & Constraints
8
9 22 jortel@redhat.com
The requirements define the minimum criteria to be satisfied by both solutions. The design constrains and goals define <span class="underline">how</span> the requirements are met.
10
11 21 jortel@redhat.com
**juniper**:
12
13
  - constraints:
14
15
>   - object oriented
16
>   - support semantic versioning
17
18
  - goals
19
20
>   - encapsulate underlying technologies
21
>   - consistent interface across downloads. standard arguments, return values and raised exceptions.
22
>   - delegation pattern for common customization:
23
>
24
>>   - handling of downloaded bits to *Writers*
25
>>   - validation delegated to *Validations*
26
>>   - optional digest and size calculation delegated to *DownloadMonitor*
27
>>   - error handling delegated to *Event* handlers.
28
>
29
>   - external participation of download process through defined event registration and callback.
30
>   - delegate concurrency to standard lib (*concurrent.futures*).
31
>   - delegate protocol implementation to client libs.
32
33
**saturn**:
34
35
  - constraints:
36
37
>   - object oriented
38
>   - support semantic versioning
39
40
  - goals
41
42
>   - direct exposure of client libs.
43
>   - minimum encapsulation of underlying technologies.
44
>   - minimum \# of first class concepts (classes) and abstractions.
45
>   - minimum \# lines of code to maintain.
46
>   - delegate concurrency to standard lib (*asyncio*).
47
>   - delegate protocol implementation to client libs.
48
49 1 jortel@redhat.com
## Use Cases
50
51 2 jortel@redhat.com
### Importer
52 1 jortel@redhat.com
53 29 jortel@redhat.com
-----
54 1 jortel@redhat.com
55 29 jortel@redhat.com
#### As an importer, I need to download single files.
56
57 9 jortel@redhat.com
**jupiter**:
58 5 jortel@redhat.com
59 15 jortel@redhat.com
~~~python
60 6 jortel@redhat.com
download = HttpDownload(
61
    url=url,
62
    writer=FileWriter(path),
63
    timeout=Timeout(connect=10, read=15),
64
    user=User(name='elmer', password='...'),
65
    ssl=SSL(ca_certificate='path-to-certificate',
66
            client_certificate='path-to-certificate',
67
            client_key='path-to-key',
68
            validation=True),
69
    proxy_url='http://user:password@gateway.org')
70 5 jortel@redhat.com
71
try:
72
    download()
73
except DownloadError:
74
    # An error occurred.
75
else:
76
   # Go read the downloaded file \o/
77
~~~
78
79 9 jortel@redhat.com
**saturn**:
80 1 jortel@redhat.com
81 15 jortel@redhat.com
~~~python
82 6 jortel@redhat.com
ssl_context = aiohttpSSLContext()
83
ssl_context.load_cert_chain('path-to-CA_certificate')
84
ssl_context.load_cert_chain('path-to-CLIENT_certificate')
85
ssl_context.load_cert_chain('path-to-CLIENT_key')
86
87
connector=aiohttp.TCPConnector(verify_ssl=True, ssl_context=ssl_context)
88
89
session = aiohttp.ClientSession(
90
    connector=connector,
91
    read_timeout=15,
92
    auth=aiohttp.BasicAuth('elmer', password='...', encoding='utf-8'))
93
94
downloader_obj = HttpDownloader(
95
    session,
96
    url,
97
    proxy='http://gateway.org',
98
    proxy_auth=aiohttp.BasicAuth('elmer', password='...', encoding='utf-8')
99
100 5 jortel@redhat.com
downloader_coroutine = downloader_obj.run()
101
loop = asyncio._get_running_loop()
102
done, not_done = loop.run_until_complete(asyncio.wait([downloader_coroutine]))
103
for task in done:
104
    try:
105 1 jortel@redhat.com
        result = task.result()  # This is a DownloadResult
106
    except aiohttp.ClientError:
107
        # An error occurred.
108 5 jortel@redhat.com
~~~
109
110 6 jortel@redhat.com
question: How can the connect timeout be set in aiohttp?
111 1 jortel@redhat.com
112
-----
113
114 29 jortel@redhat.com
#### As an importer, I can leverage all settings supported by underlying protocol specific client lib.
115 9 jortel@redhat.com
116
**jupiter**:
117
118 1 jortel@redhat.com
Commonly used settings supported by abstraction. Additional settings could be supported by subclassing.
119 9 jortel@redhat.com
120 15 jortel@redhat.com
~~~python
121
122 9 jortel@redhat.com
class SpecialDownload(HttpDownload):
123
124
    def _settings(self):
125
        settings = super()._settings()
126
        settings['special'] = <special value>
127
        return settings
128
~~~
129
130
**saturn**:
131
132
The underlying client lib arguments directly exposed.
133 1 jortel@redhat.com
134 9 jortel@redhat.com
-----
135 1 jortel@redhat.com
136 29 jortel@redhat.com
#### As an importer, I can create an Artifact with a downloaded file using the size and digests calculated during the download.
137 10 jortel@redhat.com
138 1 jortel@redhat.com
**jupiter**:
139
140 10 jortel@redhat.com
Using the optional *DownloadMonitor* to collect statistics such as size and calculate digests.
141
142 15 jortel@redhat.com
~~~python
143
144 10 jortel@redhat.com
download = HttpDownload(..)
145 14 jortel@redhat.com
monitor = DownloadMonitor(download)
146 10 jortel@redhat.com
...  # perform download.
147 14 jortel@redhat.com
artifact = Artifact(**monitor.facts())
148 10 jortel@redhat.com
artifact.save()
149
~~~
150 1 jortel@redhat.com
151
**saturn**:
152 10 jortel@redhat.com
153
The *size* and all *digests* always calculated.
154
155 15 jortel@redhat.com
~~~python
156
157 10 jortel@redhat.com
downloader_obj = HttpDownloader(...)
158
...  # perform download.
159 28 jortel@redhat.com
result = task.result()
160
artifact = Artifact(**result.artifact_attributes)
161 10 jortel@redhat.com
artifact.save()
162 1 jortel@redhat.com
~~~
163 10 jortel@redhat.com
164 11 jortel@redhat.com
-----
165
166 29 jortel@redhat.com
#### As an importer, I need to download files concurrently.
167 1 jortel@redhat.com
168 11 jortel@redhat.com
**jupiter**:
169
170
Using the *Batch* to run the downloads concurrently. Only 3 downloads in memory at once.
171
172 15 jortel@redhat.com
~~~python
173
174 11 jortel@redhat.com
downloads = (HttpDownload(...) for _ in range(10))
175
176
with Batch(downloads, backlog=3) as batch:
177
    for plan in batch():
178
        try:
179
            plan.result()
180
        except DownloadError:
181
            # An error occurred.
182
        else:
183 1 jortel@redhat.com
            # Use the downloaded file \o/
184
~~~
185 11 jortel@redhat.com
186
**saturn**:
187
188
Using the asyncio run loop. This example does not restrict the number of downloads in memory at once.
189 12 jortel@redhat.com
190 15 jortel@redhat.com
~~~python
191
192 16 jortel@redhat.com
downloaders = (HttpDownloader...) for _ in range(10))
193 11 jortel@redhat.com
194
loop = asyncio._get_running_loop()
195 16 jortel@redhat.com
done, not_done = loop.run_until_complete(asyncio.wait([d.run() for d in downloaders]))
196 11 jortel@redhat.com
for task in done:
197
    try:
198
        result = task.result()  # This is a DownloadResult
199
    except aiohttp.ClientError:
200 1 jortel@redhat.com
        # An error occurred.
201 11 jortel@redhat.com
~~~
202
203 1 jortel@redhat.com
-----
204
205 29 jortel@redhat.com
#### As an importer, I want to validate downloaded files.
206 16 jortel@redhat.com
207 1 jortel@redhat.com
**jupiter**:
208
209 28 jortel@redhat.com
Supported by adding provided or custom validations to the download. A validation error raises *ValidationError* which *IsA* *DownloadError*.
210 17 jortel@redhat.com
211 16 jortel@redhat.com
~~~python
212
213
download = HttpDownload(...)
214
download.append(DigestValidation('sha256', '0x1234'))
215
216
try:
217
    download()
218
except DownloadError:
219
    # An error occurred.
220
~~~
221
222
**saturn**:
223
224 17 jortel@redhat.com
Supported by passing the *expected_digests* dictionary and catching *DigestValidationError*.
225 16 jortel@redhat.com
226
~~~python
227
228
downloader_obj = HttpDownloader(..., expected_digests={'sha256': '0x1234'})
229
230
downloader_coroutine = downloader_obj.run()
231
loop = asyncio._get_running_loop()
232
done, not_done = loop.run_until_complete(asyncio.wait([downloader_coroutine]))
233
for task in done:
234
    try:
235
        result = task.result()  # This is a DownloadResult
236 1 jortel@redhat.com
    except (aiohttp.ClientError, DigestValidationError):
237 16 jortel@redhat.com
        # An error occurred.
238
~~~
239
240
-----
241
242 29 jortel@redhat.com
#### As an importer, I am not required to keep all content (units) and artifacts in memory to support concurrent downloading.
243 18 jortel@redhat.com
244
**jupiter**:
245
246 27 jortel@redhat.com
Using the *Batch* to run the downloads concurrently. The input to the batch can be a *generator* and the number of downloads in  
247
memory is limited by the *backlog* argument.
248
249 18 jortel@redhat.com
~~~python
250 27 jortel@redhat.com
251
downloads = (HttpDownload(...) for _ in range(10))
252
253
with Batch(downloads, backlog=3) as batch:
254
    for plan in batch():
255
        try:
256
            plan.result()
257
        except DownloadError:
258
            # An error occurred.
259
        else:
260
            # Use the downloaded file \o/
261 18 jortel@redhat.com
~~~
262
263 19 jortel@redhat.com
**saturn**:
264 27 jortel@redhat.com
265 1 jortel@redhat.com
@bmbouters: please describe and provide examples.
266 27 jortel@redhat.com
267 19 jortel@redhat.com
~~~python
268
~~~
269
270
-----
271 1 jortel@redhat.com
272 29 jortel@redhat.com
#### As an importer, I need a way to link a downloaded file to an artifact without keeping all content units and artifacts in memory.
273 19 jortel@redhat.com
274
**jupiter**:
275 1 jortel@redhat.com
276 27 jortel@redhat.com
Using the *Batch* to run the downloads concurrently and specifying the *backlog* to limit the number of downloads in memory. See other examples.
277 1 jortel@redhat.com
278 27 jortel@redhat.com
The Download.attachment provides linkage to objects like Artifacts that are related to the download.
279
280 19 jortel@redhat.com
~~~python
281
282 27 jortel@redhat.com
download = HttpDownload(...)
283
download.attachment = Artifact(..)
284 18 jortel@redhat.com
~~~
285
286
**saturn**:
287 1 jortel@redhat.com
288 27 jortel@redhat.com
@bmbouters: please describe and provide examples.
289 19 jortel@redhat.com
290 18 jortel@redhat.com
~~~python
291
~~~
292
293
-----
294
295 29 jortel@redhat.com
#### As an importer, I can perform concurrent downloading using a synchronous pattern.
296 18 jortel@redhat.com
297
**jupiter**:
298 1 jortel@redhat.com
299 19 jortel@redhat.com
Using the *Batch*. See other examples.
300 18 jortel@redhat.com
301
**saturn**:
302
303 19 jortel@redhat.com
Using either the *GroupDownloader* or asyncio loop directly. See other examples.
304 18 jortel@redhat.com
305 1 jortel@redhat.com
-----
306
307 29 jortel@redhat.com
#### As an importer, concurrent downloads must share resources such as sessions,connection pools and auth tokens across individual downloads.
308 1 jortel@redhat.com
309 18 jortel@redhat.com
**jupiter**:
310
311 19 jortel@redhat.com
The Download.context is designed to support this. The *shared* context can be used to safely share anything This includes python-requests sessions (using a Cache), auth tokens and resolved mirror lists. The sharing is done through collaboration. When it's appropriate for individual downloads to share things, an external actor like the Batch or the Streamer will ensure that all of the download  
312 1 jortel@redhat.com
objects have the same context.
313 18 jortel@redhat.com
314
**saturn**:
315
316 19 jortel@redhat.com
Each downloader could define a class attribute. This global can be used share anything. This includes python-requests sessions (using a Cache), auth tokens and resolved mirror lists. The sharing is done through collaboration. Sharing is global and unconditional.
317 1 jortel@redhat.com
318 20 jortel@redhat.com
Question: how will thread safety be provided? The streamer will have multiple twisted threads using these downloaders.
319
320 18 jortel@redhat.com
-----
321
322 29 jortel@redhat.com
#### As an importer I can customize how downloading is performed. For example, to support mirror lists
323 18 jortel@redhat.com
324
**jupiter**:
325 1 jortel@redhat.com
326 23 jortel@redhat.com
All download objects can be customized in one of two ways. First, by delegation using *events*. And, second by subclassing.
327 1 jortel@redhat.com
328 23 jortel@redhat.com
Delegation example.
329
330 1 jortel@redhat.com
~~~python
331 23 jortel@redhat.com
332
class MirrorDelegate:
333
    # Any download can delegate mirror list resolution
334
    # and hunting to this object.
335
336
    def __init__(self):
337
        self.mirrors = iter([])
338
339
    def attach(self, download):
340
        download.register(Event.PREPARED, self.on_prepare)
341
        download.register(Event.ERROR, self.on_error)
342
343
    def on_prepare(self, event):
344
        # Resolve the mirror list URL
345
        # May already be stored in the context or need to be downloaded and parsed.
346
        with event.download.context as context:
347
            try:
348
                mirrors = context.mirrors
349
            except AttributeError:
350
                download = event.download.clone()
351
                download.writer = BufferWriter()
352
                download()
353 25 jortel@redhat.com
                _list = download.writer.content()
354 23 jortel@redhat.com
                mirrors = [u.strip() for u in _list.split('\n') if u.strip()]
355
                context.mirrors = mirrors
356
        # Align retries with # of mirrors.
357
        event.download.retries = len(mirrors)
358
        self.mirrors = iter(mirrors)
359
        # Start
360
        event.download.url = next(self.mirrors)
361
362
    def on_error(self, event):
363
        try:
364
            event.download.url = next(self.mirrors)
365
        except StopIteration:
366
            # no more mirrors
367
            pass
368
        else:
369
            event.repaired = True
370
371
# importer
372
def get_download(...):
373
    download = Factory.build(...)
374
    delegate = MirrorDelegate()
375
    delegate.attach(download)
376
~~~
377
378
Subclass example.
379
380
~~~python
381
382
class MirrorDownload(HttpDownload):
383
    # Support HTTP/HTTPS mirror list downloading.
384
385
    def _prepare(self):
386
        super()._prepare()
387
        # Resolve the mirror list URL
388
        # May already be stored in the context or need to be downloaded and parsed.
389
        with self.context as context:
390
            try:
391
                mirrors = context.mirrors
392
            except AttributeError:
393
                download = self.clone()
394
                download.writer = BufferWriter()
395
                download()
396 25 jortel@redhat.com
                _list = download.writer.content()
397 23 jortel@redhat.com
                mirrors = [u.strip() for u in _list.split('\n') if u.strip()]
398
                context.mirrors = mirrors
399
        # Align retries with # of mirrors.
400
        self.retries = len(mirrors)
401
        self.mirrors = iter(mirrors)
402
        # Start
403
        self.url = next(self.mirrors)
404
405
    def _on_error(self, event):
406
        super()._on_error(event)
407
        try:
408
            self.url = next(self.mirrors)
409
        except StopIteration:
410
            # no more mirrors
411
            return False
412
        else:
413
            return True
414
415
# importer
416 1 jortel@redhat.com
def get_download(...):
417 23 jortel@redhat.com
    # Factory needs to support custom class.
418 18 jortel@redhat.com
~~~
419
420
**saturn**:
421
422
~~~python
423
~~~
424
425
-----
426
427 29 jortel@redhat.com
#### As an importer, concurrent downloading must limit the number of simultaneous connections. Downloading 5k artifacts cannot open 5k connections.
428 1 jortel@redhat.com
429 18 jortel@redhat.com
**jupiter**:
430 1 jortel@redhat.com
431 20 jortel@redhat.com
This is supported by sharing connection pools and limiting the total number of downloads in progress concurrently. See resource sharing and concurrency limiting use cases.
432 18 jortel@redhat.com
433
**saturn**:
434
435 20 jortel@redhat.com
This is supported by sharing connection pools and limiting the total number of downloads in progress concurrently. See resource sharing and concurrency limiting use cases.
436 18 jortel@redhat.com
437
-----
438 1 jortel@redhat.com
439 29 jortel@redhat.com
#### As an importer, I can terminate concurrent downlading at any point and not leak resources.
440 18 jortel@redhat.com
441
**jupiter**:
442
443 26 jortel@redhat.com
The loop using the iterator returned by *Batch* can be safely exited at any point and all resources are then free to be garbage collected.
444 18 jortel@redhat.com
445 1 jortel@redhat.com
**saturn**:
446 18 jortel@redhat.com
447 26 jortel@redhat.com
The loop using the asyncio loop can be safely exited at any point and all resources are then free to be garbage collected.
448 18 jortel@redhat.com
449
-----
450 26 jortel@redhat.com
451 29 jortel@redhat.com
#### As an importer, I can download using any protocol. Starting with HTTP/HTTPS and eventually FTP.
452 1 jortel@redhat.com
453
**jupiter**:
454
455 18 jortel@redhat.com
Classes extending *Download* may implement any protocol. HTTP/HTTPS is supported by *HttpDownload*. See other use case examples.
456
457 26 jortel@redhat.com
**saturn**:
458 18 jortel@redhat.com
459
HTTP/HTTPS is supported by *HttpDownloader*. See other use case examples.
460
461 26 jortel@redhat.com
-----
462 18 jortel@redhat.com
463
### Streamer
464
465 29 jortel@redhat.com
-----
466 1 jortel@redhat.com
467 29 jortel@redhat.com
#### As the streamer, I need to download files related to published artifacts and metadata but delegate *the implementation* (protocol, settings, credentials) to the importer. The implementation must be a black-box.
468
469 1 jortel@redhat.com
**jupiter**:
470 18 jortel@redhat.com
471 28 jortel@redhat.com
The *Download* is a callable.
472 18 jortel@redhat.com
473
~~~python
474 28 jortel@redhat.com
475
download = importer.get_downloader(...)
476 1 jortel@redhat.com
download()
477
~~~
478 18 jortel@redhat.com
479
**saturn**:
480 1 jortel@redhat.com
481 28 jortel@redhat.com
@bmbouters: please describe and provide examples.
482
483 1 jortel@redhat.com
~~~python
484 28 jortel@redhat.com
485
downloader = importer.get_downloader(...)
486
self.not_done.append(downloader.run())
487 18 jortel@redhat.com
~~~
488 16 jortel@redhat.com
489 30 jortel@redhat.com
-----
490 1 jortel@redhat.com
491 29 jortel@redhat.com
#### As the streamer, I want to validate downloaded files.
492 1 jortel@redhat.com
493
**jupiter**:
494
495 28 jortel@redhat.com
The *Download* may be configured by the importer with a list of *Validation* objects. Validation is performed on the downloaded bit stream.
496 1 jortel@redhat.com
497 28 jortel@redhat.com
~~~python
498
499 1 jortel@redhat.com
download = importer.get_downloader(...)
500 28 jortel@redhat.com
download()
501 18 jortel@redhat.com
~~~
502
503 1 jortel@redhat.com
**saturn**:
504
505 28 jortel@redhat.com
The *HttpDownloader* may be configured by the importer with expected size and expected digests. Validation is performed on the downloaded bit stream.
506
507 1 jortel@redhat.com
~~~python
508 28 jortel@redhat.com
509
downloader = importer.get_downloader(...)
510
self.not_done.append(downloader.run())
511 1 jortel@redhat.com
~~~
512
513
-----
514
515 29 jortel@redhat.com
#### As the streamer, concurrent downloads must share resources such as sessions,connection pools and auth tokens across individual downloads without having knowledge of such things.
516 1 jortel@redhat.com
517
**jupiter**:
518
519 28 jortel@redhat.com
Each download may be configured with a shared context. The download objects collaborate to share resources using the context. The streamer updates each Download provided by the importer to use the same (shared) context.
520 18 jortel@redhat.com
521 28 jortel@redhat.com
~~~python
522
523 1 jortel@redhat.com
download = importer.get_downloader(...)
524 28 jortel@redhat.com
download.context = self.context  # This is a Context.
525
download()
526 1 jortel@redhat.com
~~~
527 18 jortel@redhat.com
528
**saturn**:
529
530 28 jortel@redhat.com
Each downloader has a class attribute used to globally share resources.
531
532 1 jortel@redhat.com
~~~python
533 28 jortel@redhat.com
534
downloader = importer.get_downloader(...)
535 18 jortel@redhat.com
self.not_done.append(downloader.run())
536
~~~
537
538 1 jortel@redhat.com
-----
539 18 jortel@redhat.com
540 29 jortel@redhat.com
#### As the streamer, I need to support complex downloading such as mirror lists. This complexity must be delegated to the importer.
541 18 jortel@redhat.com
542
**jupiter**:
543
544 28 jortel@redhat.com
The downloader object provided by the importer will handle the mirror list.
545 18 jortel@redhat.com
546
**saturn**:
547
548 28 jortel@redhat.com
~~~python
549
550
downloader = importer.get_downloader(...)
551 1 jortel@redhat.com
self.not_done.append(downloader.run())
552 18 jortel@redhat.com
~~~
553
554
-----
555
556 29 jortel@redhat.com
#### As the streamer, I need to bridge the downloaded bit stream to the Twisted response. The file is not written to disk.
557 18 jortel@redhat.com
558 1 jortel@redhat.com
**jupiter**:
559
560 30 jortel@redhat.com
The Download.writer can be reassigned to the base *Writer*.
561
562 18 jortel@redhat.com
~~~python
563 1 jortel@redhat.com
564 30 jortel@redhat.com
class TwistedWriter(Writer):
565 1 jortel@redhat.com
566 30 jortel@redhat.com
    def __init__(self, request):
567
        self.request = request
568 1 jortel@redhat.com
569 30 jortel@redhat.com
    def append(self, buffer):
570
        reactor.callFromThread(self.request.write, buffer)
571 18 jortel@redhat.com
572 30 jortel@redhat.com
    def close(self):
573
        reactor.callFromThread(self.request.finish)
574 1 jortel@redhat.com
575 29 jortel@redhat.com
576 30 jortel@redhat.com
download = importer.get_downloader(...)
577
download.writer = TwistedWriter(request)
578
download()
579 1 jortel@redhat.com
~~~
580 18 jortel@redhat.com
581
**saturn**:
582
583 30 jortel@redhat.com
@bmbouters: please describe and add example.
584
585 18 jortel@redhat.com
~~~python
586 1 jortel@redhat.com
~~~
587
588
-----
589
590 30 jortel@redhat.com
#### As the streamer, I need to forward HTTP headers from the download response to the twisted response.
591 1 jortel@redhat.com
592
**jupiter**:
593
594 30 jortel@redhat.com
The headers can be forwarded using a delegate.
595
596 1 jortel@redhat.com
~~~python
597 30 jortel@redhat.com
598
class HeaderDelegate:
599
600
    def __init__(self, request):
601
        self.request = request
602
603
    def attach(download):
604
        download.register(Event.REPLIED, self.on_reply)
605
606
    def on_reply(event):
607
        for header, value in event.download.reply.headers:
608
            # do filtering here
609
            self.request.setHeader(header, value)
610
611
download = importer.get_downloader(...)
612
delegate = HeaderDelegate(request)
613
delegate.attach(download)
614
download()
615 18 jortel@redhat.com
~~~
616
617
**saturn**:
618 30 jortel@redhat.com
619
@bmbouters: please describe and add example.
620 18 jortel@redhat.com
621
~~~python
622
~~~
623
624
-----