Ticket #924 (new Defect)

Opened 5 months ago

Last modified 5 months ago

Partial Transfer Fails

Reported by: knreed Assigned to: davea
Priority: Major Milestone: 6.12
Component: Client - Daemon Version: 6.6.31
Keywords: Cc:

Description

If a project compresses files and relies on libCurl to decompress the file during transfer, then in the event that a transfer is interrupted, BOINC relies on the decompressed file size to set the Request Range header. This will be invalid and therefore fail with:

416 Requested Range Not Satisfiable

this will repeat until the timeout limit for the download.

It would be great if in the event of the failure of a partial file download, the client would then attempt a full download.

Change History

06/22/09 11:59:38 changed by Nicolas

Is the server sending Transfer-Encoding: gzip, Content-Encoding: gzip, or is the file type actually a .gz, thus sending neither encoding header?

06/22/09 12:32:27 changed by knreed

I am using the suffix gzb and setting the following header:

Content-Encoding: gzip

Ex: https://grid.worldcommunitygrid.org/boinc/download/3b6/faah7186_ZINC00054326_xmdEq_1MSN_01_ZINC00054326.pdbqt.gzb

Note that this example will be deleted eventually.

06/22/09 12:35:24 changed by Nicolas

Who is supposed to decompress that file, the science app, or are you relying on the BOINC client doing it?

06/22/09 12:39:44 changed by knreed

The boinc client decompresses the file.

06/22/09 12:49:51 changed by knreed

We were using mod_deflate as outlined here: http://boinc.berkeley.edu/trac/wiki/FileCompression

Most of that page reflects what we had been doing.

We need to conserve storage on the server side, so we have started to 'pre-compress' the files for download and we are using the suffix .gzb to indicate this. We have added the following to our httpd.conf in order to add the header:

AddEncoding? gzip gzb

06/22/09 13:04:46 changed by Nicolas

Older clients that don't support compression will fail; the server should give a compressed file only if the client says Accept-Encoding: gzip; otherwise it should give the decompressed file. I just tested your URL, and I'm sure it's sending a gzipped response always.

06/22/09 13:15:58 changed by knreed

Clients older than 5.4 no longer work with World Community Grid. We specify that in config.xml using the tags:

<min_core_client_version>

504

</min_core_client_version>

The 5.4 and more recent clients work with this correctly with the exception of partial file transfers. We have been using this for about 3 months.

06/22/09 13:48:48 changed by Nicolas

That's not the point. Do servers have a giant database saying what versions of what web browsers support compression?

If this page is incomplete or incorrect, please edit it or add it to the wiki to-do list. To do this, you must be logged in; click Login or Register above.