Ticket #443 (new Task)

Opened 10 months ago

Last modified 8 months ago

Requesting 7zip support in BOINC

Reported by: Ageless Assigned to:
Priority: Trivial Milestone: 6.4
Component: BOINC - API Version:
Keywords: 7zip compression Cc: Nicolas, Ageless

Description

For Hydrogen@Home we're busy trying out several ways to get 7zip archives to decompress through BOINC. So far without success.

Asking through email, Rom answered we'd best make use of the wrapper application. That's all fine, but doesn't the wrapper decompress natively to a slot? And then when the wrapper app ends, doesn't BOINC delete the contents of the slot? We're trying to keep the decompressed files in the project folder.

I managed to get an answer from Carl Christensen (he's still alive :-)).
self-extracting execs across platforms isn't possible with the way BOINC is setup. If you like the 7zip algorithm (which I've never heard of) I suggest making a "boinc_7zip" library similar to how I did 'boinc_zip' -- I'm sure David Anderson would welcome this addition to the BOINC CVS tree. My other suggestion is to use the built-in web compression (I think it's mod_deflate in apache) to do "transparent" gzip compression (the details are on the wiki -- basically when I redid the BOINC networking layer two years ago using libcurl this featre was made possible). I'm afraid I am all tied up to do any BOINC programming now.

So the question is, can someone make a boinc_7zip library for integration into BOINC? Would other projects be interested in this?

If you're unfamiliar with 7zip's capabilities, read its site or Wikipedia on 7zip. It does compress better than zip or gzip.

Change History

(follow-up: ↓ 3 ) 10/20/07 20:11:56 changed by Nicolas

  • cc set to Nicolas.
  • keywords changed from 7zip compression decompression to 7zip compression.
  • component changed from Server - Other to BOINC - API.
  • owner changed from davea to ballen.

Something like boinc_7zip would also decompress to the slot... I'm not sure how things work if you add files to the project directory without "letting BOINC know" the file is there.

10/20/07 20:12:40 changed by Nicolas

  • owner deleted.
  • type changed from Enhancement to Task.

ballen? Whoops.

(in reply to: ↑ 1 ; follow-up: ↓ 5 ) 10/20/07 20:58:45 changed by Ageless

Replying to Nicolas:

Something like boinc_7zip would also decompress to the slot... I'm not sure how things work if you add files to the project directory without "letting BOINC know" the file is there.

It's not tasks we're trying to zip up, just static library files and probably the executables. The difference between users having to download 49MB worth of stuff or 5MB worth of archive is huge.

But does boinc_zip decompress to a slot? I thought CPDN used it to decompress to their project folder, then copied to the slot. I asked for a boinc_7zip, but perhaps that the present boinc_zip can be adapted so 7zip is also supported. Although the neatest would be if it could be done through Apache, like mod_deflate. Oh wishes.

10/20/07 21:00:05 changed by Ageless

  • cc changed from Nicolas to Nicolas, Ageless.

Hopefully no commas are needed between CC's?

(in reply to: ↑ 3 ) 10/20/07 21:05:52 changed by Nicolas

Replying to Ageless:

It's not tasks we're trying to zip up, just static library files and probably the executables. The difference between users having to download 49MB worth of stuff or 5MB worth of archive is huge.

I don't think CPDN compresses the apps.

But does boinc_zip decompress to a slot? I thought CPDN used it to decompress to their project folder, then copied to the slot. I asked for a boinc_7zip, but perhaps that the present boinc_zip can be adapted so 7zip is also supported. Although the neatest would be if it could be done through Apache, like mod_deflate. Oh wishes.

Remember that technically you can decompress anywhere on the user's hard disk.

Is it so much of a problem to bundle 7z.exe with the Windows application, p7zip with the Linux app, etc. and use that to decompress? The "application" would consist of the extracting tool (7z or p7zip), the .7z archive itself, and a tiny wrapper that would start the extracting tool to decompress to slot and run the extracted main app. BURP does that, but with a .zip archive (bundling unzip.exe).

10/21/07 07:58:17 changed by jackygrahamez

Here is my failed one-eyed attempt to work arround the problem with a dirty solution. I used different decompressions for different platforms and I used a batch file in the case of windows to decompress to project directory if it was not decomressed already, then copy to slot. But that caused errors about unable to handle processes.

(follow-up: ↓ 8 ) 11/13/07 12:33:26 changed by Ananas

7-Zip is freesource and not only an executable binary. They have an SDK here : http://www.7-zip.org/sdk.html so there's no need for a separate .exe file as soon as the format is recognized by the core client.

On some file types (e.g. EPS) it is up to 2 times more efficient than any other compressor, usually it's only a few % though.

(in reply to: ↑ 7 ; follow-up: ↓ 9 ) 11/13/07 12:51:41 changed by Nicolas

Replying to Ananas:

7-Zip is freesource and not only an executable binary. They have an SDK here : http://www.7-zip.org/sdk.html so there's no need for a separate .exe file as soon as the format is recognized by the core client.

Adding support on the core client is a bad idea. Means all previous versions that don't support it won't be able to run that project. It's easy enough to make the science app extract the files on startup.

(in reply to: ↑ 8 ; follow-up: ↓ 10 ) 11/14/07 16:19:44 changed by Ananas

Replying to Nicolas:

... Means all previous versions that don't support it won't be able to run that project ...

Not necessarily. If the scheduler knows (by the reported version) that the client supports 7z it can send 7z compressed files in 1st place.

(in reply to: ↑ 9 ; follow-up: ↓ 11 ) 11/14/07 16:24:37 changed by Nicolas

Replying to Ananas:

Replying to Nicolas:

... Means all previous versions that don't support it won't be able to run that project ...

Not necessarily. If the scheduler knows (by the reported version) that the client supports 7z it can send 7z compressed files in 1st place.

And any older version gets bigger-than-acceptable files. Instead, if the science app does the extraction, any BOINC version would be able to work fine with 7zip, since it's not the core client the one extracting anything, it's the science app, project-provided.

Also, making the scheduler send a different file depending on the version would mean quite a bit of hacking on the scheduler code.

(in reply to: ↑ 10 ) 11/14/07 16:29:01 changed by Ageless

Replying to Nicolas:

Replying to Ananas:

Replying to Nicolas:

... Means all previous versions that don't support it won't be able to run that project ...

Not necessarily. If the scheduler knows (by the reported version) that the client supports 7z it can send 7z compressed files in 1st place.

Also, making the scheduler send a different file depending on the version would mean quite a bit of hacking on the scheduler code.

It would also mean that projects need to have two zips of each file available, going against what I asked for in the first place.

(I see we're still without assignment. No one wanting to take it? ;))

12/14/07 17:40:08 changed by Ageless

  • cc changed from Nicolas, Ageless to Nicolas, Ageless, Pepo.

12/14/07 18:10:48 changed by Ageless

  • cc changed from Nicolas, Ageless, Pepo to Nicolas, Ageless.
If this page is incomplete or incorrect, please edit it or add it to the wiki to-do list. To do this, you must be logged in; click Login or Register above.