Thread 'Temporary Files and Gzip using libz'

Message boards : Server programs : Temporary Files and Gzip using libz
Message board moderation

To post messages, you must log in.

AuthorMessage
Seth

Send message
Joined: 19 Nov 16
Posts: 63
Australia
Message 74829 - Posted: 18 Dec 2016, 10:03:49 UTC

Hello,
For my project the data files used for a WU are downloaded already gzipped.

I use libz to unzip them.

I notice when the job is done the unzipped files are not removed from the client.

I can remove them my self via code when a WU is done but was wondering if there was any existing mechanism to do this?

The tricky bit here for me is when a data file is shared by multiple work units, I'll have to figure our the wu_name and use that as part of the unzipped file name so I don't remove a file thats being used by another WU.

Cheers
Seth
ID: 74829 · Report as offensive
ChristianB
Volunteer developer
Volunteer tester

Send message
Joined: 4 Jul 12
Posts: 321
Germany
Message 74845 - Posted: 19 Dec 2016, 6:27:59 UTC

Hi,
this depends on how you unzip the files. If you unzip to the project directory you have to clean up yourself. If you unzip into the slot directory BOINC will clean up for you.
ID: 74845 · Report as offensive
Juha
Volunteer developer
Volunteer tester
Help desk expert

Send message
Joined: 20 Nov 12
Posts: 801
Finland
Message 74850 - Posted: 19 Dec 2016, 21:53:54 UTC - in response to Message 74829.  

If your application can handle uncompressed input files you can leave compressing/decompressing to BOINC. See <gzip/> in Input templates. (File compression says this is not supported for input files but it's probably outdated info.)
ID: 74850 · Report as offensive
Seth

Send message
Joined: 19 Nov 16
Posts: 63
Australia
Message 74882 - Posted: 21 Dec 2016, 13:34:46 UTC - in response to Message 74850.  

Yes Jhua but I think this doesn't work if you stage files externally like I am in an S3://bucket. In anycase I've adapted my client to unzip using the WU name for uniqueness.
ID: 74882 · Report as offensive
ChristianB
Volunteer developer
Volunteer tester

Send message
Joined: 4 Jul 12
Posts: 321
Germany
Message 74893 - Posted: 21 Dec 2016, 22:24:13 UTC - in response to Message 74882.  

Yes Jhua but I think this doesn't work if you stage files externally like I am in an S3://bucket. In anycase I've adapted my client to unzip using the WU name for uniqueness.

It should work too. Since you give your own URL in the input template either way you can just et the URL to the correct one but remove the ".gz" at the end. Then the Client (>7.x) will add the ".gz" because of the </gzip>, download and unpack the file into the slot directory. The file is kept on the computer for as long as there are tasks that need it and you don't need special handling in your application.
ID: 74893 · Report as offensive

Message boards : Server programs : Temporary Files and Gzip using libz

Copyright © 2024 University of California.
Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation.