Use HDD for BOINC work files instead of SSD

Message boards : Questions and problems : Use HDD for BOINC work files instead of SSD
Message board moderation

To post messages, you must log in.

Previous · 1 · 2

AuthorMessage
Profile el_gallo_azul

Send message
Joined: 19 Jan 14
Posts: 79
Message 53811 - Posted: 27 Apr 2014, 1:49:11 UTC - in response to Message 53805.  

You don't judge by load. You look in logs.


That makes sense, but I'm afraid that I don't know where these logs would be.

Second, #8 provided then #5, #6, #7 are completely unnecessary.


If you mean that step 8 makes steps 5, 6 and 7 unnecessary:

Step 5: This is the punchline of the whole thing, which is to move the BOINC working directory from my SSD to my HDD.

Step 6: That was just in preparation for creating the link.

Step 7: I created the link as a result of Message 53626, and BOINC Manager started running again, so I believe it to be necessary.

If that's not what you mean, then I don't understand that sentence.
ID: 53811 · Report as offensive
Profile el_gallo_azul

Send message
Joined: 19 Jan 14
Posts: 79
Message 53812 - Posted: 27 Apr 2014, 1:55:48 UTC - in response to Message 53807.  

If it still doesn't work run the following:

cd /media/el_gallo_azul/Storage/var-lib-boinc-client/boinc-client/ && sudo -u boinc boinc


That starts BOINC as user boinc. All the messages go to your terminal instead of the various files. Copy-paste the messages here, maybe there's something useful in them.


The result of that is

27-Apr-2014 11:20:12 [---] Starting BOINC client version 7.2.42 for x86_64-pc-linux-gnu
27-Apr-2014 11:20:12 [---] log flags: file_xfer, sched_ops, task
27-Apr-2014 11:20:12 [---] Libraries: libcurl/7.35.0 OpenSSL/1.0.1f zlib/1.2.8 libidn/1.28 librtmp/2.3
27-Apr-2014 11:20:12 [---] Data directory: /media/el_gallo_azul/Storage/var-lib-boinc-client/boinc-client
27-Apr-2014 11:20:12 [---] No usable GPUs found
27-Apr-2014 11:20:12 [---] Host name: W2600CR
27-Apr-2014 11:20:12 [---] Processor: 32 GenuineIntel Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz [Family 6 Model 45 Stepping 7]
27-Apr-2014 11:20:12 [---] Processor features: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx lahf_lm ida arat epb xsaveopt pln pts dtherm tpr_shadow vnmi flexpriority ept vpid
27-Apr-2014 11:20:12 [---] OS: Linux: 3.13.0-24-generic
27-Apr-2014 11:20:12 [---] Memory: 31.32 GB physical, 31.92 GB virtual
27-Apr-2014 11:20:12 [---] Disk: 916.77 GB total, 441.61 GB free
27-Apr-2014 11:20:12 [---] Local time is UTC +9 hours
27-Apr-2014 11:20:12 [---] VirtualBox version: 4.3.10_Ubuntur93012
27-Apr-2014 11:20:12 [---] Config: GUI RPCs allowed from:
27-Apr-2014 11:20:12 [pogs] URL http://pogs.theskynet.org/pogs/; Computer ID 34098; resource share 100
27-Apr-2014 11:20:12 [---] General prefs: from http://lhcathome2.cern.ch/test4theory/ (last modified 17-Nov-2013 17:08:05)
27-Apr-2014 11:20:12 [---] Host location: none
27-Apr-2014 11:20:12 [---] General prefs: using your defaults
27-Apr-2014 11:20:12 [---] Reading preferences override file
27-Apr-2014 11:20:12 [---] Preferences:
27-Apr-2014 11:20:12 [---]    max memory usage when active: 16036.07MB
27-Apr-2014 11:20:12 [---]    max memory usage when idle: 28864.93MB
27-Apr-2014 11:20:12 [---]    max disk usage: 441.54GB
27-Apr-2014 11:20:12 [---]    suspend work if non-BOINC CPU load exceeds 25%
27-Apr-2014 11:20:12 [---]    (to change preferences, visit a project web site or select Preferences in the Manager)
dir_open: Could not open directory 'slots' from '/media/el_gallo_azul/Storage/var-lib-boinc-client/boinc-client'.
27-Apr-2014 11:20:12 [---] gui_rpc_auth.cfg is empty - no GUI RPC password protection
27-Apr-2014 11:20:12 [---] Not using a proxy
27-Apr-2014 11:20:12 Initialization completed


That
Could not open directory 'slots' from '/media/el_gallo_azul/Storage/var-lib-boinc-client/boinc-client'
is pretty suspicious. I've seen it several times before. I don't understand it because there's no directory 'slots' in /media/el_gallo_azul/Storage/var-lib-boinc-client/boinc-client.
ID: 53812 · Report as offensive
Richard Haselgrove
Volunteer tester
Help desk expert

Send message
Joined: 5 Oct 06
Posts: 5082
United Kingdom
Message 53813 - Posted: 27 Apr 2014, 9:19:21 UTC - in response to Message 53812.  

That
Could not open directory 'slots' from '/media/el_gallo_azul/Storage/var-lib-boinc-client/boinc-client'
is pretty suspicious. I've seen it several times before. I don't understand it because there's no directory 'slots' in /media/el_gallo_azul/Storage/var-lib-boinc-client/boinc-client.

BOINC operates with three sub-directories below the main data directory. You have the main directory listed there:

27-Apr-2014 11:20:12 [---] Data directory: /media/el_gallo_azul/Storage/var-lib-boinc-client/boinc-client

so create these three directories within it:

notices
projects
slots

if they're not already there. Make sure that BOINC has permission to read and write to all three sub-directories, and in the case of 'projects' and 'slots' to execute files - including files in daughter directories.
ID: 53813 · Report as offensive
Juha
Volunteer developer
Volunteer tester
Help desk expert

Send message
Joined: 20 Nov 12
Posts: 801
Finland
Message 53820 - Posted: 27 Apr 2014, 20:05:48 UTC - in response to Message 53810.  

el_gallo_azul@W2600CR:~$ ps -A|grep boinc
  7372 ?        00:00:10 boincmgr


I'm afraid that I don't know what that question mark means.

It just means boincmgr doesn't have its input and output connected to any terminal.

Was boinc supposed to be running at that time? If you start the service does it run for a moment and then crash like it did in the other thread?

You don't judge by load. You look in logs.


That makes sense, but I'm afraid that I don't know where these logs would be.


Well, kern.log and syslog in/var/log/ in case of crashes (these log names vary from one system to another) and std*.txt in BOINC's data directory in case BOINC doesn't do what you want it to do.

Step 5: This is the punchline of the whole thing, which is to move the BOINC working directory from my SSD to my HDD.


Technically, you could skip the move and just tell BOINC to use the new location. For BOINC this would look like a new computer. Of course if you had any workunits on board BOINC wouldn't know about them anymore. The top level directory is still something you need to create yourself, BOINC shouldn't have permissions to do it by itself.

The result of that is

27-Apr-2014 11:20:12 [---] Starting BOINC client version 7.2.42 for x86_64-pc-linux-gnu


And this time the client didn't crash? I can't see anything weird in the messages.

That
Could not open directory 'slots' from '/media/el_gallo_azul/Storage/var-lib-boinc-client/boinc-client'
is pretty suspicious. I've seen it several times before. I don't understand it because there's no directory 'slots' in /media/el_gallo_azul/Storage/var-lib-boinc-client/boinc-client.


That error message is Debian add-on. I think it's a false alarm this time. The client is probably just looking if there is any slot directories to clean up. You can still make the directory yourself. Maybe that will make BOINC happy.

Moving the data directory should just work. I don't have any good ideas at the moment so I think I need to try the Ubuntu package myself but that will be later next week.
ID: 53820 · Report as offensive
Profile el_gallo_azul

Send message
Joined: 19 Jan 14
Posts: 79
Message 53824 - Posted: 28 Apr 2014, 1:40:57 UTC - in response to Message 53813.  

BOINC operates with three sub-directories below the main data directory. You have the main directory listed there:

27-Apr-2014 11:20:12 [---] Data directory: /media/el_gallo_azul/Storage/var-lib-boinc-client/boinc-client

so create these three directories within it:

notices
projects
slots

if they're not already there.


I had checked that BOINC was working, just before moving
/var/lib/boinc-client
to
/media/el_gallo_azul/Storage/var-lib-boinc-client/boinc-client
, and the directory (which I have not changed in any way) contains the following subdirectories:

notices
projects
.nv

I'm happy to create a slots subdirectory, bearing in mind that BOINC was working OK before I moved this directory. In order to speed this up, it would be helpful if someone could give me the correct command to create the directory and set the correct permissions.

Make sure that BOINC has permission to read and write to all three sub-directories, and in the case of 'projects' and 'slots' to execute files - including files in daughter directories.


The three subdirectories have the following permissions which I have not changed:

notices
Owner: BOINC core client
Access: Create and delete files

Group: boinc
Access: Create and delete files

projects

Owner: BOINC core client
Access: Create and delete files

Group: boinc
Access: Create and delete files

.nv

Owner: BOINC core client
Access: Create and delete files

Group: boinc
Access: None
ID: 53824 · Report as offensive
Profile el_gallo_azul

Send message
Joined: 19 Jan 14
Posts: 79
Message 53825 - Posted: 28 Apr 2014, 2:07:58 UTC - in response to Message 53820.  

It just means boincmgr doesn't have its input and output connected to any terminal.


Nope I don't know what that means (the "connected" part).

Was boinc supposed to be running at that time? If you start the service does it run for a moment and then crash like it did in the other thread?


Just now:
cd /media/el_gallo_azul/Storage/var-lib-boinc-client/boinc-client/ && sudo -u boinc boinc


followed by

ps -A|grep boinc


in another terminal gives:

  5031 pts/2    00:00:00 boinc
  5036 pts/2    00:00:00 boinc


Technically, you could skip the move and just tell BOINC to use the new location. For BOINC this would look like a new computer.


Yes but I didn't want to have to create subdirectories and permissions, hence the move.

The top level directory is still something you need to create yourself, BOINC shouldn't have permissions to do it by itself.


Which top-level directory?

And this time the client didn't crash? I can't see anything weird in the messages.


Yes if I:
1. Start BOINC Manager; then
2. Try to add a project (since my projects have disappeared), after about 10 seconds I get the message "The BOINC client has exited unexpectedly 3 times within the last 2 minutes. Would you like to restart it again?".

Before I moved the directory "boinc-client", I added a project and BOINC all worked, downloading and starting on about 40 or 50 tasks before I paused it and moved the directory.

Then I did the three things from this post:

1. Move (or create) the "boinc-client" directory; and
2. Create a symbolic link "var/lib/boinc-client" to the new "boinc-client" directory; and
3. Edit "/etc/default/boinc-client" to state the location of "boinc-client"

and now it:

1. Doesn't show any projects or tasks; and
2. Doesn't let me add any projects.

Moving the data directory should just work.


Yes I agree with this. I am sure that there is some way to use the HDD instead of the SSD for BOINC working files, but so far I haven't been able to achieve it or even to find out how.
ID: 53825 · Report as offensive
Juha
Volunteer developer
Volunteer tester
Help desk expert

Send message
Joined: 20 Nov 12
Posts: 801
Finland
Message 53962 - Posted: 5 May 2014, 21:07:44 UTC - in response to Message 53825.  

Been playing with the client for a while now.

I still don't know what happened to your slots directory during the move. Either the whole data directory should have been moved or the files and directories that couldn't be moved should have been left at the original location. But since you could create the link it's like some of the files just vanished into thin air???

Anyway. Have you noticed the text in the lower right corner of the Manager? You know, the text that says "Connected" or "Connecting" or "Disconnected" or something like that. When you say your projects and tasks have disappeared has that text in the Manager ever said "Connected"? And has it said "Connected" before you tried to add a project?

Next, I would like you to list the owner and permissions of all the directories in the data directory path, starting from /media. And not from file manager but from command line with "ls -ld /media" and so on. Please just copy-paste.

You said you set your HDD to mount automatically at OS start. How and where did you set it? What's in your /etc/fstab?

Btw, with the link in place you don't really need to edit the defaults file. Everything already goes through the link to the new location.
ID: 53962 · Report as offensive
Profile el_gallo_azul

Send message
Joined: 19 Jan 14
Posts: 79
Message 53969 - Posted: 6 May 2014, 6:27:57 UTC - in response to Message 53962.  
Last modified: 6 May 2014, 6:32:44 UTC


I still don't know what happened to your slots directory during the move. Either the whole data directory should have been moved or the files and directories that couldn't be moved should have been left at the original location. But since you could create the link it's like some of the files just vanished into thin air???


Since my last post, I have got BOINC running again as before, on my SSD (as the default installation configuration). I would rather be running BOINC now in the shorter term than spending loads of time trying to work out how to be able to run BOINC in the longer term. Hopefully it won't take too long to get a new SSD when I need one.

I see that there is a directory "slots" in /var/lib/boinc-client. That was not there after I moved the directory "boinc-client" using
sudo mv /var/lib/boinc-client/ /media/sdb1/var-lib-boinc-client/
as described in Message 53609.

I will have another go at getting it running on the HDD though, so that it doesn't get the big interruption when it's time to change the SSD.


Have you noticed the text in the lower right corner of the Manager? You know, the text that says "Connected" or "Connecting" or "Disconnected" or something like that. When you say your projects and tasks have disappeared has that text in the Manager ever said "Connected"? And has it said "Connected" before you tried to add a project?

I have seen it before, but I didn't notice what it said when I was trying to run BOINC on my HDD. That's a shame isn't it.


Next, I would like you to list the owner and permissions of all the directories in the data directory path, starting from /media. And not from file manager but from command line with "ls -ld /media" and so on. Please just copy-paste.

I'll do this, but I'll let BOINC run for a couple of weeks before I have another go.


You said you set your HDD to mount automatically at OS start. How and where did you set it?

I don't really understand the question (the "set it" part). This is the first time that I've had more than one drive in a computer. When I bought the computer, the HDD was set up to list as a directory "storage" within / and not have a disk mounted discretely, but I changed it, and now it looks the same straight after booting as an external disk does when plugged in.


What's in your /etc/fstab?

Nothing special:
# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
# / was on /dev/sda1 during installation
UUID=21e4cb98-d3af-4a9a-9cdf-aa5e92493ed8 /               ext4    errors=remount-ro 0       1
# swap was on /dev/sda5 during installation
UUID=265ae6bf-eb9e-4100-98d4-b1fbec0b658d none            swap    sw              0       0
ID: 53969 · Report as offensive
Juha
Volunteer developer
Volunteer tester
Help desk expert

Send message
Joined: 20 Nov 12
Posts: 801
Finland
Message 53982 - Posted: 6 May 2014, 21:22:35 UTC - in response to Message 53969.  
Last modified: 6 May 2014, 21:25:56 UTC

Since my last post, I have got BOINC running again as before, on my SSD (as the default installation configuration). I would rather be running BOINC now in the shorter term than spending loads of time trying to work out how to be able to run BOINC in the longer term.

That is quite understandable. :)

I see that there is a directory "slots" in /var/lib/boinc-client. That was not there after I moved the directory "boinc-client" using
sudo mv /var/lib/boinc-client/ /media/sdb1/var-lib-boinc-client/
as described in Message 53609.

Yep. I tried that and tried it again while BOINC was still running but for me the slots directory didn't disappear. Not that it matters all that much, you'll just lose any half completed work and possibly entire cache, but I don't think the missing slots directory prevented BOINC from running.

I have seen it before, but I didn't notice what it said when I was trying to run BOINC on my HDD. That's a shame isn't it.

Yeah, the Manager doesn't make it easy to notice. It really could disable the menu entries when it's not connected.


You said you set your HDD to mount automatically at OS start. How and where did you set it?

I don't really understand the question (the "set it" part).

Edit a file, click some buttons in some GUI tool, etc.

This is the first time that I've had more than one drive in a computer. When I bought the computer, the HDD was set up to list as a directory "storage" within / and not have a disk mounted discretely, but I changed it, and now it looks the same straight after booting as an external disk does when plugged in.


What's in your /etc/fstab?
Nothing special:
[...]

Based on the lack of "Storage" in fstab and your story I'm going to guess: You boot up the computer, open file manager and there is "Storage" listed under Devices. When you click the "Storage" an eject like icon appears next to it.

Guessing some more: When you changed how the HDD showed in file manager, you either removed the "Storage" from fstab or went through Disks->your HDD->partition in it->Edit Mount Options->Automatic Mount Options->On.

There is two problems with this. First, the HDD isn't mounted automatically when the OS starts. And second, when you let GNOME manage the mounting process the permissions gets set too tight so BOINC can't change it's working directory to the new data directory.

When you are ready to try again:
- Open the Disks programs and select the mount options
- Switch off the automatic options
- Select "Mount at startup"
- Don't select "Show in user interface" (you want it but it doesn't work)
- In mount options put something like "defaults,comment=x-gvfs-show" (in Mint 15 that box doesn't have label, it's the one between Icon Name and Mount Point)
- Pick a mount point and type it in Mount Point (/storage or /mnt/storage is good, /media/el_gallo_azul/... isn't, let GNOME have that directory)
- For Identify As UUID or LABEL based is good
- Set Filesystem Type as auto or ext4

The "comment=x-gvfs-show" should make the "Storage" appear in file manager. At least it does so in Mint 15...
ID: 53982 · Report as offensive
BobCat13

Send message
Joined: 6 Dec 06
Posts: 118
United States
Message 53996 - Posted: 7 May 2014, 1:26:39 UTC - in response to Message 53982.  

How about creating a 30 or 40GB partition on the HDD labeled /var and make sure it mounts at startup? That way, when installing BOINC from repositories it will automatically create the data directory on the HDD.
ID: 53996 · Report as offensive
Profile el_gallo_azul

Send message
Joined: 19 Jan 14
Posts: 79
Message 53997 - Posted: 7 May 2014, 3:59:07 UTC - in response to Message 53982.  


You boot up the computer, open file manager and there is "Storage" listed under Devices. When you click the "Storage" an eject like icon appears next to it.

Yes

When you changed how the HDD showed in file manager, you either removed the "Storage" from fstab or went through Disks->your HDD->partition in it->Edit Mount Options->Automatic Mount Options->On.

No, I used the CLI, but I can't remember now what the command line said.
ID: 53997 · Report as offensive
Profile el_gallo_azul

Send message
Joined: 19 Jan 14
Posts: 79
Message 53998 - Posted: 7 May 2014, 4:09:16 UTC

I have a funny feeling deep down inside that the problem is probably related to

First, the HDD isn't mounted automatically when the OS starts. And second, when you let GNOME manage the mounting process the permissions gets set too tight so BOINC can't change it's working directory to the new data directory.


despite the fact that the permissions for /.../boinc-client and its subfolders looked OK when I looked at them after moving the folder from /var/lib/boinc-client on the SSD to /media/el_gallo_azul/Storage/var-lib-boinc-client/boinc-client on the HDD.

The other possibility is something to do with the fact that the folder /var/lib/boinc-client/slots exists now while BOINC is working on the SSD, but when I moved the folder /var/lib/boinc-client from the SSD to the HDD, all of a sudden there was no /.../boinc-client/slots.
ID: 53998 · Report as offensive
Juha
Volunteer developer
Volunteer tester
Help desk expert

Send message
Joined: 20 Nov 12
Posts: 801
Finland
Message 54012 - Posted: 7 May 2014, 19:30:01 UTC - in response to Message 53998.  

Yes. After a reboot BOINC can't start because /media/el_gallo_azul/Storage isn't mounted. After you manually mount it BOINC still can't start because one of the directories have too tight permissions, probably /media/el_gallo_azul but that's not necessarily the only one.

I forgot to mention earlier. That .nv directory in /var/lib/boinc-client is NVIDIA's OpenGL shader cache. I can imagine that it's used to cache CUDA or OpenCL kernels too. NVIDIA drivers puts that directory in user's home directory. For user boinc, that is /var/lib/boinc-client. So you'll want to keep that directory around either as a real directory or as a symlink to the new location or change boinc's home directory to the new location.
ID: 54012 · Report as offensive
Profile el_gallo_azul

Send message
Joined: 19 Jan 14
Posts: 79
Message 54016 - Posted: 8 May 2014, 2:14:29 UTC - in response to Message 54012.  

So you'll want to keep that directory around either as a real directory or as a symlink to the new location or change boinc's home directory to the new location.

Do you mean to do something else once I've moved it?

Also, there is no "boinc's home directory" specified in /etc/default/boinc-client.
ID: 54016 · Report as offensive
Juha
Volunteer developer
Volunteer tester
Help desk expert

Send message
Joined: 20 Nov 12
Posts: 801
Finland
Message 54028 - Posted: 8 May 2014, 18:41:54 UTC - in response to Message 54016.  

So you'll want to keep that directory around either as a real directory or as a symlink to the new location or change boinc's home directory to the new location.

Do you mean to do something else once I've moved it?

Making a symlink old->new works ok, like you've done so far to make Manager work.

I just figured I'd mention it.

Also, there is no "boinc's home directory" specified in /etc/default/boinc-client.

It's a user account thing so it's in /etc/passwd, modified with usermod.
ID: 54028 · Report as offensive
Profile el_gallo_azul

Send message
Joined: 19 Jan 14
Posts: 79
Message 54153 - Posted: 18 May 2014, 5:31:11 UTC

After all that, I think I'll just leave it as is now.

According to the Disks utility, the number of bad sectors hasn't increased from 302 for two weeks or more, and I have been running BOINC during that time.

If my SSD is going to hold for a while, I'd rather than just spend the money (and the time) to replace it when the time comes than spend more time now trying to get BOINC to work nicely on my HDD.

Thanks for your help.
ID: 54153 · Report as offensive
Previous · 1 · 2

Message boards : Questions and problems : Use HDD for BOINC work files instead of SSD

Copyright © 2024 University of California.
Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation.