Debugging server components

A grab-bag of techniques for debugging BOINC server software:

Log files

Most error conditions are reported in the log files. Make sure you know where these are. If you're interested in the history of a particular WU or result, grep for WU#12345 or RESULT#12345 (12345 represents the ID) in the log files. The html/ops pages also provide an interface for this.

Database query tracing

If you uncomment the symbol SHOW_QUERIES in db/db_base.C, and recompile everything, all database queries will be written to stderr (for daemons, this goes to log files; for command-line apps it's written to your terminal). This is verbose but extremely useful for tracking down database-level problems.

Getting core dumps from the scheduler

In sched/main.C put:

#define DUMP_CORE_ON_SEGV 1

and recompile.

Running the scheduler under a debugger

The scheduler is a CGI program. It reads from stdin and writes to stdout, so you can also run it with a command-line debugger like gdb:

• Copy the "scheduler_request_X.xml" file from a client to the machine running the scheduler. (X = your project URL)
• Run the scheduler under the debugger, giving it this file as stdin, i.e.:

     gdb cgi
     (set a breakpoint)
     r < scheduler_request_X.xml
  

• You may have to doctor the database as follows:

     update host set rpc_seqno=0, rpc_time=0 where hostid=N
  

to keep the scheduler from rejecting the request.

This is useful for figuring out why your project is generating 'no work available' messages. As an alternative to this, edit handle_request.C, and put a call to debug_sched(sreq, sreply, "../debug_sched") just before sreply.write(fout). Then, after recompiling, touch a file called 'debug_sched' in the project root directory. This will cause transcripts of all subsequent scheduler requests and replies to be written to the cgi-bin/ directory with separate small files for each request. The file names are sched_request_H_R where H=hostid and R=rpc sequence number. This can be turned off by deleting the 'debug_sched' file.

MySQL interfaces

You should become familiar with MySQL tools such as

• mytop: like 'top' for MySQL
• the mysql interpreter ('mysql') and in particular the 'show processlist;' query.
• MySQLadmin: general-purpose web interface to MySQL