Debugging server components

A grab-bag of techniques for debugging BOINC server software:

Log files

Most error conditions are reported in the log files. Make sure you know where these are. If you're interested in the history of a particular WU or result, grep for WU#12345 or RESULT#12345 (where 12345 represents the ID) in the log files. The html/ops pages also provide an interface for this.

Debugging the scheduler

The scheduler is a CGI program; it reads a request from stdin and writes a reply to stdout. You can run it with a command-line debugger like gdb:

• Copy the "scheduler_request_X.xml" file from a client to the machine running the scheduler. (X = your project URL)
• Run the scheduler under the debugger, giving it this file as stdin, i.e.:

  gdb cgi
  (set a breakpoint)
  r < scheduler_request_X.xml
  

• You may have to doctor the database as follows to keep the scheduler from rejecting the request:

  update host set rpc_seqno=0, rpc_time=0 where hostid=N
  

This is useful for figuring out why your project is generating 'no work available' messages. As an alternative to this, edit handle_request.C, and put a call to debug_sched(sreq, sreply, "../debug_sched") just before sreply.write(fout). Then, after recompiling, touch a file called 'debug_sched' in the project root directory. This will cause transcripts of all subsequent scheduler requests and replies to be written to the cgi-bin/ directory with separate small files for each request. The file names are sched_request_H_R where H=hostid and R=rpc sequence number. This can be turned off by deleting the 'debug_sched' file.

To get core files for scheduler crashes, put the following line in sched/main.C, and recompile:

#define DUMP_CORE_ON_SEGV 1

MySQL interfaces

You should become familiar with MySQL tools such as

• The ​mysql interpreter and in particular the '​show processlist;' query.
• ​mytop: like 'top' for MySQL
• ​phpMyAdmin: general-purpose web interface to MySQL

Database query tracing

If you uncomment the symbol SHOW_QUERIES in db/db_base.C, and recompile everything, all database queries will be written to stderr (for daemons, this goes to log files; for command-line apps it's written to your terminal). This is verbose but extremely useful for tracking down database-level problems.