wiki:ServerDebug

Trouble-shooting a BOINC server

Trouble-shooting tools

Log files

Each server component (scheduler, feeder, transitioner, etc.) has its own log file. These files are in the log_HOSTNAME subdirectory of the project directory. Most error conditions are reported in the log files.

If you're interested in the history of a particular job, grep for WU#12345 or RESULT#12345 (where 12345 represents the ID) in the log files. The html/ops pages also provide an interface for this.

To control the verbosity of the log files:

  • Scheduler: set the desired logging options
  • File upload handler: set fuh_debug_level.
  • daemons: pass the cmdline arg "-d N" (1=least verbose, 4=most verbose) If you run server components with -d 4, their database queries will be logged. This is useful for tracking down database-level problems.

Examining the database

The admin web interface provides a web-based interface for browsing your project's database.

You can also use MySQL tools such as

Examining shared memory

The command

bin/show_shmem

will print a textual summary of the contents of the shared-memory structure that caches jobs and information about applications.

Trouble-shooting the job pipeline

  • Are workunits (jobs) getting created correctly? Examine the database to see. If you're using a work generator, check its log file.
  • Are results (job instances) getting created? Examine the database to see. If you don't see results, check the transitioner log file.
  • Are jobs getting into shared memory? Use show_shmem (see above). You should see jobs. If not, check the feeder log file.
  • Is the scheduler sending jobs? If not, check its log file, preferably with the following log flags:
    • <debug_version/>: show details of app version selection
    • <debug_send/>: show details of job assignment
    • <debug_quota/>: show details of quota enforcement
  • Are clients processing jobs correctly? Check the status and stderr output of completed jobs.
  • Are output files getting uploaded? Check the file upload handler log file.
  • Are jobs getting validated? Check the validator log file.
  • Are jobs getting assimilated? Check the assimilator log file.

Debugging the scheduler

If the scheduler is acting incorrectly or crashing, and you like mucking around in C++ source code, you can run it under a debugger like gdb. The scheduler is a CGI program; it reads a request from stdin and writes a reply to stdout. So you can debug it as follows:

  • Copy the "scheduler_request_X.xml" file from a client to the machine running the scheduler. (X = your project URL)
  • Run the scheduler under the debugger, giving it this file as stdin, i.e.:
    gdb cgi
    (set a breakpoint if desired)
    r < scheduler_request_X.xml
    
  • You may have to doctor the database as follows to keep the scheduler from rejecting the request:
    update host set rpc_seqno=0, rpc_time=0 where hostid=N
    

As an alternative to this, edit sched/handle_request.cpp, and put a call to debug_sched("debug_sched"); just before sreply.write(fout, sreq);. Then, after recompiling, touch a file called 'debug_sched' in the project root directory. This will cause transcripts of all subsequent scheduler requests and replies to be written to the cgi-bin/ directory with separate small files for each request. The file names are sched_request_H_R and sched_reply_H_R where H=hostid and R=rpc sequence number. This can be turned off by deleting the 'debug_sched' file.

To get core files for scheduler crashes, uncomment the following line in sched/sched_main.cpp, and recompile:

#define DUMP_CORE_ON_SEGV 1
Last modified 10 years ago Last modified on Jul 30, 2014, 9:28:13 AM