Other aop/tasks/channelquality_db

Table Of Contents

Previous topic

Offline Database Support Tasks

Next topic

SVN/Trac

This Page

Daya Bay Links

Content Skeleton

Channel Quality DB Maintenance

backup/transfer script

Single script runs in a daily crontab on the DB server dybdb2.ihep.ac.cn under the control of Qiumei maqm@ihep.ac.cn (Tao Lin, lint@ihep.ac.cn can help also).

The above script has an extensive docstring describing its usage, with examples. Although available in NuWa, dbsrv.py –help usage with the system python and MySQLdb is the more usual operation environment. Due to incremental operation and use of table partitioning (into 10k SEQNO chunks) each tarball transfered to remote nodes is less than 100M each.

monitoring script

Runs in a daily crontab on the target node, to check that the backups continue to arrive. The valmon.py and digestpath.py scripts used are housed in the env repository

Usage and configuration are described in the “Transfer Monitoring” section of the dbsrv.py docstring. Note that an email address must also be configured to switch on notifications when the constraints are violated.

Responsibilties for maintainer

  • know how the backup/transfer script operates, familiarity with ssh keys and passwordless automated transfers and how to debug them when they fail is required
  • maintain the daily remote cron task that monitors backup operation, and receive monitoring notification emails
  • act on irregularities, instruct/help Qiumei to fix issues on the server side
    • most commonly starting ssh-agents after reboots, which cause off-box transfers to fail.

preparatory task

Use the backup/transfer script interactively to transfer a few tables from a tmp_ DB to a remote node in a partitioned manner and recover the tables from the partitioned tarballs (using dbsrv.py on the remote node with different options).

For fast testing use options to make the backup/transfer/recover complete in seconds by controlling the partition sizes/counts.

tasks

  • work with Qiumei to change backup target to SJTU (or elsewhere), this will entail
    • add ssh config section on server identifying the remote target node
    • positioning ssh keys to allow automated scp of tarballs from server to target
    • changing cron commandline argument or envvar to point at a new target
  • setup daily target monitoring crontab that runs the monitor script,
  • maintain near continuous daily monitoring, act on monitoring notifications
  • test validity of backup system by doing a full recover of the CQDB on the target node and making comparisons against the source DB