Other aop/dbimonitor

Table Of Contents

Previous topic

Monitoring

Next topic

Env Repository : Admin Infrastructure Sources

This Page

Daya Bay Links

Content Skeleton

DbiMonitor package : cron invoked nosetests

The DbiMonitor package contains dybgaudi:Database/DbiMonitor/tests that check the updates being made to various databases, and send notification mails when update expectations are not met.

DbiMonitor.tests.test_dcs

Checking update age of tables in IHEP DCS mirror

The expectations for the table ages are set in dybgaudi:Database/DbiMonitor/tests/test_dcs.py

Usage:

DBCONF=womble nosetests -v test_dcs.py    # tests are skipped if DBConf section is not available
VERBOSE=1 nosetests -v test_dcs.py        # default is to only list OVERAGE tables, to see all use VERBOSE envvar 
python test_dcs.py                        # runs the test and presents the table of ages
nosetests -v test_dcs.py                  # runs test, only get table presentation when OVERAGE tables are found

Update Expectations

See dybgaudi:Database/DbiMonitor/tests/test_dcs.py for the current expectations. Update expectations for each table or group of tables specified by name regular expressions are defined by the maxage list of tuples structure.

class DcsUpdates(Updates):
    date_time = 'date_time' 
    abandoned = timedelta(weeks=19)
    maxage = [
               (re.compile('^AD\d_Calibration$'), timedelta(hours=1)),
               (re.compile('^\S*_ADCoverGas$'), timedelta(hours=1)), 
               (re.compile('^FARS_RPC_GAS_10[12]$'),timedelta(days=5)),
               (re.compile('^\S*_RadonMonitor$'), timedelta(days=5)), 
               (re.compile('^\S*_MUCAL$'), timedelta(days=10)),
               (re.compile('^dybalar\S*$'), timedelta(days=10)),
               (re.compile('.*'), timedelta(minutes=30)),
             ] 

DB connection config

The script uses default DBCONF section of ihep_dcs requiring config parameters within your ~/.my.cnf of form:

[ihep_dcs]
host = 202.122.37.89
database = dybdcsdb
user = dayabay    
password = %(youknowit)s

python module running

As DbiMonitor requirements does apply_pattern install_dybtests the tests are installed, allowing the following running technique:

python -m DbiMonitor.tests.test_dcs

OR with more verbosity:

VERBOSE=1 python -m DbiMonitor.tests.test_dcs

dybinst level running

./dybinst -l /dev/stdout trunk tests dbimonitor:tests/test_dcs.py 

crontab automation

This uses a feature of dybinst where in the event of test failures notification emails are sent to addresses in the MAILTO which can be a space delimited list of email addesses

SHELL=/bin/bash
CRONLOG_DIR=/home/blyth/cronlog
PATH=/home/blyth/env/bin:/usr/bin:/bin
DYBINST_DIR=/data1/env/local/dyb
#
45 20 * * * ( cd $DYBINST_DIR ; MAILTO="blyth@hep1.phys.ntu.edu.tw joe@example.com" ./dybinst -L -l $CRONLOG_DIR/test_dcs.log trunk tests dbimonitor:tests/test_dcs.py ) > $CRONLOG_DIR/test_dcs_.log 2>&1

nosetests and logging issue

When run as nosetests getting DEBUG level output despite INFO setting in setup. This issue is apparently fixed in a future version of nose:

* https://github.com/nose-devs/nose/issues/21 
* https://github.com/nose-devs/nose/pull/493

Meanwhile just comment the DEBUG logging.

DbiMonitor.tests.test_offline

Checking update age of tables in offline_db

For a more detailed description see test_dcs.py which uses the same Updates base class.

Usage:

DBCONF=womble nosetests -v test_offline.py    # tests are skipped if DBConf section is not available
VERBOSE=1 nosetests -v test_offline.py        # default is to only list OVERAGE tables, to see all use VERBOSE envvar 
python test_offline.py                        # runs the test and presents the table of ages
nosetests -v test_offline.py                  # runs test, only get table presentation when OVERAGE tables are found

The default DB checked is offline_db, that can be overridden with the DBCONF envvar pointing to a section in your ~/.my.cnf. The default is to only report problems, for more detailed summaries use the VERBOSE envvar.

[blyth@belle7 tests]$ DBCONF=offline_db1 python test_offline.py
INFO:updates:reading from DB offline_db1 
WARNING:updates:missing fields prevents age check for table DcsPmtHvVld : {'msg': 'MISSKEY', 'N': 94347L} 
DBCONF offline_db1 ie offline_db [dayabay@dybdb1.ihep.ac.cn]
table                                    count        last                 look                  : age                            maxage                         msg         
[blyth@belle7 tests]$ DBCONF=offline_db1 VERBOSE=1 python test_offline.py
INFO:updates:reading from DB offline_db1 
WARNING:updates:missing fields prevents age check for table DcsPmtHvVld : {'msg': 'MISSKEY', 'N': 94347L} 
DBCONF offline_db1 ie offline_db [dayabay@dybdb1.ihep.ac.cn]
table                                    count        last                 look                  : age                            maxage                         msg                            
DcsPmtHvVld                              94347        -                    -                     : -                              -                              MISSKEY                        
DaqRawDataFileInfoVld                    335485       2013-07-29 15:18:46  2013-07-29 15:29:23   : 0:10:37                        3 days, 0:00:00                OK                             
DcsAdLidSensorVld                        111592       2013-07-29 14:46:11  2013-07-29 15:29:23   : 0:43:12                        364 days, 0:00:00              OK                             
DcsAdTempVld                             104439       2013-07-29 14:39:30  2013-07-29 15:29:23   : 0:49:53                        364 days, 0:00:00              OK                             
DaqRunInfoVld                            38572        2013-07-29 09:57:41  2013-07-29 15:29:23   : 5:31:42                        3 days, 0:00:00                OK                             
EnergyReconVld                           2919         2013-07-29 03:24:14  2013-07-29 15:29:24   : 12:05:10                       364 days, 0:00:00              OK                             
DcsAdWpHvVld                             7877         2013-07-28 13:51:02  2013-07-29 15:29:24   : 1 day, 1:38:22                 364 days, 0:00:00              OK                             
DaqCalibRunInfoVld                       43584        2013-07-26 11:38:56  2013-07-29 15:29:23   : 3 days, 3:50:27                364 days, 0:00:00              OK                             
CalibPmtFineGainVld                      18192        2013-06-25 11:55:41  2013-07-29 15:29:22   : 34 days, 3:33:41               364 days, 0:00:00              OK                             
ReactorVld                               1152         2013-03-21 02:41:19  2013-07-29 15:29:24   : 130 days, 12:48:05             364 days, 0:00:00              OK                             
McsPosVld                                1755         2012-10-08 14:53:00  2013-07-29 15:29:24   : 294 days, 0:36:24              364 days, 0:00:00              OK                             
CalibPmtHighGainVld                      1268         2012-04-26 03:10:48  2013-07-29 15:29:22   : 459 days, 12:18:34             364 days, 0:00:00              ABANDONED                      

Technical issue regards logging and nosetests

  1. nosetests and logging not working together, getting DEBUG level despite the INFO setting in setup