Other api/dbaux

Table Of Contents

Previous topic


Next topic


This Page

Daya Bay Links

Content Skeleton



$Id: dbaux.py 17856 2012-08-22 11:40:42Z blyth $

Performs actions based on working copy at various revision points.

action notes
ls lists commit times/messages
rcmpcat compare ascii catalog with DB
rloadcat load ascii catalog into DB

Usage examples:

./dbaux.py ls      4913
./dbaux.py ls      4913:4914
./dbaux.py ls      4913:4932    
./dbaux.py ls      4913:4914   --author bv 

./dbaux.py --workingcopy ~/mgr/tmp_offline_db --baseurl file:///tmp/repos/catalog ls 2:39 
     # using non default workingcopy path and baseurl 
     #      NB baseurl must be the base of the repository 
     #    TODO: avoid duplication by extracting baseurl from the working copy, or at least assert on consistency  

./dbaux.py    rcmpcat 4913   
./dbaux.py    rcmpcat 4913:4932   
./dbaux.py -r rcmpcat 4913   

./dbaux.py         rloadcat 4913   
./dbaux.py --reset rloadcat 4913   ## -r/--reset deletes SVN working copy before `svn up` 

To select non-contiguous revisions use -a/–author to pick just that authors commits within the revision range. Test with ls.

While testing in “tmp_offline_db” return to starting point with:

./db.py offline_db dump ~/offline_db.sql 
./db.py tmp_offline_db load ~/offline_db.sql 

While performing test loads into tmp_offline_db, multiple ascii catalog revisions can be loaded into DB with a single command:

./dbaux.py -c -r rloadcat 4913:4932      
       ## -c/--cachesvnlog   improves rerun speed while testing 
       ## -r/--reset         starts from a clean revision each time, 
                             ignoring fastforward changes done by **rloadcat**

./dbaux.py -c -r rloadcat 4913:4932     
       ## a rerun will fail at the first revision and will do nothing 
       ## as the DB is detected to be ahead of the catalog

However when performing the real definitive updates into offline_db it is preferable to do things a bit differently:

./dbaux.py -c -r --dbconf offline_db  rloadcat 4913:4932 --logpath dbaux-rloadcat-4913-4932.log

       ## -s/--sleep 3 seconds sleep between revisions, avoid fastforward insert times with the same UTC second
       ##  --dbconf offline_db      target ~/.my.cnf section 

Checks after performing rloadcat(s)

Each rloadcat modifies the catalog inplace, changing the INSERTDATE times. However as are operating beneath the dybaux trunk it is not straightforward to commit these changes and record them as they are made. Instead propagate them from the database into the catalog by an rdumpcat following updates. This is also a further check of a sequence of rloadcat.

Dump the updated DB into the catalog with:

db.py offline_db rdumpcat ~/dybaux/catalog/tmp_offline_db 
db.py tmp_offline_db rdumpcat ~/dybaux/catalog/tmp_offline_db    ## when testing 

Then check the status of the catalog, only expected tables .csv should be changed:

svn st  ~/dybaux/catalog/tmp_offline_db

M       /home/blyth/dybaux/catalog/tmp_offline_db/CableMap/CableMapVld.csv     
M       /home/blyth/dybaux/catalog/tmp_offline_db/HardwareID/HardwareIDVld.csv

           ## should only be INSERTDATE changes, 
           ## the new times should be UTC now times spread out over the 
           ## rloadcat operations    
M       /home/blyth/dybaux/catalog/tmp_offline_db/tmp_offline_db.cat         

           ##  minor annoyance : changed order of entries in .cat 
           ##  ... to be fixed by standardizing order with sorted  TABLENAME 

Following a sequence of definitive commits into offline_db do an OVERRIDE commit into dybaux mentioning the revision range and author in the commit message. For example:

svn ci -m "fastforward updates following offline_db rloadcat of bv r4913:r4932 OVERRIDE  " ~/dybaux/catalog/tmp_offline_db

Logfile Checks

Using the --logpath <path> option writes a log that is nearly the same as the console output. Checks to make on the logfile:

Check all commits are covered:

grep commit dbaux-rloadcat-4913-4932.log

Look at the SEQNO being loaded, verify no gaps and that the starting SEQNO is where expected:

egrep "CableMap.*new SEQNO" dbaux-rloadcat-4913-4932.log
egrep "HardwareID.*new SEQNO" dbaux-rloadcat-4913-4932.log

Examine fastforward times:

grep fastforward dbaux-rloadcat-4913-4932.log

Manual Checks

Before loading a sequence of commits sample the ascii catalog at various revisions with eg:

svn up -r <revision> ~/dybaux/catalog/tmp_offline_db
cat ~/dybaux/catalog/tmp_offline_db/LOCALSQNO/LOCALSEQNO.csv

Verify that the LASTUSEDSEQNO value changes are as expected compared to:

mysql> select * from LOCALSEQNO ; 
| *            |             0 | 
| CalibFeeSpec |           113 | 
| CalibPmtSpec |            29 | 
| FeeCableMap  |             3 | 
| CableMap     |           440 |    
| HardwareID   |           358 | 
6 rows in set (0.00 sec)

Expectations are:

  1. incremental only ... no going back in SEQNO
  2. no SEQNO gaps

The tools perform many checks and comparisons, but manual checks are advisable also, eg:

mysql> select distinct(INSERTDATE) from CableMapVld  ;
mysql> select distinct(INSERTDATE) from HardwareIDVld  
mysql> select distinct(SEQNO) from  CableMap ;
mysql> select distinct(SEQNO) from  CableMapVld ;

rloadcat checks in various situations

Starting with r4913 and r4914 already loaded, try some operations.

  1. rloadcat r4913 again:

    ./dbaux.py rloadcat 4913 
    AssertionError: ('ERROR LASTUSEDSEQNO in target exceeds that in ascii cat HardwareID ', 42, 58)
       ## the DB is ahead of the catalog ... hence the error
  2. rloadcat r4914 again:

    ./dbaux.py rloadcat 4913 
    WARNING:DybPython.db:no updates (new tables or new SEQNO) are detected  
        ## DB and catalog are level pegging ... hence "no updates" warning


  1. same process rcmpcat checking following an rloadcat fails as has outdated idea of DB content despite cache wiping on rloadcat. A subsequent rcmpcat in a new process succeeds. .. was avoided by creating a fresh DB instance after loads, forcing re-accessing to Database


class DybPython.dbaux.Aux(args)[source]

Bases: object


Pull up a new DB instance


parse/wrap output of svn info –xml ... caution rerun on each access


Lists the revisions, author, time, commit message


Loops over revisions:

  1. svn up -r the working copy
  2. runs rcmpcat comparing the ascii catalog with DB

Loops over revisions

  1. svn up -r the working copy
  2. runs rcmpcat to verify there are some updates to be loaded
  3. invokes rloadcat loading ascii catalog into DB
  4. runs rcmpcat agsin to verify load is complete

NB no confirmation is requested, thus before doing this perform an rcmpcat to verify expected updates

Rerunning an rloadcat

./dbaux.py rloadcat 4913   ## 1st time OK
./dbaux.py rloadcat 4913   ## 2nd time was giving conflicts ... now fails with unclean error
./dbaux.py --reset rloadcat 4913   ## blow away conflicts by deletion of working copy before "svn up"

How to fix ?

  1. When testing “svn revert” the changed validity tables throwing away the fastforward times ? via parsing “svn status”

parse/wrap output of svn status –xml ... caution rerun on each access

svnup_(rev, reset=False, force=False)[source]
  • rev – revision number to bring working copy directory to
  • reset – remove the directory first, wiping away uncommitted changes/conflicts

Aug 22, 2012 moved to checkout and revert rather than priot just update as this was failing with --reset due to lack of the working copy directory, resulting in svn up skipping and subsequent assertions. The idea is to step thru pristine revisions, one by one:

svn co -r 5292 http://dayabay.ihep.ac.cn/svn/dybaux/catalog/tmp_offline_db ~/dybaux/catalog/tmp_offline_db
svn revert ~/dybaux/catalog/tmp_offline_db