Other api/dybdbi/vld

Table Of Contents

Previous topic


Next topic


This Page

Daya Bay Links

Content Skeleton


In –transfix mode copies all DBI tables from tmp_offline_db into fix_offline_db with VERSIONDATE changed to timestart floored scheme.

As yet no collision checks.

Try to back-predict versiondate for all validities in all Vld tables


./versiondate.py --help
./versiondate.py CalibPmtHighGain
./versiondate.py --transfix 
./versiondate.py --transfix             -l DEBUG
./versiondate.py --transfix HardwareID  -l DEBUG   
./versiondate.py --transfix CableMap    -l DEBUG   
./versiondate.py --transfix CalibPmtSpec -l DEBUG   

To confirm no change when timestart flooring is off:

echo select \* from tmp_offline_db.HardwareIDVld | mysql -t > tmp_HardwareID.txt 
echo select \* from fix_offline_db.HardwareIDVld | mysql -t > fix_HardwareID.txt
diff tmp_HardwareID.txt fix_HardwareID.txt
class DybDbi.vld.versiondate.VFS(field, value)[source]

Bases: list

Convenience class to format all column names in validity table and allow easy swap out of single fields


For all DBI tables in source tmp DB invoke the check_versiondate_tab


Check if the versiondate of all validities matches that divined with QueryOverlayVersionDate, via SEQNO condition to allow backdating

Probably this is just confirming that overlay versioning was used

  1. Establish coordinates of source tmp and target fix databases with safety asserts
  2. drop and recreate the target DB fix
  3. label DB instances with dbno for DBI usage

Copies all DBI tables from tmp into fix changing the VERSIONDATE to conform to TimeStartFlooredVersionDate scheme.

Hmm currently the VERSIONDATE collision avoidance implemented in the DBI writer does not come into play here : IT NEEDS TO BE SPOOFED HERE

DybDbi.vld.versiondate.transfix_tab(tmp, fix, tab, localseqno=False)[source]

SEQNO by SEQNO transfers table entries from tmp to fix databases effectively replaying table history as it grows in the fix DB. Allowing changes to the VERSIONDATE scheme to be tested.

  • tmp – source DybPython.DB instance
  • fix – target DybPython.DB instance
  • tab – payload table name


  1. create payload and validity tables using DBI, note there is no need to drop the tables first as started by dropping the fix DB
  2. loops over validity entries of tab in tmp DB in SEQNO asc order

For each vld entry calls qovd_transfix prior to transferring the vld from the tmp to fix DB allowing the VERSIONDATE to be modified to use a different scheme.

The backdated qovd comparison with versiondate could be used to detect early entries that did not use overlay versioning:

qovd.GetSec() != versiondate.GetSec()

Note, formerly used application of an SQL condition to effect a timstart floor, inside:

qovd = qovd_transfix( kls, vrec, fix.dbno ).AsString("s")

But following C++ additions can now directly use QueryOverlayVersionDate and set the additional parameter fTimeStartFlooredVersionDate


Compare validity lookup tables between different DB and option variations

Due to usage of converter.tabular.TabularData for presentation of the tables this must be run from the docs virtual python:

~/rst/bin/python vlut.py 
~/v/docs/bin/python vlut.py 


  1. top level index.rst of tables requires manual editing
  2. running single context, has habit of messing up all context indices. Workaround is rerunning the --ctx ALL

Production Run From Scratch

Clean start:

cd vld
./versiondate.py                ##  transfixion of tmp_offline_db creating fix_offline_db
rm -rf /tmp/blyth/dbiscan       ## clean start 
time ~/rst/bin/python ./vlut.py 

Full traverse takes:

real    103m41.969s
user    92m13.993s
sys     4m54.889s

Payload Digest Based Comparisons

Goal is to compare very different DB for content:

  1. tmp_offline_db with CableMap,HardwareID duplications eliminated and written with the G*Fix classes
  2. tmp_copy_db copy of offline_db


  1. payload digests should allow comparison without SEQNO correspondence
  2. TIMESTARTs should align


  1. insertdates(table update history) does not align forcing to manually

    1. opts['_insertdate_aligned'] = False
    2. common junctures might also be defined to allow more that just the last comparison

Presenting the LUTs with Sphinx


Unexplained difference between tmp_offline_db without extra ordering (formerly though to be implicit VERSIONDATE desc, SEQNO asc) and SEQNO asc

[blyth@belle7 aggno-1_simflag2_site1_subsite2_task0]$ diff ./tmp_offline_db/vlut.rst ./orderingSEQNOasc_tmp_offline_db/vlut.rst

Promising no diffs between tmp_offline_db with added “SEQNO asc” validity ordering and the fix_offline_db (for the fixed the SEQNO diddling makes no difference as all same ... at least in ctxs covered so far). This means that enforcing extra SEQNO asc validity ordering on an horrible degenerate mess of duplicated VERSIONDATEs in tmp_offline_db succeeds to give the same VLUT as the transfixed one ... with the careful timestart floored version date with no degeneracy.

This is understandable as the validity ordering becomes “VERSIONDATE desc, SEQNO asc” so the larger SEQNO of degenerate sets wins. That seems like it should be mostly correct ... are there any edge cases ?

If this pans out to all ctxs then the eagle has landed

Nginx Hookup

Hook up to nginx:

cd `nginx-htdocs`
sudo ln -s $SITEROOT/../users/$USER/dbiscan/sphinx/_build/dirhtml dbiscan 

Add location with nginx-edit:

location /dbiscan {
     autoindex on ;
     autoindex_exact_size off ;
     autoindex_localtime on ;

After restart of nginx can peruse the tables:

Machinery Issues

  1. rerun bug, not updating index have to manually: rm  /tmp/blyth/dbidigest/sphinx/CableMap/index.rst

For all tables common to the databases traverse all contexts in common, writing the vlut rst files and making comparisons.

Loads persisted scans created by DybPython.vlut.Scan and presents as LUTs (look up tables) in rst table format

Parameters:vsVlutSpec instance



time ~/rst/bin/python vsmry.py $SITEROOT/../users/$USER/dbiscan/sphinx/CableMap/smry.pc

Smry are dicts of stat dicts, keyed by the VLUT relative path, eg:

CableMap/aggno-1_simflag2_site4_subsite5_task0/tmp_offline_db_cf_fix_offline_db/vlutorderingSEQNOdesc.rst {'ndif': 15}
CableMap/aggno-1_simflag2_site4_subsite6_task0/tmp_offline_db_cf_fix_offline_db/vlut.rst {'ndif': 14}
CableMap/aggno-1_simflag2_site4_subsite6_task0/tmp_offline_db_cf_fix_offline_db/vlutorderingSEQNOasc.rst {'ndif': 0}
CableMap/aggno-1_simflag2_site4_subsite6_task0/tmp_offline_db_cf_fix_offline_db/vlutorderingSEQNOdesc.rst {'ndif': 15}
CableMap/aggno-1_simflag2_site4_subsite7_task0/tmp_offline_db_cf_fix_offline_db/vlut.rst {'ndif': 0}
CableMap/aggno-1_simflag2_site4_subsite7_task0/tmp_offline_db_cf_fix_offline_db/vlutorderingSEQNOasc.rst {'ndif': 0}
CableMap/aggno-1_simflag2_site4_subsite7_task0/tmp_offline_db_cf_fix_offline_db/vlutorderingSEQNOdesc.rst {'ndif': 3}

Down to handful of 3 pathological ctxs in 3 tables (Demo doesnt count) for whom the fix is not doing the biz:

    CRITICAL:__main__:checking smry beneath /tmp/blyth/dbiscan/sphinx levels ['ctx'] 

    CRITICAL:__main__:CableMap             aggno-1_simflag2_site32_subsite1_task0   tmp_offline_db_cf_fix_offline_db vlutorderingSEQNOasc.rst       23    ctx    EXCEPTIONAL NON-ZERO NDIF ???  
    CRITICAL:__main__: http://belle7.nuu.edu.tw/dbiscan/CableMap/aggno-1_simflag2_site32_subsite1_task0/ 

    CRITICAL:__main__:CalibFeeSpec         aggno-1_simflag1_site32_subsite1_task0   tmp_offline_db_cf_fix_offline_db vlutorderingSEQNOasc.rst       94    ctx    EXCEPTIONAL NON-ZERO NDIF ???  
    CRITICAL:__main__: http://belle7.nuu.edu.tw/dbiscan/CalibFeeSpec/aggno-1_simflag1_site32_subsite1_task0/ 
             ## this  CalibFeeSpec ctx is the only CalibFeeSpec ctx (suggesting junk?)

    CRITICAL:__main__:CalibPmtSpec         aggno-1_simflag1_site32_subsite2_task0   tmp_offline_db_cf_fix_offline_db vlutorderingSEQNOasc.rst       305   ctx    EXCEPTIONAL NON-ZERO NDIF ???  
    CRITICAL:__main__: http://belle7.nuu.edu.tw/dbiscan/CalibPmtSpec/aggno-1_simflag1_site32_subsite2_task0/ 

             ## no order flip discrep within tmp_ and fix_ ... but there is between em ?

    CRITICAL:__main__:Demo                 aggno-1_simflag1_site127_subsite0_task0  tmp_offline_db_cf_fix_offline_db vlutorderingSEQNOasc.rst       15    ctx    EXCEPTIONAL NON-ZERO NDIF ???  
    CRITICAL:__main__: http://belle7.nuu.edu.tw/dbiscan/Demo/aggno-1_simflag1_site127_subsite0_task0/ 

select * from CableMapVld where SIMMASK=2 and SITEMASK=32 and TASK=0 ; 
select * from CalibFeeSpecVld where SIMMASK=1 and SITEMASK=32 and TASK=0 ;     
select * from CalibPmtSpecVld where SIMMASK=1 and SITEMASK=32 and TASK=0 ; 



To verify are seeing the appropriate number of distinct ctxs:

mysql> select distinct(CONCAT(SITEMASK,":",SIMMASK,":",SUBSITE,":",TASK,":",AGGREGATENO)) from CalibPmtSpecVld ;
| 32:1:1:0:-1                                                         | 
| 32:1:2:0:-1                                                         | 
| 127:1:0:0:-1                                                        | 
| 127:2:0:0:-1                                                        | 
| 1:2:1:0:-1                                                          | 
| 1:1:5:0:-1                                                          | 
| 1:1:6:0:-1                                                          | 
| 1:1:1:0:-1                                                          | 
| 1:1:2:0:-1                                                          | 
9 rows in set (0.00 sec)

Dump tables for each cfdir expressing summary dif info in cfdir/tn/name hierarchy:

tn vlut.rst vlutorderingSEQNOasc.rst vlutorderingSEQNOdesc.rst
CableMap 16/35 1/35 19/35
CalibFeeSpec 1/1 1/1 1/1
CalibPmtHighGain 0/6 0/6 0/6
CalibPmtPedBias 0/1 0/1 0/1
DybDbi.vld.vsmry.dump_difctx(cfdir='tmp_offline_db_cf_fix_offline_db', name='vlut.rst')[source]

For each tn within cfdir/name Dump tables listting ctx with differnces and their ranges of difference in INSERTDATE,TIMESTART space

DybDbi.vld.vsmry.grow_cf(tn, ctx, cfdir, name, stat)[source]
  • tn – table name
  • ctx – context
  • cfdir – comparison dir
  • name – eg vlut.rst
  • stat – statistics and range dict

Collect lists of all ctx and ctx with ndif > 0 into cf dict keyed into changed hierarchy cfdir/tn/name


Currently needs manual hookup to global index.rst

DybDbi.vld.vsmry.squeeze_tab(dohd, cols, kn)[source]
  • dohd – dict of hashdicts
  • cols – presentation ordering of keys in the hashdicts
  • kn – name of the dohd key that becomes added column for gang up referencing

Suppress duplicate value entries in a table by ganging