Other sop/dbvalid

Table Of Contents

Previous topic

DB Table Creation

Next topic

DB Testing

This Page

Daya Bay Links

Content Skeleton

DB Validation

This section covers basic checks of the DBI integrity of updates done by the SVN pre-commit hook when attempting to commit into dybaux, Pre-commit enforced validation : DBI Gatekeeper and also generic DBI validation done on offline_db entries as part of the autorun nosetests, Generic Validation.

These generic checks of the DBI mechanics have been found to rarely fail. For the more critical and fragile testing of the meaning of updates see DB Testing.

Interactive Checking with ipython

DB Checking

Simple select on LOCALSEQNO table, provides same info as the .seqno methods below:

mysql> select * from LOCALSEQNO ;
+--------------+---------------+
| TABLENAME    | LASTUSEDSEQNO |
+--------------+---------------+
| *            |             0 |
| CalibFeeSpec |           113 |
| CalibPmtSpec |            50 |
| FeeCableMap  |             3 |
| HardwareID   |           372 |
| CableMap     |           460 |
| Reactor      |           372 |
+--------------+---------------+
7 rows in set (0.07 sec)
In [1]: from DybPython import DB

In [2]: db = DB("offline_db")          ## the DBCONF section name

In [3]: db.seqno          ## obtained from LOCALSEQNO table, providing LASTUSEDSEQNO values
Out[3]:
{'CableMap': 460,
 'CalibFeeSpec': 113,
 'CalibPmtSpec': 50,
 'FeeCableMap': 3,
 'HardwareID': 372,
 'Reactor': 372}

In [4]: db.fabseqno       ## fabricated from .allseqno with SEQNO counts
Out[4]:
{'CableMap': 460,
 'CalibFeeSpec': 111,
 'CalibPmtSpec': 29,
 'FeeCableMap': 3,
 'HardwareID': 372,
 'Reactor': 372}

     ## legacy tables : CalibFeeSpec CalibPmtSpec have know SEQNO irregularities
     ## .. all other tables must be consistent

In [5]: db.allseqno        ## obtained via SQL queries on validity tables
Out[5]:
{'CableMap': [1,
              2,
              3,
              4,
              5,
              6,
              7,
  ... too long to show ...
             370,
             371,
             372]}

In [6]: db.allseqno.keys()
Out[6]:
['Reactor',
 'CalibFeeSpec',
 'HardwareID',
 'CalibPmtSpec',
 'FeeCableMap',
 'CableMap']

Ascii Catalog Checking

Simple cat of LOCALSEQNO.csv table, provides same info as the .seqno methods below:

[blyth@belle7 ~]$ svnversion ~/dybaux/catalog/tmp_offline_db
4974
[blyth@belle7 ~]$ cat ~/dybaux/catalog/tmp_offline_db/LOCALSEQNO/LOCALSEQNO.csv
TABLENAME char(64),LASTUSEDSEQNO int(11),PRIMARY KEY (TABLENAME)
"*",0
"CableMap",460
"CalibFeeSpec",113
"CalibPmtSpec",50
"CoordinateAd",1
"CoordinateReactor",1
"FeeCableMap",3
"HardwareID",372
"Reactor",372
In [8]: from DybPython.asciicat import AsciiCat

In [9]: cat = AsciiCat("~/dybaux/catalog/tmp_offline_db")      ## reads all entries into memory

In [10]: cat.seqno
Out[10]:
{'CableMap': 460,
 'CalibFeeSpec': 113,
 'CalibPmtSpec': 50,
 'CoordinateAd': 1,
 'CoordinateReactor': 1,
 'FeeCableMap': 3,
 'HardwareID': 372,
 'Reactor': 372}

In [11]: cat.fabseqno
Out[11]:
{'CableMap': 460,
 'CalibFeeSpec': 111,
 'CalibPmtSpec': 29,
 'CoordinateAd': 1,
 'CoordinateReactor': 1,
 'FeeCableMap': 3,
 'HardwareID': 372,
 'Reactor': 372}

Workflow Checks

At each step of the workflow the tools, such as db.py and dbaux.py perform DBI integrity checks, checking things like:

  1. SEQNO consistency between LOCALSEQNO table and actual payload and validity tables
  2. table existance
  3. payload/validity consistency

Furthermore when performing operations such as rloadcat or rcmpcat that involve both an ascii catalog and the DB the differences present in the ascii catalog are checked to be valid DBI updates with the expected tables and SEQNO values.

Generic Validation

Packages containing tests that focus on DB access/function testing

package notes on tests
dybgaudi:DybPython operation of db.py and dbsvn.py, and workflow steps
dybgaudi:Database/DybDbi simple readonly access to a variety of tables
dybgaudi:Database/DbiTest DBI supplied C++ tests of most DBI functionality
dybgaudi:Database/DybDbiTest DybDbi equivalents of all relevant DbiTest
dybgaudi:Database/DbiValidate generic tests of DBI table structure : integers inside enums, PK etc..