Other api/dbsvn

Table Of Contents

Previous topic


Next topic


This Page

Daya Bay Links

Content Skeleton

dbsvn - DBI SVN Gatekeeper


Usage examples

./dbsvn.py --help
    ## full list of options and this help text 

./dbsvn.py ~/catdir -M
    ## check catalog and skip commit message test    

./dbsvn.py ~/catdir -m "test commit message  dybsvn:source:dybgaudi/trunk/CalibWritingPkg/DBUPDATE.txt@12000  "
    ## check catalog and commit message 

This script performs basic validations of SVN commits intended to lead to DB updates, it is used in two situations:

  1. On the SVN server as part of the pre-commit hook that allows/denies the commit
  2. On the client, to allow testing of an intended commit before actually attempting the commit as shown above

NB this script DOES NOT perform commits, it only verifies them

How this script fits into the workflow

cd ; svn co http://dayabay.ihep.ac.cn/svn/dybaux/catalog/tmp_offline_db 
      ## check out catalog containing the subset of manually updated tables

cd ; svn co http://dayabay.phys.ntu.edu.tw/repos/newtest/catalog/tmp_offline_db/
      ## test catalog at NTU 

./db.py offline_db rdumpcat ~/tmp_offline_db     
      ## rdumpcat current offline_db on top of the SVN checkout and look for diffs

svn diff ~/tmp_offline_db

./db.py tmp_joe_offline_db rdumpcat ~/tmp_offline_db       ## NB name switch
      ## write DBI catalog on top of working copy ~/tmp_offline_db 

svn diff ~/tmp_offline_db     
      ## see if changed files are as you expect

./dbsvn.py ~/tmp_offline_db 
      ## use this script to check the "svn diff" to see if looks like a valid DBI update

./dbsvn.py ~/tmp_offline_db -m "Updating dybsvn:source:dybgaudi/trunk/CalibWritingPkg/DBUPDATE.txt@12000 " 
      ## fails as annotation link refers to dummy path, no such package and no change to that file at that revision 

./dbsvn.py ~/tmp_offline_db -m "Annotation link  dybsvn:source:dybgaudi/trunk/Database/DybDbiTest/tests/README "
      ## check the "svn diff" and intended commit message, fails as no revision  

./dbsvn.py ~/tmp_offline_db -m "Annotation link  dybsvn:source:dybgaudi/trunk/Database/DybDbiTest/tests/README@10000 "
      ## fails as no change to that file at that revision 

./dbsvn.py ~/tmp_offline_db -m "Annotation link  dybsvn:source:dybgaudi/trunk/Database/DybDbiTest/tests/README@9716 " 
      ## succeeds 

svn ci ~/tmp/offline_db -m  "Updating dybsvn:source:dybgaudi/trunk/CalibWritingPkg/DBUPDATE.txt@12000 " 
      ## attempt the actual commit 

What is validated by dbsvn.py

  1. The commit message, eg “Updating dybsvn:source:dybgaudi/trunk/CalibWritingPkg/DBUPDATE.txt@12000 “
    1. must provide valid dybsvn reference which includes dybgaudi/trunk package path and revision number
  2. Which files (which represent tables) are changed
    1. author must have permission for these files/tables
    2. change must effect DBI file/tablepairs (payload, validity)
  3. What changes are made:
    1. must be additions/subtractions only (allowing subtractions is for revertions)
    2. note that LOCALSEQNO (a DBI bookkeeping table) is a special case

Rationale behind these validations

  1. valid DBI updates

  2. establish provenance and purpose

    1. what purpose for the update
    2. where it comes from (which revision of which code was used)
    3. precise link to producing code and documentation

Commit denial

This script is invoked on the SVN server by the pre-commit hook (shown below) if any directories changed by the commit start with “catalog/”. If this script exits normally with zero return code, the commit is allowed to proceed.

On the other hand, if this script returns a non-zero exit code, for example if an assert is tickled, then the commit is denied and stderr is returned to the failed committer.

OVERRIDE commits

Administrators (configured using -X option on the server) can use the string “OVERRIDE” in commit messages to short circuit validation. This is needed for non-standard operations, currently:

  1. adding/removing tables

A commit like the below from inside catalog will fail, assuming that the dayabay svn identity is not on the admin list:

svn --username dayabay ci -m "can dayabay use newtest OVERRIDE "

Deployment of pre-commit hook on SVN server

Only SVN repository administrators need to understand this section.

The below commands are an example of creating a bash pre-commit wrapper. After changing the TARGET and apache user identity, the commands can be used to prepare the hook. Note that the pre-commit script is invoked by the server in a bare environment, so any customizations must be propagated in.

Checkout/Update DybPython on SVN server node:

svn co http://dayabay.ihep.ac.cn/svn/dybsvn/dybgaudi/trunk/DybPython/python/DybPython  
svn up ~/DybPython

As root, copy in python code used by the pre-commit hook:

cd /home/scm/svn/dybaux/hooks/
ls -l
rm *.pyc  # tidy up
cp ~/DybPython/{dbsvn,svndiff,dbvld}.py .  
chown apache.apache {dbsvn,svndiff,dbvld}.py 

Creating the hook:

export TARGET=/home/scm/svn/dybaux/hooks/pre-commit        ## dybaux hooks 
DBSVN_XREF=/home/scm/svn/dybsvn python $HOME/DybPython/dbsvn.py HOOK  ## check the hook is customized as desired
DBSVN_XREF=/home/scm/svn/dybsvn python $HOME/DybPython/dbsvn.py HOOK  | sudo bash -c "cat - > $TARGET && chmod ugo+x $TARGET && chown apache.apache $TARGET " 
  1. DBSVN_XREF points to the dybsvn SVN repository, which is used to validate cross referencing links from dybaux to dybsvn
  2. user apache corresponds to the user which the SVN webserver process runs as
  3. note that the dbsvn.py option -c/--refcreds is not used for dybaux as local access to dybsvn repository is used (with svnlook)

Hook Deployment on server remote from dybsvn

The test deployed hook at NTU gets cross-referencing to dybsvn via svn log etc whereas, the real dybaux hook accesses dybsvn locally on the server using svnlook. Due to this different options are needed in hook deployment, specicially as are using the default DBSVN_XREF of http://dayabay.ihep.ac.cn/svn/dybsvn need to enter DBSVN_XREF_PASS:

## on SVN server node  
svn co http://dayabay.ihep.ac.cn/svn/dybsvn/dybgaudi/trunk/DybPython/python/DybPython  ## into $HOME
svn up ~/DybPython

export TARGET=/var/scm/repos/newtest/hooks/pre-commit ; export APACHE_USER=nobody.nobody                           
sudo bash -c "cp $HOME/DybPython/{dbsvn,svndiff,dbvld}.py $(dirname $TARGET)/  && chown $APACHE_USER  $(dirname $TARGET)/{dbsvn,svndiff,dbvld}.py "

DBSVN_XREF_PASS=youknowit python $HOME/DybPython/dbsvn.py HOOK  ## check the hook is customized as desired
DBSVN_XREF_PASS=youknowit python $HOME/DybPython/dbsvn.py HOOK  | sudo bash -c "cat - > $TARGET && chmod ugo+x $TARGET && chown $APACHE_USER $TARGET " 

Typical Problems with the Hook

Mainly for admins

If the precommit hook is mis-configured the likely result is that attempts to commit will hang. For example the dbsvn.py invokation in the hook script needs to have:

  1. a valid admin user (SVN identity)
  2. local filesystem repository path for the cross reference -r option

The default cross reference path is the dybsvn URL which might hang on the server as the user(root/nobody/...) that runs the SVN repository normally does not have user permissions to access sibling repository dybsvn. (have switched to non-interactive now)

A pre-commit hook testing harness is available in bash functions env:trunk/svn/svnprecommit.bash

Trac Config to limit large diff hangs

Only for admins

The large diffs representing DB updates that are stored in dybaux can cause Trac/apache to hang on attempting to browse them in Trac. To avoid this the default max_diff_bytes needs to be reduced, do this for dybaux with:

env-    ## env precursor
TRAC_INSTANCE=dybaux trac-edit

Modify down to 100000:

max_diff_bytes = 100000       # 10000000
max_diff_files = 0


class DybPython.dbsvn.DBIValidate(diff, msg, author, opts)[source]

Bases: list

Basic validation of commit that represents an intended DB update


Traverse the parsed diff hierarchy diff/delta/block/hunk to extract the validity diffs such as:

+30,"2010-09-22 12:26:59","2038-01-19 03:14:07",127,3,0,1,-1,"2010-09-22 12:26:59","2011-05-05 05:24:00"

deltas should have a single block for a valid update


Check the Vld table diff validity entries have valid times and conform to overlay versioning compliance.

Turns out not to be possible to check for overlay versioning compliance from a delta as in the case of updates with changed timestart the offset from the first timestart gets used, see #868

NB this has to run on SVN server without NuWa, and potentially with an ancient python, so hardcoded constants and conservative style are necessary


Current checks do not verify tail addition


Checks on the validity contextrange of updates, to verify:

  1. Presence of valid dates in all four DBI date slots
  2. Overlay versioning compliance, namely appropriate correspondence between TIMESTART and VERSIONDATE