/search.css" rel="stylesheet" type="text/css"/> /search.js">
| Classes | Job Modules | Data Objects | Services | Algorithms | Tools | Packages | Directories | Tracs |

In This Package:

dbaux.py
Go to the documentation of this file.
00001 #!/usr/bin/env python
00002 """
00003 $Id: dbaux.py 17856 2012-08-22 11:40:42Z blyth $
00004 
00005 Performs actions based on working copy at various revision points.
00006 
00007   ============  ===================================
00008     action         notes
00009   ============  ===================================
00010     ls             lists commit times/messages
00011     rcmpcat        compare ascii catalog with DB
00012     rloadcat       load ascii catalog into DB
00013   ============  ===================================
00014 
00015 Usage examples::
00016 
00017    ./dbaux.py ls      4913
00018    ./dbaux.py ls      4913:4914
00019    ./dbaux.py ls      4913:4932    
00020    ./dbaux.py ls      4913:4914   --author bv 
00021 
00022    ./dbaux.py --workingcopy ~/mgr/tmp_offline_db --baseurl file:///tmp/repos/catalog ls 2:39 
00023         #
00024         # using non default workingcopy path and baseurl 
00025         #
00026         #      NB baseurl must be the base of the repository 
00027         #    TODO: avoid duplication by extracting baseurl from the working copy, or at least assert on consistency  
00028         #
00029 
00030    ./dbaux.py    rcmpcat 4913   
00031    ./dbaux.py    rcmpcat 4913:4932   
00032    ./dbaux.py -r rcmpcat 4913   
00033 
00034    ./dbaux.py         rloadcat 4913   
00035    ./dbaux.py --reset rloadcat 4913   ## -r/--reset deletes SVN working copy before `svn up` 
00036 
00037 
00038 To select non-contiguous revisions use `-a/--author` to pick just that 
00039 authors commits within the revision range.  Test with `ls`.
00040 
00041 While testing in "tmp_offline_db" return to starting point with::
00042 
00043     ./db.py offline_db dump ~/offline_db.sql 
00044     ./db.py tmp_offline_db load ~/offline_db.sql 
00045 
00046 
00047 While performing test loads into `tmp_offline_db`, multiple 
00048 ascii catalog revisions can be loaded into DB with a single command::
00049 
00050     ./dbaux.py -c -r rloadcat 4913:4932      
00051            ## -c/--cachesvnlog   improves rerun speed while testing 
00052            ## -r/--reset         starts from a clean revision each time, 
00053                                  ignoring fastforward changes done by **rloadcat**
00054 
00055     ./dbaux.py -c -r rloadcat 4913:4932     
00056            ## a rerun will fail at the first revision and will do nothing 
00057            ## as the DB is detected to be ahead of the catalog
00058 
00059 However when performing the real definitive updates into `offline_db` it is 
00060 preferable to do things a bit differently::
00061 
00062     ./dbaux.py -c -r --dbconf offline_db  rloadcat 4913:4932 --logpath dbaux-rloadcat-4913-4932.log
00063 
00064            ## -s/--sleep 3 seconds sleep between revisions, avoid fastforward insert times with the same UTC second
00065            ##  --dbconf offline_db      target ~/.my.cnf section 
00066 
00067 
00068 Checks after performing **rloadcat(s)** 
00069 -----------------------------------------
00070 
00071 Each **rloadcat** modifies the catalog inplace, changing the INSERTDATE times. 
00072 However as are operating beneath the dybaux trunk it is not straightforward 
00073 to commit these changes and record them as they are made.  Instead propagate 
00074 them from the database into the catalog by an **rdumpcat** following updates.
00075 This is also a further check of a sequence of **rloadcat**.
00076 
00077 Dump the updated DB into the catalog with::
00078 
00079     db.py offline_db rdumpcat ~/dybaux/catalog/tmp_offline_db 
00080     db.py tmp_offline_db rdumpcat ~/dybaux/catalog/tmp_offline_db    ## when testing 
00081 
00082 Then check the status of the catalog, only expected tables .csv should be changed::
00083 
00084     svn st  ~/dybaux/catalog/tmp_offline_db
00085  
00086     M       /home/blyth/dybaux/catalog/tmp_offline_db/CableMap/CableMapVld.csv     
00087     M       /home/blyth/dybaux/catalog/tmp_offline_db/HardwareID/HardwareIDVld.csv
00088 
00089                ## should only be INSERTDATE changes, 
00090                ## the new times should be UTC now times spread out over the 
00091                ## rloadcat operations    
00092              
00093     M       /home/blyth/dybaux/catalog/tmp_offline_db/tmp_offline_db.cat         
00094 
00095                ##  minor annoyance : changed order of entries in .cat 
00096                ##  ... to be fixed by standardizing order with sorted  TABLENAME 
00097 
00098 Following a sequence of definitive commits into `offline_db` do an OVERRIDE commit 
00099 into dybaux mentioning the revision range and author in the commit message. For example::
00100 
00101     svn ci -m "fastforward updates following offline_db rloadcat of bv r4913:r4932 OVERRIDE  " ~/dybaux/catalog/tmp_offline_db
00102 
00103 
00104 
00105 Logfile Checks
00106 ---------------
00107 
00108 Using the ``--logpath <path>`` option writes a log that is nearly the same as the console output.
00109 Checks to make on the logfile:
00110 
00111 Check all commits are covered::
00112  
00113    grep commit dbaux-rloadcat-4913-4932.log
00114 
00115 
00116 Look at the `SEQNO` being loaded, verify no gaps and that the starting `SEQNO` is where expected::
00117 
00118    egrep "CableMap.*new SEQNO" dbaux-rloadcat-4913-4932.log
00119    egrep "HardwareID.*new SEQNO" dbaux-rloadcat-4913-4932.log
00120 
00121 Examine fastforward times::
00122 
00123    grep fastforward dbaux-rloadcat-4913-4932.log
00124 
00125 
00126 Manual Checks
00127 --------------
00128 
00129 Before loading a sequence of commits sample the ascii catalog at various revisions with eg::
00130 
00131     svn up -r <revision> ~/dybaux/catalog/tmp_offline_db
00132     cat ~/dybaux/catalog/tmp_offline_db/LOCALSQNO/LOCALSEQNO.csv
00133 
00134 Verify that the ``LASTUSEDSEQNO`` value changes are as expected compared to:: 
00135 
00136         mysql> select * from LOCALSEQNO ; 
00137         +--------------+---------------+
00138         | TABLENAME    | LASTUSEDSEQNO |
00139         +--------------+---------------+
00140         | *            |             0 | 
00141         | CalibFeeSpec |           113 | 
00142         | CalibPmtSpec |            29 | 
00143         | FeeCableMap  |             3 | 
00144         | CableMap     |           440 |    
00145         | HardwareID   |           358 | 
00146         +--------------+---------------+
00147         6 rows in set (0.00 sec)
00148 
00149 Expectations are:
00150 
00151 #. incremental only ... no going back in `SEQNO`
00152 #. no `SEQNO` gaps 
00153 
00154 
00155 
00156 
00157 The tools perform many checks and comparisons, but manual checks are advisable also, eg::
00158 
00159         mysql> select distinct(INSERTDATE) from CableMapVld  ;
00160         mysql> select distinct(INSERTDATE) from HardwareIDVld  
00161         mysql> select distinct(SEQNO) from  CableMap ;
00162         mysql> select distinct(SEQNO) from  CableMapVld ;
00163 
00164 
00165 rloadcat checks in various situations
00166 ---------------------------------------
00167 
00168 Starting with r4913 and r4914 already loaded, try some operations.
00169 
00170 a) rloadcat r4913 again::
00171 
00172     ./dbaux.py rloadcat 4913 
00173     ... 
00174     AssertionError: ('ERROR LASTUSEDSEQNO in target exceeds that in ascii cat HardwareID ', 42, 58)
00175        ## the DB is ahead of the catalog ... hence the error
00176 
00177 b) rloadcat r4914 again::
00178 
00179     ./dbaux.py rloadcat 4913 
00180     ..
00181     WARNING:DybPython.db:no updates (new tables or new SEQNO) are detected  
00182         ## DB and catalog are level pegging ... hence "no updates" warning
00183 
00184 
00185 AVOIDED ISSUES
00186 ---------------
00187 
00188 #. same process rcmpcat checking following an rloadcat fails as has outdated idea of DB 
00189    content despite cache wiping on rloadcat. A subsequent rcmpcat in a new process succeeds. 
00190    .. was avoided by creating a fresh DB instance after loads, forcing re-accessing to Database 
00191 
00192 
00193 """
00194 import os,sys,logging,argparse,shutil,time
00195 sys.path.insert(0, os.path.expandvars("$SITEROOT/../installation/trunk/dybinst/scripts"))
00196 from svnlog import SVNLog, Info, Status
00197 log = logging.getLogger('DybPython')
00198 
00199 pathx = lambda _:os.path.expanduser(os.path.expandvars(_))
00200 
00201 
00202 def args_():
00203     ap = argparse.ArgumentParser( description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
00204     ap.add_argument("-l","--loglevel",   help="logging level INFO, DEBUG, WARN,... Default %(default)s ")
00205     ap.add_argument(     "--logformat",  help="Used by logger. Default %(default)s ") 
00206     ap.add_argument(     "--logpath",    help="Path to write log file to. Default %(default)s ") 
00207     ap.add_argument("-b","--baseurl",    help="base url of SVN repository. Default %(default)s ")
00208     ap.add_argument("-w","--workingcopy",help="working copy of SVN repository to svn up etc.. Default %(default)s ")
00209     ap.add_argument("-a","--author",     help="select revisions to be from this author. Default %(default)s ")
00210     ap.add_argument(     "--limit",      help="limit the maximum number of revisions to avoid slow queries. Default %(default)s ")
00211     ap.add_argument(     "--dbconf",     help="DB to compare against or load into. Default %(default)s ") 
00212     ap.add_argument(     "--sleep",      help="Seconds to sleep between revisions, eliminating the remote possibility of subsecond fastforwarding time clashes. Default %(default)s ") 
00213     #
00214     ap.add_argument( "action",           help="Action to perform eg ls, rcmpcat, rloadcat  ") 
00215     ap.add_argument( "revs",             help="String representation of revision or contiguous revision ranges, eg 4913:4932 or 4913") 
00216     # 
00217     ap.add_argument("-r","--reset",       action="store_true",  help="reset SVN working copy to precisely the revision by deletion of directory prior to update ")
00218     ap.add_argument("-c","--cachesvnlog", action="store_true",  help="cache the SVN log as an xml file for speed, when you know the log for a revision or revision range is unchanging ")
00219 
00220     # vestigial option was never acted upon, fast forward commands are done manually, as noted above
00221     #ap.add_argument("-f","--ffcommit",    action="store_true",  help="perform commits of the fastforward modified INSERTDATE catalogs. Default %(default)s ")
00222 
00223     ap.set_defaults( 
00224       loglevel="info",
00225       baseurl="http://dayabay.ihep.ac.cn/svn/dybaux" ,
00226       workingcopy="~/dybaux/catalog/tmp_offline_db",
00227       dbconf="tmp_offline_db",
00228       author=None,
00229       sleep=3, 
00230       limit=50,
00231       reset=False,
00232       cachesvnlog=False,
00233       logformat='%(asctime)s %(name)s %(levelname)-8s %(message)s',
00234       logpath=None,   
00235     )
00236 
00237     args = ap.parse_args()
00238     urlleaf = os.path.basename(args.baseurl)
00239     assert urlleaf == 'dybaux', "leaf of baseurl is now restricted to be dybaux for simplicity" 
00240 
00241     loglevel = getattr( logging , args.loglevel.upper() )
00242     log.setLevel(loglevel) 
00243 
00244     sh = logging.StreamHandler() 
00245     sh.setLevel(loglevel) 
00246     fmtr = logging.Formatter(args.logformat)
00247     sh.setFormatter(fmtr)
00248     log.addHandler(sh)        
00249 
00250     if args.logpath:
00251         fh = logging.FileHandler(args.logpath,mode="w")
00252         fh.setFormatter(fmtr)
00253         fh.setLevel(loglevel) 
00254         log.addHandler(fh)
00255 
00256     return args 
00257 
00258 def db_(args):
00259     """
00260     programmatic use of db.py, to avoid logging confusion (double logging/missed logging/fmt differences):
00261        
00262     #.  `--nolog` avoids a clash of loggers
00263     #. logger name of `dbaux.py` is chosen to be 'DybPython' to hierachically contain 'DybPython.db'
00264 
00265     """
00266     from DybPython.db import main 
00267     dbconf = args.dbconf
00268     cmdline = 'db.py --noconfirm --nolog  %(dbconf)s noop' % locals()  
00269     sys.argv = cmdline.split()
00270     return main()                  
00271 
00272 class Aux(object):
00273 
00274     def __init__(self, args ):
00275         log.info( "Aux %r" % args )
00276         self.args = args
00277         
00278         opts = {'limit':args.limit } 
00279         if args.cachesvnlog:
00280             opts['xmlcache'] = self.cachesvnlog
00281 
00282         if args.action == "rloadcat" and args.dbconf == 'offline_db':
00283             log.warn( "checking options for definitive rloadcat into offline_db"  )
00284             #if not args.ffcommit: 
00285             #    log.fatal( "definitive rloadcat requires the  -f/--ffcommit option ")
00286             #    sys.exit(1)
00287         else:
00288             pass
00289 
00290         slog = SVNLog( args.baseurl, args.revs, opts )
00291         self.slog = slog 
00292         log.info( "completed the slog " )
00293         pass 
00294         self.fresh_db()
00295         info = self.info 
00296         log.info("%r ... revision %s " % ( info, info.revision) ) 
00297 
00298     cachesvnlog = property(lambda self:"/tmp/dbaux-slog-%s-%s.xml" % (os.getlogin(), self.args.revs ))
00299     info = property(lambda self:Info(self.args.workingcopy), doc="parse/wrap output of `svn info --xml` ... caution rerun on each access" )  
00300     stat = property(lambda self:Status(self.args.workingcopy), doc="parse/wrap output of `svn status --xml` ... caution rerun on each access" )  
00301 
00302     def fresh_db(self):
00303         """
00304         Pull up a new DB instance
00305         """
00306         self.db = db_( self.args )
00307 
00308     def sslog(self):
00309         for e in self.slog:
00310             if self.args.author and e.author != self.args.author:
00311                 continue 
00312             yield e 
00313 
00314     def ls_(self):
00315         """
00316         Lists the revisions, author, time, commit message
00317         """
00318         for i,e in enumerate(self.sslog()):
00319             log.info( "%-3s %6s %10s %20s %s " % (i,e.revision,e.author,e.t,e.msg) )   
00320 
00321     def svnup_(self, rev , reset=False, force=False ):
00322         """
00323         :param rev: revision number to bring working copy directory to 
00324         :param reset:  remove the directory first, wiping away uncommitted changes/conflicts 
00325 
00326 
00327         Aug 22, 2012 moved to checkout and revert rather than priot just update
00328         as this was  failing with ``--reset`` due to lack of the working copy directory,
00329         resulting in ``svn up`` skipping and subsequent assertions.  The idea is to step 
00330         thru pristine revisions, one by one::
00331 
00332             svn co -r 5292 http://dayabay.ihep.ac.cn/svn/dybaux/catalog/tmp_offline_db ~/dybaux/catalog/tmp_offline_db
00333             svn revert ~/dybaux/catalog/tmp_offline_db
00334 
00335         """ 
00336         info = self.info
00337         if info.revision == rev and not force:
00338             log.info(repr(info))
00339             log.info("svnup_ skip update as working copy is already at revision %(rev)s" % locals() )  
00340             return None
00341 
00342         wc = pathx(self.args.workingcopy)
00343         url = "%s/catalog/tmp_offline_db" % self.args.baseurl
00344         if reset:
00345             basename = os.path.basename(wc)
00346             assert os.path.isdir(wc) and len(wc) > 10, "bad wc %s " % wc 
00347             assert basename == 'tmp_offline_db'  
00348             log.info("removing working copy dir %s " % wc )
00349             shutil.rmtree(wc)
00350         else:
00351             stat = self.stat
00352             if len(stat) > 0:
00353                 log.fatal("svnup_ into unclean working copy %r " % stat )
00354                 log.fatal("\n".join([repr(e) for e in stat])) 
00355                 log.fatal("consider using --reset option to blow away the working copy before update")
00356                 sys.exit(1)
00357 
00358         svnco = "svn co -r %(rev)s %(url)s %(wc)s " % locals() 
00359         svnrv = "svn revert %(wc)s " % locals() 
00360         for svnxx in (svnco,svnrv):
00361             log.info(svnxx) 
00362             ret = os.popen(svnxx).read()
00363             log.info(ret)
00364  
00365     def rcmpcat_(self):
00366         """
00367         Loops over revisions:
00368         
00369         #. `svn up -r` the working copy  
00370         #. runs **rcmpcat**  comparing the ascii catalog with DB
00371  
00372         """
00373         stat = self.stat    
00374         if len(stat) > 0:
00375             log.warn("unclean working copy %r " % stat )
00376             log.warn("\n".join([repr(e) for e in stat])) 
00377 
00378         cat = self.args.workingcopy
00379         log.info("rcmpcat_ %r" % cat )
00380         for i,e in enumerate(self.sslog()):
00381             log.info("commit %-3s %6s %10s %20s %s " % (i,e.revision,e.author,e.t,e.msg) )   
00382             self.svnup_(e.revision, reset=self.args.reset )
00383             rcc = self.db.rcmpcat_(cat)   ## compare  
00384             log.info("rcmpcat %r" % rcc )
00385 
00386     def rloadcat_(self):
00387         """
00388         Loops over revisions
00389  
00390         #. `svn up -r` the working copy  
00391         #. runs **rcmpcat** to verify there are some updates to be loaded
00392         #. invokes **rloadcat** loading ascii catalog into DB
00393         #. runs **rcmpcat** agsin to verify load is complete 
00394 
00395 
00396         NB no confirmation is requested, thus before doing this perform an 
00397         **rcmpcat** to verify expected updates
00398 
00399         Rerunning an **rloadcat** ::
00400 
00401             ./dbaux.py rloadcat 4913   ## 1st time OK
00402             ./dbaux.py rloadcat 4913   ## 2nd time was giving conflicts ... now fails with unclean error
00403             ./dbaux.py --reset rloadcat 4913   ## blow away conflicts by deletion of working copy before "svn up"
00404 
00405         How to fix ?
00406 
00407         #.  When testing "svn revert" the changed validity tables throwing away
00408             the fastforward times ?  via parsing "svn status"  
00409 
00410         """
00411         cat = self.args.workingcopy
00412         sslog = [e for e in self.sslog()]
00413         log.info("rloadcat_ %r over %s revisions " % (cat, len(sslog)) )
00414         for i,e in enumerate(sslog):
00415             log.info( "commit %-3s %6s %10s %20s %s " % (i,e.revision,e.author,e.t,e.msg) )   
00416             self.svnup_(e.revision, reset=self.args.reset )
00417             rlc = self.db.rloadcat_(cat)                       ## compares and loads 
00418             if len(rlc) == 0:
00419                 log.info("rloadcat_ didnt see any updates for revision %s " % e.revision )
00420                 continue
00421 
00422             log.info("rloadcat %r" % rlc )
00423             self.fresh_db()
00424             rcc = self.db.rcmpcat_(cat)  
00425             log.info("after rloadcat rcmpcat %r" % rcc )
00426 
00427             assert len(rcc) == 0 , "should be no table/seqno differences after rloadcat " 
00428             if i + 1 < len(sslog):
00429                 sleep = self.args.sleep
00430                 log.info("sleeping for %s seconds" % sleep )
00431                 time.sleep(sleep)  
00432 
00433 
00434     def __call__(self):
00435         cmd = self.args.action + "_"
00436         if hasattr( self , cmd ):
00437             getattr( self, cmd )() 
00438         else:
00439             log.warn("no method %s " % cmd ) 
00440 
00441 
00442 def main():
00443     args = args_()
00444     aux = Aux(args)
00445     aux()
00446 
00447 if __name__ == '__main__':
00448     main()
00449 
00450 
| Classes | Job Modules | Data Objects | Services | Algorithms | Tools | Packages | Directories | Tracs |

Generated on Fri May 16 2014 09:55:40 for DybPython by doxygen 1.7.4