/search.css" rel="stylesheet" type="text/css"/> /search.js">
| Classes | Job Modules | Data Objects | Services | Algorithms | Tools | Packages | Directories | Tracs |

In This Package:

Public Member Functions | Public Attributes | Static Public Attributes
DybPython::dbcf::DBCF Class Reference

List of all members.

Public Member Functions

def __init__
def table
def mismatch
def present
def __call__
def args_

Public Attributes

 opts
 collect intersected tables
 dbs
 tc

Static Public Attributes

tuple args_ = classmethod( args_ )

Detailed Description

DB Schema comparison between tables in Databases identified
by a list of dbconf sectnames.
Table schema comparisons are made for tables that appear 
in all specified Databases

The basis for comparison is the tuple of fieldatt dicts returned by
MySQL-python in response to "describe TableName", the fieldattr for each 
Field has the form::

    {'Default': None,
     'Extra': '',
     'Field': 'SEQNO',
     'Key': 'PRI',
     'Null': 'NO',
     'Type': 'int(11)'}


Suspect a MySQL version difference in 'Default' reporting 
in offline_db getting '' but in belle7 getting None, resulting 
in many  mismatches (despite same creation commands): 

PhysAdVld                      TIMEEND              Default    {'tmp_nodata': None, 'offline_db': ''}  

Definition at line 24 of file dbcf.py.


Constructor & Destructor Documentation

def DybPython::dbcf::DBCF::__init__ (   self,
  dbconf,
  kwa 
)
Establish names of common set of tables that exist in all Databases

Definition at line 50 of file dbcf.py.

00051                                        :
00052          """
00053          Establish names of common set of tables that exist in all Databases
00054          """
00055          list.__init__(self, dbconf )
00056 
00057          dbs = map( DB, self )
00058 
00059          tc = set()    
00060          for db in dbs: 
00061              t = set(db.showtables)
00062              if len(tc) == 0:
00063                  tc = set(t)
00064              else:
00065                  tc.intersection_update(t)    ## collect intersected tables
00066 
00067          self.opts = kwa.get('opts', {})
00068          self.dbs = dbs
00069          self.tc = tc
00070 


Member Function Documentation

def DybPython::dbcf::DBCF::table (   self,
  t 
)
:param t: table name

Obtain the descriptions from each Database and rekey  
to return a dict-of-dict-of-dict keyed by fieldname/fieldattr/dbconf, 
The rearrangement puts the dbconf at the base of the structure 
giving access to all values in all db in one place

ddd is an intermediate dict-of-dict-of-dict keyed on dbconf/fieldname/fieldattr

Definition at line 71 of file dbcf.py.

00072                        :
00073         """
00074         :param t: table name
00075 
00076         Obtain the descriptions from each Database and rekey  
00077         to return a dict-of-dict-of-dict keyed by fieldname/fieldattr/dbconf, 
00078         The rearrangement puts the dbconf at the base of the structure 
00079         giving access to all values in all db in one place
00080 
00081         ddd is an intermediate dict-of-dict-of-dict keyed on dbconf/fieldname/fieldattr
00082         """
00083         assert t in self.tc, "table %s is not in common " % t     
00084 
00085         # convert per field tuple-of-dicts into dict-of-dicts keyed on fieldname/fieldattr
00086         rejig = lambda x:dict(map(lambda _:(_['Field'],_) , x ))
00087  
00088         dd = []      
00089         ddd = {}     
00090         for n,db in zip(self,self.dbs):
00091             d = db("describe " + t )
00092             dd.append(d)
00093             ddd[n] = rejig(d)
00094         assert len(set(map(len,dd))) == 1 , "mismatch in field length %s " % t  
00095 
00096         kk = None
00097         for n in ddd:
00098            ks = sorted(ddd[n].keys())
00099            if not kk:
00100               kk = ks
00101            else:
00102               assert ks == kk,  "field key mismatch %s \n%r \n%r " % ( t , kk, ks )  
00103 
00104         # rekey to put the sectname at base giving access 
00105         # to all values in all db in one place
00106         msm = {}  
00107         for n in ddd.keys():
00108             for fn in ddd[n]:
00109                 if not fn in msm:msm[fn] = {}
00110                 for fa in ddd[n][fn]:
00111                     if not fa in msm[fn]:msm[fn][fa] = {}
00112 
00113                     v = ddd[n][fn][fa] 
00114                     if fa == 'Default' and not self.opts.get('nofix'): 
00115                          if v == None:v = ''                        
00116 
00117                     msm[fn][fa][n] = v
00118                   
00119         # prune when have only one value in leaf dict  
00120         # NB must use longform .keys() as are mutating the dict 
00121         for fn in msm.keys():
00122             for fa in msm[fn].keys():
00123                  vs = set(msm[fn][fa].values())
00124                  if len(vs) == 1:         ## only one value ... so prune
00125                      del msm[fn][fa]
00126             if len(msm[fn]) == 0:         ## prune up here too
00127                 del msm[fn] 
00128 
00129         return msm 
00130 

def DybPython::dbcf::DBCF::mismatch (   self)
Returns dict-of-dict-of-dict-of-dict keyed by tablename/fieldname/fieldattr/dbconf, 
for example from tablename level::
       
      'SimPmtSpecVld': {'INSERTDATE': {'Default': {'offline_db': '', 'tmp_nodata': None}},
                'TIMEEND':    {'Default': {'offline_db': '', 'tmp_nodata': None},
                                   'Key': {'offline_db': 'MUL', 'tmp_nodata': ''}},
                'TIMESTART': {'Default': {'offline_db': '', 'tmp_nodata': None},
                                  'Key': {'offline_db': 'MUL', 'tmp_nodata': ''}},
              'VERSIONDATE': {'Default': {'offline_db': '', 'tmp_nodata': None}}}}

The dddd is pruned to only contain "paths" with mismatches

Definition at line 131 of file dbcf.py.

00132                       :
00133         """
00134         Returns dict-of-dict-of-dict-of-dict keyed by tablename/fieldname/fieldattr/dbconf, 
00135         for example from tablename level::
00136        
00137               'SimPmtSpecVld': {'INSERTDATE': {'Default': {'offline_db': '', 'tmp_nodata': None}},
00138                                 'TIMEEND':    {'Default': {'offline_db': '', 'tmp_nodata': None},
00139                                                    'Key': {'offline_db': 'MUL', 'tmp_nodata': ''}},
00140                                 'TIMESTART': {'Default': {'offline_db': '', 'tmp_nodata': None},
00141                                                   'Key': {'offline_db': 'MUL', 'tmp_nodata': ''}},
00142                               'VERSIONDATE': {'Default': {'offline_db': '', 'tmp_nodata': None}}}}
00143 
00144         The dddd is pruned to only contain "paths" with mismatches
00145         """
00146         msm = {}
00147 
00148         ot = self.opts.get('table',None)
00149         op = self.opts.get('skip',None)
00150         skip = op.split(",") if op else [] 
00151 
00152         for t in self.tc:
00153             if ot and t != ot:
00154                 continue  
00155             if t in skip:
00156                 log.warn("skipping %s as has fieldname mismatch needing correction before can apply other checks " % t )
00157             else:
00158                 msm[t] =  self.table( t )
00159         return msm
00160     

def DybPython::dbcf::DBCF::present (   self,
  msm 
)

Definition at line 161 of file dbcf.py.

00162                           :
00163 
00164         of = self.opts.get('field',None)
00165         oa = self.opts.get('attr', None)
00166 
00167         for tn in msm:
00168             for fn in msm[tn]:
00169                 if of and fn != of:continue 
00170                 for fa in msm[tn][fn]:
00171                     if oa and fa != oa:continue 
00172                     print "%-30s %-20s %-10s %r " % ( tn,fn,fa, msm[tn][fn][fa] )

def DybPython::dbcf::DBCF::__call__ (   self)

Definition at line 173 of file dbcf.py.

00174                       :
00175         msm = self.mismatch()
00176         #print pformat(msm)
00177         self.present(msm)
00178         return msm

Definition at line 179 of file dbcf.py.

00180                   :
00181         ap = argparse.ArgumentParser( description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
00182         ap.add_argument('dbconf',   nargs="+", help='Two or more dbconf section names from ~/.my.cnf file')
00183         ap.add_argument('-l','--loglevel',     help='logging level INFO,WARN,DEBUG... Default %(default)s ')
00184         ap.add_argument('-t','--table',        help='Restrict comparison to just this table. Default %(default)s ')
00185         ap.add_argument('-f','--field',        help='Restrict comparison to just this field. Default %(default)s ')
00186         ap.add_argument('-a','--attr',         help='Restrict comparison to just this attribute. Default %(default)s ')
00187         ap.add_argument('-s','--skip',         help='Comma delimited list of tables to skip. Default %(default)s ')
00188         ap.add_argument('-N','--nofix', action="store_true"  ,  help='Dont fix "Default" reporting.  Default %(default)s ')
00189         ap.set_defaults(
00190              loglevel="INFO",
00191                table=None,
00192                field=None,
00193                attr=None,
00194                skip=None,
00195                nofix=False,
00196          )
00197         args = ap.parse_args()
00198         loglevel = getattr(logging,args.loglevel.upper())
00199         logging.basicConfig( level=loglevel )
        return args

Member Data Documentation

tuple DybPython::dbcf::DBCF::args_ = classmethod( args_ ) [static]

Definition at line 200 of file dbcf.py.

collect intersected tables

Definition at line 53 of file dbcf.py.

Definition at line 53 of file dbcf.py.

Definition at line 53 of file dbcf.py.


The documentation for this class was generated from the following file:
| Classes | Job Modules | Data Objects | Services | Algorithms | Tools | Packages | Directories | Tracs |

Generated on Fri May 16 2014 09:55:40 for DybPython by doxygen 1.7.4