/search.css" rel="stylesheet" type="text/css"/> /search.js">
| Classes | Job Modules | Data Objects | Services | Algorithms | Tools | Packages | Directories | Tracs |

In This Package:

Public Member Functions | Public Attributes
DybDbi::cleaner::DbiCleaner Class Reference

List of all members.

Public Member Functions

def __init__
def reset_target
def __call__

Public Attributes

 db
 src
 tgt

Detailed Description

Usage::

     db = DB("tmp_offline_db")
     assert db.sect == os.environ['DBCONF']
     dc = DbiCleaner( db , GCalibPmtSpec, GCalibPmtSpecCleaner )
     dc.reset_target()
     dc()

Currently just duplicates for fidelity testing  

Definition at line 51 of file cleaner.py.


Constructor & Destructor Documentation

def DybDbi::cleaner::DbiCleaner::__init__ (   self,
  db,
  src,
  tgt 
)
:param db: ``DybPython.DB`` instance
:param src: source DybDbi class, eg GTableName
:param tgt: target DybDbi class, eg GTableNameCleanup

Target is just being used for table creation and table name

Definition at line 64 of file cleaner.py.

00065                                        :
00066         """
00067         :param db: ``DybPython.DB`` instance
00068         :param src: source DybDbi class, eg GTableName
00069         :param tgt: target DybDbi class, eg GTableNameCleanup
00070 
00071         Target is just being used for table creation and table name
00072 
00073         """
00074         assert db.sect == os.environ['DBCONF']
00075         self.db = db
00076         self.src = src         
00077         self.tgt = tgt          
00078         self['tname'] = tgt.__name__[1:]   # G prefix convention 


Member Function Documentation

def DybDbi::cleaner::DbiCleaner::reset_target (   self)
Drop the target ``GTableNameCleanup`` tables and ``LOCALSEQNO`` refernence   
harmless warning regarding non-existing tables on first run 

Definition at line 79 of file cleaner.py.

00080                           :
00081         """
00082         Drop the target ``GTableNameCleanup`` tables and ``LOCALSEQNO`` refernence   
00083         harmless warning regarding non-existing tables on first run 
00084         """  
00085         assert self['tname'][-4:] == 'Fake' or self['tname'][-7:] == 'Cleanup' , "SAFETY MEASURE : target classname must end with 'Fake' or 'Cleanup' unlike %(tname)s  " % self
00086         self.db("drop table if exists %(tname)s, %(tname)sVld" % self )      
00087         self.db("delete from LOCALSEQNO where TABLENAME='%(tname)s'" % self )
00088         self.tgt().CreateDatabaseTables(0,  self['tname'] )

def DybDbi::cleaner::DbiCleaner::__call__ (   self,
  seqnos 
)
Traverse all validity records in source table, ordered by VERSIONDATE desc::

mysql> select * from CalibFeeSpecVld order by VERSIONDATE desc ;
+-------+---------------------+---------------------+----------+---------+---------+------+-------------+---------------------+---------------------+
| SEQNO | TIMESTART           | TIMEEND             | SITEMASK | SIMMASK | SUBSITE | TASK | AGGREGATENO | VERSIONDATE         | INSERTDATE          |
+-------+---------------------+---------------------+----------+---------+---------+------+-------------+---------------------+---------------------+
|   113 | 2010-06-18 02:18:24 | 2038-01-19 03:14:07 |       32 |       1 |       1 |    0 |          -1 | 2010-06-18 10:23:24 | 2010-06-23 05:40:30 | 
|   112 | 2010-06-18 02:18:24 | 2038-01-19 03:14:07 |       32 |       1 |       1 |    0 |          -1 | 2010-06-18 10:22:24 | 2010-06-22 11:37:48 | 
|   111 | 2010-06-18 02:18:24 | 2038-01-19 03:14:07 |       32 |       1 |       1 |    0 |          -1 | 2010-06-18 10:21:24 | 2010-06-22 11:36:10 | 
|   110 | 2010-06-18 02:18:24 | 2038-01-19 03:14:07 |       32 |       1 |       1 |    0 |          -1 | 2010-06-18 10:20:24 | 2010-06-22 11:35:22 | 
|   109 | 2010-06-18 02:18:24 | 2038-01-19 03:14:07 |       32 |       1 |       1 |    0 |          -1 | 2010-06-18 10:19:24 | 2010-06-22 10:01:04 | 
|   108 | 2010-06-18 02:18:24 | 2038-01-19 03:14:07 |       32 |       1 |       1 |    0 |          -1 | 2010-06-18 10:18:24 | 2010-06-22 10:00:47 | 
|

which means the created SEQNO will get changed unless can sort on SEQNO  ... but there are gaps so they must change anyhow

Succeeds to do a full copy BUT differences:

#. INSERTDATE not propagated : expected
#. VERSIONDATE is 8hr shifted  ... some DBI TZ handling emerging ?
#. SEQNO do not match : makes comparisons difficult 
#. SEQNO mismatch because ordering is VERSIONDATE desc ... could sort the vrec by SEQNO but there are generally gaps anyhow
 

Definition at line 89 of file cleaner.py.

00090                                 :
00091         """
00092         Traverse all validity records in source table, ordered by VERSIONDATE desc::
00093 
00094                 mysql> select * from CalibFeeSpecVld order by VERSIONDATE desc ;
00095                 +-------+---------------------+---------------------+----------+---------+---------+------+-------------+---------------------+---------------------+
00096                 | SEQNO | TIMESTART           | TIMEEND             | SITEMASK | SIMMASK | SUBSITE | TASK | AGGREGATENO | VERSIONDATE         | INSERTDATE          |
00097                 +-------+---------------------+---------------------+----------+---------+---------+------+-------------+---------------------+---------------------+
00098                 |   113 | 2010-06-18 02:18:24 | 2038-01-19 03:14:07 |       32 |       1 |       1 |    0 |          -1 | 2010-06-18 10:23:24 | 2010-06-23 05:40:30 | 
00099                 |   112 | 2010-06-18 02:18:24 | 2038-01-19 03:14:07 |       32 |       1 |       1 |    0 |          -1 | 2010-06-18 10:22:24 | 2010-06-22 11:37:48 | 
00100                 |   111 | 2010-06-18 02:18:24 | 2038-01-19 03:14:07 |       32 |       1 |       1 |    0 |          -1 | 2010-06-18 10:21:24 | 2010-06-22 11:36:10 | 
00101                 |   110 | 2010-06-18 02:18:24 | 2038-01-19 03:14:07 |       32 |       1 |       1 |    0 |          -1 | 2010-06-18 10:20:24 | 2010-06-22 11:35:22 | 
00102                 |   109 | 2010-06-18 02:18:24 | 2038-01-19 03:14:07 |       32 |       1 |       1 |    0 |          -1 | 2010-06-18 10:19:24 | 2010-06-22 10:01:04 | 
00103                 |   108 | 2010-06-18 02:18:24 | 2038-01-19 03:14:07 |       32 |       1 |       1 |    0 |          -1 | 2010-06-18 10:18:24 | 2010-06-22 10:00:47 | 
00104                 |
00105 
00106         which means the created SEQNO will get changed unless can sort on SEQNO  ... but there are gaps so they must change anyhow
00107 
00108         Succeeds to do a full copy BUT differences:
00109 
00110         #. INSERTDATE not propagated : expected
00111         #. VERSIONDATE is 8hr shifted  ... some DBI TZ handling emerging ?
00112         #. SEQNO do not match : makes comparisons difficult 
00113         #. SEQNO mismatch because ordering is VERSIONDATE desc ... could sort the vrec by SEQNO but there are generally gaps anyhow
00114  
00115         """
00116         sqlcontext, task, subsite = "1=1", -1, -1   ## wideopen DBI querying 
00117         vrb = self.src.GetTableProxy().MakeValidityRecBuilder( sqlcontext, subsite, task )
00118         nvr = vrb.GetNumValidityRec()
00119         gap = vrb.GetValidityRec(0)
00120         assert gap.IsGap() 
00121 
00122         npr = 0
00123         for ivr in reversed(range(1,nvr)):   ## reverse for closer SEQNO correspondence 
00124             vrec = vrb.GetValidityRec(ivr)
00125             if len(seqnos) > 0 and vrec.seqno not in seqnos:continue
00126 
00127             npr += 1
00128             # writer for each vrec 
00129             wrt = self.src.Wrt().Clone() 
00130             wrt.ctx( contextrange=vrec.contextrange, subsite=vrec.subsite, task=vrec.task, versiondate=vrec.versiondate, tablename=self['tname'] ) ## make a fresh writer ... dont use vrec alone
00131 
00132             # wideopen reader for all payloads within the vrec (aka SEQNO)
00133             rpt = self.src.Rpt().Clone()
00134             rpt.ctx( validityrec=vrec )  ## vrec reader does " select * from T where SEQNO= 113 "   etc...
00135             n = len(rpt)
00136             log.debug("ivr %s vrec seqno %s n %s " % (ivr, vrec.seqno, n ))
00137 
00138             # write all source payloads to target 
00139             for row in rpt:
00140                 wrt.Write(row)    
00141             assert wrt.Close()
00142             pass
00143         log.info("from %s attempted to propagate %s " % (nvr-1, npr) )   ## slot 0 is Gap 
00144 
00145 
00146 
00147 


Member Data Documentation

Definition at line 71 of file cleaner.py.

Definition at line 71 of file cleaner.py.

Definition at line 71 of file cleaner.py.


The documentation for this class was generated from the following file:
| Classes | Job Modules | Data Objects | Services | Algorithms | Tools | Packages | Directories | Tracs |

Generated on Fri May 16 2014 09:57:24 for DybDbi by doxygen 1.7.4