/search.css" rel="stylesheet" type="text/css"/> /search.js">
| Classes | Job Modules | Data Objects | Services | Algorithms | Tools | Packages | Directories | Tracs |

In This Package:

Public Member Functions | Public Attributes
Scraper::dq::CQScraper::CQScraper Class Reference

List of all members.

Public Member Functions

def __init__
def readsrc
def propagate
def check

Public Attributes

 src
 tgt
 packer
 ordering

Detailed Description

...

Definition at line 195 of file CQScraper.py.


Constructor & Destructor Documentation

def Scraper::dq::CQScraper::CQScraper::__init__ (   self,
  src,
  tgt 
)
:param src: ``CQTable`` instance of source 
:param tgt: ``CQTable`` instance of target

Definition at line 197 of file CQScraper.py.

00198                                  :
00199         """
00200         :param src: ``CQTable`` instance of source 
00201         :param tgt: ``CQTable`` instance of target
00202         """
00203         self.src = src
00204         self.tgt = tgt          
00205         log.info("src: %s " % src.summary()) 
00206         log.info("tgt: %s " % tgt.summary()) 
00207         self.packer = CQPacker()
00208         self.ordering = CQOrdering() 
00209 
00210 
       

Member Function Documentation

def Scraper::dq::CQScraper::CQScraper::readsrc (   self)

Definition at line 212 of file CQScraper.py.

00213                      :
00214         """ 
00215         """
00216         lastseqno = self.tgt.lastseqno
00217         log.debug("readsrc starting with SEQNO following %s  " % (lastseqno) )
00218         recs = self.src.getEntries(lastseqno)
00219         return lastseqno, recs
       
def Scraper::dq::CQScraper::CQScraper::propagate (   self,
  recs 
)
Obtains a tranche of records `src.getEntries(tgt.lastseqno)` from source db.table
using MySQL-python access
packs them into mask integers and writes these compressed records 
to the target db.table using DybDbi 

Note that ordering requires access to the CableMap which it accesses
via DybDbi this means the target table has to contain the CableMap.
When testing into a tmp_offline_db ensure to include that table.

:return: (lastwrite, counter)

Definition at line 221 of file CQScraper.py.

00222                              :
00223         """ 
00224         Obtains a tranche of records `src.getEntries(tgt.lastseqno)` from source db.table
00225         using MySQL-python access
00226         packs them into mask integers and writes these compressed records 
00227         to the target db.table using DybDbi 
00228 
00229         Note that ordering requires access to the CableMap which it accesses
00230         via DybDbi this means the target table has to contain the CableMap.
00231         When testing into a tmp_offline_db ensure to include that table.
00232 
00233         :return: (lastwrite, counter)
00234         """
00235         lastwrite,counter = 0,0
00236         for rec in recs:
00237             timestart   = TimeStamp(calendar.timegm(rec['timestart'].timetuple()))
00238             timeend     = TimeStamp(calendar.timegm(rec['timeend'].timetuple()))
00239 
00240             channels    = self.ordering.channels(rec['sitemask'], rec['subsite'], rec['timestart'])  # canonical ordering of channels
00241             dstatus     = rec['dstatus']
00242             assert set(dstatus.keys()) == set(channels), 'inconsistent channel list between DqChannelStatus and GCableMap!'
00243             lstatus     = map(lambda x: str(dstatus[x]), channels)                          # order determined from cable map
00244             masks       = self.packer.pack(lstatus)                                                  # pack pmt status into 7 int
00245             assert self.packer.unpack(masks)==lstatus, 'pack/unpack consistency check failed!' 
00246 
00247             wrt = self.tgt.kls.Wrt()
00248             assert rec['simmask'] == SimFlag.kData
00249             cr = ContextRange( rec['sitemask'], rec['simmask'], timestart, timeend )
00250             wrt.ctx(contextrange=cr, versiondate=TimeStamp(0,0), subsite=rec['subsite'])
00251 
00252             record = self.tgt.kls.Create(   RunNo   = rec['runno'],
00253                                             FileNo  = rec['fileno'],
00254                                             Mask0   = masks[0],
00255                                             Mask1   = masks[1],
00256                                             Mask2   = masks[2],
00257                                             Mask3   = masks[3],
00258                                             Mask4   = masks[4],
00259                                             Mask5   = masks[5],
00260                                             Mask6   = masks[6])
00261             wrt.Write(record)
00262             wseqno = wrt.Close()
00263             assert wseqno == rec['seqno'], ("read/write seqno misaligned", rec['seqno'], wseqno )
00264             lastwrite = wseqno
00265             counter += 1
00266             if counter % 100 == 0:
00267                 log.info("wseqno %d propagated count %s " % ( wseqno,counter))
00268             pass
00269         log.info("propagated %s upto lastwrite SEQNO %s  " % (counter, lastwrite) )
00270         return lastwrite, counter

def Scraper::dq::CQScraper::CQScraper::check (   self,
  lastseqno 
)
Compare record dicts obtained from unpacked source and packed target tables.
Due to use of `pop=True` the technical extraneous fields such as the masks
are eliminated allowing direct dict comparisons to be made.

:param lastseqno: base seqno from which to validate
:return: validation count

Definition at line 272 of file CQScraper.py.

00273                               :
00274         """
00275         Compare record dicts obtained from unpacked source and packed target tables.
00276         Due to use of `pop=True` the technical extraneous fields such as the masks
00277         are eliminated allowing direct dict comparisons to be made.
00278 
00279         :param lastseqno: base seqno from which to validate
00280         :return: validation count
00281         """
00282         log.debug("check starting with SEQNO following %s  " % (lastseqno) )
00283         sen = self.src.getEntries(lastseqno, pop=True)
00284         ten = self.tgt.getPackedEntries(lastseqno, pop=True)
00285 
00286         nsrc = len(sen)
00287         ntgt = len(ten)
00288         ncom = min(nsrc,ntgt)   # potentially not equal, as src may continue to be written to during the slow target writing        
00289         log.info("check nsrc %s ntgt %s ncom %s " %  (nsrc,ntgt,ncom))
00290 
00291         counter = 0 
00292         for s, t in zip(sen[0:ncom],ten[0:ncom]):
00293             assert s['seqno'] == t['seqno'], "seqno discrepancy"   # actually this is redundant as dict comparison compares the lot 
00294             assert s == t, ("src/tgt dict discrepancy", s,t)
00295             counter += 1
00296             if counter % 100 == 0:
00297                 log.debug("seqno %d validated count %s " % ( s['seqno'],counter))
00298         log.debug("checked %s starting with SEQNO following %s  " % (counter, lastseqno) )
00299         return counter
00300 


Member Data Documentation

Definition at line 200 of file CQScraper.py.

Definition at line 200 of file CQScraper.py.

Definition at line 200 of file CQScraper.py.

Definition at line 200 of file CQScraper.py.


The documentation for this class was generated from the following file:
| Classes | Job Modules | Data Objects | Services | Algorithms | Tools | Packages | Directories | Tracs |

Generated on Fri May 16 2014 09:50:03 for Scraper by doxygen 1.7.4