/search.css" rel="stylesheet" type="text/css"/> /search.js">
| Classes | Job Modules | Data Objects | Services | Algorithms | Tools | Packages | Directories | Tracs |

In This Package:

Public Member Functions | Public Attributes | Properties | Private Member Functions
history::History Class Reference

List of all members.

Public Member Functions

def __init__
def __repr__
def dump
def paydump
def fillgaps
def shunt
def writecopy

Public Attributes

 sitemask
 ugly subsite repetition
 subsite
 task
 wtask
 sqlcontext
 seqnocut
 target_tablename
 dupefirst
 kls
 tn
 vrb
 gaps
 seqno2ik
 kets

Properties

 digests = property(lambda self:[self[t]['digest'] for t in sorted(self)], doc="payload digests in timestart seconds order")
 seqnos = property(lambda self:[self[t]['seqno'] for t in sorted(self)] , doc="seqnos in timestart seconds order")

Private Member Functions

def _collect
def _index
def _findgaps

Detailed Description

Validity history dict keyed on timestart seconds

After gap filling SEQNO ordering will not match time ordering, hence moved to 
keying on timestart seconds : which should also be unique within an SS block
(other tasks, simmasks would break that but would need to change blocking 
in that case anyhow)

Definition at line 15 of file history.py.


Constructor & Destructor Documentation

def history::History::__init__ (   self,
  kls,
  sitemask,
  subsite,
  wtask = 0,
  task = -1,
  paydict_ = lambda rpt:{},
  seqnocut = 3867,
  target_tablename = None,
  dupefirst = True 
)

:param task: reading task
:param wtask: writing task

:param seqnocut: presentational only, SEQNO higher than this are highlighted
:param target_tablename:
:param dupefirst: duplicate the first entry to retain same SEQNO count, otherwise looses the first entry 


Collects `Rec` instances into this dict keyed by TIMESTART seconds

.. warn:: ensure the chunk does not have duplicate TIMESTARTs via sitemask, subsite, task choices

Definition at line 24 of file history.py.

00024                                                                                      :{}, seqnocut=3867 , target_tablename=None, dupefirst=True ):
00025         """
00026 
00027         :param task: reading task
00028         :param wtask: writing task
00029 
00030         :param seqnocut: presentational only, SEQNO higher than this are highlighted
00031         :param target_tablename:
00032         :param dupefirst: duplicate the first entry to retain same SEQNO count, otherwise looses the first entry 
00033 
00034 
00035         Collects `Rec` instances into this dict keyed by TIMESTART seconds
00036 
00037         .. warn:: ensure the chunk does not have duplicate TIMESTARTs via sitemask, subsite, task choices
00038 
00039         """
00040         pass
00041         sqlcontext = "SITEMASK=%(sitemask)s and SUBSITE=%(subsite)s " % locals() 
00042         vrb = Vrb(kls, sqlcontext=sqlcontext, subsite=subsite, task=task )       ## ugly subsite repetition
00043         pass
00044         self.sitemask = sitemask
00045         self.subsite = subsite
00046         self.task = task
00047         self.wtask = wtask
00048         self.sqlcontext = sqlcontext 
00049         self.seqnocut = seqnocut
00050         self.target_tablename = target_tablename
00051         self.dupefirst = dupefirst
00052         pass
00053         self.kls = kls
00054         self.tn = kls.__name__[1:] 
00055         self.vrb = vrb
00056         self.gaps = []
00057         self._collect(paydict_)
00058         self.seqno2ik = {}        
00059         self._index()
00060         self._findgaps()
00061 
00062 

Member Function Documentation

def history::History::__repr__ (   self)

Definition at line 63 of file history.py.

00064                       :
00065         return "%s %s nvrec %s gaps %s " % ( self.__class__.__name__, self.sqlcontext, len(self.vrb), len(self.gaps) )

def history::History::_collect (   self,
  paydict_ 
) [private]
Collect Rec instances into this dict keyed on timestart seconds

:param paydict_: function of rpt instance that returns a dict

Definition at line 66 of file history.py.

00067                                 :
00068         """
00069         Collect Rec instances into this dict keyed on timestart seconds
00070 
00071         :param paydict_: function of rpt instance that returns a dict
00072         """
00073         for seqno,vrec in self.vrb:
00074             cr = vrec.contextrange
00075             rpt = self.kls.Rpt().Clone()
00076             rpt.ctx( validityrec=vrec )
00077             dpay = paydict_(rpt)   # tuck away table specifics in the passed in paydict_ function
00078             npay = len(dpay) 
00079             pass 
00080             rec = Rec(seqno=seqno,vrec=vrec,timestart=cr.timestart,timeend=cr.timeend,npay=npay,payload=dpay,digest=rpt.digest)
00081             rec['ftimestart'] = rec['timestart'].AsString('s')
00082             rec['ftimeend']   = rec['timeend'].AsString('s')
00083             rec['msg'] = "***" if seqno > self.seqnocut else "   "
00084             pass
00085             key = cr.timestart.sec 
00086             self[key] = rec 
            pass
def history::History::_index (   self) [private]
Construct mapping from SEQNO to timestart ordered index

Definition at line 87 of file history.py.

00087                     : 
00088         """
00089         Construct mapping from SEQNO to timestart ordered index
00090         """
00091         kets = sorted(self.keys()) 
00092         self.kets = kets
00093         for ik in range(0,len(kets)):
00094             ck = kets[ik]
00095             cv = self[ck]
00096             seqno = cv['seqno']
00097             self.seqno2ik[seqno] = ik
00098         pass 
00099 
def history::History::_findgaps (   self) [private]
Comparison of Rec with their priors, collecting any breaks 
in chain::

    [ts-te][ts-te]

Definition at line 104 of file history.py.

00105                        :
00106         """
00107         Comparison of Rec with their priors, collecting any breaks 
00108         in chain::
00109 
00110             [ts-te][ts-te]
00111 
00112         """
00113         for ik in range(1,len(self.kets)):
00114             ck = self.kets[ik]
00115             pk = self.kets[ik-1] 
00116             cv = self[ck]
00117             pv = self[pk] 
00118             pass
00119             assert ck == cv['timestart'].sec
00120             assert pk == pv['timestart'].sec
00121             assert cv['npay'] == pv['npay'], (cv, pv)
00122             #assert cv['seqno'] > pv['seqno'], (cv, pv)    ## no longer the case after gap filling
00123             assert cv['timestart'].sec >= pv['timeend'].sec, (cv, pv)
00124             assert cv['timestart'].sec >= pv['timeend'].sec, (cv, pv)
00125             pass
00126             if cv['timestart'].sec != pv['timeend'].sec:
00127                 gap = Gap(seqbef=pv['seqno'],seqaft=cv['seqno'],timestart=pv['timeend'], timeend=cv['timestart'] )             
00128                 self.gaps.append(gap)
 
def history::History::dump (   self)
Dump validities, highlighting those with SEQNO greater than cut 

Definition at line 129 of file history.py.

00130                   :
00131         """
00132         Dump validities, highlighting those with SEQNO greater than cut 
00133         """
00134         for i,t in enumerate(sorted(self)):
00135             rec = self[t]
00136             log.info("%4s %s %r" % (i,t,rec))

def history::History::paydump (   self,
  paykey = 'L2C2R8' 
)

Definition at line 137 of file history.py.

00138                                       :
00139         for i,t in enumerate(sorted(self)):
00140             rec = hist[t]
00141             pay = rec['payload']
00142             pav = pay.get(paykey,None)
00143             log.info("%4s %s %r %s" % (i,t,rec,pav ))
 
def history::History::fillgaps (   self)
Filling gaps via duplication of prior entry with timerange shifted

Definition at line 144 of file history.py.

00145                       :
00146         """
00147         Filling gaps via duplication of prior entry with timerange shifted
00148         """
00149         wseqs=[] 
00150         for gap in self.gaps:
00151             seqbef = gap['seqbef']
00152             wseq=self.writecopy( seqbef, gap['timestart'], gap['timeend'] , task=self.wtask )
00153             wseqs.append(wseq)
00154         log.info("wrote seqnos %r " % wseqs )

def history::History::shunt (   self,
  seqno 
)
:param seqno: if 0 all entries are shunted, otherwise only the specified SEQNO

Shunting payload forward, '''pay it forward'''

Originally the SEQNO were not aligned due to the
treatment via chunks of (SITEMASK, SUBSITE).
Effectively the deck was unshuffled into sorted manner.  

To rectify this the source validity table (SEQNO, SITEMASK, SUBSITE)
sequence is used to retrace the steps of the scraper and address multiple
instances of this history class for each (SITEMASK,SUBSITE) chunk.
This allowed the deck to be shuffled according to the source shuffle.    

#. 1st rec untouched
#. 2nd rec gets payload from 1st 
#. 3rd rec gets payload from 2nd
#. etc
#. nth rec gets payload from (n-1)th
#. last gets payload from penultimate

Definition at line 155 of file history.py.

00156                           :
00157         """
00158         :param seqno: if 0 all entries are shunted, otherwise only the specified SEQNO
00159 
00160         Shunting payload forward, '''pay it forward'''
00161 
00162         Originally the SEQNO were not aligned due to the
00163         treatment via chunks of (SITEMASK, SUBSITE).
00164         Effectively the deck was unshuffled into sorted manner.  
00165 
00166         To rectify this the source validity table (SEQNO, SITEMASK, SUBSITE)
00167         sequence is used to retrace the steps of the scraper and address multiple
00168         instances of this history class for each (SITEMASK,SUBSITE) chunk.
00169         This allowed the deck to be shuffled according to the source shuffle.    
00170 
00171         #. 1st rec untouched
00172         #. 2nd rec gets payload from 1st 
00173         #. 3rd rec gets payload from 2nd
00174         #. etc
00175         #. nth rec gets payload from (n-1)th
00176         #. last gets payload from penultimate
00177 
00178         """
00179         assert len(self.gaps) == 0, "must fill gaps before shunting "
00180         assert self.target_tablename , "must define target table when shunting "
00181 
00182         if seqno == 0:
00183             log.info("shunt into %s %s " %  ( len(self.kets),self.target_tablename) )
00184             iks = range(0,len(self.kets))
00185         else:
00186             iks = [self.seqno2ik[seqno]] 
00187 
00188         for ik in iks:
00189             ck = self.kets[ik]
00190             cv = self[ck]
00191             if ik == 0 and self.dupefirst:
00192                 if seqno > 0:
00193                     assert cv['seqno'] == seqno 
00194                 self.writecopy( cv['seqno'], cv['timestart'], cv['timeend'], task=self.wtask , target_tablename=self.target_tablename )
00195                 continue            
00196             pk = self.kets[ik-1] 
00197             pv = self[pk] 
00198             pass
00199             assert ck == cv['timestart'].sec
00200             assert pk == pv['timestart'].sec
00201             assert cv['npay'] == pv['npay'], (cv, pv)
00202             assert cv['timestart'].sec >= pv['timeend'].sec, (cv, pv)
00203             assert cv['timestart'].sec >= pv['timeend'].sec, (cv, pv)
00204             assert cv['timestart'].sec == pv['timeend'].sec , "gaps need fixin 1st "
00205             pass 
00206             if seqno > 0:
00207                 assert cv['seqno'] == seqno 
00208             self.writecopy( pv['seqno'], cv['timestart'], cv['timeend'], task=self.wtask , target_tablename=self.target_tablename )
00209 

def history::History::writecopy (   self,
  seqno,
  timestart = None,
  timeend = None,
  task = 0,
  target_tablename = None 
)
Copy validity rec and payload contents corresponding to seqno 
into a new entry, optionally changing timestart and timeend        

:param vrec: ValidityRec instance
:param timestart: TimeStamp instance
:param timeend: TimeStamp instance
:param task:
:param target_tablename:

Definition at line 210 of file history.py.

00211                                                                                             :
00212         """
00213         Copy validity rec and payload contents corresponding to seqno 
00214         into a new entry, optionally changing timestart and timeend        
00215 
00216         :param vrec: ValidityRec instance
00217         :param timestart: TimeStamp instance
00218         :param timeend: TimeStamp instance
00219         :param task:
00220         :param target_tablename:
00221         """ 
00222         from DybDbi import ContextRange, SimFlag, TimeStamp
00223 
00224         vrec = self.vrb(seqno)
00225         assert vrec.subsite == self.subsite
00226         assert vrec.contextrange.sitemask == self.sitemask
00227         #assert vrec.contextrange.simmask == self.simmask
00228 
00229         if not target_tablename:
00230             target_tablename = self.tn
00231         if not timestart:
00232             timestart = vrec.contextrange.timestart
00233         if not timeend:
00234             timeend = vrec.contextrange.timeend
00235 
00236         wrt = self.kls.Wrt().Clone()
00237         cr = ContextRange( self.sitemask,  SimFlag.kData , timestart, timeend )
00238         overlay = TimeStamp(0,0)
00239         wrt.ctx( contextrange=cr, subsite=vrec.subsite, task=task, versiondate=overlay, tablename=target_tablename )
00240         rpt = self.kls.Rpt().Clone()
00241         rpt.ctx( validityrec=vrec )
00242         n = len(rpt)
00243         for row in rpt:
00244             wrt.Write(row)
00245         wseqno = wrt.Close()
00246         return wseqno
00247 
00248 


Member Data Documentation

ugly subsite repetition

Definition at line 39 of file history.py.

Definition at line 39 of file history.py.

Definition at line 39 of file history.py.

Definition at line 39 of file history.py.

Definition at line 39 of file history.py.

Definition at line 39 of file history.py.

Definition at line 39 of file history.py.

Definition at line 39 of file history.py.

Definition at line 39 of file history.py.

Definition at line 39 of file history.py.

Definition at line 39 of file history.py.

Definition at line 39 of file history.py.

Definition at line 39 of file history.py.

Definition at line 89 of file history.py.


Property Documentation

history::History::digests = property(lambda self:[self[t]['digest'] for t in sorted(self)], doc="payload digests in timestart seconds order") [static]

Definition at line 100 of file history.py.

history::History::seqnos = property(lambda self:[self[t]['seqno'] for t in sorted(self)] , doc="seqnos in timestart seconds order") [static]

Definition at line 101 of file history.py.


The documentation for this class was generated from the following file:
| Classes | Job Modules | Data Objects | Services | Algorithms | Tools | Packages | Directories | Tracs |

Generated on Fri May 16 2014 09:50:03 for Scraper by doxygen 1.7.4