/search.css" rel="stylesheet" type="text/css"/> /search.js">
| Classes | Job Modules | Data Objects | Services | Algorithms | Tools | Packages | Directories | Tracs |

In This Package:

Public Member Functions | Public Attributes | Properties | Private Member Functions | Private Attributes
Scraper::dq::CQTable::CQTable Class Reference

List of all members.

Public Member Functions

def __init__
def __repr__
def summary
def getLastSeqno
def getEntries
def deprecated_dqChannelStatus
def deprecated_dqChannelJudge
def getPackedEntries
def __call__

Public Attributes

 db
 tn
 lim
 judge
 packer
 ordering

Properties

 kls = property(_get_kls, doc="lazy access to the genDbi kls")
 lastseqno = property(_get_lastseqno, doc="cached last SEQNO obtained from validity table")

Private Member Functions

def _get_kls
def _get_lastseqno
def _dqChannelPacked

Private Attributes

 _kls
 _lastseqno

Detailed Description

Definition at line 17 of file CQTable.py.


Constructor & Destructor Documentation

def Scraper::dq::CQTable::CQTable::__init__ (   self,
  db,
  tn = "DqChannelStatus",
  lim = 1000 
)
:param db:`DybPython.DB` instance to be queried
:param tn: table name 
:param lim: limit to number of records returned

Definition at line 18 of file CQTable.py.

00019                                                            :
00020         """
00021         :param db:`DybPython.DB` instance to be queried
00022         :param tn: table name 
00023         :param lim: limit to number of records returned
00024         """
00025         self.db = db
00026         self.tn = tn
00027         self.lim = lim
00028         self._kls = None
00029         self._lastseqno = None
00030         self.judge = Judge(table=tn, lastSeqno="%s", limit=lim)
00031         self.packer = CQPacker()
00032         from CQOrdering import CQOrdering   # deferred import to here in order to avoid root `--help` hijack
00033         self.ordering = CQOrdering() 


Member Function Documentation

def Scraper::dq::CQTable::CQTable::_get_kls (   self) [private]
Lazy accessor for the kls, as annoyingly slow during non-DBI development

Definition at line 34 of file CQTable.py.

00035                       :
00036         """
00037         Lazy accessor for the kls, as annoyingly slow during non-DBI development
00038         """
00039         if not self._kls:
00040             import DybDbi
00041             self._kls = getattr(DybDbi,'G'+self.tn)  
        return self._kls
def Scraper::dq::CQTable::CQTable::_get_lastseqno (   self) [private]
Lazy accessor, caution caches the prior value. So will be stale after a write.

Definition at line 44 of file CQTable.py.

00045                             :
00046         """
00047         Lazy accessor, caution caches the prior value. So will be stale after a write.
00048         """
00049         if not self._lastseqno:
00050             self._lastseqno = self.getLastSeqno()
        return self._lastseqno 
def Scraper::dq::CQTable::CQTable::__repr__ (   self)

Definition at line 54 of file CQTable.py.

00055                       :
00056         return "%s %s.%s " % (self.__class__.__name__, self.db.sect, self.tn )

def Scraper::dq::CQTable::CQTable::summary (   self)

Definition at line 57 of file CQTable.py.

00058                      :
00059         return "%-50s %s " % ( repr(self), "lastseqno:%s" % self.lastseqno )

def Scraper::dq::CQTable::CQTable::getLastSeqno (   self)
:return: queries for last seqno in the Vld table 
     
If the table does not exist create it, 
the cached `_lastseqno` value is retained but the cache is 
not consulted, ie queries are always performed.

Definition at line 60 of file CQTable.py.

00061                           :
00062         """ 
00063         :return: queries for last seqno in the Vld table 
00064      
00065         If the table does not exist create it, 
00066         the cached `_lastseqno` value is retained but the cache is 
00067         not consulted, ie queries are always performed.
00068         """
00069         checktb = self('show tables like "%s"' % self.tn )
00070         if len(checktb)==0:
00071             log.warn("creating Database Tables for %s " % self.tn )
00072             self.kls().CreateDatabaseTables(0, self.tn )
00073             last = 0
00074         else:
00075             records = self('select seqno from %sVld order by seqno desc limit 1' % self.tn )
00076             if len(records)==0:
00077                 last = 0 
00078             else:
00079                 last = records[0]['seqno']
00080         log.debug('last seqno in %s.%s : %d' % (self.db.sect, self.tn, last ))
00081         self._lastseqno = last
00082         return last

def Scraper::dq::CQTable::CQTable::getEntries (   self,
  lastSeqno = 0,
  pop = False 
)
MySQL-python (not DybDbi) access

The uncompressed payload level status bits are read together with the 
validity context via a `group by seqno` query againts a payload validity join.
A string dict is composed at mysql level and `eval` brings it into python 
after sanity checks.

:param lastSeqno: cut
:param pop: gets rid of record entries `sstatus` `row_counter` `n`
:return: list of dicts with SEQNO greater than `lastSeqno` restricted to the `limit`

Definition at line 83 of file CQTable.py.

00084                                                  :
00085         """
00086         MySQL-python (not DybDbi) access
00087 
00088         The uncompressed payload level status bits are read together with the 
00089         validity context via a `group by seqno` query againts a payload validity join.
00090         A string dict is composed at mysql level and `eval` brings it into python 
00091         after sanity checks.
00092 
00093         :param lastSeqno: cut
00094         :param pop: gets rid of record entries `sstatus` `row_counter` `n`
00095         :return: list of dicts with SEQNO greater than `lastSeqno` restricted to the `limit`
00096         """
00097 
00098         prepdefault = self.judge.prepdefault
00099         query = self.judge.default   # query appropriate to the table
00100 
00101         self('set group_concat_max_len=4096')
00102         self(prepdefault)
00103         ptndict = re.compile("^{[ 01234567890,:]*}$")   # ensure string really is just a dict of integers, protection against SQL attack
00104         recs = self(query % lastSeqno)
00105         log.debug("getEntries %s recs " % len(recs)) 
00106         for rec in recs:
00107             sstatus     = rec['sstatus']
00108             assert ptndict.match(sstatus), "pre-eval sanity check failure for sstatus %s " % sstatus
00109             rec['dstatus'] = eval(sstatus)  # dict  {channelid:status,...}
00110             assert rec['n'] == rec['row_counter'] == 192, (rec['seqno'],rec['n'], rec['row_counter'] )
00111             if pop:
00112                 rec.pop('sstatus')
00113                 rec.pop('row_counter')
00114                 rec.pop('n')
00115         pass
00116         return recs

def Scraper::dq::CQTable::CQTable::deprecated_dqChannelStatus (   self)
:return: SQL for DBI join query with payload group by SEQNO, the lastSeqno is left as '%s'

Definition at line 117 of file CQTable.py.

00118                                         :
00119         """
00120         :return: SQL for DBI join query with payload group by SEQNO, the lastSeqno is left as '%s'
00121         """
00122         tn = self.tn
00123         assert tn == 'DqChannelStatus'
00124         lastSeqno = "%s"
00125         limit = self.lim
00126         pcol = "row_counter runno fileno".split()
00127         vcol = "timestart timeend sitemask simmask subsite task".split()
00128         vcol_ = lambda field:"v.%(field)s %(field)s" % locals()
00129         pcol_ = lambda field:"max(p.%(field)s) %(field)s" % locals()
00130         cols = ["p.seqno seqno", "count(*) n"]
00131         cols += map(vcol_,vcol )
00132         cols += map(pcol_,pcol )       
00133         cols += ['concat("{",group_concat(concat(p.channelid,":",p.status) order by p.channelid),"}") sstatus']
00134         from_ = "from %(tn)s p inner join %(tn)sVld v on p.seqno = v.seqno where p.seqno > %(lastSeqno)s  " % locals()
00135         tail_ = "group by p.seqno order by p.seqno limit %(limit)s "  % locals()
00136         sql = " ".join(["select", ",".join(cols) , from_, tail_ ])
00137         return sql

def Scraper::dq::CQTable::CQTable::deprecated_dqChannelJudge (   self)
On the fly construct the CQ status judgement from ingredients via a nested case query 
with join to. The idea for the query was developed in :file:`judge.py`

:return: SQL for DBI join query with payload group by SEQNO, the lastSeqno is left as '%s'

Definition at line 138 of file CQTable.py.

00139                                        :
00140         """
00141         On the fly construct the CQ status judgement from ingredients via a nested case query 
00142         with join to. The idea for the query was developed in :file:`judge.py`
00143 
00144         :return: SQL for DBI join query with payload group by SEQNO, the lastSeqno is left as '%s'
00145         """
00146         tn = self.tn
00147         assert tn == 'DqChannel'
00148         judge = Judge(table=tn, lastSeqno="%s", limit=self.lim)
00149         sql = judge.default
00150         return sql

def Scraper::dq::CQTable::CQTable::getPackedEntries (   self,
  lastSeqno = 0,
  pop = False 
)
The compressed status bits read from the table are 
used to populate a channelid keyed status dict `dstatus`
using the channelid ordering discerned from the context of the record.
The ordering is obtained via a CableMap lookup.

:param lastSeqno: cut
:return: list of dicts with SEQNO greater than `lastSeqno` restricted to the `limit`

Definition at line 151 of file CQTable.py.

00152                                                       :
00153         """
00154         The compressed status bits read from the table are 
00155         used to populate a channelid keyed status dict `dstatus`
00156         using the channelid ordering discerned from the context of the record.
00157         The ordering is obtained via a CableMap lookup.
00158 
00159         :param lastSeqno: cut
00160         :return: list of dicts with SEQNO greater than `lastSeqno` restricted to the `limit`
00161         """ 
00162         query = self._dqChannelPacked()
00163         recs = self(query % lastSeqno)
00164         nmask = 7
00165         pass
00166         for rec in recs:
00167             channels = self.ordering.channels(rec['sitemask'], rec['subsite'], rec['timestart'])  
00168             masks = map(lambda _:rec['mask%d'%_], range(nmask))    
00169             lstatus  = map(int,self.packer.unpack(masks))
00170             assert len(lstatus) == len(channels), 'channels inconsistency'
00171             rec['dstatus'] = dict(zip(channels,lstatus))
00172             assert rec['n'] == rec['row_counter'] == 1
00173             if pop:
00174                  rec.pop('row_counter')
00175                  rec.pop('n')
00176                  for i in range(nmask):
00177                      rec.pop('mask%d' % i)
00178         pass 
00179         return recs

def Scraper::dq::CQTable::CQTable::_dqChannelPacked (   self) [private]
:return: SQL for DBI join query with payload group by SEQNO, the lastSeqno is left as '%s'

Definition at line 180 of file CQTable.py.

00181                               :
00182         """
00183         :return: SQL for DBI join query with payload group by SEQNO, the lastSeqno is left as '%s'
00184         """
00185         tn = self.tn
00186         assert tn == 'DqChannelPacked'
00187         lastSeqno = "%s"
00188         limit = self.lim
00189         pcol = "row_counter runno fileno mask0 mask1 mask2 mask3 mask4 mask5 mask6".split()
00190         vcol = "timestart timeend sitemask simmask subsite task".split()
00191         vcol_ = lambda field:"v.%(field)s %(field)s" % locals()
00192         pcol_ = lambda field:"p.%(field)s %(field)s" % locals()
00193         cols = ["p.seqno seqno", "count(*) n"]
00194         cols += map(vcol_,vcol )
00195         cols += map(pcol_,pcol )       
00196         from_ = "from %(tn)s p inner join %(tn)sVld v on p.seqno = v.seqno where p.seqno > %(lastSeqno)s  " % locals()
00197         tail_ = "group by p.seqno order by p.seqno limit %(limit)s "  % locals()
00198         sql = " ".join(["select", ",".join(cols) , from_, tail_ ])
00199         return sql

def Scraper::dq::CQTable::CQTable::__call__ (   self,
  query 
)

Definition at line 200 of file CQTable.py.

00201                              :
00202         log.debug(query)
00203         return self.db(query)
00204 
00205 
00206 
00207 
00208  
00209 


Member Data Documentation

Definition at line 22 of file CQTable.py.

Definition at line 22 of file CQTable.py.

Definition at line 22 of file CQTable.py.

Definition at line 22 of file CQTable.py.

Definition at line 22 of file CQTable.py.

Definition at line 22 of file CQTable.py.

Definition at line 22 of file CQTable.py.

Definition at line 22 of file CQTable.py.


Property Documentation

Scraper::dq::CQTable::CQTable::kls = property(_get_kls, doc="lazy access to the genDbi kls") [static]

Definition at line 42 of file CQTable.py.

Scraper::dq::CQTable::CQTable::lastseqno = property(_get_lastseqno, doc="cached last SEQNO obtained from validity table") [static]

Definition at line 51 of file CQTable.py.


The documentation for this class was generated from the following file:
| Classes | Job Modules | Data Objects | Services | Algorithms | Tools | Packages | Directories | Tracs |

Generated on Fri May 16 2014 09:50:03 for Scraper by doxygen 1.7.4