/search.css" rel="stylesheet" type="text/css"/> /search.js">
| Classes | Job Modules | Data Objects | Services | Algorithms | Tools | Packages | Directories | Tracs |

In This Package:

Public Member Functions | Public Attributes | Static Public Attributes
chunks::Chunks Class Reference

List of all members.

Public Member Functions

def stakeout
def __init__
def count
def __call__
def splitcount
def spawn_payload

Public Attributes

 db
 tn
 payidcol

Static Public Attributes

tuple stakeout = classmethod(stakeout)

Detailed Description

Definition at line 15 of file chunks.py.


Constructor & Destructor Documentation

def chunks::Chunks::__init__ (   self,
  db,
  args 
)

Definition at line 46 of file chunks.py.

00047                                 :
00048         dict.__init__(self)
00049     
00050         dbsect = db.sect 
00051         tn = args.table
00052         payidcol = args.payidcol
00053 
00054         self.db = db
00055         self.tn = tn
00056         self.payidcol = payidcol
00057 
00058         log.info("dbsect %(dbsect)s table %(tn)s " %  locals() )
00059 
00060         path = "%(tn)s.%(payidcol)s.pc" % locals()
00061         if os.path.exists(path): 
00062             log.warn("loading Chunks from cache %s " % path )
00063             d = load(path)
00064             self.update(d)
00065         else:
00066             d = Chunks.stakeout(db, tn, payidcol)
00067             self.update(d)
00068             save(dict(self), path)


Member Function Documentation

def chunks::Chunks::stakeout (   cls,
  db,
  tn,
  payidcol 
)
Split the payload, validity tables into (SITEMASK,SUBSITE,"payid") chunks 

:param db:
:param tn:
:param payidcol:  payload column acting as an identity, such as locationId 

:return: list of dict containing the coordinated of each stake

Definition at line 16 of file chunks.py.

00017                                         :
00018         """
00019         Split the payload, validity tables into (SITEMASK,SUBSITE,"payid") chunks 
00020 
00021         :param db:
00022         :param tn:
00023         :param payidcol:  payload column acting as an identity, such as locationId 
00024 
00025         :return: list of dict containing the coordinated of each stake
00026         """
00027         sss = []
00028         for sitemask in map(lambda _:_['SITEMASK'], db("select distinct(SITEMASK) as SITEMASK from %(tn)sVld" % locals() )):
00029             for subsite in map(lambda _:_['SUBSITE'],  db("select distinct(SUBSITE) as SUBSITE from %(tn)sVld where SITEMASK=%(sitemask)s" % locals() )):
00030                 sss.append( dict(tn=tn,sitemask=sitemask,subsite=subsite,payidcol=payidcol) )
00031         log.info(" distinct (sitemask, subsite) : %s " % len(sss)) 
00032 
00033         ssls = []
00034         for iss,ss in enumerate(sss): 
00035             join = "%(tn)s p join %(tn)sVld v on p.SEQNO=v.SEQNO "
00036             sql = "select distinct(p.%(payidcol)s) as payid from " + join + " where v.SITEMASK=%(sitemask)s and v.SUBSITE=%(subsite)s" 
00037             sql = sql % ss
00038             log.info("iss %s : %s " % ( iss, sql )) 
00039             for payid in map(lambda _:_['payid'], db(sql)): 
00040                 ssl = dict(ss)
00041                 ssl['payid'] = payid 
00042                 ssls.append(ssl) 
00043         log.info(" distinct (sitemask, subsite, payid ) : %s " % len(ssls)) 
        return dict(sss=sss,ssls=ssls) 
def chunks::Chunks::count (   self)

Definition at line 69 of file chunks.py.

00070                    :
00071         tn = self.tn
00072         return self.db("select count(*) as C from %(tn)s" % locals() )[0]['C']
00073 

def chunks::Chunks::__call__ (   self,
  limit = -1 
)
:param limit: -1 for no limit otherwith restrict range for test running
:return: tuple of (sql, ssl ) 
     where sql is payload/validity join chunked sitemask/subsite/locationId string
     and ssl is dict with ingredients of the sql  

Definition at line 74 of file chunks.py.

00075                                 :
00076         """
00077         :param limit: -1 for no limit otherwith restrict range for test running
00078         :return: tuple of (sql, ssl ) 
00079              where sql is payload/validity join chunked sitemask/subsite/locationId string
00080              and ssl is dict with ingredients of the sql  
00081 
00082         """
00083         for issl, ssl in enumerate(self['ssls']):
00084             ssl['index'] = issl
00085             log.debug( "ssl %r " % ssl )
00086             join = "%(tn)s p join %(tn)sVld v on p.SEQNO=v.SEQNO "
00087             where = "v.SITEMASK=%(sitemask)s and v.SUBSITE=%(subsite)s and p.%(payidcol)s='%(payid)s' " 
00088             sql = join + "where " +  where 
00089             if limit == -1 or issl < limit:
00090                 yield sql % ssl, ssl
              
def chunks::Chunks::splitcount (   self)
Check the split counts, to ensure have staked out the full payload table

Definition at line 91 of file chunks.py.

00092                          :
00093         """  
00094         Check the split counts, to ensure have staked out the full payload table
00095         """
00096         db = self.db
00097         tn = self.tn 
00098         splits = [] 
00099         for issl,ssl in enumerate(self['ssls']):
00100             join = "%(tn)s p join %(tn)sVld v on p.SEQNO=v.SEQNO "
00101             where = "v.SITEMASK=%(sitemask)s and v.SUBSITE=%(subsite)s and p.%(payidcol)='%(payid)s' " 
00102             sql = "select count(*) as N from " + join + "where " +  where 
00103             sql = sql % ssl
00104             N = map(lambda _:int(_['N']), db(sql) )[0]
00105             if issl % 100 == 0:
00106                 log.info("issl %s : %s : %s " % ( issl, N, sql )) 
00107             splits.append(N)
00108         return splits 
00109 

def chunks::Chunks::spawn_payload (   self,
  ext 
)
Drop/create a table named with passed extension 

:param ext: table name extension

Definition at line 110 of file chunks.py.

00111                                    :
00112         """
00113         Drop/create a table named with passed extension 
00114 
00115         :param ext: table name extension
00116         """
00117         db = self.db
00118         tn = self.tn 
00119         tnx = "%s%s" % (tn, ext )
00120         drop   = "drop table if exists %(tnx)s " % locals()
00121         create = "create table %(tnx)s like %(tn)s " % locals()
00122         for cmd in (drop, create):
00123             ret = cmd
00124             log.info("doing %s returned %s " % ( cmd, ret )) 
00125         pass
00126         return tnx
00127 
00128 


Member Data Documentation

tuple chunks::Chunks::stakeout = classmethod(stakeout) [static]

Definition at line 44 of file chunks.py.

Definition at line 46 of file chunks.py.

Definition at line 46 of file chunks.py.

Definition at line 46 of file chunks.py.


The documentation for this class was generated from the following file:
| Classes | Job Modules | Data Objects | Services | Algorithms | Tools | Packages | Directories | Tracs |

Generated on Fri May 16 2014 09:50:03 for Scraper by doxygen 1.7.4