/search.css" rel="stylesheet" type="text/css"/> /search.js">
| Classes | Job Modules | Data Objects | Services | Algorithms | Tools | Packages | Directories | Tracs |

In This Package:

Public Member Functions | Public Attributes | Static Public Attributes | Properties | Private Attributes
DybPython::asciicat::AsciiCat Class Reference

List of all members.

Public Member Functions

def parse_relpath
def findcat
def __init__
def compare
def __repr__
def read
def read_catalog
def fake_write
def read_seqno
def check_seqno
def get_seqno
def clear_cache
def read_allseqno
def check_allseqno
def get_allseqno
def get_fabseqno
def write
def write_catalog
def resolve
def updates
def seqno_updates
def allseqno_updates
def diff

Public Attributes

 skip_pay_check
 allow_partial
 dir
 v
 p

Static Public Attributes

tuple ptn = re.compile("(?P<tdir>[^/]*)/(?P<tnam>\S*)\.csv$")

Properties

 showtables = property( lambda self:sorted(self.keys()))
 orderpk = property(lambda self:sorted(self))
 seqno = property( get_seqno , doc=get_seqno.__doc__ )
 allseqno = property( get_allseqno , doc=get_allseqno.__doc__ )
 fabseqno = property( get_fabseqno, doc=get_fabseqno.__doc__ )

Private Attributes

 _seqno
 _allseqno
 _fabseqno

Detailed Description

DBI Ascii catalog (a tablename keyed dict holding AsciiCSV representing each table), usage::

    import logging
    logging.basicConfig(level=logging.INFO)

    cat = AsciiCat("~/dybaux/catalog/tmp_offline_db")  

    for tn,csv in cat.items():
        print tn, csv

    for tn,lastseqno in cat.seqno.items():
        print cat[tn+'Vld'][lastseqno]         

    for tn in cat.orderpk:
        csv = cat[tn]
        print tn, csv.abspath, csv.tabname, csv.tabbase, csv.orderpk[0:100]


Definition at line 312 of file asciicat.py.


Constructor & Destructor Documentation

def DybPython::asciicat::AsciiCat::__init__ (   self,
  dir,
  skip_pay_check = False,
  allow_partial = False 
)
:param dir:
:param skip_pay_check: when True skip loading payload tables in memory 

Definition at line 356 of file asciicat.py.

00357                                                                          :
00358         """
00359         :param dir:
00360         :param skip_pay_check: when True skip loading payload tables in memory 
00361         """
00362         dir = os.path.normpath(pathx(dir))
00363         assert os.path.isdir(dir) , dir
00364         catpath = self.findcat( dir ) 
00365 
00366         log.debug("AsciiCat catpath %s skip_pay_check %s allow_partial %s " % (catpath,skip_pay_check,allow_partial) )
00367 
00368         self.skip_pay_check = skip_pay_check   
00369         self.allow_partial = allow_partial   
00370         self.dir = dir
00371         self.v = {}
00372         self.p = {}
00373         pass
00374         self.read( catpath )


Member Function Documentation

def DybPython::asciicat::AsciiCat::parse_relpath (   self,
  relpath 
)

Definition at line 335 of file asciicat.py.

00336                                     :
00337         m = self.ptn.match(relpath)
00338         assert m, "failed to match %s " % relpath 
00339         d = m.groupdict()
00340         tab = d['tdir']
00341         d['tab'] = tab
00342         if d['tdir'] == d['tnam']:
00343             d['type'] = "pay"
00344         elif "%sVld" % d['tdir'] == d['tnam']:
00345             d['type'] = "vld"
00346         return dict(d, relpath=relpath) 

def DybPython::asciicat::AsciiCat::findcat (   cls,
  dir 
)

Definition at line 351 of file asciicat.py.

00351                          : 
00352         catz = filter(lambda n:n[-4:] == ".cat" , os.listdir(dir) )
00353         assert len(catz) == 1, "must be only one .cat in dir %s " % dir
00354         return os.path.join( dir, catz[0] )
00355 
def DybPython::asciicat::AsciiCat::compare (   self,
  other 
)
Checking cat comparison, setup old and new catalogs::

    rm -rf ~/dybaux/catalog/tmp_offline_db
    svn up -r 4968 ~/dybaux/catalog/tmp_offline_db      ## clean r4968 just prior to ordering standardization
    rm -rf ~/tmp_offline_db ; db.py offline_db rdumpcat ~/tmp_offline_db   ## same named catalog with the stanbdard rdumpcat ordering 

.. code-block:: ipython

In [15]: old = AsciiCat("~/dybaux/catalog/tmp_offline_db")
INFO:__main__:read /home/blyth/dybaux/catalog/tmp_offline_db/tmp_offline_db.cat 
INFO:__main__:done AsciiCat [13   ] /home/blyth/dybaux/catalog/tmp_offline_db {'CalibPmtSpec': 50, 'FeeCableMap': 3, 'HardwareID': 372, 'CalibFeeSpec': 113, 'CableMap': 460}  

In [16]: new = AsciiCat("~/tmp_offline_db")
INFO:__main__:read /home/blyth/tmp_offline_db/tmp_offline_db.cat 
INFO:__main__:done AsciiCat [13   ] /home/blyth/tmp_offline_db {'CalibPmtSpec': 50, 'FeeCableMap': 3, 'HardwareID': 372, 'CalibFeeSpec': 113, 'CableMap': 460}  

In [17]: old == new
Out[17]: True

In [18]: old['CableMap'][(1,1)]['CHANNELID']
Out[18]: '16908545'

In [19]: old['CableMap'][(1,1)]['CHANNELID'] += "changed"

In [20]: old == new
Out[20]: False

In [21]: old['CableMap'][(1,1)]['CHANNELID'] = '16908545'

In [22]: old == new
Out[22]: True

In [25]: old.diff(new)
INFO:__main__:diffing with "diff -r --brief /home/blyth/dybaux/catalog/tmp_offline_db /home/blyth/tmp_offline_db | grep -v .svn "
INFO:__main__:diff Files /home/blyth/dybaux/catalog/tmp_offline_db/CalibPmtSpec/CalibPmtSpec.csv and /home/blyth/tmp_offline_db/CalibPmtSpec/CalibPmtSpec.csv differ
INFO:__main__:diff Files /home/blyth/dybaux/catalog/tmp_offline_db/CalibPmtSpec/CalibPmtSpecVld.csv and /home/blyth/tmp_offline_db/CalibPmtSpec/CalibPmtSpecVld.csv differ
INFO:__main__:diff Files /home/blyth/dybaux/catalog/tmp_offline_db/LOCALSEQNO/LOCALSEQNO.csv and /home/blyth/tmp_offline_db/LOCALSEQNO/LOCALSEQNO.csv differ
INFO:__main__:diff Files /home/blyth/dybaux/catalog/tmp_offline_db/tmp_offline_db.cat and /home/blyth/tmp_offline_db/tmp_offline_db.cat differ

In [27]: old.write()
WARNING:__main__:overwriting catalog into dir /home/blyth/dybaux/catalog/tmp_offline_db 

In [28]: old.diff(new)
INFO:__main__:diffing with "diff -r --brief /home/blyth/dybaux/catalog/tmp_offline_db /home/blyth/tmp_offline_db | grep -v .svn "

Definition at line 375 of file asciicat.py.

00376                             :
00377         """
00378         Checking cat comparison, setup old and new catalogs::
00379 
00380             rm -rf ~/dybaux/catalog/tmp_offline_db
00381             svn up -r 4968 ~/dybaux/catalog/tmp_offline_db      ## clean r4968 just prior to ordering standardization
00382             rm -rf ~/tmp_offline_db ; db.py offline_db rdumpcat ~/tmp_offline_db   ## same named catalog with the stanbdard rdumpcat ordering 
00383 
00384         .. code-block:: ipython
00385 
00386                 In [15]: old = AsciiCat("~/dybaux/catalog/tmp_offline_db")
00387                 INFO:__main__:read /home/blyth/dybaux/catalog/tmp_offline_db/tmp_offline_db.cat 
00388                 INFO:__main__:done AsciiCat [13   ] /home/blyth/dybaux/catalog/tmp_offline_db {'CalibPmtSpec': 50, 'FeeCableMap': 3, 'HardwareID': 372, 'CalibFeeSpec': 113, 'CableMap': 460}  
00389 
00390                 In [16]: new = AsciiCat("~/tmp_offline_db")
00391                 INFO:__main__:read /home/blyth/tmp_offline_db/tmp_offline_db.cat 
00392                 INFO:__main__:done AsciiCat [13   ] /home/blyth/tmp_offline_db {'CalibPmtSpec': 50, 'FeeCableMap': 3, 'HardwareID': 372, 'CalibFeeSpec': 113, 'CableMap': 460}  
00393 
00394                 In [17]: old == new
00395                 Out[17]: True
00396 
00397                 In [18]: old['CableMap'][(1,1)]['CHANNELID']
00398                 Out[18]: '16908545'
00399 
00400                 In [19]: old['CableMap'][(1,1)]['CHANNELID'] += "changed"
00401 
00402                 In [20]: old == new
00403                 Out[20]: False
00404 
00405                 In [21]: old['CableMap'][(1,1)]['CHANNELID'] = '16908545'
00406 
00407                 In [22]: old == new
00408                 Out[22]: True
00409 
00410                 In [25]: old.diff(new)
00411                 INFO:__main__:diffing with "diff -r --brief /home/blyth/dybaux/catalog/tmp_offline_db /home/blyth/tmp_offline_db | grep -v .svn "
00412                 INFO:__main__:diff Files /home/blyth/dybaux/catalog/tmp_offline_db/CalibPmtSpec/CalibPmtSpec.csv and /home/blyth/tmp_offline_db/CalibPmtSpec/CalibPmtSpec.csv differ
00413                 INFO:__main__:diff Files /home/blyth/dybaux/catalog/tmp_offline_db/CalibPmtSpec/CalibPmtSpecVld.csv and /home/blyth/tmp_offline_db/CalibPmtSpec/CalibPmtSpecVld.csv differ
00414                 INFO:__main__:diff Files /home/blyth/dybaux/catalog/tmp_offline_db/LOCALSEQNO/LOCALSEQNO.csv and /home/blyth/tmp_offline_db/LOCALSEQNO/LOCALSEQNO.csv differ
00415                 INFO:__main__:diff Files /home/blyth/dybaux/catalog/tmp_offline_db/tmp_offline_db.cat and /home/blyth/tmp_offline_db/tmp_offline_db.cat differ
00416 
00417                 In [27]: old.write()
00418                 WARNING:__main__:overwriting catalog into dir /home/blyth/dybaux/catalog/tmp_offline_db 
00419 
00420                 In [28]: old.diff(new)
00421                 INFO:__main__:diffing with "diff -r --brief /home/blyth/dybaux/catalog/tmp_offline_db /home/blyth/tmp_offline_db | grep -v .svn "
00422 
00423         """
00424         return self == other  

def DybPython::asciicat::AsciiCat::__repr__ (   self)

Definition at line 425 of file asciicat.py.

00426                       :
00427         return "%s [%-5d] %s %r " % ( self.__class__.__name__, len(self), self.dir, self.seqno )

def DybPython::asciicat::AsciiCat::read (   self,
  catpath 
)
Parses Vld and LOCALSEQNO tables by default, reading in the 
csv content into memory.  Only reads payload tables when `skip_pay_check=False`
Reading payload tables for larger tables is slow and memory expensive. 

Definition at line 428 of file asciicat.py.

00428                            : 
00429         """
00430         Parses Vld and LOCALSEQNO tables by default, reading in the 
00431         csv content into memory.  Only reads payload tables when `skip_pay_check=False`
00432         Reading payload tables for larger tables is slow and memory expensive. 
00433 
00434         """
00435         log.info("read %s " % catpath ) 
00436         self.read_catalog( catpath )
00437 
00438         for tn in self.orderpk:
00439             read = False
00440             if tn[-3:] == 'Vld' or tn == 'LOCALSEQNO':
00441                 read = True 
00442             else:
00443                 read = not self.skip_pay_check
00444             if read:
00445                 log.info("reading table %s " % tn ) 
00446                 self[tn].read()
00447             else:
00448                 log.debug("NOT reading table %s " % tn ) 
00449  
00450         self.read_seqno()
00451         self.check_seqno()
00452         log.info("done %r " % self ) 
00453 
def DybPython::asciicat::AsciiCat::read_catalog (   self,
  catpath 
)
Reads ascii catalog into memory.  

This just reads the small .cat file and sets up lightweight 
AsciiCSV objects without reading in csv content.

Definition at line 454 of file asciicat.py.

00455                                     :
00456         """
00457         Reads ascii catalog into memory.  
00458 
00459         This just reads the small .cat file and sets up lightweight 
00460         AsciiCSV objects without reading in csv content.
00461         """
00462         content = open( catpath , "r" ).read().strip()
00463         relpaths = content.split("\n")
00464         assert relpaths[0] == "name", "unexpected catalog header line %s " % relpaths[0]
00465         for d in map(lambda _:self.parse_relpath(_), relpaths[1:] ):
00466             typ, tab = d.get('type',None), d.get('tab', None)
00467             assert typ and tab 
00468             csv = AsciiCSV(self, d['relpath'], **d )
00469             if typ == 'pay': 
00470                 self.p[tab] = csv   
00471             elif typ == 'vld':
00472                 self.v[tab] = csv
00473             pass
00474             self[csv.pk] = csv   ## keyed by tablename
00475             pass
00476 
00477 

def DybPython::asciicat::AsciiCat::fake_write (   self,
  tn,
  npay = 1 
)
Fakes a single validity entry and ``npay`` payload entries, by copying the 
last entries in validity and payload tables and with a 
fabricated SEQNO or (SEQNO,ROW_COUNTER) pks.  The `tn` entry in the LOCALSEQNO
table is also changed to mimic a real DBI write.

:param tn: payload table name
:param npay: number of payload rows to write

Usage::

    cat = AsciiCat("~/dybaux/catalog/tmp_offline_db")
    cat.fake_write('HardwareID', 10 )
    cat.write()                       ## CAUTION inplace overwriting

Definition at line 478 of file asciicat.py.

00479                                        :
00480         """
00481         Fakes a single validity entry and ``npay`` payload entries, by copying the 
00482         last entries in validity and payload tables and with a 
00483         fabricated SEQNO or (SEQNO,ROW_COUNTER) pks.  The `tn` entry in the LOCALSEQNO
00484         table is also changed to mimic a real DBI write.
00485 
00486         :param tn: payload table name
00487         :param npay: number of payload rows to write
00488 
00489         Usage::
00490 
00491             cat = AsciiCat("~/dybaux/catalog/tmp_offline_db")
00492             cat.fake_write('HardwareID', 10 )
00493             cat.write()                       ## CAUTION inplace overwriting
00494 
00495         """
00496         p,v,m = self[tn], self[tn+'Vld'],self['LOCALSEQNO']
00497         assert len(p) > 1 and len(v) > 1 and len(m) > 1, ("cannot fake into empty tables ", len(p),len(v),len(m))
00498 
00499         a = v.last.clone()            ## pk ordered
00500         lastseqno = a.pk              ## get_pk
00501 
00502         l = p.last                    ## l.pk is tuple with SEQNO,ROW_COUNTER values
00503         assert l.pk[0] == lastseqno == int(m[tn]['LASTUSEDSEQNO']) , ( "lastseqno mismatch ", l, l.pk, lastseqno , m[tn] )
00504 
00505         fakeseqno = lastseqno + 1 
00506         a.pk = fakeseqno              ## set_pk     
00507         v[a.pk] = a                   ## fake validity entry 
00508 
00509         for n in range(npay):
00510            b = l.clone()
00511            b.pk = (fakeseqno, n+1 )   ## set_pk 
00512            p[b.pk] = b                ## fake payload entries
00513 
00514         m[tn]['LASTUSEDSEQNO'] = str(fakeseqno)  
00515 
00516         log.info("fake_write %s %s added LASTUSEDSEQNO %s " % ( tn,npay,fakeseqno )) 
00517 
00518         self.clear_cache()
00519 
00520 

def DybPython::asciicat::AsciiCat::read_seqno (   self,
  tab = "LOCALSEQNO" 
)
Reads the LASTUSEDSEQNO entries from the LOCALSEQNO table into seqno dict

Definition at line 521 of file asciicat.py.

00522                                           :
00523         """
00524         Reads the LASTUSEDSEQNO entries from the LOCALSEQNO table into seqno dict
00525         """ 
00526         self._seqno = {}
00527         for pk,row in self[tab].items():
00528             t = row['TABLENAME']
00529             n = int(row['LASTUSEDSEQNO'])
00530             if t == "*":continue
00531             self._seqno[t] = n       
        log.debug( "LASTUSEDSEQNO %r FROM %s  " % ( self._seqno, tab)  )
def DybPython::asciicat::AsciiCat::check_seqno (   self)
Checks correspondence between the max SEQNO in the Vld and Pay tables 
with that recorded in the LOCALSEQNO table

Note that without the very expensive db.py option: `--EXTRA_PAYLOAD_CHECK`
which caused setting of `self.skip_pay_check` only the validity tables are checked. 

The allseqno called inside the maxseqno property is very expensive, causing 
MemoryError at IHEP

Definition at line 532 of file asciicat.py.

00532                          :  
00533         """
00534         Checks correspondence between the max SEQNO in the Vld and Pay tables 
00535         with that recorded in the LOCALSEQNO table
00536 
00537         Note that without the very expensive db.py option: `--EXTRA_PAYLOAD_CHECK`
00538         which caused setting of `self.skip_pay_check` only the validity tables are checked. 
00539 
00540         The allseqno called inside the maxseqno property is very expensive, causing 
00541         MemoryError at IHEP
00542         """ 
00543         for t,n in self._seqno.items():
00544             v = self.get(t+'Vld',[])
00545             p = self.get(t,[])
00546             if len(v) > 0:
00547                 vmax = v.maxseqno
00548                 assert vmax == n , ( vmax , n )
00549             if len(p) > 0:
00550                 pmax = p.maxseqno
00551                 assert pmax == n , ( pmax , n )
00552 
def DybPython::asciicat::AsciiCat::get_seqno (   self)
Tablename keyed dict of LASTUSEDSEQNO values obtained from LOCALSEQNO table

Definition at line 553 of file asciicat.py.

00554                        :
00555         """
00556         Tablename keyed dict of LASTUSEDSEQNO values obtained from LOCALSEQNO table
00557         """ 
00558         if hasattr(self,'_seqno'):
00559             return self._seqno
00560         self.read_seqno()
00561         self.check_seqno()
        return self._seqno
def DybPython::asciicat::AsciiCat::clear_cache (   self)
delete cached attributes forcing recalc on next access ...
needed after faking for example  

Definition at line 565 of file asciicat.py.

00566                          :
00567         """
00568         delete cached attributes forcing recalc on next access ...
00569         needed after faking for example  
00570         """
00571         if hasattr(self, '_fabseqno' ):
00572             del self._fabseqno
00573         if hasattr(self, '_seqno' ):
00574             del self._seqno
00575         if hasattr(self, '_allseqno' ):
00576             del self._allseqno 

def DybPython::asciicat::AsciiCat::read_allseqno (   self)
Populates the table name keyed `self._allseqno` dict with results
from the `.allseqno` property applied to individual `AsciiCSV` for each Vld table.  

The `self.seqno` dict is populated by reading the LOCALSEQNO.csv so when the 
tables represented in that do not correspond to those of the catalog. 

Definition at line 577 of file asciicat.py.

00578                            :
00579         """
00580         Populates the table name keyed `self._allseqno` dict with results
00581         from the `.allseqno` property applied to individual `AsciiCSV` for each Vld table.  
00582 
00583         The `self.seqno` dict is populated by reading the LOCALSEQNO.csv so when the 
00584         tables represented in that do not correspond to those of the catalog. 
00585 
00586         """ 
00587         self._allseqno = {}
00588         for tn,lus in self.seqno.items():
00589             tnv = "%sVld"%tn
00590             csvv = self.get(tnv, None)
00591             if not csvv and self.allow_partial:continue  
00592             assert csvv, "read_allseqno failed to find AsciiCSV for %s : consider use of -P,--ALLOW_PARTIAL option to allow dealing in partial catalogs " % tnv 
            self._allseqno[tn] = csvv.allseqno
def DybPython::asciicat::AsciiCat::check_allseqno (   self)

Definition at line 593 of file asciicat.py.

00594                             :
00595         if self.allow_partial:return
00596         assert sorted(self._allseqno.keys()) == sorted(self._seqno.keys()), "seqno keys mismatch "
        pass
def DybPython::asciicat::AsciiCat::get_allseqno (   self)
Populates, checks and returns the `self._allseqno` dict which is keyed 
on tablename and holds the collected allseqno from the indicdual AsciiCSV

Definition at line 597 of file asciicat.py.

00598                           :
00599         """
00600         Populates, checks and returns the `self._allseqno` dict which is keyed 
00601         on tablename and holds the collected allseqno from the indicdual AsciiCSV
00602         """ 
00603         if hasattr(self,'_allseqno'):
00604             return self._allseqno
00605         self.read_allseqno()
00606         self.check_allseqno()
        return self._allseqno
def DybPython::asciicat::AsciiCat::get_fabseqno (   self)
Summarizes ``cat.allseqno``, by fabricating a dict keyed by table name containing 
the number of Vld SEQNO (from length of values in ``cat.allseqno``) 

This dict can be compared with ``cat.seqno``, which is obtained from 
the LASTUSEDSEQNO entries in the ``LOCALSEQNO`` table::
Assuming kosher DBI handling of tables this fabricated dict ``cat.fabseqno`` should 
match ``cat.seqno``, meaning that SEQNO start from 1 and have no gaps.

       .. code-block:: ipython          

In [1]: from DybPython import AsciiCat

In [2]: cat = AsciiCat("~/dybaux/catalog/tmp_offline_db")

In [3]: db.seqno   ## queries the LOCALSEQNO table in DB
Out[3]: 
{'CableMap': 213,
 'CalibFeeSpec': 113,
 'CalibPmtSpec': 29,
 'FeeCableMap': 3,
 'HardwareID': 172}

In [4]: db.fabseqno    ## a summarization of db.allseqno
Out[4]: 
{'CableMap': 213,
 'CalibFeeSpec': 111,
 'CalibPmtSpec': 8,
 'FeeCableMap': 3,
 'HardwareID': 172}

In [5]: db.miscreants()   ## assertions avoided by miscreant status
Out[5]: ('CalibPmtSpec', 'CalibFeeSpec')


Definition at line 609 of file asciicat.py.

00610                           :
00611         """
00612         Summarizes ``cat.allseqno``, by fabricating a dict keyed by table name containing 
00613         the number of Vld SEQNO (from length of values in ``cat.allseqno``) 
00614 
00615         This dict can be compared with ``cat.seqno``, which is obtained from 
00616         the LASTUSEDSEQNO entries in the ``LOCALSEQNO`` table::
00617         Assuming kosher DBI handling of tables this fabricated dict ``cat.fabseqno`` should 
00618         match ``cat.seqno``, meaning that SEQNO start from 1 and have no gaps.
00619 
00620        .. code-block:: ipython          
00621 
00622                 In [1]: from DybPython import AsciiCat
00623 
00624                 In [2]: cat = AsciiCat("~/dybaux/catalog/tmp_offline_db")
00625 
00626                 In [3]: db.seqno   ## queries the LOCALSEQNO table in DB
00627                 Out[3]: 
00628                 {'CableMap': 213,
00629                  'CalibFeeSpec': 113,
00630                  'CalibPmtSpec': 29,
00631                  'FeeCableMap': 3,
00632                  'HardwareID': 172}
00633 
00634                 In [4]: db.fabseqno    ## a summarization of db.allseqno
00635                 Out[4]: 
00636                 {'CableMap': 213,
00637                  'CalibFeeSpec': 111,
00638                  'CalibPmtSpec': 8,
00639                  'FeeCableMap': 3,
00640                  'HardwareID': 172}
00641 
00642                 In [5]: db.miscreants()   ## assertions avoided by miscreant status
00643                 Out[5]: ('CalibPmtSpec', 'CalibFeeSpec')
00644 
00645 
00646         """
00647         if hasattr(self, '_fabseqno' ):
00648             return self._fabseqno
00649         self._fabseqno =  dict(map(lambda(k,v):(k,len(v)),self.allseqno.items()))
        return self._fabseqno
def DybPython::asciicat::AsciiCat::write (   self,
  xdir = None 
)
:param xdir: write directory 

Writes catalog into the ``xdir`` directory named after directory basename

Definition at line 653 of file asciicat.py.

00654                                :
00655         """
00656         :param xdir: write directory 
00657 
00658         Writes catalog into the ``xdir`` directory named after directory basename
00659         """
00660         if xdir==None:
00661             xdir = self.dir
00662             log.warn("overwriting catalog into dir %s " % xdir )
00663         else:
00664             xdir = pathx(xdir)
00665             assert xdir != self.dir
00666         self.write_catalog( base=xdir )
00667         for tn in self.orderpk: 
00668             csv = self[tn]
00669             csv.write(base=xdir)

def DybPython::asciicat::AsciiCat::write_catalog (   self,
  base 
)
writes the .cat file into directory `base` the name of the `.cat` is 
obtained from the basename of `base` 

Definition at line 670 of file asciicat.py.

00671                                   :
00672         """
00673         writes the .cat file into directory `base` the name of the `.cat` is 
00674         obtained from the basename of `base` 
00675         """
00676         if not os.path.exists(base):
00677             log.info("creating %s " % base ) 
00678             os.makedirs(base)
00679         outpath = os.path.join( base , "%s.cat" % os.path.basename(base) )
00680         log.debug("write_catalog to %s " % outpath )
00681 
00682         fp = open(outpath,"w")
00683         fp.write("\n".join(["name"] + map(lambda tn:self[tn].path, self.orderpk)) + "\n" ) 
00684         fp.close()           

def DybPython::asciicat::AsciiCat::resolve (   self,
  relpath,
  base = None 
)
Resolve a relative path wrt the base

Definition at line 685 of file asciicat.py.

00686                                           :
00687         """Resolve a relative path wrt the base""" 
00688         if base == None:
00689             base = self.dir 
00690         return os.path.join( base , relpath )
00691 

def DybPython::asciicat::AsciiCat::updates (   self,
  target,
  tselect = [],
  fastforward = False 
)
:param target: DBI store (AsciiCat or DB) but usually the target DB to be updated

Return tablename (payload only) keyed dict listing new SEQNO 
and prepares the ascii catalog for loading into DB, by fastforwarding INSERTDATE to UTC now. 

Definition at line 692 of file asciicat.py.

00693                                                              :
00694         """
00695         :param target: DBI store (AsciiCat or DB) but usually the target DB to be updated
00696 
00697         Return tablename (payload only) keyed dict listing new SEQNO 
00698         and prepares the ascii catalog for loading into DB, by fastforwarding INSERTDATE to UTC now. 
00699 
00700         """
00701         log.debug("updates tselect %r " % tselect )
00702         upls = self.seqno_updates(target, tselect=tselect)
00703         upas = self.allseqno_updates(target, tselect=tselect )
00704         assert upls.keys() == upas.keys(), ("updates keys mismatch ", upls.keys(), upas.keys() )
00705 
00706         # checking correspondence between LASTUSEDSEQNO and the last of the lists of allseqno
00707         for tn in upas:
00708             assert upls[tn] == upas[tn][-1] , ("seqno vs allseqno mismatch %s" % tn, upls[tn], upas[tn][-1] ) 
00709             #print tn,  upls[tn], upas[tn][-1] , upas[tn][0] 
00710 
00711         if fastforward: 
00712             now = datetime.utcnow()
00713             now = now.strftime("%Y-%m-%d %H:%M:%S")
00714             for tn in upas:
00715                 n = 0 
00716                 vld = self[tn+'Vld']
00717                 for sq in upas[tn]:
00718                     n += 1
00719                     vld[sq]['INSERTDATE'] = now
00720                 if n > 0: 
00721                     log.info("fastforward %s validity rows of %s to %s " % (n,tn,now) )
00722                     vld.write()
00723         else: 
00724             pass
00725         return upas
00726 

def DybPython::asciicat::AsciiCat::seqno_updates (   self,
  target,
  tselect = [] 
)
:param target: DBI store (AsciiCat or DB) but usually the target DB to be updated
:param tselect: restrict comparisons to payload tables in tselect, if tselect list is populated 

This operates by comparing LASTUSEDSEQNO entries in the LOCALSEQNO tables, via 
the ``.seqno`` 

Provides dict keyed by table names of LASTUSEDSEQNO values (payload names only) for 
new or changed LASTUSEDSEQNO entries in the ascii catalog.  

For example this AsciiCat might contain::

      {'Noodles': 15, 'CalibPmtSpec': 29, 'FeeCableMap': 3, 'HardwareID': 172, 'CalibFeeSpec': 113, 'CableMap': 213}

Whereas the target DB contains::
   
      {'Noodles': 10, 'CalibPmtSpec': 29, 'FeeCableMap': 3, 'CalibFeeSpec': 113}  
 
This would return::

      {'Noodles':15 ,'HardwareID': 172 ,'CableMap':213}


Note the use of SEQNO 0 to mark not-present


This way of checking for updates works fine without the payload tables in memory
as it is just using the LASTUSEDSEQNO counts hailing from LOCALSEQNO.csv
BUT make sure are rcmpcat comparing against a propa full DB not the decoupled subset of tables
that tmp_offline_db often is.

Definition at line 727 of file asciicat.py.

00728                                                  :
00729         """
00730         :param target: DBI store (AsciiCat or DB) but usually the target DB to be updated
00731         :param tselect: restrict comparisons to payload tables in tselect, if tselect list is populated 
00732 
00733         This operates by comparing LASTUSEDSEQNO entries in the LOCALSEQNO tables, via 
00734         the ``.seqno`` 
00735 
00736         Provides dict keyed by table names of LASTUSEDSEQNO values (payload names only) for 
00737         new or changed LASTUSEDSEQNO entries in the ascii catalog.  
00738 
00739         For example this AsciiCat might contain::
00740 
00741               {'Noodles': 15, 'CalibPmtSpec': 29, 'FeeCableMap': 3, 'HardwareID': 172, 'CalibFeeSpec': 113, 'CableMap': 213}
00742 
00743         Whereas the target DB contains::
00744            
00745               {'Noodles': 10, 'CalibPmtSpec': 29, 'FeeCableMap': 3, 'CalibFeeSpec': 113}  
00746  
00747         This would return::
00748 
00749               {'Noodles':15 ,'HardwareID': 172 ,'CableMap':213}
00750         
00751 
00752         Note the use of SEQNO 0 to mark not-present
00753 
00754 
00755         This way of checking for updates works fine without the payload tables in memory
00756         as it is just using the LASTUSEDSEQNO counts hailing from LOCALSEQNO.csv
00757         BUT make sure are rcmpcat comparing against a propa full DB not the decoupled subset of tables
00758         that tmp_offline_db often is.
00759         """
00760         upls = {}
00761 
00762         
00763         ttabs = filter(lambda t:len(tselect) == 0 or t in tselect, target.seqno )
00764         log.debug("target tables %r " % ttabs )         
00765         missing = list(set(ttabs).difference(set(self.seqno)))
00766         assert len(missing) == 0, "tables in target that are not in ascii catalog %r \n try restricting table selection with -t option " % missing
00767         
00768         log.debug("seqno_updates self.seqno %r " % self.seqno )         
00769         log.debug("seqno_updates target.seqno %r " % target.seqno )         
00770 
00771         alldbi = list(set(self.seqno).union(set(target.seqno)))
00772         for tn in alldbi:
00773             if len(tselect) > 0 and tn not in tselect:
00774                 continue
00775             acls = self.seqno.get(tn,0)     ## ascii catalog LASTUSEDSEQNO for table
00776             tgls = target.seqno.get(tn,0)   ## LASTUSEDSEQNO in target db 
00777             #log.info("seqno_updates acls %s tgls %s " % (acls,tgls) )
00778             assert acls >= tgls, ("ERROR LASTUSEDSEQNO in target exceeds that in ascii cat %s " % tn, acls, tgls)
00779             if acls > tgls:
00780                 upls[tn] = acls
00781             else:
00782                 pass  
00783         log.info("seqno_updates : ascii catalog LASTUSEDSEQNO changes relative to target : %r " % upls )
00784         return upls

def DybPython::asciicat::AsciiCat::allseqno_updates (   self,
  target,
  tselect = [] 
)
:param target: DBI store (AsciiCat or DB) but usually the target DB to be updated

This operates by comparing all SEQNO in the tables

Definition at line 785 of file asciicat.py.

00786                                                    :
00787         """
00788         :param target: DBI store (AsciiCat or DB) but usually the target DB to be updated
00789         
00790         This operates by comparing all SEQNO in the tables
00791 
00792         """ 
00793         upas = {}
00794         alldbi = list(set(self.allseqno).union(set(target.allseqno)))
00795         for tn in alldbi:
00796             if len(tselect) > 0 and tn not in tselect:
00797                 continue
00798             acsq = set(self.allseqno.get(tn,[]))         
00799             tgsq = set(target.allseqno.get(tn,[]))      
00800             #log.info("allseqno_updates tn %s acsq %s tgsq %s " % (tn,acsq,tgsq))
00801             assert len(tgsq.difference(acsq)) == 0 , "ERROR SEQNO in db that are not present in ascii catalog table %s  " % tn
00802             upsq = sorted(list(acsq.difference(tgsq)))     ## SEQNO in ascii catalog but not target DB
00803             if len(upsq) > 0:
00804                 upas[tn] = upsq 
00805             else:
00806                 pass
00807         for tn,seq in upas.items():
00808             contiguous = seq[-1] - seq[0] + 1  == len(seq) 
00809             if not contiguous:
00810                 msg = "non contiguous SEQNO in tn %s %s %s %s " % ( tn,  seq[0], seq[-1], len(seq)  )
00811                 if tn in ('CalibPmtSpec',):    ## WHY NOT NORMALLY TICKLED ?
00812                     log.info(msg)
00813                 else:
00814                     log.fatal(msg)       
00815                     assert contiguous, msg 
00816 
00817         #log.debug("allseqno_updates %r " % upas )
00818         return upas
00819 

def DybPython::asciicat::AsciiCat::diff (   self,
  otherdir 
)

Definition at line 820 of file asciicat.py.

00821                             :
00822         if type(otherdir) == AsciiCat:
00823             otherdir = otherdir.dir  
00824         cmd = "diff -r --brief %s %s | grep -v .svn " % ( self.dir, otherdir )
00825         log.info("diffing with \"%s\"" % cmd )
00826         for line in os.popen(cmd).readlines():
00827             log.info("diff %s" % line.strip())
00828 


Member Data Documentation

tuple DybPython::asciicat::AsciiCat::ptn = re.compile("(?P<tdir>[^/]*)/(?P<tnam>\S*)\.csv$") [static]

Definition at line 333 of file asciicat.py.

Definition at line 359 of file asciicat.py.

Definition at line 359 of file asciicat.py.

Definition at line 359 of file asciicat.py.

Definition at line 359 of file asciicat.py.

Definition at line 359 of file asciicat.py.

Definition at line 523 of file asciicat.py.

Definition at line 584 of file asciicat.py.

Definition at line 644 of file asciicat.py.


Property Documentation

DybPython::asciicat::AsciiCat::showtables = property( lambda self:sorted(self.keys())) [static]

Definition at line 347 of file asciicat.py.

DybPython::asciicat::AsciiCat::orderpk = property(lambda self:sorted(self)) [static]

Definition at line 348 of file asciicat.py.

DybPython::asciicat::AsciiCat::seqno = property( get_seqno , doc=get_seqno.__doc__ ) [static]

Definition at line 562 of file asciicat.py.

DybPython::asciicat::AsciiCat::allseqno = property( get_allseqno , doc=get_allseqno.__doc__ ) [static]

Definition at line 607 of file asciicat.py.

DybPython::asciicat::AsciiCat::fabseqno = property( get_fabseqno, doc=get_fabseqno.__doc__ ) [static]

Definition at line 650 of file asciicat.py.


The documentation for this class was generated from the following file:
| Classes | Job Modules | Data Objects | Services | Algorithms | Tools | Packages | Directories | Tracs |

Generated on Fri May 16 2014 09:55:40 for DybPython by doxygen 1.7.4