/search.css" rel="stylesheet" type="text/css"/> /search.js">
| Classes | Job Modules | Data Objects | Services | Algorithms | Tools | Packages | Directories | Tracs |

In This Package:

Public Member Functions | Public Attributes | Properties
DybPython::asciicat::AsciiCSV Class Reference

List of all members.

Public Member Functions

def __init__
def __repr__
def compare
def parse_hdr
def __call__
def resolve
def read
def merged
def compare
def content
def write

Public Attributes

 cat
 path
 kwa
 tabbase
 tabname
 pk
 seqnos_read
 determine fields, pk etc...
 pks_read
 hdr
 fields
 dbtypes
 fdict
 pkfields
 pktypes
 intpk
 quote
 all integer pk

Properties

 pks = property(lambda self:self.pks_read, doc="for all tables" )
 allseqno = property(lambda self:self.seqnos_read, doc="for payload/validity tables" )
 maxseqno = property(lambda self:max(self.allseqno), doc="for payload/validity tables")
 minseqno = property(lambda self:min(self.allseqno), doc="for payload/validity tables")
 abspath = property(lambda self:self.cat.resolve(self.path))
 pretty = property(lambda self:pformat(dict(self)), doc="pretty print the AsciiCSV dict, only useful for small CSVs such as LOCALSEQNO" )
 lastusedseqno = property(lambda self:dict(map(lambda k:(k,int(self[k]['LASTUSEDSEQNO'])) , self )), doc="for LOCALSEQNO tables" )
 smry = property(lambda self:"\n".join( [self.pretty, pformat(self.lastusedseqno) ]), doc="for LOCALSEQNO tables" )
 orderpk = property(lambda self:sorted(self))
 lastpk = property(lambda self:self.orderpk[-1])
 last = property(lambda self:self[self.lastpk])

Detailed Description

AsciiCSV is a dict keyed by PK with values which are dicts keyed by fieldname
which have string values. The PKs and field names are determined from the 
header line of the csv file. For example::

    csv = AsciiCSV( None, "~/dybaux/catalog/tmp_offline_db/CalibPmtSpec/CalibPmtSpec.csv")()
          ## standalone use without a catalog

    csv[(50,167)] == {'PMTAFTERPULSE': '0',
                      'PMTDARKRATE': '0',
                      'PMTDESCRIB': 'DayaBayOWS-wall09-spot23-in',
                      'PMTEFFIC': '1',
                       ...
                      'ROW_COUNTER': '167',
                      'SEQNO': '50'}

    csv[(50,167)]['PMTEFFIC'] == '1'

    for pk,v in csv.items():
        print pk, v

Typical PKs:
      
#. (SEQNO,ROW_COUNTER) integer tuple for payload tables
#. SEQNO integer for validity tables
#. TABLENAME string for LOCALSEQNO tables 

Definition at line 75 of file asciicat.py.


Constructor & Destructor Documentation

def DybPython::asciicat::AsciiCSV::__init__ (   self,
  cat,
  path,
  kwa 
)

Definition at line 104 of file asciicat.py.

00105                                          :
00106         dict.__init__(self)
00107         self.cat = cat
00108         self.path = path
00109         self.kwa = kwa
00110         self.tabbase = kwa.get('tdir',None)
00111         self.tabname = kwa.get('tnam',None)
00112         self.pk = kwa.get('tnam',None)
00113         self.seqnos_read = []
00114         self.pks_read = []
 

Member Function Documentation

def DybPython::asciicat::AsciiCSV::__repr__ (   self)

Definition at line 135 of file asciicat.py.

00136                       :
00137         return "%s [%-5d] %s  " % ( self.__class__.__name__, len(self), self.path  )

def DybPython::asciicat::AsciiCSV::compare (   self,
  other 
)
Equality operator for dict-of-dict works as would want : 
recursive same keys with same values. As demonstrated::
 
    old = AsciiCSV( None, "~/dybaux/catalog/tmp_offline_db/CalibPmtSpec/CalibPmtSpec.csv.old")()
    new = AsciiCSV( None, "~/dybaux/catalog/tmp_offline_db/CalibPmtSpec/CalibPmtSpec.csv.new")()

    old == new   ## True

    keep = old[(50,167)]['PMTEFFIC'] 
    old[(50,167)]['PMTEFFIC'] = keep + "changed"
    old == new                                     ## False after changing a single value 

    old[(50,167)]['PMTEFFIC'] = keep
    old == new   ## True                           ## back to equality 

    list(old) == list(new)       ## False          ## key order cannot be relied on 
    sorted(old) == sorted(new)   ## True
    old.orderpk == new.orderpk   ## True


The .old which matches dybaux r4968 is from before standardized order was established...
 
    [blyth@belle7 CalibPmtSpec]$ diff CalibPmtSpec.csv CalibPmtSpec.csv.old
    [blyth@belle7 CalibPmtSpec]$ diff CalibPmtSpec.csv CalibPmtSpec.csv.new | wc
    2470    4934  222803

Standardize it by ``old.write()`` after which::

    [blyth@belle7 CalibPmtSpec]$ diff CalibPmtSpec.csv.old  CalibPmtSpec.csv.new
    [blyth@belle7 CalibPmtSpec]$ 

Definition at line 138 of file asciicat.py.

00139                             :
00140         """
00141         Equality operator for dict-of-dict works as would want : 
00142         recursive same keys with same values. As demonstrated::
00143  
00144             old = AsciiCSV( None, "~/dybaux/catalog/tmp_offline_db/CalibPmtSpec/CalibPmtSpec.csv.old")()
00145             new = AsciiCSV( None, "~/dybaux/catalog/tmp_offline_db/CalibPmtSpec/CalibPmtSpec.csv.new")()
00146 
00147             old == new   ## True
00148 
00149             keep = old[(50,167)]['PMTEFFIC'] 
00150             old[(50,167)]['PMTEFFIC'] = keep + "changed"
00151             old == new                                     ## False after changing a single value 
00152 
00153             old[(50,167)]['PMTEFFIC'] = keep
00154             old == new   ## True                           ## back to equality 
00155 
00156             list(old) == list(new)       ## False          ## key order cannot be relied on 
00157             sorted(old) == sorted(new)   ## True
00158             old.orderpk == new.orderpk   ## True
00159 
00160 
00161         The .old which matches dybaux r4968 is from before standardized order was established...
00162  
00163             [blyth@belle7 CalibPmtSpec]$ diff CalibPmtSpec.csv CalibPmtSpec.csv.old
00164             [blyth@belle7 CalibPmtSpec]$ diff CalibPmtSpec.csv CalibPmtSpec.csv.new | wc
00165             2470    4934  222803
00166 
00167         Standardize it by ``old.write()`` after which::
00168 
00169             [blyth@belle7 CalibPmtSpec]$ diff CalibPmtSpec.csv.old  CalibPmtSpec.csv.new
00170             [blyth@belle7 CalibPmtSpec]$ 
00171 
00172         """
00173         return self == other
00174 

def DybPython::asciicat::AsciiCSV::parse_hdr (   self,
  hdr 
)
Make sense of hdr line like:: 

   SEQNO int(11),ROW_COUNTER int(11),SENSORID int(11),CHANNELID int(11),PRIMARY KEY (SEQNO,ROW_COUNTER)

Definition at line 175 of file asciicat.py.

00176                              :
00177         """
00178         Make sense of hdr line like:: 
00179 
00180            SEQNO int(11),ROW_COUNTER int(11),SENSORID int(11),CHANNELID int(11),PRIMARY KEY (SEQNO,ROW_COUNTER)
00181 
00182         """
00183         fdesc,pkdesc = hdr.split(",PRIMARY KEY ")
00184         assert pkdesc[0] == '(' and pkdesc[-1] == ")"
00185         
00186         self.hdr = hdr 
00187         self.fields  =  map(lambda _:_.split()[0], fdesc.split(",") )
00188         self.dbtypes =  map(lambda _:_.split()[1], fdesc.split(",") )
00189         self.fdict = dict(zip(self.fields, self.dbtypes))
00190         self.pkfields = pkdesc[1:-1].split(",")
00191         self.pktypes =  map(lambda k:self.fdict[k], self.pkfields)
00192         self.intpk = len(filter( lambda k:k.startswith('int'), self.pktypes )) == len(self.pktypes)   ## all integer pk
00193         qtyps = ("char","varc","date",) 
00194         self.quote = map( lambda _:self.fdict[_][0:4] in qtyps, self.fields )    ## the type is quoted
00195 

def DybPython::asciicat::AsciiCSV::__call__ (   self,
  content = None 
)

Definition at line 196 of file asciicat.py.

00197                                     :
00198         return self.read(content)

def DybPython::asciicat::AsciiCSV::resolve (   self,
  base = None 
)

Definition at line 199 of file asciicat.py.

00200                                  :
00201         if self.cat:
00202             path = self.cat.resolve( self.path, base )
00203         else:
00204             path = pathx(self.path)
00205         return path

def DybPython::asciicat::AsciiCSV::read (   self,
  content = None 
)
Parse the csv storing as dict of dict keyed by pk 

Typical DBI validity and payload tables are ordered by SEQNO/(SEQNO,ROW_COUNTER), but 
the standard order is not followed by:

#. LOCALSEQNO, not surprising as TABLENAME keyed and follows historic ordering 
#. CalibPmtSpec, CalibPmtSpecVld  **SURPISE : EVIDENCE FOR NON-DBI HANDLING**

Definition at line 206 of file asciicat.py.

00207                                 :
00208         """
00209         Parse the csv storing as dict of dict keyed by pk 
00210 
00211         Typical DBI validity and payload tables are ordered by SEQNO/(SEQNO,ROW_COUNTER), but 
00212         the standard order is not followed by:
00213 
00214         #. LOCALSEQNO, not surprising as TABLENAME keyed and follows historic ordering 
00215         #. CalibPmtSpec, CalibPmtSpecVld  **SURPISE : EVIDENCE FOR NON-DBI HANDLING**
00216 
00217         """
00218         if content:
00219             pass
00220         else:
00221             path = self.resolve()
00222             log.debug("parse csv %s " % path ) 
00223             content = open( path , "r" ).read()
00224 
00225         feol = content.find("\n")
00226         self.parse_hdr( content[:feol] )  ## determine fields, pk etc...
00227         rdr = DictReader( StringIO(content[feol+1:]), fieldnames=self.fields )
00228 
00229         sqn=[] 
00230         pks=[]
00231 
00232         for i,d in enumerate(rdr):
00233             r = AsciiRow.Create(d, self )
00234             rpk = r.pk
00235             self[rpk] = r 
00236             pks.append(rpk)
00237 
00238             # collect seqno during filling to avoid memory expense of grabbing latter 
00239             seqno = int(r.get('SEQNO',0))
00240             if seqno:
00241                 sqn.append(seqno)
00242 
00243             assert len(self) == i + 1, ("error non-unique pk %r for row %r " % ( rpk, r ) )  
00244             pass
00245 
00246         self.seqnos_read = list(set(sqn))
00247         self.pks_read = list(set(pks))
00248         return self       
00249 

def DybPython::asciicat::AsciiCSV::merged (   self,
  update,
  postpend = "" 
)
Return a new instance that merges `update` into `self`, changes in
the `update` override things in self 
       
NB `self` is not changed by this 

:param update: instance of `AsciiCSV` 
:return merged instance:
:rtype dict of dict:

Definition at line 250 of file asciicat.py.

00251                                            :
00252         """
00253         Return a new instance that merges `update` into `self`, changes in
00254         the `update` override things in self 
00255        
00256         NB `self` is not changed by this 
00257 
00258         :param update: instance of `AsciiCSV` 
00259         :return merged instance:
00260         :rtype dict of dict:
00261 
00262         """
00263         log.debug("merge csv %r " % update ) 
00264         mrgd = deepcopy(self)
00265         if mrgd.path:
00266             mrgd.path += postpend
00267 
00268         allpk = set(self).union(set(update))
00269         for pk in allpk:
00270             if pk in mrgd and pk in update:
00271                 m = deepcopy(self[pk])
00272                 m.update( update[pk] )      
00273                 log.debug("merged of pk %r combines self %r with update %r yielding %r  " % ( pk, mrgd[pk],update[pk],m) )
00274                 mrgd[pk] = m
00275             elif pk in mrgd:
00276                 log.debug("merged of pk %r keeps entry %r " % ( pk, mrgd[pk] ) )
00277                 pass
00278             elif pk in update:
00279                 log.debug("merged of pk %r adds update entry %r " % ( pk, update[pk] ) )
00280                 mrgd[pk] = deepcopy(update[pk])
00281         return mrgd

def DybPython::asciicat::AsciiCSV::compare (   self,
  other 
)
Return a dict with the differences  

Definition at line 282 of file asciicat.py.

00283                             :
00284         """
00285         Return a dict with the differences  
00286         """
00287         pass

def DybPython::asciicat::AsciiCSV::content (   self)

Definition at line 288 of file asciicat.py.

00289                      :
00290         content = self.hdr + "\n"
00291         for pk in sorted(self):
00292             content += str(self[pk]) + "\n"
00293         return content

def DybPython::asciicat::AsciiCSV::write (   self,
  base = None 
)

Definition at line 294 of file asciicat.py.

00295                               :
00296         path = self.resolve( base=base )
00297         if base == None:
00298             log.warn("inplace overwriting %s " % path )        
00299         else:
00300             log.debug("writing to %s " % path )
00301 
00302         wdir = os.path.dirname( path )
00303         if not os.path.exists(wdir):
00304             os.makedirs(wdir)
00305 
00306         fp = open(path, "w")
00307         fp.write( self.content() )
00308         fp.close()
00309 
00310 
00311 


Member Data Documentation

Definition at line 104 of file asciicat.py.

Definition at line 104 of file asciicat.py.

Definition at line 104 of file asciicat.py.

Definition at line 104 of file asciicat.py.

Definition at line 104 of file asciicat.py.

Definition at line 104 of file asciicat.py.

determine fields, pk etc...

Definition at line 104 of file asciicat.py.

Definition at line 104 of file asciicat.py.

Definition at line 180 of file asciicat.py.

Definition at line 180 of file asciicat.py.

Definition at line 180 of file asciicat.py.

Definition at line 180 of file asciicat.py.

Definition at line 180 of file asciicat.py.

Definition at line 180 of file asciicat.py.

Definition at line 180 of file asciicat.py.

all integer pk

Definition at line 181 of file asciicat.py.


Property Documentation

DybPython::asciicat::AsciiCSV::pks = property(lambda self:self.pks_read, doc="for all tables" ) [static]

Definition at line 119 of file asciicat.py.

DybPython::asciicat::AsciiCSV::allseqno = property(lambda self:self.seqnos_read, doc="for payload/validity tables" ) [static]

Definition at line 120 of file asciicat.py.

DybPython::asciicat::AsciiCSV::maxseqno = property(lambda self:max(self.allseqno), doc="for payload/validity tables") [static]

Definition at line 121 of file asciicat.py.

DybPython::asciicat::AsciiCSV::minseqno = property(lambda self:min(self.allseqno), doc="for payload/validity tables") [static]

Definition at line 122 of file asciicat.py.

DybPython::asciicat::AsciiCSV::abspath = property(lambda self:self.cat.resolve(self.path)) [static]

Definition at line 125 of file asciicat.py.

DybPython::asciicat::AsciiCSV::pretty = property(lambda self:pformat(dict(self)), doc="pretty print the AsciiCSV dict, only useful for small CSVs such as LOCALSEQNO" ) [static]

Definition at line 127 of file asciicat.py.

DybPython::asciicat::AsciiCSV::lastusedseqno = property(lambda self:dict(map(lambda k:(k,int(self[k]['LASTUSEDSEQNO'])) , self )), doc="for LOCALSEQNO tables" ) [static]

Definition at line 128 of file asciicat.py.

DybPython::asciicat::AsciiCSV::smry = property(lambda self:"\n".join( [self.pretty, pformat(self.lastusedseqno) ]), doc="for LOCALSEQNO tables" ) [static]

Definition at line 129 of file asciicat.py.

DybPython::asciicat::AsciiCSV::orderpk = property(lambda self:sorted(self)) [static]

Definition at line 131 of file asciicat.py.

DybPython::asciicat::AsciiCSV::lastpk = property(lambda self:self.orderpk[-1]) [static]

Definition at line 132 of file asciicat.py.

DybPython::asciicat::AsciiCSV::last = property(lambda self:self[self.lastpk]) [static]

Definition at line 133 of file asciicat.py.


The documentation for this class was generated from the following file:
| Classes | Job Modules | Data Objects | Services | Algorithms | Tools | Packages | Directories | Tracs |

Generated on Fri May 16 2014 09:55:40 for DybPython by doxygen 1.7.4