/search.css" rel="stylesheet" type="text/css"/> /search.js">
| Classes | Job Modules | Data Objects | Services | Algorithms | Tools | Packages | Directories | Tracs |

In This Package:

Public Member Functions | Public Attributes | Properties | Private Member Functions | Private Attributes | Static Private Attributes
Scraper::daq::configinfo::ConfigInfo Class Reference
Inheritance diagram for Scraper::daq::configinfo::ConfigInfo:
Inheritance graph
[legend]
Collaboration diagram for Scraper::daq::configinfo::ConfigInfo:
Collaboration graph
[legend]

List of all members.

Public Member Functions

def __init__
def is_object_offline
def qconfig
def objects
def tst_insert
def __repr__
def __call__

Public Attributes

 msgs

Properties

 versions = property( _get_versions, _set_versions )
 is_offline = property( _is_offline , doc=_is_offline.__doc__ )
 okselect = property(_okselect, doc=_okselect.__doc__ )

Private Member Functions

def _set_versions
def _get_versions
def _is_offline
def _okselect

Private Attributes

 _schemaVersion
 _dataVersion
 _baseVersion

Static Private Attributes

list _configClassList = ['Partition', 'Segment', 'ROSApplication', 'ROSConfiguration', 'Detector', 'Crate', 'LTB%', 'FEE%']

Detailed Description

Gathers from DAQ tables and propagates new configs to the non-DBI offline tables ``DaqRunConfig``::

    mysql> describe DaqRunConfig ;
    +-----------------+--------------+------+-----+---------+-------+
    | Field           | Type         | Null | Key | Default | Extra |
    +-----------------+--------------+------+-----+---------+-------+
    | schemaVersion   | mediumint(9) | NO   | PRI | 0       |       | 
    | dataVersion     | mediumint(9) | NO   | PRI | 0       |       | 
    | creationVersion | mediumint(9) | YES  |     | NULL    |       | 
    | className       | varchar(64)  | NO   | PRI |         |       | 
    | objectId        | varchar(64)  | NO   | PRI |         |       | 
    | name            | varchar(128) | NO   | PRI |         |       | 
    | parentPosition  | mediumint(9) | NO   | PRI | 0       |       | 
    | intValue        | bigint(20)   | YES  |     | NULL    |       | 
    | floatValue      | float        | YES  |     | NULL    |       | 
    | stringValue     | varchar(64)  | YES  |     | NULL    |       | 
    +-----------------+--------------+------+-----+---------+-------+
    10 rows in set (0.01 sec)

Source DAQ tables:

    ==============     ======================================
     table               notes
    ==============     ======================================
     OKSOBJECT
     OKSDATA 
     OKSDATAVAL
     OKSDATAREL
    ==============     ======================================

Definition at line 9 of file configinfo.py.


Constructor & Destructor Documentation

def Scraper::daq::configinfo::ConfigInfo::__init__ (   self,
  source,
  target,
  cfg = {} 
)

Definition at line 44 of file configinfo.py.

00044                                             {} ):
00045         BaseInfo.__init__(self, source, target , cfg  )
00046 
00047         self._schemaVersion = None
00048         self._dataVersion = None
00049         self._baseVersion = None
00050 
00051         self.msgs = []
00052 

Member Function Documentation

def Scraper::daq::configinfo::ConfigInfo::_set_versions (   self,
  sdv 
) [private]

Definition at line 53 of file configinfo.py.

00054                                  :
00055         assert len(sdv) == 3, sdv
        self._schemaVersion, self._dataVersion, self._baseVersion = sdv 
def Scraper::daq::configinfo::ConfigInfo::_get_versions (   self) [private]

Definition at line 56 of file configinfo.py.

00057                            :
        return  self._schemaVersion, self._dataVersion, self._baseVersion
def Scraper::daq::configinfo::ConfigInfo::_is_offline (   self) [private]
Query non-DBI offline table ``DaqRunConfig`` to see if the version set (schemaVersion,dataVersion) 
has already been collected offline

Definition at line 60 of file configinfo.py.

00061                          :
00062         """
00063         Query non-DBI offline table ``DaqRunConfig`` to see if the version set (schemaVersion,dataVersion) 
00064         has already been collected offline
00065         """
00066         off = self.target()
00067         kdrc  = self.target.kls_("DaqRunConfig")
00068         n = off.query(kdrc).filter(and_(kdrc.schemaVersion == self._schemaVersion,kdrc.dataVersion == self._dataVersion)).count()
        return n>0 
def Scraper::daq::configinfo::ConfigInfo::is_object_offline (   self,
  creationVersion,
  className,
  objectId 
)
Query non-DBI offline table ``DaqRunConfig`` to see if object corresponding to arguments  
has already been collected

Definition at line 71 of file configinfo.py.

00072                                                                        :
00073         """
00074         Query non-DBI offline table ``DaqRunConfig`` to see if object corresponding to arguments  
00075         has already been collected
00076         """
00077         off = self.target()
00078         kdrc  = self.target.kls_("DaqRunConfig")
00079         obc = and_(
00080                   kdrc.schemaVersion == self._schemaVersion,
00081                   kdrc.creationVersion == creationVersion,
00082                   kdrc.className == className,
00083                   kdrc.objectId == objectId,
00084                  )
00085         n = off.query(kdrc).filter(obc).count()
00086         return n>0 

def Scraper::daq::configinfo::ConfigInfo::_okselect (   self) [private]
Based on ``check_newConfig``

Selects config entries matching version arguments and corresponding 
to config classes of interest from DAQ tables OKSOBJECT, OKSDATA

Definition at line 87 of file configinfo.py.

00088                          :
00089         """
00090         Based on ``check_newConfig``
00091 
00092         Selects config entries matching version arguments and corresponding 
00093         to config classes of interest from DAQ tables OKSOBJECT, OKSDATA
00094         """
00095         t1 = self.source.table_("OKSOBJECT")
00096         t2 = self.source.table_("OKSDATA")
00097 
00098         likes = [t1.c.CLASSNAME.like(_) for _ in  self._configClassList ] 
00099         lkc = or_( *likes )
00100         dvc = or_( 
00101                  t1.c.DATAVERSION == self._dataVersion , 
00102                  and_(
00103                       t1.c.DATAVERSION == self._baseVersion , 
00104                       t1.c.CREATIONVERSION <= self._dataVersion 
00105                      )
00106                  )
00107         top =  and_( 
00108               t1.c.SCHEMAVERSION == self._schemaVersion ,  
00109               t1.c.SCHEMAVERSION == t2.c.SCHEMAVERSION , 
00110               t1.c.DATAVERSION == t2.c.VERSION , 
00111               dvc , 
00112               lkc ,
00113               )
00114         s = select([t1.c.CLASSNAME,t1.c.ID,t1.c.CREATIONVERSION], top )
00115         result = self.source.engine.execute(s)
00116         return result 

def Scraper::daq::configinfo::ConfigInfo::qconfig (   self,
  kls,
  creationVersion,
  className,
  objectId 
)
Can be used for both OKSDATAVAL and OKSDATAREL as they share these attributes

Definition at line 120 of file configinfo.py.

00121                                                                      :
00122         """ 
00123         Can be used for both OKSDATAVAL and OKSDATAREL as they share these attributes
00124         """
00125         plk = and_(
00126                     kls.SCHEMAVERSION == self._schemaVersion,
00127                     kls.CREATIONVERSION == creationVersion,
00128                     kls.CLASSNAME == className,
00129                     kls.OBJECTID == objectId,
00130                   )
00131         daq = self.source()
00132         return daq.query(kls).filter(plk)

def Scraper::daq::configinfo::ConfigInfo::objects (   self,
  creationVersion,
  className,
  objectId 
)
:param creationVersion:
:param className:
:param objectId:

Finds entries from ``OKSDATAVAL`` and ``OKSDATAREL`` corresponding to 
the arguments.

The ``ODR`` entries are splayed out into two ``ODV`` entries with 
some name changes:: 
 
mysql> describe OKSDATAVAL ;
+-----------------+--------------+------+-----+---------+-------+
| Field           | Type         | Null | Key | Default | Extra |
+-----------------+--------------+------+-----+---------+-------+
| SCHEMAVERSION   | mediumint(9) | NO   | PRI | 0       |       | 0
| DATAVERSION     | mediumint(9) | NO   | PRI | 0       |       | 1
| CREATIONVERSION | mediumint(9) | NO   |     | 0       |       | 2
| CLASSNAME       | varchar(64)  | NO   | PRI |         |       | 3
| OBJECTID        | varchar(64)  | NO   | PRI |         |       | 4
| NAME            | varchar(128) | NO   | PRI |         |       | 5
| PARENTPOSITION  | mediumint(9) | NO   | PRI | 0       |       | 6
| INTEGERVALUE    | bigint(20)   | YES  |     | NULL    |       | 7
| NUMBERVALUE     | double       | YES  |     | NULL    |       | 8
| STRINGVALUE     | text         | YES  |     | NULL    |       | 9
+-----------------+--------------+------+-----+---------+-------+
10 rows in set (0.01 sec)

mysql> describe OKSDATAREL ;
+-----------------+--------------+------+-----+---------+-------+
| Field           | Type         | Null | Key | Default | Extra |
+-----------------+--------------+------+-----+---------+-------+
| SCHEMAVERSION   | mediumint(9) | NO   | PRI | 0       |       | 0  
| DATAVERSION     | mediumint(9) | NO   | PRI | 0       |       | 1
| CREATIONVERSION | mediumint(9) | NO   |     | 0       |       | 2
| CLASSNAME       | varchar(64)  | NO   | PRI |         |       | 3
| OBJECTID        | varchar(64)  | NO   | PRI |         |       | 4
| NAME            | varchar(128) | NO   | PRI |         |       | 5
| PARENTPOSITION  | mediumint(9) | NO   | PRI | 0       |       | 6
| VALUECLASSNAME  | varchar(64)  | NO   |     |         |       | 7
| VALUEOBJECTID   | varchar(64)  | NO   |     |         |       | 8
| VALUEVERSION    | mediumint(9) | YES  |     | NULL    |       | 9
+-----------------+--------------+------+-----+---------+-------+
10 rows in set (0.00 sec)

Target table is similar to ``OKSDATAVAL``::

mysql> describe DaqRunConfig ;
+-----------------+--------------+------+-----+---------+-------+
| Field           | Type         | Null | Key | Default | Extra |
+-----------------+--------------+------+-----+---------+-------+
| schemaVersion   | mediumint(9) | NO   | PRI | 0       |       | 
| dataVersion     | mediumint(9) | NO   | PRI | 0       |       | 
| creationVersion | mediumint(9) | YES  |     | NULL    |       | 
| className       | varchar(64)  | NO   | PRI |         |       | 
| objectId        | varchar(64)  | NO   | PRI |         |       | 
| name            | varchar(128) | NO   | PRI |         |       | 
| parentPosition  | mediumint(9) | NO   | PRI | 0       |       | 
| intValue        | bigint(20)   | YES  |     | NULL    |       | 
| floatValue      | float        | YES  |     | NULL    |       | 
| stringValue     | varchar(64)  | YES  |     | NULL    |       | 
+-----------------+--------------+------+-----+---------+-------+

Duplicates the legacy splaying:: 

      #print row
      childClassName, childObjectId = (row[7], row[8])
      # change to OKSDATAVAL format
      rowval = (row[0], row[1], row[2], row[3], row[4], 'childClassName_' + row[5], row[6], None, None, childClassName)
      records.append(rowval)
      rowval = (row[0], row[1], row[2], row[3], row[4], 'childObjectId_' + row[5], row[6], None, None, childObjectId)
      records.append(rowval)


Definition at line 133 of file configinfo.py.

00134                                                                :
00135         """
00136         :param creationVersion:
00137         :param className:
00138         :param objectId:
00139 
00140         Finds entries from ``OKSDATAVAL`` and ``OKSDATAREL`` corresponding to 
00141         the arguments.
00142 
00143         The ``ODR`` entries are splayed out into two ``ODV`` entries with 
00144         some name changes:: 
00145  
00146                 mysql> describe OKSDATAVAL ;
00147                 +-----------------+--------------+------+-----+---------+-------+
00148                 | Field           | Type         | Null | Key | Default | Extra |
00149                 +-----------------+--------------+------+-----+---------+-------+
00150                 | SCHEMAVERSION   | mediumint(9) | NO   | PRI | 0       |       | 0
00151                 | DATAVERSION     | mediumint(9) | NO   | PRI | 0       |       | 1
00152                 | CREATIONVERSION | mediumint(9) | NO   |     | 0       |       | 2
00153                 | CLASSNAME       | varchar(64)  | NO   | PRI |         |       | 3
00154                 | OBJECTID        | varchar(64)  | NO   | PRI |         |       | 4
00155                 | NAME            | varchar(128) | NO   | PRI |         |       | 5
00156                 | PARENTPOSITION  | mediumint(9) | NO   | PRI | 0       |       | 6
00157                 | INTEGERVALUE    | bigint(20)   | YES  |     | NULL    |       | 7
00158                 | NUMBERVALUE     | double       | YES  |     | NULL    |       | 8
00159                 | STRINGVALUE     | text         | YES  |     | NULL    |       | 9
00160                 +-----------------+--------------+------+-----+---------+-------+
00161                 10 rows in set (0.01 sec)
00162 
00163                 mysql> describe OKSDATAREL ;
00164                 +-----------------+--------------+------+-----+---------+-------+
00165                 | Field           | Type         | Null | Key | Default | Extra |
00166                 +-----------------+--------------+------+-----+---------+-------+
00167                 | SCHEMAVERSION   | mediumint(9) | NO   | PRI | 0       |       | 0  
00168                 | DATAVERSION     | mediumint(9) | NO   | PRI | 0       |       | 1
00169                 | CREATIONVERSION | mediumint(9) | NO   |     | 0       |       | 2
00170                 | CLASSNAME       | varchar(64)  | NO   | PRI |         |       | 3
00171                 | OBJECTID        | varchar(64)  | NO   | PRI |         |       | 4
00172                 | NAME            | varchar(128) | NO   | PRI |         |       | 5
00173                 | PARENTPOSITION  | mediumint(9) | NO   | PRI | 0       |       | 6
00174                 | VALUECLASSNAME  | varchar(64)  | NO   |     |         |       | 7
00175                 | VALUEOBJECTID   | varchar(64)  | NO   |     |         |       | 8
00176                 | VALUEVERSION    | mediumint(9) | YES  |     | NULL    |       | 9
00177                 +-----------------+--------------+------+-----+---------+-------+
00178                 10 rows in set (0.00 sec)
00179 
00180         Target table is similar to ``OKSDATAVAL``::
00181 
00182                 mysql> describe DaqRunConfig ;
00183                 +-----------------+--------------+------+-----+---------+-------+
00184                 | Field           | Type         | Null | Key | Default | Extra |
00185                 +-----------------+--------------+------+-----+---------+-------+
00186                 | schemaVersion   | mediumint(9) | NO   | PRI | 0       |       | 
00187                 | dataVersion     | mediumint(9) | NO   | PRI | 0       |       | 
00188                 | creationVersion | mediumint(9) | YES  |     | NULL    |       | 
00189                 | className       | varchar(64)  | NO   | PRI |         |       | 
00190                 | objectId        | varchar(64)  | NO   | PRI |         |       | 
00191                 | name            | varchar(128) | NO   | PRI |         |       | 
00192                 | parentPosition  | mediumint(9) | NO   | PRI | 0       |       | 
00193                 | intValue        | bigint(20)   | YES  |     | NULL    |       | 
00194                 | floatValue      | float        | YES  |     | NULL    |       | 
00195                 | stringValue     | varchar(64)  | YES  |     | NULL    |       | 
00196                 +-----------------+--------------+------+-----+---------+-------+
00197         
00198         Duplicates the legacy splaying:: 
00199 
00200               #print row
00201               childClassName, childObjectId = (row[7], row[8])
00202               # change to OKSDATAVAL format
00203               rowval = (row[0], row[1], row[2], row[3], row[4], 'childClassName_' + row[5], row[6], None, None, childClassName)
00204               records.append(rowval)
00205               rowval = (row[0], row[1], row[2], row[3], row[4], 'childObjectId_' + row[5], row[6], None, None, childObjectId)
00206               records.append(rowval)
00207 
00208 
00209         """
00210         asdict = lambda i:dict([(p.key,getattr(i,p.key)) for p in class_mapper(i.__class__).iterate_properties])  ## mapped instance as clean dict, without SA instrumentation 
00211 
00212         kodv = self.source.kls_("OKSDATAVAL")
00213         qodv = self.qconfig( kodv, creationVersion, className, objectId )
00214         odvs = map(asdict, qodv.all())
00215 
00216         kodr = self.source.kls_("OKSDATAREL")
00217         qodr = self.qconfig( kodr, creationVersion, className, objectId )
00218 
00219         for odr in qodr.all():
00220             a,b = asdict(odr), asdict(odr)    
00221 
00222             a['NAME'] = 'childClassName_%s' % odr.NAME
00223             a['VALUECLASSNAME'] = None
00224             a['VALUEOBJECTID'] = None
00225             a['VALUEVERSION'] =  odr.VALUECLASSNAME
00226             odvs.append(a) 
00227 
00228             b['NAME'] = 'childObjectId_%s' % odr.NAME
00229             b['VALUECLASSNAME'] = None
00230             b['VALUEOBJECTID'] = None
00231             b['VALUEVERSION'] =  odr.VALUEOBJECTID
00232             odvs.append(b) 
00233             
00234         return odvs

def Scraper::daq::configinfo::ConfigInfo::tst_insert (   cls,
  odvs 
)
Create tmp_test.DaqRunConfig by creating DB and  `~/.my.cnf` section then creating empty DB and table:: 

    mysql> create database tmp_test ;
    mysql> create table if not exists tmp_test.DaqRunConfig like tmp_offline_db.DaqRunConfig ;

Custom mapping creates ``DaqRunConfig`` ``kls`` with attributes that correspond 
to the default attributes(==columns) of the basis ``OKSDATAVAL``.
Allows ``ODV`` dicts to be consumed by the ``kls`` ctor.
  

Definition at line 236 of file configinfo.py.

00237                              :
00238         """
00239         Create tmp_test.DaqRunConfig by creating DB and  `~/.my.cnf` section then creating empty DB and table:: 
00240 
00241             mysql> create database tmp_test ;
00242             mysql> create table if not exists tmp_test.DaqRunConfig like tmp_offline_db.DaqRunConfig ;
00243 
00244         Custom mapping creates ``DaqRunConfig`` ``kls`` with attributes that correspond 
00245         to the default attributes(==columns) of the basis ``OKSDATAVAL``.
00246         Allows ``ODV`` dicts to be consumed by the ``kls`` ctor.
00247           
00248         """
00249         mapping = dict(
00250                     schemaVersion="SCHEMAVERSION",
00251                     dataVersion="DATAVERSION",
00252                     creationVersion="CREATIONVERSION",
00253                     className="CLASSNAME",
00254                     objectId="OBJECTID",
00255                     name="NAME",
00256                     parentPosition="PARENTPOSITION",
00257                     intValue="INTEGERVALUE",
00258                     floatValue="NUMBERVALUE",
00259                     stringValue="STRINGVALUE",
00260                    )
00261         rapping = dict((v,k) for (k,v) in mapping.items())
00262         kls  = tstm.kls_("DaqRunConfig", mapping=rapping )    ## once only 
00263 
00264         tst = tstm()
00265         for odv in odvs:
00266             drc = kls()    ## generated kwa ctor is a feature of declarative base that is not provided by classical mapping
00267             for k,v in odv.items():
00268                 setattr( drc, k , v )
00269             tst.add(drc)
00270         tst.commit()
00271 

def Scraper::daq::configinfo::ConfigInfo::__repr__ (   self)

Definition at line 272 of file configinfo.py.

00273                       :
00274         return "%s(schemaVersion=%s,dataVersion=%s,baseVersion=%s)" % ( self.__class__.__name__ , self._schemaVersion, self._dataVersion, self._baseVersion)

def Scraper::daq::configinfo::ConfigInfo::__call__ (   self)
Early returns if:

#. if this version combination is already offline
 

Definition at line 275 of file configinfo.py.

00276                       :
00277         """
00278         Early returns if:
00279 
00280         #. if this version combination is already offline
00281  
00282         """ 
00283         if self.is_offline:
00284             msg = " %s if offline already " % self 
00285             log.warn(msg)
00286             self.msgs.append(msg)    
00287             if self.cfg.get('test',False):
00288                 log.warn("proceeding due to test option")
00289             else: 
00290                 return None  
00291 
00292         result = self.okselect
00293         log.info("okselect rowcount %s " % result.rowcount ) 
00294 
00295         if result.rowcount == 0:
00296             msg = "no config rows from okselect "
00297             log.warn(msg)
00298             self.msgs.append(msg)    
00299             return None  
00300 
00301         ## NASTY DOUBLY NESTED QUERYING
00302         for className, objectId, creationVersion in result:
00303             ioo = self.is_object_offline( creationVersion , className , objectId )
00304             odvs = self.objects( creationVersion=creationVersion, className=className, objectId=objectId )
00305             for odv in odvs:
00306                 print odv
00307             ConfigInfo.tst_insert( odvs ) 
00308 
00309 


Member Data Documentation

list Scraper::daq::configinfo::ConfigInfo::_configClassList = ['Partition', 'Segment', 'ROSApplication', 'ROSConfiguration', 'Detector', 'Crate', 'LTB%', 'FEE%'] [static, private]

Definition at line 42 of file configinfo.py.

Definition at line 44 of file configinfo.py.

Definition at line 44 of file configinfo.py.

Definition at line 44 of file configinfo.py.

Definition at line 44 of file configinfo.py.


Property Documentation

Scraper::daq::configinfo::ConfigInfo::versions = property( _get_versions, _set_versions ) [static]

Definition at line 58 of file configinfo.py.

Scraper::daq::configinfo::ConfigInfo::is_offline = property( _is_offline , doc=_is_offline.__doc__ ) [static]

Definition at line 69 of file configinfo.py.

Scraper::daq::configinfo::ConfigInfo::okselect = property(_okselect, doc=_okselect.__doc__ ) [static]

Definition at line 117 of file configinfo.py.


The documentation for this class was generated from the following file:
| Classes | Job Modules | Data Objects | Services | Algorithms | Tools | Packages | Directories | Tracs |

Generated on Fri May 16 2014 09:50:03 for Scraper by doxygen 1.7.4