/search.css" rel="stylesheet" type="text/css"/> /search.js">
| Classes | Job Modules | Data Objects | Services | Algorithms | Tools | Packages | Directories | Tracs |

In This Package:

Public Member Functions | Static Public Attributes
Scraper::base::parser::Parser Class Reference
Inheritance diagram for Scraper::base::parser::Parser:
Inheritance graph
[legend]
Collaboration diagram for Scraper::base::parser::Parser:
Collaboration graph
[legend]

List of all members.

Public Member Functions

def __init__
def config
def __call__

Static Public Attributes

tuple timedeltas = ('interval','maxage','sleep','heartbeat',)
tuple config = classmethod(config)

Detailed Description

To see all the available options and defaults for a particular config sections::

    scr.py --help 
    scr.py -s adtemp_scraper --help
    scr.py -s pmthv_scraper --help

Performs two stage parsing, with the first stage driven by ``-s/--sect`` option to specify the 
section name within a configuration file. The path at which a config file is read from  
can be controlled by :envvar:`SCRAPER_CFG`, with default value::

    echo $SCRAPER_CFG      ## path of default config file 
        -->  $SITEROOT/dybgaudi/Database/Scraper/python/Scraper/.scraper.cfg
        -->  $SCRAPERROOT/python/Scraper/.scraper.cfg


Note that the first stage of parsing occurs in the ``AParser.__init__`` which:

#. provides config section name and path 
#. primes the base ``AParser`` dict with defaults read from that section 

The 2nd stage parse typically does nothing, as it is preferable to 
keep config at defaults read from file. This commandline control is mainly for 
testing/debugging.

Note the configparser/argparser mismatch in boolean handling:

#. argparse typically has "store_true/store_false" actions for convenient/brief commandline control
#. configparser and config file understandability requires True/False strings  

Have sided with configparser as the commandline interface beyond the `-s` is mainly for developer usage. 
However some options, such as `--dryrun` which make little sense in config files, buck this tendency.

Definition at line 74 of file parser.py.


Constructor & Destructor Documentation

def Scraper::base::parser::Parser::__init__ (   self,
  args,
  kwargs 
)

Definition at line 110 of file parser.py.

00111                                        :
00112 
00113         AParser.__init__(self, *args, **kwargs )
00114 
00115         ## the code to run 
00116         self.add_argument( '-n', '--dryrun',       help="Just dump the configuration.  Default %(default)s ", action="store_true" )
00117         self.add_argument( '--ALLOW_DROP_CREATE_TABLE', help='allow dropping and creating target tables (USE FOR INITIAL MANUAL SEEDING ONLY : NOT FOR CONFIGURING). Default %(default)s  ', action="store_true" )
00118         self.add_argument( '--DROP_TARGET_TABLES',      help='DROPS TARGET TABLES  (USE IN DEVELOPMENT ONLY). Default %(default)s  ', action="store_true" )
00119 
00120         self.add_argument( '--regime',             help="modpath:class spec of Regime class eg Scraper.adtemp:AdTemp (NOT FOR MANUAL SWITCHING... CHANGE VIA SECT) Default %(default)s " )
00121         self.add_argument( '--kls',                help="name of genDbi target class                                 (NOT FOR MANUAL SWITCHING... CHANGE VIA SECT) Default %(default)s " )
00122         self.add_argument( '--mode',               help="mode of operation : scraper or faker                        (NOT FOR MANUAL SWITCHING... CHANGE VIA SECT) Default %(default)s ")
00123 
00124         ## from where to where
00125         self.add_argument( '--source',             help="name of source database config section. Default %(default)s ")
00126         self.add_argument( '--target',             help="name of target database config section. Default %(default)s ")
00127 
00128         ## scraper parameters
00129         self.add_argument( '--maxiter',     type=int,  help="maximum iterations, or 0 for no limit. Default %(default)s ")
00130         self.add_argument( '--task',        type=int,  help="DBI contextrange task. Default %(default)s ")
00131         self.add_argument( '--offset',      type=int,  help="source skipping using SQL offsets. Default %(default)s ")
00132         self.add_argument( '--tunesleepmod', type=int, help="Modulo control for adjusting sleep time of scraper iteration, 1 means tune following every propagation. Default %(default)s ")
00133 
00134         self.add_argument( '--sleep',                  help="string timedelta eg 10s or 0s : strategic sleep between scraper cycles. Default %(default)s ")
00135         self.add_argument( '--cooldownsec',type=float, help="floating point minimum number of seconds eg 0.1 between samples. Default %(default)s ")
00136         self.add_argument( '--interval',               help="string timedelta eg 3m : time cursor increment.  Default %(default)s ")
00137         self.add_argument( '--maxage',                 help="string timedelta eg 1h : age beyond which unchanged quantities are written.  Default %(default)s  ")
00138         self.add_argument( '--heartbeat',              help="string timedelta eg 30s : guidance to scraper on source beat update period, used for constraining other parameters. Default %(default)s  ")
00139 
00140         self.add_argument( '--aggregate',              help="comma delimited aggregate function list for example avg,min,max,std or None for no aggregation.   Default %(default)s  ")
00141 
00142         self.add_argument( '--threshold',  type=float, help="delta threshold.  Default %(default)s ")
00143         self.add_argument( '--seed_target_tables', type=mybool,    help='switch on target seeding, creating tables and writing seed entry. Default %(default)s  ' )
00144         self.add_argument( '--seed_timestart',         help='seed contextrange start time in localtime. Default %(default)s  ', action=TimeAction )
00145         self.add_argument( '--seed_timeend',           help='seed contextrange end time in localtime. Default %(default)s ', action=TimeAction )
00146         self.add_argument( '--timefloor',              help='string time for limiting expense of initial validity query. Default %(default)s ', action=TimeAction )
00147         
00148         ## faker parameters 
00149         self.add_argument( '--faker_dropsrc',    type=mybool, help="faker: drop create and load source DB from dumpfile names after `src` config section.  Default %(default)s ")
00150         self.add_argument( '--faker_timestart', help='faker start time in localtime. Default %(default)s  ', action=TimeAction )
00151         
00152         ## common
00153         self.add_argument( '--timeformat',      help='strptime format for datetimes. Default %(default)s ')
00154  
00155         defaults = dict()
00156         defaults.update( kwargs )
00157         self.set_defaults( **defaults )
00158 


Member Function Documentation

def Scraper::base::parser::Parser::config (   cls,
  defsect = "adtemp_scraper",
  defpath = None,
  noargs = False 
)
Conveniuece classmethod for config access

Definition at line 159 of file parser.py.

00160                                                                            :
00161         """
00162         Conveniuece classmethod for config access
00163         """
00164         if not defpath:
00165             defpath = os.environ['SCRAPER_CFG']
00166         prs = cls(defpath=defpath,defsect=defsect)
00167         cfg = prs(noargs=noargs)
        return cfg
def Scraper::base::parser::Parser::__call__ (   self,
  args,
  kwa 
)
Fix up parsing of datestrings that are hailing from defaults 
rather than cmdline that appear to be skipped by the TimeAction 

Definition at line 170 of file parser.py.

00171                                      :
00172         """
00173         Fix up parsing of datestrings that are hailing from defaults 
00174         rather than cmdline that appear to be skipped by the TimeAction 
00175         """
00176         d = AParser.__call__( self, *args, **kwa )
00177         cfg = Cfg()
00178         for k,v in d.items():
00179            if k.find('timestart') > -1 or k.find('timeend') > -1 or k.find('timefloor') > -1:
00180                if type(v) == str:
00181                    v = parsetime( v, d['timeformat'] ) 
00182            if k in self.timedeltas:
00183                if type(v) == str:
00184                    v = timedelta_from_string(v) 
00185 
00186            if k != 'description':
00187                cfg[k] = v 
00188                log.debug("SParser %-20s %r " % ( k, v ) ) 
00189         return cfg
00190 


Member Data Documentation

tuple Scraper::base::parser::Parser::timedeltas = ('interval','maxage','sleep','heartbeat',) [static]

Definition at line 109 of file parser.py.

tuple Scraper::base::parser::Parser::config = classmethod(config) [static]

Definition at line 168 of file parser.py.


The documentation for this class was generated from the following file:
| Classes | Job Modules | Data Objects | Services | Algorithms | Tools | Packages | Directories | Tracs |

Generated on Fri May 16 2014 09:50:03 for Scraper by doxygen 1.7.4