/search.css" rel="stylesheet" type="text/css"/> /search.js">
| Classes | Job Modules | Data Objects | Services | Algorithms | Tools | Packages | Directories | Tracs |

In This Package:

Public Member Functions | Public Attributes | Private Member Functions
DybPython::Control::NuWa Class Reference

List of all members.

Public Member Functions

def __init__
def cmdline
def known_input_type
def add_input_file
def configure_python_features
def configure_ipython
def configure_framework
def configure_random
def configure_run
def configure_dbconf
def configure_dbi
def configure_job
def configure_geometry
def configure_optmods
def configure_args
def configure_mod
def configure_user
def configure_post_user
def configure_visualization
def configure_dyb_services
def configure
def add_service_redirect
def spade_files
def run_post_user
def run
def finalize

Public Attributes

 DATETIME_FORMAT
 post_user_algs
 initAlgs
 spadeSvc
 opts
 args
 use_AES
 AES_window
 ipshell
 theApp
 input_files
 Uncommenting the following leads to a service-initialization loop.
 evtmgr
 io
 daqIO
 rawLoad
 modules

Private Member Functions

def _configure_trimio_output
def _configure_archive_output

Detailed Description

This is the main program to run NuWa offline jobs.

It provides a job with a minimal, standard setup.  Non standard
behavior can made using command line options or providing additional
configuration in the form of python files or modules to load.

Usage::
   
  nuwa.py --help 
  nuwa.py [options] [-m|--module "mod.ule --mod-arg ..."] \\
          [config1.py config2.py ...] \\
          [mod.ule1 mod.ule2 ...] \\
          [[input1.root input2.root ...] or [input1.data ...]] \\

Python modules can be specified with -m|--module options and may
include any per-module arguments by enclosing them in shell quotes
as in the above usage.  Modules that do not take arguments may
also be listed as non-option arguments.  Modules may supply the
following functions:

#. ``configure(argv=[])`` - if exists, executed at configuration time
#. ``run(theApp)`` - if exists, executed at run time with theApp set to the AppMgr.

Additionally, python job scripts may be specified.

Modules and scripts are loaded in the order they are specified on
the command line.

Finally, input ROOT files may be specified.  These will be read in
the order they are specified and will be assigned to supplying
streams not specificially specified in any input-stream map.

The listing of modules, job scripts and/or ROOT files may be
interspersed but must follow all options.

In addition to the command line, arguments can be given in a text
file with one line per argument.  This file can then be given to
nuwa.py on the command line prefaced with an '@' or a '+'.

Definition at line 14 of file Control.py.


Constructor & Destructor Documentation

def DybPython::Control::NuWa::__init__ (   self)

Definition at line 57 of file Control.py.

00058                       :
00059         'Create a NuWa instance.'
00060 
00061         self.DATETIME_FORMAT = '%Y-%m-%dT%H:%M:%S'
00062         self.post_user_algs = []
00063         self.initAlgs = None # Used to correctly-order algorithms
00064         
00065         # If the spade service (SpadeAlgo) is used it will register itself.
00066         self.spadeSvc = None;
00067         return


Member Function Documentation

def DybPython::Control::NuWa::cmdline (   self,
  argv 
)

Definition at line 68 of file Control.py.

00069                           :
00070         'Parse command line'
00071         #from DybPython.cmdline import optparser as cmdline_parser
00072         from DybPython.cmdline import argparser as cmdline_parser
00073 
00074         (options,args) = cmdline_parser(args=argv, docstr = self.__doc__)
00075 
00076         self.opts = options
00077         self.args = args
00078 
00079         # convert string to dictionary object
00080         from Tools import mapify
00081         self.opts.input_streams = mapify(self.opts.input_streams)
00082         self.opts.output_streams = mapify(self.opts.output_streams)
00083         if self.opts.output_stats:
00084             self.opts.output_stats = mapify(self.opts.output_stats)
00085         if self.opts.input_stats:
00086             self.opts.input_stats = mapify(self.opts.input_stats)
00087         self.opts.job_info = mapify(self.opts.job_info)
00088 
00089         # Convert formatted time to seconds since Epoch
00090         from time import gmtime, mktime, strftime, strptime, timezone
00091         if -1 != self.opts.time.find('T'):
00092             self.opts.time = int(mktime(strptime(self.opts.time,
00093                                                  self.DATETIME_FORMAT))
00094                                  -timezone)
00095             # Note: mktime adds the timezone, so we need to subtract it.
00096         else:
00097             self.opts.time = int(self.opts.time)
00098 
00099 
00100         from Tools import mapify, unitify
00101         if self.opts.aes_trim_window.lower() == "none":
00102             self.use_AES = False
00103         else:
00104             self.use_AES = True
00105             self.AES_window = unitify(self.opts.aes_trim_window,"second")
00106             if self.AES_window >=0 :
00107                 print "AES is turned on and trimming window is set to ",\
00108                     self.AES_window/units.second," seconds"
00109                 # valid
00110                 pass
00111             else :
00112                 print "error: invalid AES trimming window"
00113                 exit()
00114 
00115         return

def DybPython::Control::NuWa::known_input_type (   self,
  fname 
)

Definition at line 116 of file Control.py.

00117                                     :
00118         'Return True if file name has a recognized extension.'
00119         for ext in ['.root','.data','.rraw','.list']:
00120             if fname[-len(ext):] == ext:
00121                 return True
00122             continue
00123         return False

def DybPython::Control::NuWa::add_input_file (   self,
  fname 
)
Add file name or list of file names to self.input_files,
expanding if it is a .list file.

Definition at line 124 of file Control.py.

00125                                   :
00126         '''Add file name or list of file names to self.input_files,
00127         expanding if it is a .list file.'''
00128 
00129         if not fname: return
00130 
00131         #print 'add_input_file(%s)'%fname
00132 
00133         # recurse if list
00134         if isinstance(fname,list):
00135             for name in fname:
00136                 self.add_input_file(name)
00137                 continue
00138             return
00139 
00140         if not self.known_input_type(fname):
00141             msg = 'Got unknown input file type: "%s"'%fname
00142             print msg
00143             raise TypeError,msg
00144 
00145         if fname[-5:] != '.list':
00146             self.input_files.append(fname)
00147             return
00148 
00149         # process as a list file
00150         fp = open(fname)
00151         for line in fp.readlines():
00152             line = line.strip()
00153             if len(line) == 0: continue
00154             if line[0] == '#': pass
00155             if not line: pass
00156             if not self.known_input_type(line):
00157                 print ' unknown file:',line
00158                 continue
00159             self.add_input_file(line)
00160             continue
00161         fp.close()
00162         return
00163 

def DybPython::Control::NuWa::configure_python_features (   self)

Definition at line 164 of file Control.py.

00165                                        :
00166         'Set up python features'
00167         if not os.getcwd() in sys.path:
00168             sys.path = [ os.getcwd() ] + sys.path
00169 
00170         if not '' in sys.path:
00171             sys.path = [ '' ] + sys.path
00172 
00173         sys.ps1 = 'nuwa> '
00174         sys.ps2 = '. ... '
00175 
00176 
00177         if self.opts.interactive:
00178             try:
00179                 import rlcompleter, readline, atexit
00180             except ImportError:
00181                 print 'No readline available.'
00182             else:
00183                 readline.parse_and_bind('tab: complete')
00184                 readline.parse_and_bind('set show-all-if-ambiguous On')
00185             
00186                 fhistory = os.path.expanduser( '~/.nuwa_history' )
00187                 if os.path.exists(fhistory):
00188                     readline.read_history_file(fhistory)
00189                 readline.set_history_length(1024)
00190                 atexit.register(readline.write_history_file,fhistory)
00191         else:                   # batch
00192             if os.isatty(sys.stdin.fileno()):
00193                 os.close(sys.stdin.fileno())
00194             pass
00195         
00196 
00197         if self.opts.leak_check_execute:
00198             try:
00199                 import Hephaestus.MemoryTracker as memtrack
00200             except ImportError:
00201                 print "ERROR: Hephaestus is not available."
00202                 sys.exit(1)
00203             else:
00204                 methods = []
00205                 if "Algorithm::sysExecute" not in self.opts.leak_check_method:
00206                     methods.append("Algorithm::sysExecute")
00207                 methods += self.opts.leak_check_method
00208                 for m in methods:
00209                     print "Will leak check in",m
00210                     memtrack.trace(m)
00211                 memtrack.start()
00212 
00213         pyloglevel = getattr( logging, self.opts.pyloglevel.upper() )
00214         logging.basicConfig( level=pyloglevel )
00215         return
00216 

def DybPython::Control::NuWa::configure_ipython (   self)
     If ipython not available or are already inside ipython, setup a dummy 
     embedded ipython ipshell function, otherwise setup the real thing.

Definition at line 217 of file Control.py.

00218                                :
00219         """
00220              If ipython not available or are already inside ipython, setup a dummy 
00221              embedded ipython ipshell function, otherwise setup the real thing.
00222         """
00223         if not self.opts.interactive:
00224             return
00225 
00226         def ipshell(*args): pass
00227         self.ipshell = ipshell
00228 
00229         try:
00230             import IPython
00231         except ImportError:
00232             print "No ipython available."
00233             return
00234 
00235         try:
00236             __IPYTHON__
00237         except NameError:
00238             from IPython.Shell import IPShellEmbed
00239             irgs = [''] 
00240             banner = "entering ipython embedded shell, within the scope of the NuWa instance... try the locals() command " 
00241             self.ipshell = IPShellEmbed(irgs, banner=banner, exit_msg="exiting ipython" )
00242 
00243 

def DybPython::Control::NuWa::configure_framework (   self)

Definition at line 244 of file Control.py.

00245                                  :
00246         'Set up framework level defaults'
00247 
00248         # Fudge to make sure classes are loaded for ROOT
00249         from GaudiPython import gbl
00250         gbl.gROOT.ProcessLine("v = vector<float>();")
00251         t = gbl.TimeStamp()
00252         # end fudge
00253         
00254         from Gaudi.Configuration import ApplicationMgr
00255         self.theApp = ApplicationMgr()
00256         self.theApp.OutputLevel = self.opts.log_level
00257 
00258         if self.opts.audit:
00259             from Gaudi.Configuration import AuditorSvc, ChronoAuditor
00260             audSvc = AuditorSvc()
00261             audSvc.Auditors.append(ChronoAuditor())
00262             self.theApp.AuditAlgorithms = True
00263             ## Uncommenting the following leads to a service-initialization loop.
00264             ##self.theApp.AuditServices = True
00265             self.theApp.AuditTools = True
00266             self.theApp.ExtSvc.append(audSvc)
00267             pass
00268 
00269         # Setup history service 
00270         if self.opts.history.lower() not in ["off","no","none","n","false"] \
00271                 and self.opts.executions != 0:
00272             from GaudiSvc.GaudiSvcConf import HistorySvc
00273             hs = HistorySvc()
00274             hs.OutputLevel = 1
00275             self.theApp.ExtSvc.append(hs)
00276             self.theApp.ActivateHistory = True
00277             if self.opts.history.lower() in ["on","yes","y","true"]:
00278                 hs.Dump = True
00279             else:
00280                 hs.OutputFile = self.opts.history
00281 
00282 
00283 
00284         # Deal with I/O.
00285         try:
00286             import DybPython.Catalog as Catalog
00287             print 'Embedded Catalog is available with the following settings:'
00288             print Catalog.getSettings()
00289         except ImportError:
00290             print 'No Embedded Catalog available'
00291 
00292         # Input files can be determined in several ways
00293         self.input_files = []
00294 
00295         # First add files that are specified on the command line
00296         left_overs = []
00297         for arg in self.args:
00298             if not self.known_input_type(arg):
00299                 left_overs.append(arg)
00300                 continue
00301             self.add_input_file(arg)
00302             continue
00303         self.args = left_overs
00304         del(left_overs)
00305 
00306         # Next, WTF is this code supposed to do?  The first half
00307         # requires a Python file to exist but then strips off the .py
00308         # and loads it as a module.  This unnecessarily limiting use
00309         # to a .py file in the current directory and won't allow use
00310         # of modules that are found via sys.path.  The second half
00311         # duplicates the functionality of just specifying input files
00312         # on the command line.
00313         if self.opts.file_list:
00314             if (os.path.exists(self.opts.file_list) and self.opts.file_list[-3:] == '.py'):
00315                 # If python file, assume it is a List module
00316                 listModule = self.opts.file_list[:-3]
00317                 try:
00318                     exec('import ' + listModule)
00319                 except Exception,err:
00320                     import traceback
00321                     traceback.print_exc()
00322                     print 'Failed to load', listModule
00323                     sys.exit(1)
00324                 print "Loaded \"%s\""%listModule
00325                 self.add_input_file(eval(listModule + '.list'))
00326 
00327             else:
00328                 # Otherwise get list by evaluation the expression.
00329                 fnames = eval(str(self.opts.file_list))
00330                 # if evaluation is not a real list try and force it to be one.
00331                 if not isinstance(fnames, type([])):
00332                     fnames = eval(str(fnames))
00333                 self.add_input_file(fnames)
00334 
00335 
00336         # Set up the event stores and  event loop manager(s).
00337         import DybEventMgr
00338         self.evtmgr = DybEventMgr.Configure(self.use_AES, self.opts.event_builder, 
00339                                             len(self.input_files))
00340 
00341         # Check that all input files are of the same type
00342         inputFiletype = None
00343         if self.input_files:
00344             firstFile = self.input_files[0]
00345             inputFiletype = firstFile[-5:]
00346             for filename in self.input_files[1:]:
00347                 if filename[-5:] != inputFiletype:
00348                     msg = "nuwa.py ERROR: Incorrect file type:",file
00349                     raise TypeError,msg
00350                     pass
00351                 continue
00352             pass
00353 
00354         # Should check that the files exist unless they are 
00355         # xrootd files
00356         for fname in self.input_files:
00357             if fname[:7]=='root://' :
00358                 if self.input_files.index( fname ) < 1 :
00359                     print 'Cannot test for existence of xrootd file. Here is first file',fname
00360             else:
00361                 if not os.path.exists(fname):
00362                     msg = 'Warning: input file does not exist "%s"'%fname
00363                     raise RuntimeError,msg
00364                 continue
00365 
00366         # For ROOT input, prepare streams
00367         if self.input_files and inputFiletype == '.root':
00368             try:
00369                 default = self.opts.input_streams['default']
00370             except KeyError:
00371                 default = self.input_files
00372             self.opts.input_streams['default'] = default
00373             pass
00374             
00375         if self.opts.input_streams and inputFiletype == '.root':
00376             # Enable reading of Run Data from ROOT input file
00377             from RunDataSvc.RunDataSvcConf import RunDataSvc
00378             runDataSvc = RunDataSvc()
00379             runDataSvc.ReadFromFile = True
00380             self.theApp.ExtSvc.append( runDataSvc )
00381 
00382             # Enable reading of Job Info from ROOT input file
00383             from JobInfoSvc.JobInfoSvcConf import JobInfoSvc
00384             jobInfoSvc = JobInfoSvc()
00385             jobInfoSvc.ReadFromFile = True
00386             self.theApp.ExtSvc.append( jobInfoSvc )
00387 
00388         # Merge default output files if needed
00389         if self.opts.output:
00390             self.opts.output_streams['default'] = self.opts.output
00391 
00392         if self.opts.output_streams:
00393             # Add Run Data to ROOT output file
00394             from RunDataSvc.RunDataSvcConf import RunDataWriterAlg
00395             runDataWriterAlg = RunDataWriterAlg()
00396             runDataWriterAlg.WriteToFile = True
00397 
00398             # Add Job Info to ROOT output file
00399             from JobInfoSvc.JobInfoSvcConf import JobInfoWriterAlg
00400             jobInfoWriterAlg = JobInfoWriterAlg()
00401             jobInfoWriterAlg.WriteToFile = True
00402 
00403                     
00404 
00405 
00406         import RootIOSvc
00407         self.io = RootIOSvc.Configure(self.opts.input_streams,
00408                                       self.opts.output_streams,
00409                                       self.opts.force_readout)
00410 
00411 
00412         # Set up DaqFormat handling
00413         import DaqReadoutSvc
00414         daqSvc=DaqReadoutSvc.DaqReadoutSvc()
00415         # For ActionOnInvalid: 0 = exit, 1 = skip, 2 = process
00416         daqSvc.ActionOnInvalid = 1
00417         if self.opts.skip_raw_rpc:
00418             # For ActionOnRawRpc: 0 = process, 1 = skip
00419             daqSvc.ActionOnRawRpc = 1
00420         if self.opts.daq == 'on':
00421             daqSvc.Generating=True
00422         if self.opts.daq == 'off':
00423             daqSvc.Generating=False
00424         if self.opts.repack_rpc == 'on':
00425             daqSvc.RepackRpcEvent=True
00426         if self.opts.repack_rpc == 'off':
00427             daqSvc.RepackRpcEvent=False
00428         if self.opts.time_alignment == 'on':
00429             daqSvc.TimeAlignment=True
00430         if self.opts.time_alignment== 'off':
00431             daqSvc.TimeAlignment=False
00432         daqSvc.RunNumber = self.opts.run
00433         daqSvc.SamplingInterval = 1
00434 
00435         # Enable Raw File I/O if given raw data files.
00436         if inputFiletype == '.data' or inputFiletype == '.rraw':
00437             if self.opts.daq == 'on' or self.opts.raw_load == 'daq':
00438                 import DaqFormatIO
00439                 self.daqIO = DaqFormatIO.eventSelector()
00440                 self.daqIO.InputFiles = self.input_files
00441 
00442             elif self.opts.raw_load.startswith('daq-'):
00443                 from DaqFormatModules.DaqFormatModulesConf import DybDaq__DaqFormatInput
00444                 self.rawLoad = DybDaq__DaqFormatInput()
00445                 self.theApp.TopAlg += [self.rawLoad]
00446                 self.rawLoad.InputFiles = self.input_files
00447                 if self.opts.raw_load =='daq-repackage':
00448                     self.rawLoad.Repackage = True
00449                 else:
00450                     self.rawLoad.Repackage = False
00451                     if self.opts.raw_load in ['daq-both','daq-packed']:
00452                         self.rawLoad.UsePacked = True
00453                     else:
00454                         self.rawLoad.UsePacked = False
00455                     if self.opts.raw_load in ['daq-both','daq-unpacked']:
00456                         self.rawLoad.UseUnpacked = True
00457                     else:
00458                         self.rawLoad.UseUnpacked = False
00459             else:
00460                 from DaqFormatModules.DaqFormatModulesConf import DybDaq__DaqFormatInput
00461                 self.rawLoad = DybDaq__DaqFormatInput()
00462                 self.theApp.TopAlg += [self.rawLoad]
00463                 self.rawLoad.InputFiles = self.input_files
00464                 if self.opts.raw_load =='daq-repackage':
00465                     self.rawLoad.Repackage = True
00466                 else:
00467                     self.rawLoad.Repackage = False
00468                     if self.opts.raw_load in ['both','packed']:
00469                         self.rawLoad.UsePacked = True
00470                     else:
00471                         self.rawLoad.UsePacked = False
00472                     if self.opts.raw_load in ['both','unpacked']:
00473                         self.rawLoad.UseUnpacked = True
00474                     else:
00475                         self.rawLoad.UseUnpacked = False
00476 
00477 
00478         # Configure Statistics Service for histogram I/O
00479         if self.opts.output_stats or self.opts.input_stats:
00480             print 'output_stats=',self.opts.output_stats
00481             print 'input_stats=',self.opts.input_stats
00482             from StatisticsSvc.StatisticsSvcConf import StatisticsSvc
00483             statsSvc = StatisticsSvc()
00484             if self.opts.output_stats:
00485                 statsSvc.Output = self.opts.output_stats
00486             if self.opts.input_stats:
00487                 statsSvc.Input = self.opts.input_stats 
00488             self.theApp.ExtSvc.append( statsSvc )
00489             pass
00490 
00491         ninput = len(self.input_files)
00492         for k,v in self.opts.input_streams.iteritems():
00493             ninput += len(v)
00494         if self.opts.executions == -1 and ninput == 0:
00495             err = 'Unbounded running with no input files, probably not what you wanted.\n'
00496             sys.stderr.write(err)
00497             sys.exit(1)
00498             pass
00499         
00500         return

def DybPython::Control::NuWa::configure_random (   self)

Definition at line 501 of file Control.py.

00502                               :
00503 
00504         if self.opts.random == 'on':
00505             from DybAlg import RandomSeeder
00506             rs = RandomSeeder(hostid = self.opts.hostid,
00507                               run = self.opts.run,
00508                               execcountoffset = self.opts.execution-1,
00509                               engine='HepRndm::Engine<HepRootRandom>',
00510                               burncount=0);
00511         return

def DybPython::Control::NuWa::configure_run (   self)

Definition at line 512 of file Control.py.

00513                            :
00514         # Catch command-line run number and wall time for simulation jobs
00515         # Send run number and wall time to RunDataSvc
00516         from RunDataSvc.RunDataSvcConf import RunDataSvc
00517         runDataSvc = RunDataSvc()
00518         runDataSvc.SimRunNumber = self.opts.run
00519         runDataSvc.SimStartTime = self.opts.time
00520         self.theApp.ExtSvc.append( runDataSvc )
00521         return
00522 

def DybPython::Control::NuWa::configure_dbconf (   self)
Existance of DBCONF envvar is used as a signal to switch between Static and DB services, 
so pull it out separate for clarity

Definition at line 523 of file Control.py.

00524                               :
00525         """
00526         Existance of DBCONF envvar is used as a signal to switch between Static and DB services, 
00527         so pull it out separate for clarity
00528         """
00529         if self.opts.dbconf:
00530             os.environ.update( DBCONF=self.opts.dbconf )
00531             log.debug("Defining DBCONF to %s " % os.environ.get('DBCONF') )
00532  

def DybPython::Control::NuWa::configure_dbi (   self)
For motivation for DbiSvc level configuration,  see :dybsvn:`ticket:842`

Definition at line 533 of file Control.py.

00534                            :
00535         """
00536         For motivation for DbiSvc level configuration,  see :dybsvn:`ticket:842`
00537         """
00538         dbconf = os.environ.get('DBCONF')
00539         if not dbconf:
00540             log.debug("dbconf is not defined so skipping DbiSvc setup")
00541             return 
00542 
00543         from DbiSvc.DbiSvcConf import DbiSvc
00544         dbiSvc = DbiSvc()
00545 
00546         if self.opts.dbilevel:
00547             log.debug("dbi OutputLevel %s " % self.opts.dbilevel )
00548             dbiSvc.OutputLevel = self.opts.dbilevel
00549 
00550         if self.opts.dbirollback:
00551             log.debug("dbi RollbackDates %s " % self.opts.dbirollback )
00552             dbiSvc.RollbackDates = ",".join(self.opts.dbirollback)
00553 
00554         if self.opts.dbiextracondition:
00555             log.debug("dbi ExtraConditions %s " % self.opts.dbiextracondition )
00556             dbiSvc.ExtraConditions = "@".join(self.opts.dbiextracondition)
00557 
00558         log.debug("dbi OrderContextQuery %s " % self.opts.dbiordercontextquery )
00559         dbiSvc.OrderContextQuery = self.opts.dbiordercontextquery
00560 
00561         self.theApp.ExtSvc.append( dbiSvc )
00562         return

def DybPython::Control::NuWa::configure_job (   self)

Definition at line 563 of file Control.py.

00564                            :
00565         # Catch job configuration info and feed to Job Info Service
00566         from JobInfoSvc.JobInfoSvcConf import JobInfoSvc
00567         jobInfoSvc = JobInfoSvc()
00568         # Initialize current job info
00569         import sys, time, os, socket
00570         nuwaPath = os.path.realpath(os.path.expandvars('$DYBRELEASEROOT/../../'))
00571         # Get SVN revision number from file (faster than calling svnversion)
00572         revision = "Unknown"
00573         revPath = os.path.expandvars('$DYBPYTHONROOT/share/revision.txt')
00574         if os.path.isfile(revPath):
00575             revisionFile = open(revPath)
00576             if revisionFile:
00577                 revLine = revisionFile.readline()
00578                 if revLine != "":
00579                     revision = revLine.strip()
00580         # Get full domain name of the host
00581         hostName = socket.getfqdn()
00582             
00583         jobConfigList = {'command':' '.join(sys.argv),
00584                          'revision':revision,
00585                          'nuwaPath':nuwaPath,
00586                          'cmtConfig':os.getenv('CMTCONFIG'),
00587                          'hostid':str(self.opts.hostid),
00588                          'hostname':hostName,
00589                          'jobTime':time.strftime("%a, %d %b %Y %H:%M:%S +0000", time.gmtime()),
00590                          'username':os.getenv('LOGNAME')}
00591         jobInfoSvc.JobId = self.opts.jobid
00592         # Process user-defined job information
00593         for key in self.opts.job_info.keys():
00594             print "User-defined job info: "+key+" = "+self.opts.job_info[key]
00595             jobConfigList[key] = self.opts.job_info[key]
00596         jobInfoSvc.JobConfig = jobConfigList
00597         self.theApp.ExtSvc.append( jobInfoSvc )
00598         return

def DybPython::Control::NuWa::configure_geometry (   self)

Definition at line 599 of file Control.py.

00600                                 :
00601         import XmlDetDesc
00602         if self.opts.detector:
00603             XmlDetDesc.Configure(self.opts.detector)
00604             print "Note: using non-default geometry in " \
00605                 + self.opts.detector + " is loaded."
00606         else:
00607             XmlDetDesc.Configure()
00608         return
00609 

def DybPython::Control::NuWa::configure_optmods (   self)
load and configure() "-m" modules here   

Definition at line 610 of file Control.py.

00611                                :
00612         """ load and configure() "-m" modules here   """
00613         if self.opts.module == None: return
00614 
00615         from DybPython.cmdline import quotedParse
00616 
00617         for modcmd in self.opts.module:
00618             argv = quotedParse(modcmd)
00619             modname = argv[0]
00620             if modname[-3:] == '.py':
00621                 print 'Fixing assumed typo in module name from "%s" to "%s"'%(modname,modname[:-3])
00622                 modname = modname[:-3]
00623             modargs = argv[1:]
00624             self.configure_mod( modname, modargs )
00625 

def DybPython::Control::NuWa::configure_args (   self)
spin over all non-option arguments 

Definition at line 626 of file Control.py.

00627                             :
00628         """ spin over all non-option arguments """
00629         # At this point there should be no input files left in the args.
00630         for cfg in self.args:
00631             self.configure_mod( cfg )
00632         return
    
def DybPython::Control::NuWa::configure_mod (   self,
  modname,
  modargs = None 
)
Configure this module, add to job 

Definition at line 633 of file Control.py.

00634                                                   :
00635         """ Configure this module, add to job """
00636         mod = None
00637         # First, check for module type
00638         if modname[-3:] == '.py':
00639             # This is a script file, not a true module
00640             mod = ModuleHandle()
00641             mod.topAlg = []
00642             cfgf = open(modname)
00643             try:
00644                 exec(cfgf)
00645             except Exception,err:
00646                 import traceback
00647                 traceback.print_exc()
00648                 print 'Failed to load',modname
00649                 sys.exit(1)
00650             print "Loaded \"%s\""%modname
00651             mod.__name__ = modname
00652             cfgf.close()
00653         else:
00654             # This is a true module
00655             if modargs is None:
00656                 print 'Importing module "%s"' % modname
00657             else:
00658                 print 'Importing module "%s" ["%s"]' % (modname, '", "'.join(modargs))
00659                 pass
00660             exec('import %s'%modname) # Don't catch potential exception!
00661             mod = eval(modname)
00662             mod.topAlg = []
00663         
00664             print "Trying to call configure() on",modname
00665             try:
00666                 if modargs:
00667                     mod.configure(argv=modargs)
00668                 else:
00669                     mod.configure()
00670             except AttributeError,err:
00671                 if "object has no attribute 'configure" not in str(err):
00672                     raise
00673             pass
00674         # Catch list of top algs added by this module
00675         propertyNames = self.theApp.getValuedProperties()
00676         newAlgs = []
00677         if propertyNames.has_key("TopAlg"):
00678             newAlgs = propertyNames["TopAlg"]
00679         mod.topAlg += newAlgs[:]
00680         self.theApp.TopAlg = []
00681         self.modules.append( mod )
00682 

def DybPython::Control::NuWa::configure_user (   self)

Definition at line 683 of file Control.py.

00684                             :
00685         from DybPython.Include import include, IncludeError
00686         include.setShowIncludes(self.opts.show_includes)
00687 
00688         try:
00689             include("$HOME/.nuwarc")
00690             print "Included ~/.nuwarc"
00691         except IncludeError,err:
00692             pass
00693 
00694         self.modules = []
00695         # First, catch any pre-module algorithms, by name
00696         propertyNames = self.theApp.getValuedProperties()
00697         currentAlgNames = []
00698         if propertyNames.has_key("TopAlg"):
00699             currentAlgNames = propertyNames["TopAlg"]
00700         self.initAlgs = currentAlgNames[:]
00701         self.theApp.TopAlg = []
00702         # Process user modules
00703         self.configure_optmods()
00704         self.configure_args()
00705         
00706         return

def DybPython::Control::NuWa::_configure_trimio_output (   self) [private]

Definition at line 707 of file Control.py.

00708                                       :
00709         'Configure TrimIO based output'
00710         print "Using DybTrimIO to trigger output and do AES trimming"
00711         # add DybTrimIO
00712         from DybAlg.DybAlgConf import DybTrimIO
00713         trimIO = DybTrimIO()
00714         self.post_user_algs.append(trimIO)
00715         # add Trimming service
00716         from DybEventMgr.DybEventMgrConf import ArchiveTrimSvc
00717         trimSvc = ArchiveTrimSvc()
00718         self.theApp.ExtSvc.append(trimSvc)
00719         trimSvc.DefaultWindowSeconds = self.AES_window/units.second
00720 
00721         # check whether io is involed
00722         if self.opts.output_streams:
00723             # add SaveB4TrimAesSvc
00724             from DybEventMgr.DybEventMgrConf import SaveB4TrimAesSvc
00725             svb4t = SaveB4TrimAesSvc()
00726             self.theApp.ExtSvc.append(svb4t)
00727             trimIO.SaveFlag=True;
00728         else:
00729             trimIO.SaveFlag=False;
00730         return

def DybPython::Control::NuWa::_configure_archive_output (   self) [private]

Definition at line 731 of file Control.py.

00732                                        :
00733         'Configure DybArchiveOutput to drive output storage and trimming'
00734         print "Using DybArchiveOutput to trigger output and do AES trimming"
00735         from DybAlg.DybAlgConf import DybArchiveOutput
00736         dao = DybArchiveOutput()
00737         dao.WindowSeconds = self.AES_window/units.second
00738         if self.opts.output_streams:
00739             dao.SaveOutput = True;
00740         self.post_user_algs.append(dao)
00741         return

def DybPython::Control::NuWa::configure_post_user (   self)

Definition at line 742 of file Control.py.

00743                                  :
00744         if not self.use_AES:
00745             if self.opts.output_streams:
00746                 print "Using DybStoreAlg to trigger output."
00747                 from RootIOTest.RootIOTestConf import DybStoreAlg
00748                 dsa = DybStoreAlg()
00749                 self.post_user_algs.append(dsa)
00750         else:
00751             if self.opts.executions != 0:
00752                 #self._configure_trimio_output()
00753                 self._configure_archive_output()
00754 
00755         # Add Run Data Writer to the end of the list of algorithms
00756         from RunDataSvc.RunDataSvcConf import RunDataWriterAlg
00757         self.post_user_algs.append(RunDataWriterAlg())
00758         from JobInfoSvc.JobInfoSvcConf import JobInfoWriterAlg
00759         self.post_user_algs.append(JobInfoWriterAlg())
00760         return
00761 
  
def DybPython::Control::NuWa::configure_visualization (   self)

Definition at line 762 of file Control.py.

00763                                      :
00764         'Configure for "quanjing/panoramix" visualization'
00765         gui = os.getenv("QUANJINGROOT")
00766         if not gui:
00767             print "Warning: no QUANJINGROOT variable, no default gui, failure likely."
00768             return
00769         gui += "/xml/gui.onx"
00770         
00771         xmlvis = os.getenv("XMLVISROOT",None)
00772         if xmlvis is None:
00773             from VisSvc.VisSvcConf import VisualizationSvc
00774             vis = VisualizationSvc()
00775             vis.ColorDbLocation = os.path.dirname(gui) + '/gui/vis/colors.xml'
00776 
00777         from OnXSvc.OnXSvcConf import OnXSvc
00778         onx = OnXSvc()
00779         onx.Toolkit = "NATIVE"
00780         #onx.Toolkit = "Qt"
00781         onx.OutputToTerminal = True
00782         onx.File = gui
00783 
00784         # pre v19r5, suggestion from Pere
00785         import GaudiPython as gaudi
00786         appMgr = gaudi.AppMgr( selfoptions={'Runable':'OnXSvc'} )
00787         appMgr.config()
00788         return

def DybPython::Control::NuWa::configure_dyb_services (   self)

Definition at line 789 of file Control.py.

00790                                     :
00791         'Configure common Dyb services'
00792 
00793         dbconf = os.environ.get('DBCONF')
00794         if dbconf:              # use DBI services where available
00795             svcmap = {
00796                 'CableSvc':'DybCableSvc',
00797                 'SimDataSvc':'StaticSimDataSvc', # only static currently available
00798                 'CalibDataSvc':'DbiCalibDataSvc', # DBI version still in progress
00799                 }
00800 
00801         else:
00802             svcmap = {
00803                 'CableSvc':'StaticCableSvc',
00804                 'SimDataSvc':'StaticSimDataSvc',
00805                 'CalibDataSvc':'StaticCalibDataSvc',
00806                 }
00807 
00808         print 'Configuring Dyb services:'
00809         for alias, svc in svcmap.iteritems():
00810             self.add_service_redirect(alias,svc)
00811             print '\t%s is %s' % (alias,svc)
00812             continue
00813 
00814         return
00815         

def DybPython::Control::NuWa::configure (   self)

Definition at line 816 of file Control.py.

00817                        :
00818         self.configure_dbconf()
00819         self.configure_python_features()
00820         self.configure_ipython()
00821         self.configure_framework()
00822         self.configure_dyb_services()
00823         self.configure_random()
00824         self.configure_run()
00825         self.configure_dbi()
00826         self.configure_job()
00827         self.configure_geometry()
00828         self.configure_user()
00829         self.configure_post_user()
00830         return

def DybPython::Control::NuWa::add_service_redirect (   self,
  alias,
  name 
)
Make alias an alias for given service.  
Should be called during configuration only

Definition at line 831 of file Control.py.

00832                                              :
00833         '''
00834         Make alias an alias for given service.  
00835         Should be called during configuration only
00836         '''
00837 
00838         from Gaudi.Configuration import ApplicationMgr
00839         theApp = ApplicationMgr()
00840 
00841         # convert from ['Name/Alias', 'Name/Alias'] to Alias->Name dictionary
00842         sm_map = dict([(x,y) for (y,x) in map(lambda z: z.split('/'), 
00843                                               theApp.SvcMapping)])
00844         for a,n in sm_map.iteritems():
00845             if a == alias:
00846                 try:
00847                     theApp.ExtSvc.remove('%s/%s' % (n,a))
00848                 except ValueError:
00849                     pass
00850                 pass
00851 
00852         sm_map.update({ alias: name })
00853 
00854         # convert back to Gaudi formated list
00855         theApp.SvcMapping = ['%s/%s' % (x,y) for y,x in sm_map.iteritems()]
00856         theApp.ExtSvc.append('%s/%s' % (name,alias))
00857         return

def DybPython::Control::NuWa::spade_files (   self)

Definition at line 858 of file Control.py.

00859                          :
00860         if not self.spadeSvc: return []
00861         return self.spadeSvc.marked

def DybPython::Control::NuWa::run_post_user (   self,
  appMgr 
)
Run time addition of Python Algs so they are in correct module-order 

Definition at line 862 of file Control.py.

00863                                    :
00864         '''
00865         Run time addition of Python Algs so they are in correct module-order 
00866         '''
00867 
00868         # Add framework algorithms after user algorithms
00869         # Catch post-config algs
00870         postConfigAlgs = appMgr.topAlg[:]
00871         appMgr.topAlg = []
00872         # Run any modules with a run() function
00873         for mod in self.modules:
00874             try:
00875                 print "Trying to call run() on",mod.__name__
00876                 mod.run(appMgr)
00877             except AttributeError,err:
00878                 if "object has no attribute 'run'" not in str(err):
00879                     raise
00880                 pass
00881             # Catch list of top algs added by this module
00882             newAlgs = appMgr.topAlg[:]
00883             mod.topAlg += newAlgs
00884             appMgr.topAlg = []
00885 
00886         # Correctly order user-module algorithms
00887         newAlgOrder = []
00888         if self.initAlgs:
00889             newAlgOrder += self.initAlgs
00890         for mod in self.modules:
00891             newAlgOrder += mod.topAlg
00892         newAlgOrder += postConfigAlgs
00893         appMgr.topAlg = newAlgOrder
00894 
00895         for alg in self.post_user_algs:
00896             appMgr.addAlgorithm(alg.getFullJobOptName())
00897         return
00898         

def DybPython::Control::NuWa::run (   self)

Definition at line 899 of file Control.py.

00900                  :
00901 
00902         if self.opts.visualize:
00903             self.configure_visualization()
00904 
00905         from GaudiPython import AppMgr
00906         appMgr = AppMgr()
00907 
00908         if self.opts.color == "dark":
00909             #print 'setting up for dark color terminals'
00910             msg = appMgr.property('MessageSvc')
00911 
00912             msg.OutputLevel = self.opts.log_level
00913             msg.useColors=True
00914             msg.fatalColorCode=['red','white']
00915             msg.errorColorCode=['red']
00916             msg.warningColorCode=['yellow']
00917             msg.debugColorCode=['blue']
00918             msg.verboseColorCode=['cyan']
00919             pass
00920         if self.opts.color == "light":
00921             #print 'setting up for light color terminals'
00922             msg = appMgr.property('MessageSvc')
00923             msg.OutputLevel = self.opts.log_level
00924             msg.useColors=True
00925             msg.fatalColorCode=['red','white']
00926             msg.errorColorCode=['red']
00927             msg.warningColorCode=['yellow']
00928             msg.debugColorCode=['blue']
00929             msg.verboseColorCode=['cyan']
00930             pass
00931 
00932         # Muck with the msg formatting to give more room to show component names
00933         msg = appMgr.property('MessageSvc')
00934         msg.Format="%% F%%%dW%%S%%7W%%R%%T %%0W%%M"%35
00935 
00936         msgSvc = appMgr.service('MessageSvc','IMessageSvc')
00937         if self.opts.msg_stream_level:
00938             for namelevel in self.opts.msg_stream_level:
00939                 name,level = namelevel.split(',')
00940                 msgSvc.setOutputLevel(name,int(level))
00941                 continue
00942             pass
00943 
00944         self.run_post_user(appMgr)
00945 
00946         print 'Starting run with these top algorithms:',appMgr.TopAlg
00947 
00948         numExecs = self.opts.executions
00949         if numExecs < 0: numExecs = -1
00950         appMgr.EvtMax = numExecs
00951         #appMgr.initialize()
00952 
00953         # Run a main looper
00954         if self.opts.visualize:
00955             import Quanjing
00956             Quanjing.start()
00957             Quanjing.toui()
00958         elif self.opts.external_loop:
00959             print 'Delegating main loop to',self.opts.external_loop
00960             exec ("import %s"%self.opts.external_loop)
00961             mod = eval (self.opts.external_loop)
00962             mod.start(appMgr)
00963         else:
00964             appMgr.run(numExecs)
00965             pass
00966 
00967         if self.opts.interactive:
00968             import DybPython.Interactive
00969             self.ipshell() 
00970 
00971         return

def DybPython::Control::NuWa::finalize (   self)

Definition at line 972 of file Control.py.

00973                       :
00974         from GaudiPython import AppMgr
00975         appMgr = AppMgr()
00976         appMgr.exit()
00977 
00978         for fn in self.spade_files():
00979             base,ext = os.path.splitext(fn)
00980             meta = base + '.meta.xml'
00981             if not os.path.exists(meta):
00982                 print 'Skipping non existent SPADE meta data file: "%s"'%meta
00983                 continue
00984             sem = base + '.sem'
00985             os.rename(meta,sem)
00986             continue


Member Data Documentation

Definition at line 57 of file Control.py.

Definition at line 57 of file Control.py.

Definition at line 57 of file Control.py.

Definition at line 57 of file Control.py.

Definition at line 68 of file Control.py.

Definition at line 68 of file Control.py.

Definition at line 68 of file Control.py.

Definition at line 68 of file Control.py.

Definition at line 220 of file Control.py.

Definition at line 244 of file Control.py.

Uncommenting the following leads to a service-initialization loop.

self.theApp.AuditServices = True

Definition at line 246 of file Control.py.

Definition at line 246 of file Control.py.

Definition at line 246 of file Control.py.

Definition at line 246 of file Control.py.

Definition at line 246 of file Control.py.

Definition at line 683 of file Control.py.


The documentation for this class was generated from the following file:
| Classes | Job Modules | Data Objects | Services | Algorithms | Tools | Packages | Directories | Tracs |

Generated on Fri May 16 2014 09:55:40 for DybPython by doxygen 1.7.4