Other analysis/nuwarecipes

Table Of Contents

Previous topic

NuWa Basics

Next topic

Cheat Sheets

This Page

Daya Bay Links

Content Skeleton

NuWa Recipes

Many NuWa analysis tasks rely on a standard or familiar approach. This section provides a list of recipes for common analysis tasks such as,

See the history of a NuWa File

Before using a NuWa data file, you may want to see what processing has already been done on the file. The following command will print the history of all NuWa jobs that have been run to produce this file:

shell> nuwa.py -n 0 --no-history -m"JobInfoSvc.Dump"
           recon.NoTag.0005773.Physics.SAB-AD2.SFO-1._0001.root

You will see much information printed to the screen, including the following sections which summarize the NuWa jobs that have been run on this file:

Cached Job Information:
{  jobId : daf3a684-6190-11e0-82f7-003048c51482
 cmtConfig : x86_64-slc4-gcc34-opt
 command : /eliza7/dayabay/scratch/dandwyer/NuWa-trunk-opt/dybgaudi/InstallArea/scripts/nuwa.py
           -n 0 --no-history -mJobInfoSvc.Dump
           recon.NoTag.0005773.Physics.SAB-AD2.SFO-1._0001.root
 hostid : 931167014
 jobTime : Fri, 08 Apr 2011 03:32:40 +0000
 nuwaPath : /eliza16/dayabay/users/dandwyer/installs/trunk_2011_03_30_opt/NuWa-trunk
 revision : 11307:11331
 username : dandwyer
}


Cached Job Information:
{  jobId : 6f5c02f4-6190-11e0-897b-003048c51482
 cmtConfig : x86_64-slc4-gcc34-opt
 command : /eliza7/dayabay/scratch/dandwyer/NuWa-trunk-opt/dybgaudi/InstallArea/scripts/nuwa.py
           -A None -n -1 --no-history --random=off -mQuickstart.DryRunTables
           -mQuickstart.Calibrate -mQuickstart.Reconstruct
           -o recon.NoTag.0005773.Physics.SAB-AD2.SFO-1._0001.root
           daq.NoTag.0005773.Physics.SAB-AD2.SFO-1._0001.root
 hostid : 931167014
 jobTime : Fri, 08 Apr 2011 03:29:39 +0000
 nuwaPath : /eliza16/dayabay/users/dandwyer/installs/trunk_2011_03_30_opt/NuWa-trunk
 revision : 11307:11331
 username : dandwyer
}


Cached Job Information:
{  jobId : 22c6620e-6190-11e0-84ac-003048c51482
 cmtConfig : x86_64-slc4-gcc34-opt
 command : /eliza7/dayabay/scratch/dandwyer/NuWa-trunk-opt/dybgaudi/InstallArea/scripts/nuwa.py
           -A None -n -1 --no-history --random=off -mProcessTools.LoadReadout
           -o daq.NoTag.0005773.Physics.SAB-AD2.SFO-1._0001.root
           /eliza7/dayabay/data/exp/dayabay/2010/TestDAQ/NoTag/0922/daq.NoTag.0005773.Physics.SAB-AD2.SFO-1._0001.data
 hostid : 931167014
 jobTime : Fri, 08 Apr 2011 03:27:31 +0000
 nuwaPath : /eliza16/dayabay/users/dandwyer/installs/trunk_2011_03_30_opt/NuWa-trunk
 revision : 11307:11331
 username : dandwyer
}

The jobs are displayed in reverse-chronological order. The first job converted the raw daq .data file to a NuWa .root file. The second job ran an example calibration and reconstruction of the raw data. The final job (the current running job) is printing the job information to the screen.

Tag Events in a NuWa File

Event tags are used to identify a subset of events. These can be used to separate events into classes such as muons, inverse-beta decay, noise, etc. In general, tags be used to identify any set of events of interest.

The job module dybgaudi:Tagging/UserTagging/python/UserTagging/UserTag/DetectorTag.py is a simple example of tagging readouts by detector type. The tag can be applied by adding the module to a NuWa job:

shell> nuwa.py -n -1 --no-history -m"UserTagging.UserTag.DetectorTag"
           daq.NoTag.0005773.Physics.SAB-AD2.SFO-1._0001.root

To add your own tag, follow the steps for modifing an existing python module (section Write a Python analysis Algorithm.) Use dybgaudi:Tagging/UserTagging/python/UserTagging/UserTag/DetectorTag.py as a starting point. You should add your own tag in the initTagList function:

self.addTag('MySpecialEvent' , '/Event/UserTag/MySpecialEvent')

In the check function, you should retrieve event data and decide if you want to tag it:

# Get reconstructed data
recHdr = evt["/Event/Rec/AdSimple"]
# Add your calculation / decision here
# ...
#
if tagThisEvent:
    # Keep track of the reconstructed data you are tagging
    self.getTag('MySpecialEvent').setInputHeaders( [recHdr] )
    self.tagIt('MySpecialEvent')

Once a tag has been set, it can be used by later analysis algorithms in the current job, or saved to the output file and used at a later time. Here is a Python example of checking the tag:

# Check tag
tag = evt["/Event/UserTag/MySpecialEvent"]
if tag:
    # This event is tagged.  Do something.
    # ...

Tags can also be used to produce filtered data sets, as shown in section Write Tagged Data to a New File.

Add Variables to a NuWa File

A common task is to add a new user-defined variable for each event. For example, the time since the previous trigger can be calculated and added to each event. This is a task for UserData.

The example job module dybgaudi:Tutorial/Quickstart/python/Quickstart/DtData.py shows the example of adding the time since the previous trigger to each event. This example can be run:

shell> nuwa.py -n -1 --no-history -m"Quickstart.DtData"
           -o daqPlus.NoTag.0005773.Physics.SAB-AD2.SFO-1._0001.root
           daq.NoTag.0005773.Physics.SAB-AD2.SFO-1._0001.root

After completion, the output file can be opened in ROOT and the new data variables can be viewed and histogrammed (Fig fig:userdata.) The file can also be read back into another NuWa job, and the user data will still be accessible.

../../_images/userDataBrowser.png

fig:userdata

../../_images/dtLastTrigger.png

fig:userdata

Example of browsing and histogramming user-defined data in ROOT.

To add your own variables, copy and modify the module dybgaudi:Tutorial/Quickstart/python/Quickstart/DtData.py. See section Write a Python analysis Algorithm for general advice on modifying an existing job module. Currently single integers, single floating-point decimal numbers, and arrays of each can be added as user-defined variables.

Adding User-defined Variables to Tagged Events

The dybgaudi:Tagging/UserTagging package provides some convenient tools for simultaneously applying tags and adding user data for those tagged events. Following the example described in section Tag Events in a NuWa File, user data can be added in parallel to an event tag. In the initTagList function, you can define user data associated with the tag:

myTag = self.addTag('MySpecialEvent' , '/Event/UserTag/MySpecialEvent')
myData = myTag.addData('MySpecialData','/Event/UserData/MySpecialData')
myData.addInt('myInt')

In the check function, you should set the variable value before calling tagIt:

if tagThisEvent:
    # Keep track of the reconstructed data you are tagging
    self.getTag('MySpecialEvent').setInputHeaders( [recHdr] )
    myData = self.getTag('MySpecialEvent').getData('MySpecialData')
    myData.set('myInt',12345)
    self.tagIt('MySpecialEvent')

Copy Data Paths to a New File

There may be situations where you would like to filter only some paths of data to a smaller file. The job module SimpleFilter.Keep can be used for this purpose. The following example shows how to create an output file which contains only the AdSimple reconstructed data:

shell> nuwa.py -n -1 -m"SimpleFilter.Keep /Event/Rec/AdSimple"
            -o adSimple.NoTag.0005773.Physics.SAB-AD2.SFO-1._0001.root
            recon.NoTag.0005773.Physics.SAB-AD2.SFO-1._0001.root

This module can take multiple arguments to save more paths to the same file:

shell> nuwa.py -n -1 -m"SimpleFilter.Keep /Event/Rec/AdSimple /Event/Rec/AdQmlf"
            -o myRecData.NoTag.0005773.Physics.SAB-AD2.SFO-1._0001.root
            recon.NoTag.0005773.Physics.SAB-AD2.SFO-1._0001.root

Write Tagged Data to a New File

There may be situations where you would like to filter only some events to a smaller data file. The SmartFilter package provides some tools for this purpose. The first step is to define your own tag for the events you wish to keep, as discussed in section Tag Events in a NuWa File. The following example shows how to create an output file which contains only the events you have tagged as MySpecialEvents:

shell> nuwa.py -n -1 -m"MySpecialTagger" -m"SmartFilter.Keep /Event/UserTag/MySpecialEvents"
            -o mySpecialEvents.NoTag.0005773.Physics.SAB-AD2.SFO-1._0001.root
            recon.NoTag.0005773.Physics.SAB-AD2.SFO-1._0001.root

The output file will contain your tag /Event/UserTag/MySpecialEvents, plus any data that your tag refers to such as /Event/Rec/AdSimple, /Event/Readout/ReadoutHeader, etc.

To create more advanced data filters, copy and modify the job module dybgaudi:Filtering/SmartFilter/python/SmartFilter/Example.py.

Change an Existing Job Module

This section describes how to change an existing module with name PACKAGE.MODULE. First copy this Job Module to your local directory. You can locate a module using the environment variable $ PACKAGE ROOT,

shell> mkdir mywork
shell> cd mywork
shell> cp $<PACKAGE>ROOT/python/<PACKAGE>/<MODULE>.py myModule.py

Once you have a copy of the Job Module, open it with your favorite text editor. The module is written in the Python language (http://www.python.org); see the Python website for a good tutorial on this language. Job Modules are composed of two functions: configure() and run(),

def configure( argv=[] ):
    """A description of your module here
    """
    # Most job configuration commands here
    return

def run(app):
    """Specific run-time configuration"""
    # Some specific items must go here (Python algorithms, add libraries, etc.)
    pass

For advice on what lines to modify in the module, send your request to the offline software mailing list: theta13-offline@dayabay.lbl.gov.

To run your modified version of the module, call it in the nuwa.py command without the PACKAGE. prefix in the module name. With no prefix, modules from the current directory will be used.

shell> ls
myModule.py
shell> nuwa.py -n -1 -m"myModule" recon.NoTag.0005773.Physics.SAB-AD2.SFO-1._0001.root

Write a Python analysis Algorithm

If you wish to add your own algorithm to NuWa, a good place to start is by writing a prototype algorithm in Python. Writing your algorithm in Python is much easier than C++, and does not require you to compile.

To get started, copy the example template Python algorithm to your local directory:

shell> mkdir mywork
shell> cd mywork
shell> cp $QUICKSTARTROOT/python/Quickstart/Template.py myAlg.py

Alternatively, you can copy PrintRawData.py, PrintCalibData.py, or PrintReconData.py if you want to specifically process the readout, calibrated, or reconstructed data. Each of these files is a combination of a Python algorithm and a nuwa Python Job Module. To run this module and algorithm, you can call it in the following way:

shell> nuwa.py -n -1 -m"myAlg" recon.NoTag.0005773.Physics.SAB-AD2.SFO-1._0001.root

Inside this file, you can find a Python algorithm. It is a Python class that defines three key functions:

  • initialize(): Called once at job start
  • execute(): Called once for each event
  • finalize(): Called once at job end

You should edit these functions so that the algorithm will do the task you want. There are a few common tasks for algorithms. One is to print to the screen some data from the event:

def execute(self):
    evt = self.evtSvc()
    reconHdr = evt["/Event/Rec/RecHeader"]
    print "Energy [MeV] = ", reconHdr.recResult().energy() / units.MeV

Another common task is to histogram some data from the event:

def initialize(self):
    # Define the histogram
    self.stats["/file1/myhists/energy"] = TH1F("energy",
                                               "Reconstructed energy for each trigger",
                                                100,0,10)

def execute(self):
    evt = self.evtSvc()
    reconHdr = evt["/Event/Rec/RecHeader"]
    if reconHdr.recResult().energyStatus() == ReconStatus.kGood:
        #Fill the histogram
        self.stats["/file1/myhists/energy"].Fill(reconHdr.recResult().energy() / units.MeV)

Although these examples are simple, algorithms can perform complex calculations on the data that are not possible directly from ROOT. For cheat-sheets of the data available in NuWa, see the following sections: Readout data [Readout data in NuWa], Calibrated hit data [Calibrated data in NuWa], Reconstructed data [Reconstructed data in NuWa].

Remember to commit your new algorithm to SVN! The wiki section wiki:SVN_Repository#Guidelines provides some tips on committing new software to SVN.

Write a C++ analysis Algorithm

A drawback of using Python algorithms is that they will usually run slower than an algorithm written in C++. If you wish to run your algorithm as part of data production, or if you just want it to run faster, then you should convert it to C++.

Adding a C++ algorithm to Gaudi is a more complex task. The first step is to create your own Project. Your own Project allows you to write and run your own C++ analysis software with NuWa. See section Making your own Project for how to prepare this.

Once you have your own project, you should prepare your own package for your new algorithm. A tool has been provided to help you with this. The following commands will set up your own package:

shell> cd myNuWa
shell> svn export http:/ /dayabay.ihep.ac.cn/svn/dybsvn/people/wangzhe/Start
shell> svn export http:/ /dayabay.ihep.ac.cn/svn/dybsvn/people/wangzhe/ProjRename
shell> ProjRename Start MyNewAlg
shell> ls
MyNewAlg  ProjRename
shell> emacs MyNewAlg/src/components/MyNewAlg.cc &

At this point you should edit the empty algorithm in MyNewAlg/src/components/MyNewAlg.cc. In particular, you should add your analysis code into the initialize(), execute(), and finalize() functions.

To compile your new algorithm, you should do the following in a new clean shell:

shell> pushd NuWa-trunk
shell> source setup.sh
shell> export CMTPROJECTPATH=/path/to/myProjects:${CMTPROJECTPATH}
shell> popd
shell> cd myNuWa/MyNewAlg/cmt
shell> cmt config; cmt make;

Now you should setup a separate ‘running’ shell for you to run and test your new algorithm. Staring with a clean shell, run the following:

shell> pushd NuWa-trunk
shell> source setup.sh
shell> export CMTPROJECTPATH=/path/to/myProjects:${CMTPROJECTPATH}
shell> cd dybgaudi/DybRelease/cmt
shell> source setup.sh
shell> popd
shell> pushd myNuWa/MyNewAlg/cmt
shell> source setup.sh; source setup.sh;

Now you should be set up and ready to run your new NuWa algorithm in this shell:

shell> nuwa.py -n -1 -m"MyNewAlg.run" recon.NoTag.0005773.Physics.SAB-AD2.SFO-1._0001.root

Remember to commit your new algorithm to SVN!

Modify Part of NuWa

Sometimes you may want to modify an existing part of NuWa and test the changes you have made. First, you must setup your own Project as shown in section Making your own Project.

Next, you should checkout the package into your Project:

shell> cd myNuWa
shell> svn checkout http:/ /dayabay.ihep.ac.cn/svn/dybsvn/dybgaudi/trunk/Reconstruction/CenterOfChargePos
shell> ls
CenterOfChargePos
shell> emacs CenterOfChargePos/src/components/CenterOfChargePosTool.cc &

After you have made your changes, you should compile and test your modifications. To compile the modified package, you should run the following commands in a clean shell:

shell> pushd NuWa-trunk
shell> source setup.sh
shell> export CMTPROJECTPATH=/path/to/myProjects:${CMTPROJECTPATH}
shell> popd
shell> cd myNuWa/CenterOfChargePos/cmt
shell> cmt config; cmt make;

To make NuWa use your modified package, run the following commands in a new clean shell:

shell> pushd NuWa-trunk
shell> source setup.sh
shell> export CMTPROJECTPATH=/path/to/myProjects:${CMTPROJECTPATH}
shell> cd dybgaudi/DybRelease/cmt
shell> source setup.sh
shell> popd
shell> pushd myNuWa/CenterOfChargePos/cmt
shell> source setup.sh; source setup.sh;

This shell will now use your modified code instead of the original version in NuWa:

shell> nuwa.py -n -1 -m"Quickstart.Calibrate" -m"Quickstart.Reconstruct"
          -o recon.NoTag.0005773.Physics.SAB-AD2.SFO-1._0001.root
          daq.NoTag.0005773.Physics.SAB-AD2.SFO-1._0001.root

After you have verified that your changes are correct, you can commit your changes:

shell> cd CenterOfChargePos
shell> svn diff
(Review the changes you have made.)
shell> svn commit -m"I fixed a bug!"

Using Services

Another advantage of using NuWa is that it provides a set of useful Services. Services give you access to other data in addition to the event data, such as cable mappings, calibration parameters, geometry information, etc. Services can also provide other useful tasks. Table Some Common Services gives lists some common services. Section NuWa Services gives detailed descriptions of the common services.

Some Common Services
ICableSvc Electronics cable connection maps and hardware serial numbers
ICalibDataSvc PMT and RPC calibration parameters
ISimDataSvc PMT/Electronics input parameters for simulation
IJobInfoSvc NuWa Job History Information (command line, software version, etc)
IRunDataSvc DAQ Run information (run number, configuration, etc.)
IPmtGeomInfoSvc Nominal PMT positions
IStatisticsSvc Saving user-defined histograms, ntuples, trees, etc. to output files

Multiple versions of the same service can exists. For example, StaticCalibDataSvc loads the PMT calibration parameters from a text table, while DbiCalibDataSvc loads the PMT calibration parameters from the database. To access a Service from a Python algorithm, you should load the service in the initialize() function:

self.calibDataSvc = self.svc('ICalibDataSvc','StaticCalibDataSvc')
if self.calibDataSvc == None:
    self.error("Failed to get ICalibDataSvc: StaticCalibDataSvc")
    return FAILURE

When requesting a service, you provide the type of the service (ICalibDataSvc) followed by the specific version you wish to use (StaticCalibDataSvc).

Loading the service in C++ is similar:

ICalibDataSvc* calibDataSvc = svc<ICalibDataSvc>("StaticCalibDataSvc", true);
if( !calibDataSvc ) {
  error() << "Failed to get ICalibDataSvc: StaticCalibDataSvc" << endreq;
  return StatusCode::FAILURE;
}