Daya Bay Offline User Manual¶

Version:	22909
Date:	May 16, 2014
PDF:	`OfflineUserManual.pdf` (via reStructuredText and Sphinx)
Old PDF:	`main.pdf` (direct from latex)

Introduction
- Intended Audience
- Document Organization
- Contributing
- Building Documentation
- Typographical Conventions
Quick Start
- Offline Infrastructure
- Installation and Working with the Source Code
- Offline Framework
- Data Model
- Detector Description
- Kinematic Generators
- Detector Simulation
- Quick Start with Truth Information
- Electronics Simulation
- Trigger Simulation
- Readout
- Event Display
  - A Plain Event Display: EvtDsp
    - Simple Mode
    - Advance Mode
- Reconstruction
- Database
Analysis Basics
- Introduction
- Daya Bay Data Files
- NuWa Basics
  - The nuwa.py Command
  - NuWa Job Modules
- NuWa Recipes
- Cheat Sheets
- Hands-on Exercises
Offline Infrastructure
- Mailing lists
- DocDB
- Wikis
- Trac bug tracker
Installation and Working with the Source Code
- Using pre-installed release
  - Basic setup
  - Setup the dybgaudi project
- Instalation of a Release
- Anatomy of a Release
  - Release, Projects and Packages
  - Personal Projects
- Version Control Your Code
  - Using SVN to Contribute to a Release
  - Using GIT with SVN
- Technical Details of the Installation
  - LCGCMT
    - Builders
Offline Framework
- Introduction
- Framework Components and Interfaces
- Common types of Components
- Writing your own component
- Properties and Configuration
Data Model
- Overview
  - HeaderObject
- Times
- Examples of using the Data Model objects
  - Tutorial examples
Data I/O
- Goal
- Features
- Packages
- I/O Related Job Configuration
- How the I/O Subsystem Works
- Adding New Data Classes
Detector Description
- Introduction
  - Volumes
    - Logical
  - Physical
  - Touchable
- Conventions
  - AD PMTs
  - Pool PMTs
  - RPC
- Coordinate System
- XML Files
- Transient Detector Store
- Configuring the Detector Description
- PMT Lookups
- Visualization
Kinematic Generators
- Introduction
- Generator output
- Generator Tools
- Generator Packages
- Types of GenTools
  - GenTools package
  - GenDecay Package
- Configuration
- MuonProphet
Detector Simulation
- Introduction
- Configuring DetSim
- Truth Information
- Truth Parameters
Electronics Simulation
- Introduction
- Algorithms
- Tools
  - Hit Tools
  - FEE Tool: EsIdealFeeTool
    - Multiplicity Generation and TDC
    - ADC Generation
- Simulation Constant
Trigger Simulation
- Introduction
- Configuration
- Current Triggers
  - TsMultTriggerTool
  - TsExternalTriggerTool
- Adding a new Trigger
Readout
- Introduction
- ReadoutHeader
- SimReadoutHeader
- Readout Algorithms
  - ROsSequencerAlg
  - ROsReadoutAlg
- Readout Tools
Simulation Processing Models
- Introduction
- Fifteen
Reconstruction
Database
- Database Interface
- Concepts
- Running
  - Setting up the Environment
  - Configuring
- Accessing Existing Tables
- Creating New Tables
  - Choosing Table Names
  - Creating Detector Descriptions
- Filling Tables
- ASCII Flat Files and Catalogues
  - Overview
  - Flat Files
    - Format
    - URL
  - Catalogues
    - Format
    - URL
  - Example
- MySQL Crib
- Performance
Database Maintanence
- Introduction
- Building and Running dbmjob
  - Building
  - Running
    - Exporting Data
    - Importing Data
Bibliography
Testing Code With Nose
- Nosetests Introduction
- Using Test Attributes
- Running Tests Using dybinst
  - Informing dybinst about tests
  - Getting the slaves to auto run package tests
- Testing nose plugins
Standard Operating Procedures
- DB Definitions
  - Database Topology Diagram
    - offline_db replication data flow
  - Database names
- DBI Very Briefly
  - Validity Tables
  - Validity Table Timestamps
  - Using Rollback to Debug/Workaround problem DB entries
  - What is TASK for ?
  - Features of Clean Validity Tables
    - Overlay Versioning
  - DBI Q and A
- Rules for Code that writes to the Database
  - Scope of Rules
  - DB Writing Code Management
  - Rationale for dybsvn rule
  - Recording how DB updates were prepared
  - Verification of reproducibility
  - Testing DB writing code
- Configuring DB Access
  - Create Simple DB configuration file
    - Standardized Section Names
    - Section dependent testing
  - DBCONF envvar
  - N ways to set an envvar
  - Background Information
    - What is a mysql dump file ?
    - What is a DBI ascii catalog ?
  - Hands-On Exercise 1 : Troubleshooting DB connection configuration
  - Hands On Exercise 2 : Interactive DybDBI
- DB Table Updating Workflow
  - Objectives
  - Workflow Outline
  - Rationale for this workflow
  - Planning Update Size and Frequency
  - workflow steps in detail
  - Exceptional Operating Procedures for Major Changes
  - Hands On Exercise 3 : Copy Offline DB
  - Nosetests of workflow steps
  - Development History of Workflow
- Table Specific Instructions
  - CalibPmtFineGain
- DB Table Writing
  - Using DybDbi to write to tmp_offline_db
  - Hands On Exercise 4 : Write $DBWRITERROOT/share/DYB_MC_AD1.txt into CalibPmtSpec
  - Assigning Applicability of Constants
  - Many more examples of DB writing with DybDbi
  - Using DBWriter to write to tmp_offline_db
- DB Table Reading
  - DB Reading with DybDbi
  - Using mysql client
  - Hands On Exercise 5 : Read from DB with varying context
  - Hands On Exercise 6 : Read run timestart/timeend from DaqRunInfo table
    - Default Context Query
    - Modify to use wideopen validity context
  - Fixing this page if it breaks
- Debugging unexpected parameters
  - Configuration check
  - ipython DybDbi session
  - DbiDataSvc test
  - mysql client session
  - Following the mysql tail
  - GDB Debugging of Template Laden DBI
- DB Table Creation
  - Workflow Outline for Adding Tables
  - Design Tables
  - Prepare .spec File
  - Generate Row Classes from .spec
    - Ensuring Consistency When Changing Spec
  - Copy offline_db to tmp_offline_db
  - Create New Tables in tmp_offline_db
  - Populate New Table With Dummy Data
  - Verify tables using the mysql client
- DB Validation
  - Interactive Checking with ipython
    - DB Checking
    - Ascii Catalog Checking
  - Workflow Checks
  - Generic Validation
- DB Testing
  - Nosetesting reminder
  - When and where
  - Standardized testing
  - Responsibility for maintaining tests
  - Responsibility for running tests
  - Proof of testing
  - Dealing with mistakes
  - Checking entries make sense
  - Status of packages with tests
  - Comparing python datetimes with DBI TimeStamps
- DB Administration
  - Temporary DB Setup by MySQL Administrators
- Custom DB Operations
  - Tools to manipulate mysqldump files
  - Preparing and Comparing Dump files
  - Download mysqldump file and load into DB
  - Copying a few DBI tables between DBs using rdumpcat, rloadcat
    - Non-decoupled rdumpcat into empty folder
    - Loading of partial ascii catalog into target DB with fastforwarding of INSERTDATEs
  - Jumpstarting offline_db.DqChannelPacked table
  - CQScraper testing
- DB Services
  - User Interfaces to DBI Data
  - Tables which are Missing something
- DCS tables grouped/ordered by schema
  - SAB_TEMP
  - DBNS_ACU_HV_SlotTemp
  - DBNS_Temp
  - AD1_TEMP
  - DBNS_HALL5_TEMP
  - config_table
  - DYBAlarm
  - DBNS_AD1_HV_Imon
  - DBNS_AD2_HV_Imon
  - SAB_AD1_HV_Imon
  - SAB_AD2_HV_Imon
  - SAB_AD2_HV_SlotTemp
  - DBNS_AD1_HV_SlotTemp
  - DBNS_AD2_HV_SlotTemp
  - SAB_AD1_HV_SlotTemp
  - DBNS_SAB_TEMP
  - site_table
  - DBNS_MUON_PMT_HV_Imon
  - DBNS_MUON_PMT_HV_SlotTemp
  - status_table
  - DBNS_AD_HV_SlotTemp
  - dyb_muoncal
  - DBNS_ACU_HV_Pw
  - DBNS_ACU_HV_Imon
  - DBNS_ACU_HV_Vmon
  - EH1_ENV_RadonMonitor
  - DBNS_AD1_LidSensor
  - DBNS_AD2_LidSensor
  - DBNS_AD1_VME
  - DBNS_AD2_VME
  - DBNS_IW_VME
  - DBNS_Muon_PMT_VME
  - DBNS_OW_VME
  - DBNS_RPC_VME
  - SAB_AD1_VME
  - DBNS_AD1_HV
  - DBNS_AD2_HV
  - SAB_AD1_HV_Vmon
  - SAB_AD2_HV_Vmon
  - SAB_AD2_HV_Pw
  - DBNS_AD1_HVPw
  - DBNS_AD2_HV_Pw
  - SAB_AD1_HV_Pw
  - DBNS_MUON_PMT_HV_Vmon
  - DBNS_MUON_PMT_HV_Pw
  - DBNS_AD_HV_Imon
  - DBNS_AD_HV_Vmon
  - DBNS_AD_HV_Pw
- Non DBI access to DBI and other tables
  - Summary of Non DBI approaches
  - SQLAlchemy access to DBI tables with NonDbi
- Scraping source databases into offline_db
  - Generic Scraping Introduction
  - Scraper Status
  - DCS peculiarities
    - Time zones and scraping
  - TODO
    - Framework Level
    - Specific Regimes
  - Running Scrapers
    - Dybinst Level
    - Package Level
  - Implementing Scrapers
  - Configuring Scrapers
  - Testing Scraper Operation
  - Continuous running under supervisord
    - Initial Setup
    - Supervisorctl CLI
  - Steps to Deployment
  - Development Tips
- DBI Internals
  - Overlay versioning implementation
  - Overlay overriding problem
  - Fix attempt A
  - Write single entry into empty table
  - Write two entries at same validity range ts:EOT into empty table
  - Write 3 entries for different runs into empty table
  - Delving into overlay detection and DbiValidityRecBuilder
  - fGap : special vrec holding trim results
  - Trimming in Builder ctor
- DBI Overlay Versioning Bug
  - Background
  - Current Intended code/SOP changes
    - Planned rollout of DBI modifications
      - Extra Ordering Fix
      - Timestart Flooring and Collision Avoidance
  - Intended Migration of Existing DB entries
    - Rebuild Approach
    - Fixup Validity Approach
      - Current Transfixion Approach
  - VLUT Comparisons
  - Alternative to Recreat Rather Than Fix Approach
    - CableMap/HardwareID
  - DBI Validity Ordering Change
  - Payload Digest Rather than SEQNO comparison
    - CableMap
    - HardwareID
- DBI from C++
  - Primer for C++ usage of DBI
Administrative Operating Procedures for SVN/Trac/MySQL
- Tasks Summary
- SVN/Trac
- Backups Overview
- Monitoring
  - crontabs
- DbiMonitor package : cron invoked nosetests
  - DbiMonitor.tests.test_dcs
    - Checking update age of tables in IHEP DCS mirror
  - DbiMonitor.tests.test_offline
    - Checking update age of tables in offline_db
      - Technical issue regards logging and nosetests
- Env Repository : Admin Infrastructure Sources
- Dybinst : Dayabay Offline Software Installer
- Trac+SVN backup/transfer
- SSH Setup For Automated transfers
  - Debugging Blocked SSH
  - ssh-agent process monitoring
- Offline DB Backup
- DBSVN : dybaux SVN pre-commit hook
- Bitten Debugging
  - Extract trac.db from backup tarball
  - Examining bitten tables
    - table counts
    - bitten_build
- MySQL DB Repair
Python API
- DB
  - DybPython.db
  - DybPython.db.DB
- DBAUX
  - DybPython.dbaux
  - DybPython.dbaux.Aux
- DBConf
  - DybPython.dbconf
  - DBConf
- DBCas
  - DybPython.dbcas
- dbsvn - DBI SVN Gatekeeper
  - DybPython.dbsvn
  - DBIValidate
- DBSRV
  - DybPython.dbsrv
    - dbsrv : MySQL Server Utilities
  - DybPython.dbsrv.DB
- DybDbiPre
  - Tab
- DybDbi
- DybPython
- DybPython.Control
- DybPython.dbicnf
- DbiDataSvc
  - DbiDataSvc
- NonDbi
  - NonDbi
    - SQLAlchemy Ecosystem
- Scraper
- DybTest
Documentation
- About This Documentation
- Todolist
- References

Unrecognized latex commands¶

None

Other index

Table Of Contents

Next topic

This Page

Daya Bay Links

Content Skeleton

Daya Bay Offline User Manual¶

Unrecognized latex commands¶

Indices and tables¶

Navigation

Quick search

Other index

Table Of Contents

Next topic

This Page

Daya Bay Links

Content Skeleton

Daya Bay Offline User Manual¶

Unrecognized latex commands¶

Indices and tables¶

Navigation