Other sop/dbcode

Table Of Contents

Previous topic

DBI Very Briefly

Next topic

Configuring DB Access

This Page

Daya Bay Links

Content Skeleton

Rules for Code that writes to the Database

Scope of Rules

All code that writes into the Offline Database is required to abide by the regulations. This includes:

  1. Calibration writers
  2. Automated Scrapers

Warning

code that prepares the parameters is also covered by the rules

DB Writing Code Management

  • code must be reviewed by DB Managers or their delegates

Code reviewers must verify :

  • code is housed(and developed) in dybsvn repository CMT packages

  • packages have nosetests that can be run by everyone (including the slaves)

  • uses DBI (either directly or via DybDbi), no raw SQL

  • all times in UTC

  • context range end validity times, standardized far future time as TimeStamp::GetEOT()

  • uses enums rather than bare integers
    • if enums do not exist they need to be created
  • ... ( more suggestions )

Rationale for dybsvn rule

Housing and developing code in dybsvn has several advantages:

  • allows referencing the the state of the code with a single integer : the revision number
  • easy for all collaborators to see the code that prepared the update now and in the future

Abiding by this rule is a crucial requirement for the creation of reproducible calibration parameters (and by extension reproducible physics results).

Recording how DB updates were prepared

Good practices to adopt to record how an update was prepared

  • include documentation (in any text based format) alongside code in SVN to provide a record of the algorithm used and any changes to it
  • include simple “no argument” scripts in SVN that run your flexible scripts or executables in order to capture the arguments used. These “no argument” simple scripts can be named after the update and will prove useful for subsequent updates.
  • ensure that your final values are created with a clean revision (svnversion needs to report an integer without an “M” for modified).
  • record the revision number

Verification of reproducibility

Although time consuming the best way to ensure that results are reproducible is to test this by

  • requesting collaborators from another cluster/continent to duplicate results using just what was obtained from an SVN checkout (at a defined revision) and data files (which presumably have standard naming that allows them to be accessed from different clusters).

Testing DB writing code

  • development against offline_db is prohibited
  • developing against local copies of offline_db is recommended

Follow the example provided by dybgaudi:Database/DBWriter/tests which demonstrate best working practices for testing DB Writing code, where every step is fully controlled.

  • starts from the vacuum
  • creates an empty DB
  • creates tables descriptions in the DB
  • populates DB with pre-requisite entries (a DaqRunInfo row in this case) using DybDbi
  • invokes the DBWriter script in a separate process, using dybtest.Run
  • does reference comparisons on the output of the script
  • does reference comparisons on the mysqldumped database that results from the running of the script

This approach allows frequent automated running of the test