Other aop/tasks/svntrac

Table Of Contents

Previous topic

Tasks Summary

Next topic

Offline Database Support Tasks

This Page

Daya Bay Links

Content Skeleton

Subversion/Trac/Autobuild/test Support

Overview

The routine operations are generally not time-consuming, and several of them could be automated further to reduce time consumption further. Maintaining near continuous operation of remote servers and heeding and acting upon monitoring emails are the most demanding aspects. This page is arranged in three main sections.

Responsibilities Summary

Lin Tao (IHEP)

Mainly responsible for technical/sys-admin aspects.

  • migrations (successfully migrated dybsvn+dybaux repositories in December)
  • backups setup and monitoring

tasks

  • setup LBNL as a backup tarball target
  • setup Trac/SVN on LBNL machine and test via recovery of backup tarball (possibly using python virtualenv, to avoid need for root access)

Jimmy Ngai (HKU)

Responsible for “Dayabay aspects”

  • followup problems reported on mailing lists OR in trac tickets, and work with Tao to resolve them
  • help new users of Trac/SVN
  • familiarity with dybinst operation, which is the basis for the auto-testing
  • familiarity with bitten operation, how the recipes work
  • have some idea of which test failures can be ignored and which need to be chased and who to chase

tasks

  • Migrate Aberdeen repository (identical machinery to dybsvn) from NTU to HKU by ~end Feb
  • setup repository backups from HKU to CUHK with monitoring
  • setup/test a bitten slave at HKU,
  • Tao can help with configuring the Trac master and giving needed permissions through the Trac master web interface
  • include generation of the OUM docs (Sphinx generated documentation) done as one of the build steps

resources

  • bitten server setup/usage
  • Trac instance covering all non-dybsvn code used for infrastructure
  • are of env with backup and transfer scripts, especially scm-backup.bash, altbackup.py
  • highly configurable value monitoring with email notifications, allows to add new monitoring very quickly and in a standardized manner

Routine Operations

Subversion/Trac maintenance

  • occasionally review timeline http://dayabay.ihep.ac.cn/tracs/dybsvn/timeline look for signs of mis-use, advise on proper SVN usage via private emails initially, escalating to public ones. Look for mis-usage:
    • committing large binary files
    • not including informative commit messages
    • excessive use of dayabay account
    • adding files when copy is more appropriate

Automation Suggestion

An svn pre-commit hook that checks file names and sizes within an intended commit and which disallows commits that exceed the limits would allow prevention of user mistakes (eg committing a.out .exe .so .a .o etc..) and do this maintenance task better than it has ever been done.

Bitten slave autobuild/test system

  • occasional monitoring of builds
  • request slave managers to investigate when their slave builds have not completed cleanly for more than a few weeks
  • help the slave managers to debug issues that arise
  • suggest reref OR clean commit messages where appropriate
  • bump the base revisions in bitten admin web interface, following slave hiatus
  • occasional help with changing the tests that are standardly run

Automation Suggestion

The trac.db is available for SQLite querying on the backup node. Queries could be developed that monitor the state of the builds, looking for such things as no recent builds, or no clean builds for a few weeks.

Maintain dybinst

The Bitten build/test system is integrated to NuWa via the dybinst installation script, requiring the autobuild/test system maintainer to also maintain dybinst Dybinst : Dayabay Offline Software Installer

Offline User Manual Mechanics

The mechanics of the Offline User Manual (OUM) latex to reStructured text conversion and subsequent Sphinx builds of HTML and PDF versions. The content of the OUM documentation needs to be maintained by relevant subsystem experts.

Maintain remote monitor node (non-NuWa)

The remote monitor node (currently cms01.phys.ntu.edu.tw), performs multiple cron monitoring tasks daily. This node does not need a NuWa installation, only an env checkout is needed. Checks include:

  • SVN+Trac backup tarballs have arrived
  • offline_db database tarballs arrived and are valid
  • channelquality_db large database segmented tarballs have arrived and are valid
  • env server continues to respond.

Error conditions result in emails being sent, which must be investigated to ensure the ongoing operation. Details in Monitoring

Maintain remote source node

The remote source node (currently http://dayabay.phys.ntu.edu.tw/tracs/env), is a Trac/SVN instance that houses the source code for installation/customization of the Trac/SVN instances dybsvn and dybaux together with monitoring and backup scripts for the instances and mysql databases offline_db and channelquality_db.

This node runs the same versions of Trac and dependencies as those used at IHEP. The aberdeen SVN+Trac repository is colocated on this node. Backups are performed daily Backups Overview.

Exceptional Tasks

Debugging bitten build system

Glitches have occured with problems unfixed for long periods. Part of the blame for this is the primitive web interface, that is tedious to to use for many types of checks. Improving the monitoring of the build via queries against the daily backup SQLite trac.db would allow more automation. Some example queries in Bitten Debugging

Migrations

Moving servers to new hardware requires extensive preparation and testing. The backup/restore system is used to test for incompatibility problems on test instances prior to making the actual migration. Retaining precisely the same versions is highly preferable, although sometimes this is too inconvenient and version migrations are forced.

Bloated Trac SQLite DB

Possible adverse implications from the increasing size of the dybsvn/dybaux SQLite instances suggest that remedial trimming of the fat would be advisable. Non-trivial development work is required.

Backup

  1. note Trac tendency to fail to return pages during backups, should be investigated