Other api/dybdbi/source

Previous topic


Next topic


This Page

Daya Bay Links

Content Skeleton


class DybDbi.Source(f, delimiter='t', prefix='#Table', descmarker='#[]', synth='srcline', fields=[])

Bases: list

Behaves like a file and holds the original text of the CSV. Applies some fixes to make readable as CSV:

  1. removes lines that begin with the prefix argument, default #Table
  2. determines the description line by looking for all characters of the descmarker argument, default #[]
  3. normalize the description line to conform to the delimiter

Normally used internally via CSV, but can be useful to debug broken .csv files interactively:

In [2]: import os

In [3]: from DybDbi import Source

In [4]: src=Source(open(os.path.expandvars("$DATASVCROOT/share/feeCableMap_MDC09a.txt")), delimiter=" ", descmarker="#" )

In [5]: for _ in src:print _   ## have to interate to populate 

In [7]: src
Source stat:{'descline': 1, 'prefix': 0, 'total': 1585, 'payload': 1584} 
 cols:['srcline', 'ChannelID', 'Description', 'ElecHardwareId', 'Description', 'SensorID', 'Description', 'SensorHardwareID', 'Description']  

In [8]: src.descmarker
Out[8]: '#'

In [9]: src.descline( src[0] )   ## debugging the field extraction from the description line
Out[9]: ' ChannelID Description ElecHardwareId Description SensorID Description SensorHardwareID Description'

Note the severely invalid .csv (4 fields with the same name) workaround until the .csv can be fixed is to externally impose the fields:

fields = 'ChannelID Description0 ElecHardwareId Description1 SensorID Description2 SensorHardwareID Description3'.split()
src=Source(open(os.path.expandvars("$DATASVCROOT/share/feeCableMap_MDC09a.txt")), delimiter=" ", descmarker="#", fields=fields ) 
  • delimiter – csv field divider
  • prefix – string start of lines to be ignored
  • descmarker – strings used to identify the field description line
  • synth – when defined, add extra field with this name to hold the csv source line number
  • fields – when defined overrides the content of the descline

shrink multiple spaces to a single space, and strip head and tail whitespace


Remove the descmarker characters from the description line,


Checks if line contains all of the description markers


On iterating though this “synthetic” file fixes and additions are made to render the “physicist-csv” as real csv