woo.batch

class woo.batch.TableParamReader(file, firstLine=- 1)

Class for reading simulation parameters from text file.

Each parameter is represented by one column, each parameter set by one line. Colums are separated by blanks (no quoting).

First non-empty line contains column titles (without quotes). You may use special column named ‘title’ to describe this parameter set; if such colum is absent, title will be built by concatenating column names and corresponding values (param1=34,param2=12.22,param4=foo)

• from columns ending in ! (the ! is not included in the column name)

• from all columns, if no columns end in !.

• columns containing literal - (minus) will be ignored

Empty lines within the file are ignored (although counted); # starts comment till the end of line. Number of blank-separated columns must be the same for all non-empty lines.

A special value = can be used instead of parameter value; value from the previous non-empty line will be used instead (works recursively); in XLS, empty cell is treated the same as =.

This class is used by woo.utils.readParamsFromTable.

>>> tryData=[
...   ['head1','important2!','head3','...','...','...','!OMP_NUM_THREADS!','abcd'],
...   [1,1.1, '1','.','1','5', 1.2,1.3,],
...   ['a','b','HE','AD','_','3','c','d','###','comment'],
...   ['# empty line'],
...   [1,'=','=','=','=','=','=','g']
... ]
>>> import woo
>>> tryFile=woo.master.tmpFilename()
>>> # write text
>>> f1=tryFile+'.txt'
>>> txt=open(f1,'w')
>>> for ll in tryData: n=txt.write(' '.join([str(l) for l in ll])+'\n') # set n to suppress output in doctest under py3k
>>> txt.close()
>>> 
>>> # write xls
>>> import xlwt,itertools
>>> f2=tryFile+'.xls'
>>> xls=xlwt.Workbook(); sheet=xls.add_sheet('test')
>>> for r in range(len(tryData)):
...   for c in range(len(tryData[r])):
...     sheet.write(r,c,tryData[r][c])
>>> xls.save(f2)
>>> 
>>> from pprint import *
>>> pprint(TableParamReader(f1).paramDict())
{2: {'!OMP_NUM_THREADS': '1.2',
     'abcd': '1.3',
     'head1': '1',
     'head3': '1.15',
     'important2': '1.1',
     'title': 'important2=1.1,OMP_NUM_THREADS=1.2'},
 3: {'!OMP_NUM_THREADS': 'c',
     'abcd': 'd',
     'head1': 'a',
     'head3': 'HEAD_3',
     'important2': 'b',
     'title': 'important2=b,OMP_NUM_THREADS=c'},
 5: {'!OMP_NUM_THREADS': 'c',
     'abcd': 'g',
     'head1': '1',
     'head3': 'HEAD_3',
     'important2': 'b',
     'title': 'important2=b,OMP_NUM_THREADS=c__line=5__'}}
>>> pprint(TableParamReader(f2).paramDict())
{2: {'!OMP_NUM_THREADS': '1.2',
     'abcd': '1.3',
     'head1': '1',
     'head3': '1.15',
     'important2': '1.1',
     'title': 'important2=1.1,OMP_NUM_THREADS=1.2'},
 3: {'!OMP_NUM_THREADS': 'c',
     'abcd': 'd',
     'head1': 'a',
     'head3': 'HEAD_3',
     'important2': 'b',
     'title': 'important2=b,OMP_NUM_THREADS=c'},
 5: {'!OMP_NUM_THREADS': 'c',
     'abcd': 'g',
     'head1': '1',
     'head3': 'HEAD_3',
     'important2': 'b',
     'title': 'important2=b,OMP_NUM_THREADS=c__line=5__'}}

Setup the reader class, read data into memory. firstLine determines the number of the first line; if negative, 1 is used for XLS files and 0 for text files. The reason is that spreadsheets number lines from 1 whereas text editors number lines from zero, and having the numbering the same as the usual UI for editing that format is convenient.

paramDict()

Return dictionary containing data from file given to constructor. Keys are line numbers (which might be non-contiguous and refer to real line numbers that one can see in text editors), values are dictionaries mapping parameter names to their values given in the file. The special value ‘=’ has already been interpreted, ! (bangs) (if any) were already removed from column titles, title column has already been added (if absent).

woo.batch.cartProdParamTable(params, out, same='')

Write parameter table (as XLS) where all parameters in pp (which is a dictionary, or collections.OrderedDict) are traversed.

Parameters

• same – content of repeated cellls; if None, repeated cells are filled with the repeated value. Other useful values are '=' and '' (empty cell)

• out – XLS file to write to

• params – dictionary-like with parameter values; keys may be n-tuples, which will span multiple columns – in that case, values must also be n-tuples, and will also span those columns

Returns

total number of lines written

>>> import collections, woo.batch 
>>> pp=collections.OrderedDict() # use OrderedDict for predictable column ordering
>>> pp['pattern']=['ortho','hexa']
>>> pp['radius','...']=[(r,'*woo.unit["mm"]') for r in (1,2,3)]    # use continuation columns for unit specification
>>> pp['gravity','...','...']=[('(0,0,',g,')') for g in (9.81,20)] # use continuation columns for concatenation of expression
>>> xls=woo.master.tmpFilename()+'.xls'
>>> woo.batch.cartProdParamTable(params=pp,out=xls)
12
>>> import pprint
>>> pprint.pprint(TableParamReader(xls).paramDict())
{2: {'gravity': '(0,0,9.81)',
     'pattern': 'ortho',
     'radius': '1*woo.unit["mm"]',
     'title': 'pattern=ortho,radius=1mm,gravity=(0,0,9.81)'},
 3: {'gravity': '(0,0,20)',
     'pattern': 'ortho',
     'radius': '1*woo.unit["mm"]',
     'title': 'pattern=ortho,radius=1mm,gravity=(0,0,20)'},
 4: {'gravity': '(0,0,9.81)',
     'pattern': 'ortho',
     'radius': '2*woo.unit["mm"]',
     'title': 'pattern=ortho,radius=2mm,gravity=(0,0,9.81)'},
 5: {'gravity': '(0,0,20)',
     'pattern': 'ortho',
     'radius': '2*woo.unit["mm"]',
     'title': 'pattern=ortho,radius=2mm,gravity=(0,0,20)'},
 6: {'gravity': '(0,0,9.81)',
     'pattern': 'ortho',
     'radius': '3*woo.unit["mm"]',
     'title': 'pattern=ortho,radius=3mm,gravity=(0,0,9.81)'},
 7: {'gravity': '(0,0,20)',
     'pattern': 'ortho',
     'radius': '3*woo.unit["mm"]',
     'title': 'pattern=ortho,radius=3mm,gravity=(0,0,20)'},
 8: {'gravity': '(0,0,9.81)',
     'pattern': 'hexa',
     'radius': '1*woo.unit["mm"]',
     'title': 'pattern=hexa,radius=1mm,gravity=(0,0,9.81)'},
 9: {'gravity': '(0,0,20)',
     'pattern': 'hexa',
     'radius': '1*woo.unit["mm"]',
     'title': 'pattern=hexa,radius=1mm,gravity=(0,0,20)'},
 10: {'gravity': '(0,0,9.81)',
      'pattern': 'hexa',
      'radius': '2*woo.unit["mm"]',
      'title': 'pattern=hexa,radius=2mm,gravity=(0,0,9.81)'},
 11: {'gravity': '(0,0,20)',
      'pattern': 'hexa',
      'radius': '2*woo.unit["mm"]',
      'title': 'pattern=hexa,radius=2mm,gravity=(0,0,20)'},
 12: {'gravity': '(0,0,9.81)',
      'pattern': 'hexa',
      'radius': '3*woo.unit["mm"]',
      'title': 'pattern=hexa,radius=3mm,gravity=(0,0,9.81)'},
 13: {'gravity': '(0,0,20)',
      'pattern': 'hexa',
      'radius': '3*woo.unit["mm"]',
      'title': 'pattern=hexa,radius=3mm,gravity=(0,0,20)'}}

Table 2 Generated cartesian product parameter table (XLS)

pattern	radius	…	gravity	…	…
ortho	1	*woo.unit[“mm”]	(0,0,	9.81	)
				20
	2			9.81
				20
	3			9.81
				20
hexa	1			9.81
				20
	2			9.81
				20
	3			9.81
				20

woo.batch.dbReadResults(db, basicTypes=False)

Return list of dictionaries, representing database contents.

Parameters

basicTypes – don’t reconstruct Woo objects from JSON (keep those as dicts) and don’t return data series as numpy arrays.

Todo

Nested (grouped) series are not read correctly from HDF5. Should be fixed either by flattening the hiearchy (like we do in dbToSpread and stuffing it into returned dict; or by reflecting the hierarchy in the dict returned.

woo.batch.dbToJSON(db, **kw)

Return simulation database as JSON string.

Parameters

kw – additional arguments passed to json.dumps.

woo.batch.dbToSpread(db, out=None, dialect='xls', rows=False, series=True, ignored=('plotData', 'tags'), sortFirst=('title', 'batchtable', 'batchTableLine', 'finished', 'sceneId', 'duration'), selector=None)

Select simulation results (using selector) stored in batch database db, flatten data for each simulation, and dump the data in the CSV format (using dialect: ‘excel’, ‘excel-tab’, ‘xls’) into file out (standard output if not given). If rows, every simulation is saved into one row of the CSV file (i.e. attributes are in columns), otherwise each simulation corresponds to one column and each attribute is in one row.

If out ends with ‘.xls’, the ‘xls’ dialect is forced regardless of the value given. The ‘xls’ format will refuse to write to standard output (out must be given).

ignored fields are used to exclude large data from the dump: either database column of that name, or any attribute of that name. Attributes are flattened and path separated with ‘.’.

series determines whether the series field will be written to a separate sheet, named by the sceneId. This is only supported with the xls dialect and raises error otherwise (unless series field is empty).

Fields are sorted in their natural order (i.e. alphabetically, but respecting numbers), with sortFirst fields coming at the beginning.

woo.batch.hasBatchTable()

Tell whether an external batch table is given or not (scripts may be run in script also without batch tables)

woo.batch.inBatch()

Tell whether we are running inside the batch or separately.

woo.batch.mayHaveStaleLock(db)

woo.batch.readParamsFromTable(scene, under='table', noTableOk=True, unknownOk=False, **kw)

Read parameters from a file and assign them to woo.core.Scene.lab under the under pseudo-module (e.g. Scene.lab.table.foo and so on. This function is used for scripts (as opposed to preprocessors) running in a batch. The file format is described in TableParamReader (CSV or XLS).

Assigned tags (the title column is synthesized if absent,see woo.utils.TableParamReader):

S=woo.master.scene
S.tags['title']=…                                            # assigns the title column; might be synthesized
S.tags['params']="name1=val1,name2=val2,…"                   # all explicitly assigned parameters
S.tags['defaultParams']="unassignedName1=defaultValue1,…"    # parameters that were left at their defaults
S.tags['d.id']=s.tags['id']+'.'+s.tags['title']
S.tags['id.d']=s.tags['title']+'.'+s.tags['id']

Parameters

• tableFile – text file (with one value per blank-separated columns)

• under – name of pseudo-module under S.lab to save all values to (table by default)

• tableLine (int) – number of line where to get the values from

• noTableOk (bool) – if False, raise exception if the file cannot be open; use default values otherwise

• unknownOk (bool) – do not raise exception if unknown column name is found in the file, and assign it as well

Returns

None

woo.batch.runPreprocessor(pre, preFile=None)

Execute given Preprocessor, modifying its attributes from batch (if running in batch). Each column from the batch table (except of environment variables starting with !) must correspond to a preprocessor’s attribute.

Nested attributes are allowed, e.g. with woo.pre.horse.FallingHorse, a column named mat.tanPhi will modify horse’s material’s friction angle, using the default material object.

woo.batch.wait()

If running inside a batch, start the master simulation (if not already running) and block until it stops by itself. Typically used at the end of script so that it does not finish prematurely in batch mode (the execution would be ended in such a case). Does nothing outisde of batch.

woo.batch.writeResults(scene, defaultDb='woo-results.hdf5', syncXls=True, dbFmt=None, series=None, quiet=False, postHooks=[], **kw)

Write results to batch database. With syncXls, corresponding excel-file is re-generated. Series is a dicionary of 1d arrays written to separate sheets in the XLS. If series is None (default), S.plot.data are automatically added. All other **kw arguments are serialized in the misc field, which then appears in the main XLS sheet.

All data are serialized using json so that they can be read back in a language-independent manner.

postHooks is list of functions (taking a single argument - the database name) which will be called once the database has been updated. They can be used in conjunction with woo.batch.dbReadResults to write aaggregate results from all records in the database.

Tip

Report issues or inclarities to github.