Now so far as you have a simple requirement of independent configuration variables, you are good to go right away. But if you are used to shell scripting, and are spoilt by variable resolution there (one variable resolving from another previously defined variable, like this - $URL=$PROTOCOL://$DOMAIN:$PORT/), you sure are not satisfied. And if you are spoilt by python's opulence, you sure cannot just let it go and do with whatever is available.
I couldn't either. I had a complex configuration file, and I just couldn't think doing without variable resolution. Here is a subset of what I wanted. I had a common configuration section about a server and different set of URLs which I wanted to fire HTTP requests at (get/post/any such), on the server. Now these domains would be listed in a different 'urls' section, while server details will be listed in the 'server' section. Something like this -
[server]
domain = www.pyarabola.in
port = 80
[request-config]
protocol = http
type = get
[urls]
konkan = ${request-config.protocol}://${sever.domain}:${server.port}/coastal-konkan-day01.html
apple = ${request-config.protocol}://${sever.domain}:${server.port}/applyhypepotatopoteto.html
[requests]
get_konkan = GET ${urls.konkan}
get_apple = GET ${urls.apple}Note: I know there will be other (better) ways to get the URLs constructed, but this is a made-up example just to make the use-case clear.
The requirement is quite straight-forward - if I want to change the server or port number or the type of request I am making, I dont want to end up changing all [requests].
As you can see, the parameters are constructed by back-referencing other parameters. And the back-referencing is multi-level - one parameter referencing another, which in turn is referencing yet another, and so on, as can be seen for the get_konkan parameter in [requests] section.
As far as I know, the current ConfigParser module (or its ConfigParser.SafeConfigParser class) doesn't allow such referencing in the INI file it parses. Enter NewConfigParser.
NewConfigParser
So I ended up extending the ConfigParser.SafeConfigParser class and overriding its get(...) method to make space for such back-referencing.
Features -
- Define-anywhere-use-anywhere freedom, while defining dependent options
- Caching of resolved parameters
- Any amount of depth in dependancies (*only limited by global recursion limit)
- Detection of circular dependancies
- Same usage semantics as SafeConfigParser, i.e. variable resolution is transparant to the user
- No section means current section
This means even if the [requests] section in the above example, is defined at the top of the INI file, before any other parameter definition, it will not complain. This is contrary to the way linux shell variables are resolved. There the used variable must have been defined apriori.
A small optimization. It caches any resolved parameters. As a result, if get_konkan request, from above example, is resolved before get_apple request, the NewSafeConfigParser.get("requests", "get_apple") call wont again resolve the same protocol, domain or port. It would directly use it from cache.
The parameters' dependancies are resolved recursively. So the depth of dependancies is only limited by python's global recursion limit.
All circular dependancies are detected and they result into a CircularDependancyException. It also prints the dependancy graph for you to make hay. Take a look at the circular dependacy example in the Examples section below.
Since NewSafeConfigParser is extended from SafeConfigParser, the usage semantics are same as SafeConfigParser. You would call a NewSafeConfigParser.get(...) method in the same way you would call a SafeConfigParser.get(...) method. All the referencing and parameter resolution and caching and circular dependancy detection happens behind the closed curtains. User No Bother.
Back-references are to be specified in ${section.parameter} manner for resolution. If no section is specified, e.g. new_param = ${param}, then current section (same as the new_param) is assumed, and the back referenced parameter - param - is resolved in the same section as that of new_param.
Check the Examples section below for more details.
Limitations
- Works only for strings as of now
- Only SafeConfigParser class extended as of now. Not implemented for ConfigParser.
Code: Where and How
The code is available on GitHub - https://github.com/shreyas/NewConfigParser
I was too lazy to pack it into a legitimate module and post it on PyPI for pip install (it's only a smal tweak really). So it's just a single file (as of now). You might want to copy it into you tools folder and import it as a module to start using it.
License
This was done quite long back, only to realise that I need it quite often. So thought it might be useful to someone else as well. One shouldn't just consume open-source, but contribute to it too. So am releasing it in the public domain under Apache Software License version 2.0.
Do what you want with it, just maintain that copyright notice at the top, despite the (obvious) disclaimer that I wont be responsible if your rocket hits someone's ass instead of the moon just because you resolved its controller config using this parser.
Examples
shreyas@tochukasui:~/devel/python$ python
Python 2.7.3 (default, Aug 1 2012, 05:16:07)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from NewConfigParser import SafeConfigParser as NewSafeConfigParser
>>> from ConfigParser import SafeConfigParser
>>> newparser = NewSafeConfigParser()
>>> origparser = SafeConfigParser()
>>> newparser.read("config.ini")
['config.ini']
>>> origparser.read("config.ini")
['config.ini']
>>> newparser.get("request", "request_type")
'get'
>>> origparser.get("request", "request_type")
'get'
>>> newparser.get("request", "request");
'get http://pyarabola.blogspot.in'
>>> origparser.get("request", "request");
'${request_type} ${server.url}'
>>> newparser.get("server", "url");
'http://pyarabola.blogspot.in'
>>> origparser.get("server", "url");
'${protocol}://${website}'
>>> origparser.get("server", "protocol");
'http'
>>> origparser.get("server", "website");
'${subdomain}.${host}.${tld}'
>>> newparser.get("server", "website");
'pyarabola.blogspot.in'
>>> Circular Dependancy Detection
shreyas@tochukasui:~/devel/python$ cat circdep.ini
[humans]
drinking_water = ${govt.pipeline}
[animals]
drinking_water = ${natural.reservoire}
[govt]
pipeline = ${natural.reservoire}
[natural]
reservoire = ${river}
river = ${heavens.rains}
[heavens]
rains = ${clouds}
clouds = ${vapor}
vapor = ${natural.reservoire}
shreyas@tochukasui:~/devel/python$ cat cdep.py
#!/usr/bin/python
import NewConfigParser
p = NewConfigParser.SafeConfigParser()
p.read("circdep.ini")
try:
p.get("humans", "drinking_water")
except Exception as e:
print e
shreyas@tochukasui:~/devel/python$ ./cdep.py
Cicrular dependancy detected while resolving parameter 'natural.reservoire': natural.reservoire -> natural.river -> heavens.rains -> heavens.clouds -> heavens.vapor -> natural.reservoire
shreyas@tochukasui:~/devel/python$
