Thursday, June 7, 2012

Calling scripts from a python script dynamically


This is also an interesting thought in architecture. My scenario was that I needed to go about creating a program that gets data from various sources and returns a set of structured information regardless of the source. Each source may or may not have the same data or the data available may need modifying in order to get it into the structure needed. Every source was unique in the way the data should be called. And over time the source could change the way it gave its data meaning that a break in retrieving information for a single source could happen unexpectedly and therefore care had to be taken to build an architecture where the system would not break because of a single fail point.

A key point to note is that once the information was obtained, the way it would be further filtered, organized and stored would be a common process that would not be dependent on any source of information.

So I figured the best way to build this would be to have a separate script that works with the information provided which could advance further over time. And then build a separate script for each source or multiple sources depending on whether or not the method was common enough and these scripts would be added to some kind of config file where they should be called. So there are two ways to do this config file. One would be a separate python script, something like ‘’ which would look something like,

import scriptA
import scriptB
import scriptC
#import the rest of your scripts here

#declare a single method in the class to actually go through all the scripts in which there is a common method to call. Probably specified by documentation
def commonFunction():

Once you’ve done that you’d simply add an import statement and a method call for every new script here. The problem here is that there’s extra code that has to actually be written and overall it’s pretty annoying in a small way that makes you just wish there was a more elegant solution.

So the second config ‘file’ isn’t actually a file. But you basically get the program to check for more scripts in the folder and import them all in at run time and execute a common method that has to be there. So your main script would probably go something like,

def findAndExecute():
   #find the scripts. For each script located,
   script = __import__(‘scriptName’) #strip the .py out
   #call whatever other methods in the framework to organize data

The key here is the __import__ statement. Discovered it today and the documentation says that most python programs won’t need it. So I’m not sure if my architecture is correct but it works pretty darn well right now. So I’m rolling with it right now.