Si Sapis, Sis Apis

I could not find a clean solution to this problem anywhere, so I am posting mine here.

Problem: because Python's xmlrpclib module uses blocking sockets, if XML-RPC server you are talking to gets stuck and never responds, your program gets stuck, too.

One way to fix that was to use a global shim for socket module, timeoutsocket. This works for very simple cases, but in real-life situations you do not want to switch all your sockets globally into the same mode.

A correct solution is to subclass xmlrpclib.Transport, replace underlying connection with your subclass of httplib.HTTP, which in turn should use a customized subclass of httplib.HTTPConnection, which finally sets timeout on socket.

In the client code, almost nothing changes. Instead of "import xmlrpclib", I say "import timeout_xmlrpclib as xmlrpclib", and the rest stays as before.

Inside timeout_xmlrpclib module, define function Server that will return server proxy:

def Server(url, *args, **kwargs):
   t = TimeoutTransport()
   t.timeout = kwargs.get('timeout', 20)
   if 'timeout' in kwargs:
       del kwargs['timeout']
   kwargs['transport'] = t
   server = xmlrpclib.Server(url, *args, **kwargs)
   return server

Note that timeout can be specified as a keyword argument. If it is not specified, it is set to 20 seconds. As socket documentation says, specifying None will make socket blocking and specifying 0 will make it non-blocking.

TimeoutTransport overrides make_connection:

class TimeoutTransport(xmlrpclib.Transport):

   def make_connection(self, host):
       conn = TimeoutHTTP(host)
       conn.set_timeout(self.timeout)
       return conn

And HTTP and Connection subclasses:

class TimeoutHTTPConnection(httplib.HTTPConnection):

   def connect(self):
       httplib.HTTPConnection.connect(self)
       self.sock.settimeout(self.timeout)


class TimeoutHTTP(httplib.HTTP):
   _connection_class = TimeoutHTTPConnection

   def set_timeout(self, timeout):
       self._conn.timeout = timeout

Neat. Now my XML-RPC client does not get stuck talking to the server that got into some sort of deadlock.

Labels: customization, python, xmlrpc

So you've written your newest fantastic TurboGears application. And you want it to be easy to install for Windows users. You do not want them to have to know about Python, eazy_install, eggs and all that. They should be able to double-click Setup.exe, and boom! -- their shiny new system is ready.

Python has had a solution for this kind of problem for a while: py2exe (or py2app for Mac). For simple Python applications, it works like magic. But there are problems. Py2exe does not grok Python eggs. Some setup_utils features do not work when you're running from py2exe-created executable. Things will not work out of the box. Even though "batteries included", they are not the right voltage, so to say.

About a year ago, when working on installer for my first TG-based application (targeted primarily for Windows), with the help of kind folks on the TurboGears mailing list, I managed to put together a quick-and-dirty workaround: I packaged the whole set of dependencies, including Python interpreter and DLLs, into the installation directory, and it could run as if you just installed a new copy of Python. Fred Lin took it a step further and created tg2exe (the name is a bit misleading, since the tool does not generate a single executable, it simply gathers all dependencies into a single directory). So one solution is to use tg2exe (or similar custom script) and then package this pile of Python files with InnoSetup or something like that.

But what if you really need a single executable, say, to get a nice and simple software protection with a 3rd-party dongle/encryption kit? The only solution I've found so far is to make some modifications to the Python libraries.

Make a copy of the whole Python directory (e.g. C:\Python25) in your project folder and start customizing it without fear.

First, you'll need to unzip eggs and put their contents into site_packages, so that they do not depend on *.pth files. The eggs are just regular Python packages, plus some extra metadata. The metadata will be lost, and we'll come back to it in a few seconds, because some of it is important. The eggs that were not packed should be simply moved up, so that instead of site-packages/foo-XXX.egg/foo you have site-packages/foo.

Next, you'll need to fix TurboGears and related libraries in every place where pkg_resources.resource_filename is used -- it won't work inside an executable, so you'll need to use os.path.join instead. For instance, in turbogears/config.py,

modfile = pkg_resources.resource_filename(packagename,
                                          modname + '.cfg')

becomes

modfile = os.path.join(packagename.replace('.', '\\'),
                       modname + '.cfg')

Another thing that has to go is pkg_resources.require. Because you have full control over your distribution, do not require, just import.

Finally, we get to the eggs metadata. TurboGears uses entry points in egg_info to figure out what template engines are supported. Because you do not have the entry point information without eggs, you'll need to hard-code the engines needed in your project. For example, to enable kid and json, I added the following to the function load_engines() in turbogears/view/base.py:

if 'kid' not in engines:
   from turbokid.kidsupport import KidSupport
   engines['kid'] = KidSupport(stdvars, engine_options)
if 'json' not in engines:
   from turbojson.jsonsupport import JsonSupport
   engines['json'] = JsonSupport(stdvars, engine_options)

And this is basically it! You've just changed TurboGears into a state usable by py2exe. The documentation for py2exe will tell you how to do the rest.

Labels: installation, python, turbogears, windows