Switching between virtualenvs with tab completion

EDIT 17 April 2015: I originally wrote this post a month or two before I found out about virtualenvwrapper. The original content follows, but I strongly recommend using virtualenvwrapper to manage your virtualenvs :-)

Often I find myself working on (or maintaining) multiple python projects that require different virtual environments. The need to switch between them arises frequently enough that I thought it would be nice to create a shortcut for doing so. That’s how I wound up with this in my .bashrc:

ENVSDIR="/home/me/PROJECTS/envs"
function switchenv {
    deactivate 2> /dev/null
    source "${ENVSDIR}/${1}/bin/activate"
}
_switchenv() {
    local cur=${COMP_WORDS[COMP_CWORD]}
    COMPREPLY=( $(compgen -W "`ls -1 ${ENVSDIR}`" -- $cur) )
}
complete -F _switchenv switchenv

Basically, all of my environments are stored in the $ENVSDIR. They have names like $ENVSDIR/Django-1.6, $ENVSDIR/Python3, etc. Any time I want to switch to one, I just type switchenv [environment]. The usual tab completion rules apply; to list all environments, I type switchenv [TAB] [TAB].

Just a simple hack, but I’ve found it to be quite convenient. Enjoy!

Deploying multiple Django apps in a single Apache VirtualHost

Recently I needed to deploy a few small Django apps on an Apache server within a single VirtualHost. This isn’t terribly different from deploying each app within its own VirtualHost but the minor differences are what matter the most. Each app needs to run in it’s own “namespace” so to speak. Also, I prefer to use WSGI’s “daemon mode” because this runs each app within its own process which helps guard against individual WSGI apps interfering with each other. Here is how I accomplish this:

First, I create a VirtualHost just like I would normally:

<VirtualHost *:443>
    ServerAdmin admin@domain.tld
    ServerName apps.someserver.local

    Include /opt/apps/app1/apache/app1.conf
    Include /opt/apps/app2/apache/app2.conf
    
    [... other server directives ...]

</VirtualHost>

Note the two Include directives. You don’t have to do it this way, but I prefer to keep the apache config within the app’s directory hierarchy. Next, I create an apache config for each individual app; here’s the configuration for app1 (the config for app2 will be mostly the same as app1):

Alias /app1/static/ /opt/apps/app1/static/
Alias /app1/robots.txt /opt/apps/app1/static/robots.txt
Alias /app1/favicon.ico /opt/apps/app1/static/images/icons/favicon.ico

WSGIDaemonProcess app1 \
    python-path=/opt/apps/app1:/opt/virtualenvs/Django-1.6/lib/python2.7/site-packages/ \
    user=app1user group=app1group processes=1 threads=15 maximum-requests=10000
WSGIScriptAlias /app1 /opt/apps/app1/wsgi.py process-group=app1
WSGIScriptReloading On

<Directory /opt/apps/app1/static>
        Order deny,allow
        Deny from all
        Allow from 192.168.
        Allow from 127.0.0.1
        Options -Indexes FollowSymLinks
</Directory>

<Directory /opt/apps/app1>
    <Files wsgi.py>
        Order deny,allow
        Deny from all
        Allow from 192.168.
        Allow from 127.0.0.1
    </Files>
</Directory>

There’s a whole lot going on up there but the basic thing you’ll want to remember is to substitute “app1” for the name of the django app you’re configuring. I’ve also included some File/Directory directives but yours will undoubtedly look different from mine. The most important lines are those containing the Alias and WSGI directives. The configuration above uses a virtualenv and it also serves static files from the same server that the app resides on. If you aren’t already using virtualenvs, then now is the time to start. For more information on virtualenvs, see the following links:

http://www.virtualenv.org/en/latest/

http://www.devinterface.com/blog/en/2010/08/how-to-create-multiple-django-environments-using-virtualenv/

For more information about serving static files for a django app, see the following links:

https://docs.djangoproject.com/en/dev/howto/static-files/deployment/#serving-the-site-and-your-static-files-from-the-same-server

https://docs.djangoproject.com/en/dev/howto/deployment/wsgi/modwsgi/#serving-files

And finally, for more information on WSGI; specifically the directives used in the above example, see the following links:

https://code.google.com/p/modwsgi/wiki/ConfigurationDirectives

http://blog.dscpl.com.au/2012/10/requests-running-in-wrong-django.html

Simple configuration file parser (python)

Python’s ConfigParser works well for INI files but what about config files that are simple key/value options? For example, ConfigParser will not work with files like this:


#Example configuration file
#Foo Bar-er v1.0
foo_dir="/var/lib/foo"
bar_dir="/var/lib/bar"
foo_all_the_bars="1"
bar_all_the_foos = "yes, do it"

Granted, the example above is a poorly written config file however I’m using it as an example to demonstrate the flexibility of configuration parser. So here’s the code:


# SimpleConfigParser
# Inspired by:
# http://www.decalage.info/fr/python/configparser
class SimpleConfigParser():

    def __init__(self, comment_char = '#', option_char = '=', allow_duplicates = False, strip_quotes = True):
        self.comment_char = comment_char
        self.option_char = option_char
        self.allow_duplicates = allow_duplicates
        self.strip_quotes = True

    def parse_config(self, filename):
        self.options = {}
        config_file = open(filename)
        for line in config_file:
            if self.comment_char in line:
                line, comment = line.split(self.comment_char, 1)
            if self.option_char in line:
                option, value = line.split(self.option_char, 1)
                option = option.strip()
                value = value.strip()
                value = value.strip('"\'')
                if self.allow_duplicates:
                    if option in self.options:
                        if not type(self.options[option]) == list:
                            old_value = self.options[option]
                            self.options[option] = [value] + [old_value]
                        else:
                            self.options[option] += [value]
                    else:
                        self.options[option] = value
                else:
                    self.options[option] = value
        config_file.close()
        return self.options

And here’s an example of how you’d use this (assuming you saved the code in a file named ‘simpleconfig.py’ and you saved the example config as ‘example.cfg’):


Python 2.7.3 (default, Aug 1 2012, 05:14:39)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from simpleconfig import SimpleConfigParser
>>> scp = SimpleConfigParser()
>>> scp.parse_config('example.cfg')
{'bar_dir': '/var/lib/bar', 'foo_all_the_bars': '1', 'foo_dir': '/var/lib/foo', 'bar_all_the_foos': 'yes, do it'}
>>>

Also, if you have a config file that has duplicate “options” (nagios.cfg comes to mind), you can do the following:

Python 2.7.3 (default, Aug  1 2012, 05:14:39) 
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from simpleconfig import SimpleConfigParser
>>> scp = SimpleConfigParser(allow_duplicates = True)
>>> nagios_config = scp.parse_config('/etc/nagios3/nagios.cfg')
>>> nagios_config['cfg_dir']
['/etc/nagios3/conf.d', '/etc/nagios-plugins/config']
>>> 

Instead of a key/value pair, you’ll get a key with a list as it’s value for each identically named configuration option.

jqueryFileTree connector script for python cgi

If you use jquery and haven’t messed with the jqueryFileTree plugin, I’d highly recommend you do so. It’s pretty neat, and there are connector scripts already written in various languages for use with this plugin. However, there wasn’t one for python cgi (there was one for django however). So, I wrote my own! Try it out:

#!/usr/bin/python
# jqueryFileTree connector script for python cgi 
# Version: 1.0 / 10 August 2010
# Author: Charles Hamilton / musashi@nefaria.com
# Released under the GNU GPLv3
# Modifications and improvements are welcome

import os, cgi, cgitb, urllib, re
cgitb.enable()
form = cgi.FieldStorage()

print 'Content-Type: text/html\n\n'
print '<ul class="jqueryFileTree" style="display: none;">'

path = urllib.unquote(form['dir'].value)

dirs = []
files = []

filelist = sorted(os.listdir(path))

for object in filelist:
    if os.path.isfile(path + '/' + object):
       ext = os.path.splitext(object)
        files.append('<li class="file ext_' + re.sub('\.', '', ext[1]) + '"><a href="#" rel="'+ path + object + '">'+ object + '</a></li>')
    elif os.path.isdir(path + '/' + object):
        dirs.append('<li class="directory collapsed"><a href="#" rel="'+ path + object +'/">' + object + '</a></li>')

for d in dirs:
    print d

for f in files:
    print f
print '</ul>'

Python + Reportlab: example #1

So I’ve been using reportlab lately and I have to say, it’s pretty neat. This post will (hopefully) be the first of many to follow. It’s just a simple example that shows how to take input from a web form and insert it into a PDF. First, the code:

 1 #!/usr/bin/python
 2
 3 import cgi, sys
 4
 5 form = cgi.FieldStorage()
 6
 7 if not "namein form:
 8     print """Content-Type: text/html\n\n
 9 <html>
10     <head>
11         <title>Reportlab Example</title>
12     </head>
13     <body>
14         <form action="index2.cgi" method="post" enctype="multipart/form-data">
15             <fieldset>
16                 <legend>Personal Info:</legend>
17                 Name: <input type="text" name="name"><br />
18                 Photo: <input type="file" name="photo"><br />
19                 <input type="submit">
20             </fieldset>
21         </form>
22     </body>
23 </head>"""
24
25 else:
26
27     from reportlab.platypus import *
28     from reportlab.lib.styles import getSampleStyleSheet
29     from reportlab.lib.units import inch
30     from reportlab.lib import colors
31     doc = SimpleDocTemplate(sys.stdout)
32     styles = getSampleStyleSheet()
33     content = []
34     
35     if form[‘photo‘].filename:
36         image = Image(form[‘photo‘].file)
37         image.drawHeight = 2*inch*image.drawHeight / image.drawWidth
38         image.drawWidth = 2*inch
39
40     if form[‘name‘].value:
41         text1 = Paragraph(form[‘name‘].value, styles[‘Heading1‘])
42         text2 = Paragraph(form[‘name‘].value, styles[‘Heading2‘])
43         text3 = Paragraph(form[‘name‘].value, styles[‘Heading3‘])
44
45     content.append(text1)
46     content.append(text2)
47     content.append(text3)
48     content.append(image)
49     print "Content-Type: application/pdf"
50     print "Content-Disposition: attachment; filename=example.pdf\n\n"
51     doc.build(content)

Now the explanation:

Lines #1 – #5 handle specifying the interpreter, importing modules, and initializing the “FieldStorage” dictionary (as the variable ‘form’).

 1 #!/usr/bin/python
 2
 3 import cgi, sys
 4
 5 form = cgi.FieldStorage()

Line #7 tests whether the form has been submitted by checking to see if the ‘name’ field has been filled out. There’s many better ways to test for form submission, but for the purposes of our example, this will work just fine.

 7 if not "namein form:

Lines #8 – #23 print the form

 8     print """Content-Type: text/html\n\n
 9 <html>
10     <head>
11         <title>Reportlab Example</title>
12     </head>
13     <body>
14         <form action="index2.cgi" method="post" enctype="multipart/form-data">
15             <fieldset>
16                 <legend>Personal Info:</legend>
17                 Name: <input type="text" name="name"><br />
18                 Photo: <input type="file" name="photo"><br />
19                 <input type="submit">
20             </fieldset>
21         </form>
22     </body>
23 </head>"""

Lines #26 – #29 import some more modules (i.e., reportlab related stuff).

26     from reportlab.platypus import *
27     from reportlab.lib.styles import getSampleStyleSheet
28     from reportlab.lib.units import inch
29     from reportlab.lib import colors

Now line #30 is important, this is where we decide where we want to write the output. We can either save the output to a file, or we can dump it to stdout (i.e., back to the web browser). In this example, we’re going to send the output back to the web browser.

30     doc = SimpleDocTemplate(sys.stdout)

Line #31 handles getting the style sheet that we’re going to use to format our text.

31     styles = getSampleStyleSheet()

In line #32, we initialize the ‘content’ dictionary — this is where we’re going to keep the elements of our PDF until we’re ready to write it to stdout.

32     content = []

In lines #34 – #37, we test for the ‘photo’ field — if it has been submitted, we create an Image object out of it. We’re assuming that the user is going to submit a photo, but in reality, the user could submit anything so some further “hardening” of this form would be required in order to ensure that the only things that actually get submitted are image files.

34     if form[‘photo‘].filename:
35         image = Image(form[‘photo‘].file)
36         image.drawHeight = 2*inch*image.drawHeight / image.drawWidth
37         image.drawWidth = 2*inch

Lines #39 – #42 test to see if the ‘name’ field has been submitted; if it has, it creates some text objects to insert into our PDF.

39     if form[‘name‘].value:
40         text1 = Paragraph(form[‘name‘].value, styles[‘Heading1‘])
41         text2 = Paragraph(form[‘name‘].value, styles[‘Heading2‘])
42         text3 = Paragraph(form[‘name‘].value, styles[‘Heading3‘])

Lines #44 – #47 append all the objects that we want to appear in our PDF, to the ‘content’ dictionary we created earlier.

44     content.append(text1)
45     content.append(text2)
46     content.append(text3)
47     content.append(image)

Lines #48 – #49 send the appropriate headers to the web browser, before we send our completed PDF file.

48     print "Content-Type: application/pdf"
49     print "Content-Disposition: attachment; filename=example.pdf\n\n"

And finally, line #50 builds our PDF and sends the output to stdout.

50     doc.build(content)

Test it out; you should end up with a PDF that looks something like this:

reportlab_example1_img1

Of course, this example doesn’t even scratch the surface of what you can do with reportlab. Hopefully I’ll post some more examples later on :-)