Using pgBouncer to speed up requests… dramatically.. well kinda…

In this particular project, because of client constraints we were confined to using Windows. We ended using the following stack:

• IIS 7
• HeliconTech Zoo (for Python)
• Django
• PostgreSQL

When I used Apache JMeter to stress test the app. Wasn’t too impressed with the results. When I threw 500 concurrent requests at the server, the server was averaging 15s per request. Not to mention max response time that were nearing a minute. This request involved probably about 2 or 3 db calls if so many. Wasn’t sure why it should take that long. The server by the way was a:

• Windows 2k8 Server
• Intel Xeon E5 2.2 GHz
• 64 GB RAM
• Too much HD space

Anyway, so I decided to try out pgBouncer, which is supposed to be a connection pooler. Once I set it up I realized a DRAMATIC decrease in the ramp up time in JMeter. Usually, in JMeter you would have to wait for each request to build up, because each was requesting it’s own connection, but now that there’s connection pooling in JMeter it skipped straight to waiting for the DB to respond with the results and the good part is, it was really easy to setup.

Start by opening up the Stack Builder and installing pgBouncer (it’s under Add-ons Tools & Utilities). After it’s installed:

1. Open up C:\Program Files (x86)\PgBouncer\etc\userlist.txt and add the users that will be connecting to your database in a line by itself like so: “username” “password”. Use the quotes.
2. Open up C:\Program Files (x86)\PgBouncer\share\pgbouncer.ini and the following lines under the [databases] section.
   * = host=127.0.0.1
   listen_addr = 127.0.0.1
   listen_port = 6432
3. Tweak the max_client_conn, default_pool_size and reserve_pool_size to your liking. I used 1500, 1000 and 500 respectively.
4. Restart the “pgbouncer” service.
5. Change the port in settings.py to 6432 (or whatever port you specified in the pgBouncer install)
6. That’s it!

WAMP and the Apache alias problem

So WAMP gave a bit of trouble with aliasing /static/ to C:\…\static_root properly. This link helped out alot. The solution I used was to just add an alias using the WAMP directives. WAMP > Apache > Alias directories > Add an alias.

Geonode Chronicles: Adding a flag to show the country of origin for the data

This is going to be another stupid simple tutorial, given you’ve read the first in this Geonode Chronicles series (https://perspectiveunspoken.wordpress.com/2012/09/09/using-fabric-to-deploy-custom-geonode-installation/) then this one is going to basically going to be a follow up. For this tutorial, I’ll be adding another Django app and adding a template tag to go along with it to show a flag beside the country from which the data originated from.  This is again based on v1.2.

I’m assuming that you have a custom project built on Ariel’s template project and you’re ready to go. First, we’ll start by creating a templatetags directory inside the uwinode/uwinode directory. Also we’ll enter the Geonode environment as well.

# Go into the Geonode folder
cd ~/web/geonode
source bin/activate # activate the Geonode env

cd ~/web/uwinode/uwinode # go right outside the uwinode directory (project dir)

mkdir templatetags

cp __init__.py templatetags # copy the __init__.py so python will recognize this folder as package

# go into the templatetags folder

cd templatetags

Next, use your favourite editor and create a file called “layer_extras.py” in the templatetags folder. We’ll be adding some code to that. Now use the following starter code to populate your new layer_extras.py file.

def get_layer_bbox(layer):
 """ Returns the bounding box for a layer in the format: (lon1 lon2 lat1 lat2)
 """
 resource = layer.resource
 bbox = {}

 for i in range(0,4):
 bbox[i] = float(resource.latlon_bbox[i])

 dx = float(bbox[1]) - float(bbox[0])
 dy = float(bbox[3]) - float(bbox[2])

return bbox

def get_layer_center(layer):
 """ Returns the center
 """
 bbox = get_layer_bbox(layer)
 if bbox:
 lat = (bbox[2] + bbox[3]) / 2.0
 lng = (bbox[0] + bbox[1]) / 2.0
 return {
 "lat" : lat,
 "lng" : lng,
 "x" : lng,
 "y" : lat
 }
 return None

So what I’ve done there is start us off with two functions. The get_layer_bbox just gets the bbox in a way we can pass it to get_layer_center to get the lat, lng coordinates of what we think to be the center of the layer. Simple enough… let’s move on. We need a Python package called “Countries”, add countries to settings.py and we need to download the flags. Let’s start with Django-countries.

pip install countries

# Open up settings and add countries to INSTALLED_APPS

# Also add COUNTRIES_FLAG_PATH = STATIC_URL + 'flags/%s.png' near the bottom

./manage.py validate # make sure all is well

./manage.py syncdb

./manage.py migrate

Next download the zip file from http://www.famfamfam.com/lab/icons/flags/famfamfam_flag_icons.zip, and copy the png folder into the static directory in uwinode. I usually unzip stuff in some temp directory.

mkdir ~/tmp

cd ~/tmp

wget http://www.famfamfam.com/lab/icons/flags/famfamfam_flag_icons.zip

unzip famfamfam_flag_icons.zip

mkdir -p ~/web/uwinode/uwinode/static # ensure the destination directory exists

mv png flags # rename the png dir to flags

mv flags ~/web/uwinode/uwinode/static/ # copy over the flags dir

Once that is done you should be able to try http://localhost:8000/static/flags/jm.png and the Jamaican flag should show. Now since we have that part set up nicely, basically what we have left to do are two things.

• Integrate Google geocode so we’ll know what country we’re in based on the coordinate
• Make a template tag that will automatically show the flag given a layer.

Let’s get to doing that shall we… we need to add some more code to layer_extras.py. Consider the code below:

</pre>
from googlemaps import GoogleMaps
from django import template
from countries.models import *
from django.template.defaultfilters import stringfilter
from django.conf import settings
from django.utils.safestring import mark_safe
register = template.Library()

@register.filter()
def layer_flag(layer):
 if layer:
 center = get_layer_center(layer)
 gmaps = GoogleMaps(settings.GOOGLE_API_KEY)
 lat = center.get('lat',None)
 lng = center.get('lng', None)
 if lat and lng:
 addr = gmaps.latlng_to_address(lat, lng)
 pieces = addr.split(",")
 country = pieces[-1].strip()
 flag_location = country_flag(country)
 if flag_location:
 flag_location = flag_location.lower()
 return mark_safe("<img src='%s'/>" % flag_location)
 return ""
def country_flag(country):
 """
 Returns a full path to the ISO 3166-1 alpha-2 country code flag image based on the country name
 """
 if not country:
 return u''
 result = Country.objects.filter(name__icontains=country)
 if result:
 c = result[0]
 iso = c.iso
 flag_location = settings.COUNTRIES_FLAG_PATH % iso
 return flag_location
 return u''

Let’s quickly assess what’s going on here. First of all, let’s start with the layer_flag function. That function expects to be passed a layer. Now the first thing that we do is get the center of the layer. Next, we use initialize the GoogleMaps suite. We use the API_KEY that’s in settings. Next, we strip out the lat and lng from the dictionary. If both or those are good, then we just call the latlng_to_address function to get back an address. When this function runs it will return something of the form “Road Name, Community, Parish, Country” and all we’re interested in is the country. So in the next line, we split the addr by comma and only save the last piece. Then we pass the name of the country to the country_flag function which returns us the location of the flag. Then we return HTML from that function.

The country_flag function is also pretty straight forward, it makes sure we have a country passed in first then it checks the database to see if the country exists and we get the ISO code to the settings string we had saved and construct an image tag and return it.

Just to make sure I didn’t leave out anything, the entire file should look like this (complete with imports at the top):

from googlemaps import GoogleMaps
from django import template
from countries.models import *
from django.template.defaultfilters import stringfilter
from django.conf import settings
from django.utils.safestring import mark_safe
register = template.Library()

@register.filter()
def layer_flag(layer):
 if layer:
 center = get_layer_center(layer)
 gmaps = GoogleMaps(settings.GOOGLE_API_KEY)
 lat = center.get('lat',None)
 lng = center.get('lng', None)
 if lat and lng:
 addr = gmaps.latlng_to_address(lat, lng)
 pieces = addr.split(",")
 country = pieces[-1].strip()
 flag_location = country_flag(country)
 if flag_location:
 flag_location = flag_location.lower()
 return mark_safe("<img src='%s'/>" % flag_location)
 return ""
def country_flag(country):
 """
 Returns a full path to the ISO 3166-1 alpha-2 country code flag image based on the country name
 """
 if not country:
 return u''
 result = Country.objects.filter(name__icontains=country)
 if result:
 c = result[0]
 iso = c.iso
 flag_location = settings.COUNTRIES_FLAG_PATH % iso
 return flag_location
 return u''

Now let’s quickly open up our template layer.html (uwinode/uwinode/templates/maps/layer.html) and use our template tags. We’re going to just insert the flag below the title of the layer. Edit the file so that it looks like what we have below:

<div id="description">
 <h3>Admin Boundaries</h3>

{% load layer_extras %}

{{ layer|layer_flag }}

The load layer_extras loads in our template tag and makes all the registered filters available. In our case, the only filter we made available was the layer_flag filter. By simply inserting this line we’ve now included an image of our flag.

In closing, I’ll quickly mention that this is not an efficient way to do this at all. Google’s geocode is something that takes a while, so you’ll notice the page might take a little while to load the first time around. A better way to do this would be to use AJAX to asynchronously load in the image. I’ll get to that in another tutorial though! Happy Geonoding all!

Ajax Uploads with Django

I was looking for a nice tutorial to get this done and I found this one here done by Alex Kuhl. It uses Valum’s AJAX Upload and has the server side Django code to process the file upload. Nicely done. Just made a quick addition though. Alex had suggested changing the fileuploader.js file to get the params included for the CSRF token. That’s not necessary anymore. When declaring the FileUploader just declare it like so…

var uploader = new qq.FileUploader( {
 action: url,
 element: $('#file-uploader')[0],
 onComplete: function( id, fileName, responseJSON ) {
 /* you probably want to handle the case when responseJSON.success is false,
 which happens when the Django view could not save the file */
 if( responseJSON.success )
 successHandler( responseJSON ) ;
 },
 customHeaders : {
 "X-CSRFToken" : "{{ csrf_token }}"
 }
 debug: true
 });

Emphasis is on the customHeaders part of the options dictionary. You can set the header there. I’m assuming you’ve got a csrf_token varible supplied by the view. This can be done easily as well…

def test(request):
ctx = {}
ctx.update(csrf(request))
return render_to_response("test.html", ctx)

Easy stuff… oh and remember to include the CSS for the fileuploader, else you’ll end up like me… thinking that the upload was failing even though the script returned {success:true}. The default styles will hide the element that says it failed.

Hope this helps someone!

Geonode Chronicles: Building a custom Geonode (Adding Disqus)

This article is going to  be the first of a series of articles on my experience with Geonode. This article’s gonna cover part of my experience of extending Geonode. I’m going to cover adding Disqus to replace the default comment system. I’m using this examples because it covers changing templates and adding stuff via Pip. This is a Geonode installation on Ubuntu 12.04, based on the Geonode 1.2 release with the Bootstrap forms (I think it’s from or was… Eldarion’s repos). I forked the Geonode repos some months ago so I have my own copy of that project. I’m not sure what has changed in the main project. I’ll cover packaging this project in a later article. DISCLAIMER I am but a mere n00b… I’m still very new to Python and to Geonode and Django and so on. Forgive me if anything seems naive, outright dumb or if I miss a step or so. 🙂 My company was asked to do an installation of Geonode for the Caribbean. You can have a look at it, it’s up and running at http://cariska.mona.uwi.edu(codenamed “uwinode”). Many, many thanks to Ariel Nunez and the team over at the geonode-dev google group for all their help and guidance. Great guys! Here’s my outline of what I’m gonna try and talk about:

• Preliminaries
• Checking out Geonode (my forked repo)
• Creating a custom Geonode project
• Making changes to your custom project
• Packaging your custom project
• Deploying the custom project

Preliminaries

Just some house keeping matters before we get started and also to let everyone in on any conventions I use so that things may make sense where I failed to be clearer.

• Have a working directory in which you’ll store Geonode and your custom project. I generally have a ~/web folder in which I would have the Geonode install (mine was ~/web/geonode) and my custom project (this was ~/web/uiwnode).
• “Uwinode” was the codename of the project. It’s our custom app built using Geonode.
• Python virtual environments are AWESOME! You should use them. It allows you to have several projects on your system with completely different sets of modules and so on. It’s great and easy. You just have to say “virtualenv directory_name” and python will create a Virtual Environment in that directory. It’ll create the dir if it doesn’t exists. It creates a bin folder that has some binaries in it. “activate” is one of the critical ones. To enter the virtual environment  (“source”) you can just type “source bin/activate” to activate the virtual env. Once in the virtual env, anything you pip install will only be in that virtual env. Then just type “deactivate” to exit the virtual env.

Checking out GeoNode

So to start off this process, you need Geonode in development mode. You can follow the steps on the Geonode Docs to get this done. Nothing to see here really, except…

• I check out from my repos (jaywhy13) instead of the GeoNode repos.
• I installed openjdk-7-jdk instead of openjdk-6.
• I changed this in my repo, but I’ll mention it. If you’re using Ubuntu 12.04 you have to change pavement.py at about line 236 where it says sh(“mvn clean compile”). Change that to sh(“ant zip”). Some java issue on Ubuntu 12.04.

sudo apt-get install -y --force-yes openjdk-7-jdk
sudo apt-get install -y vim zip unzip subversion git-core binutils build-essential python-dev python-setuptools python-imaging gdal-bin libproj-dev libgeos-dev python-urlgrabber python-nose pep8 python-virtualenv python-gdal python-pastescript postgresql-contrib libpq-dev gettext python-psycopg2 libxml2-dev libxslt1-dev
sudo apt-get install -y --force-yes ant maven2 --no-install-recommends
sudo apt-get install git -y # Install GIT for version control
mkdir ~/web # make a directory called "web" in your home directory
cd ~/web # go into the directory

## RUN the Geonode V1.0 install procedure (this is different from V2.0)
git clone git://github.com/jaywhy13/geonode.git geonode
cd geonode
git submodule update --init
python bootstrap.py --no-site-packages
source bin/activate # activate the geonode virtual env
# We made a change line 236 to pavement.py if you're using Ubuntu 12.04 (see bullet #2 above)
paver build

Paver is going to take a while to build, so while that’s going on, we can move on…

Creating a custom Geonode project (uwinode)

Ariel has a nice template geonode project that you can clone from Github. The template project is used to help get us started with a custom geonode project. So we can pull that project and create a Django project based on his project. Let’s go ahead and do that. So I would have run these commands in my ~/web directory.

# in the ~/web dir, make sure the Geonode env is still active
git clone git://github.com/ingenieroariel/geonode-project.git geonode-template
django-admin.py startproject --template=geonode-template -epy,rst uwinode
pip install -e uwinode # makes a link to your project in the Geonode virtual env

Just a note here, so pip install -e creates an “editable” link to your project. In essence, since we’re going to be working on our project, that -e option adds our project to the Python path (in the virtual env). Awesome stuff! Ok, hopefully paver is done building by now. You can go back and create an admin user if you want and then we can move on.

Making changes to your custom project and to Geonode

First go into the folder (“uwinode” in our case) and edit uwinode/settings.py.  We need to make a couple of changes. Ariel seems to have updated his template project, so it’s important to note that our templates are added above Geonode’s.. the same goes for static file dirs. Also, the root url conf is set to projectname.urls by default.

• Add imports for our projects (mynode)
• For our project we added Disqus to replace the default comments system they have in Geonode. We needed some settings for this.

Here are the parts we added… just find the different parts in the file and modify or add where necessary. Ensure that     ‘geonode.maps.context_processors.resource_urls’ and “geonode.maps.context_processors.resource_urls”   is in the list of TEMPLATE_PROCESSORS.

# imports near the top...
import uwinode

# add your app to list of those installed
INSTALLED_APPS = (
# whatever was before

# Ariel didn't put this one in the template project...
'taggit',

# mynode additions
'django_forms_bootstrap', # bootstrap forms.. YAY!!!
'uwinode',
'disqus', # the Disqus plugin
)

# these need to go in as well...</pre>
DB_DATASTORE = False
# Database datastore connection settings
DB_DATASTORE_NAME = ''
DB_DATASTORE_USER = ''
DB_DATASTORE_PASSWORD = ''
DB_DATASTORE_HOST = ''
DB_DATASTORE_PORT = ''
DB_DATASTORE_TYPE = ''
AUTH_PROFILE_MODULE = 'maps.Contact'

SITEURL = "http://localhost:8000/"
# make sure this value is as below... search the file for it...
GEOSERVER_BASE_URL = "http://localhost:8001/geoserver/"

###### These changes are already made... but make note though
STATICFILES_DIRS = [
os.path.join(PROJECT_ROOT, "static"),
os.path.join(GEONODE_ROOT, "static"),

]

TEMPLATE_DIRS = (
os.path.join(PROJECT_ROOT, "templates"),
os.path.join(GEONODE_ROOT, "templates"),
)

ROOT_URLCONF = 'uwinode.urls' # use our urls instead

# Also add this to the TEMPLATE_CONTEXT_PROCESSORS</pre>
TEMPLATE_CONTEXT_PROCESSORS = [

...

"django.core.context_processors.request", # <==== Insert this below media

...

"geonode.maps.context_processors.resource_urls" # Insert this at the end

]
# We also need to add back in the middleware settings...

MIDDLEWARE_CLASSES = (
 'django.middleware.common.CommonMiddleware',
 'django.contrib.sessions.middleware.SessionMiddleware',
 'django.contrib.messages.middleware.MessageMiddleware',
 # The setting below makes it possible to serve different languages per
 # user depending on things like headers in HTTP requests.
 'django.middleware.locale.LocaleMiddleware',
 'django.middleware.csrf.CsrfViewMiddleware',
 'django.contrib.auth.middleware.AuthenticationMiddleware',
)

 

We also need to add some requirements to Geonode's requirements.txt file in the geonode/shared directory. We could just pip install these but it's necessary if we want Geonode to install our application (I'll cover this is another tutorial). Moving on though... When we run bootstrap, it installs all the requirements in shared/requirements.txt. Now we just need to uncomment some stuff I already added for us there. Edit the file so that everything below the notice about "Uncomment below here" is uncommented, it should resemble below.
-e git+https://github.com/jaywhy13/uwinode.git#egg=mynode
django-disqus
googlemaps
django-forms-bootstrap

So once you've edited the shared/requirements.txt file you can go ahead and install the requirements using pip. Note that this just mimics what the Geonode installer would do when installing our custom app (not covered in this article), so alternatively, you could have just pip installed each package individually.
cd geonode
source bin/activate
pip install -r shared/requirements.txt

Run ./manage.py validate from inside the uwinode directory. We should get 0 errors found. Let's now create the db for our project.
./manage.py validate # this should say "0 error(s) found"
./manage.py syncdb # you can create the admin user if you want
./manage.py migrate
./manage.py createsuperuser

Next we bring over the shared folder from the geonode dir and make some  quick changes to it. They use Paster (multi-thread, better alternative to manage.py runserver) to run start up geonode and proxy for geoserver. Open up shared/dev-paste.ini after you've copied it over and change geonode.settings to uwinode.settings. It should be on line 31. Next, we need to replace the urls.py in our uwinode project because it's outdated. We can use the one from GeoNode. Note we also need to start up Geonode, we might need two terminal windows.
# from the ~/web dir
cp -r geonode/shared uwinode/

# open up uwinode/shared/dev-paste.ini and change geonode.settings to uwinode.settings

cp geonode/src/GeoNodePy/geonode/urls.py uwinode/uwinode

# then run paster and check what's happening at localhost:8000
cd uwinode

paster serve shared/dev-paste.ini

# also start up geoserver
cd geonode/src/geoserver-geonode-ext/
./startup.sh # run the startup geoserver script

Once that works, login at localhost:8000 and upload a layer, we'll need it. Ok, now.. after ALLL that.. 🙂 Now it's time to actually integrate Disqus. Let's re-open uwinode/settings.py and add two variables at the end.

Setup Django Key
Head over to Disqus.com and create an account, once logged in you can hit http://disqus.com/api/get_my_key/ to get the key you need to put in settings.py file.

</pre>
</div>
<pre id="LC371">DISQUS_API_KEY = '<GET THIS FROM DISQUS>' # once logged in, your key is here: <a href="http://disqus.com/api/get_my_key/">http://disqus.com/api/get_my_key/</a></pre>
<pre id="LC372">DISQUS_WEBSITE_SHORTNAME = '<SOME SITE NAME>'

Now let's start copying over and editing some templates.
# Go into the main web directory
cd ~/web

mkdir -p uwinode/uwinode/templates/maps
# copy over the layer.html template that shows a particular layer...
cp geonode/src/GeoNodePy/geonode/templates/maps/layer.html uwinode/uwinode/templates/maps/

Swap out Geonode's comment system

Now open layer.html (uwinode/uwinode/templates/maps/layer.html) up and search for comments_container, delete the contents of the div and instead insert:
{% load disqus_tags %}
{% disqus_dev %}
{% disqus_show_comments %}

 
The load disqus tags loads in Disqus tags, the Disqus Dev puts us in developer mode and show comments... that does the magic! Had this been production, we would have needed to do the following:

{% load disqus_tags %}

{% set_disqus_identifier "layer_" layer.pk %}

{% set_disqus_url request.build_absolute_uri %}

{% disqus_show_comments %}


And that pretty much ties up this tutorial, at this point, you should be able to see the following clear things:

How we go about creating a custom new project that will use Geonode
How we add Django based apps to a custom project
An idea of the template that shows the layer data template.

							

			

				

				

					
													Comment