Yeah, way more than you can think. In this case I’m talking about the e-cidadania permission system. We all know that django provides some basic permission system at model level, but in the case of e-cidadania it wasn’t enough. We needed very detailed object level permissions.

First of all let me tell you that the e-cidadania project is run by only one person: me. At the time of designing the base of the project the object permission systems available were really outdated or didn’t even work, so I had to make a workaround that at the moment seemed a good idea, even after trying some example cases, it was based on three fields in what I call the “father” model, a model that doesn’t control everything but is the base of all the rest of the modules (if you want to take a look, it’s the Space module).

As I was developing everything went fine, then the class-based views came out and I couldn’t make the proper validations, so I had to make another workaround in the get_object method. It turned out to be “consistent” but I didn’t had enough time to test it, and I continued developing the platform, in some cases with the help of some people that joined temporarily the project (GSoC 2012, pre-GSoC 2013) and everything seemed fine, until a week ago.

Let me tell you, e-cidadania has almost 60Ksloc (stats here) and currently has 12 modules, there are still a couple of modules to develop. I was so centered in development that totally forgot to test all the user roles in the platform (it would take me 4 times the time it was taking) and I convinced my workmates  (which are not related to IT, that’s a different story) to make some basic usage testing. What we discovered was that most of the permissions failed, and the validations didn’t work as expected, allowing in one of the use cases an anonymous user to enter in private spaces and even interact in them!

And I had t redesign the permission system. From scratch. Luckily django-guardian is up-to-date now and we’re using it, but I didn’t know all the work we had to do until I finished planning the new design. Not only we had to redo all the views of the platform, we also had to modify all the permissions in the models, all the checks in the templates, everything from ground up! And that will take quite a while, and we planned to release e-cidadania 0.1.9 on April, and there’s a lot of work to do yet, and a lot to polish, and only a pair of hands is working on it.

So be careful with sofware design, it can cost you more than you think.

Hi! It has been some time since the last post…

As you know I’m developing an open source e-democracy platform called e-cidadania. The strong point of the platform is the debate system. It’s nothing like the forums we are used to see in the internet, this time is an ordered debate system.

The thing is, I’ve never developed something that complex and I’m not used to ajax interactions, but after our partners dissapeared (they were supposed to do that module) I had to start the development by myself. The debate system is very much like a kanban or scrum board (in fact I plan to revamp it for that) and I’m almost finishing the frontend. Since I have no experience with this maybe I’ve got the wrong approach, but “it works” for now (if you want to give me some advice you’re very welcome).

The current frontend is based on a table, jQuery sortables and a bunch of javascript to give the table the needed flexibility. Here goes a screenshot!

One of the modules for e-cidadania project was a basic event calendar, meaning that the event will be marked on the respective modules (proposals, spaces, debates, etc.) and will be gathered by the calendar module. In the first moment i thought to use jquery directly, but after some research I found that it was easier to generate the calendar locally and serve it to the client, so here it is, a multilanguage, event-driven HTML calendar:

The basic explanation is that we create a class inheriting the LocaleHTMLCalendar (a localized version of the HTMLCalendar module) and override some functions like formatmonth() then in the view we stablish the current user locale, some basic protection (month number) and convert the month and year numbers to integers, since they come as strings from the URL.

models.py (I put it on models for convenience but it really should be on views.py)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45

import calendar
from datetime import date
from itertools import groupby
 
from django.utils.html import conditional_escape as esc
 
class EventCalendar(calendar.LocaleHTMLCalendar):
 
    """
    Event calendar is a basic calendar made with HTMLCalendar module.
    """
    def __init__(self, events, *args, **kwargs):
        self.events = self.group_by_day(events)
        super(EventCalendar, self).__init__(*args, **kwargs)
 
    def formatday(self, day, weekday):
        if day != 0:
            cssclass = self.cssclasses[weekday]
            if date.today() == date(self.year, self.month, day):
                cssclass += ' today'
            if day in self.events:
                cssclass += ' filled'
                body = ['<ul>']
                for event in self.events[day]:
                    body.append('<li>')
                    body.append('<a href="%s">' % event.get_absolute_url())
                    body.append(esc(event.title))
                    body.append('</a></li>')
                body.append('<ul>')
                return self.day_cell(cssclass, '%d %s' % (day, ''.join(body)))
            return self.day_cell(cssclass, day)
        return self.day_cell('noday', '&nbsp;')
 
    def formatmonth(self, year, month):
        self.year, self.month = year, month
        return super(EventCalendar, self).formatmonth(year, month)
 
    def group_by_day(self, events):
        field = lambda event: event.meeting_date.day
        return dict(
            [(day, list(items)) for day, items in groupby(events, field)]
        )
 
    def day_cell(self, cssclass, body):
        return '<td class="%s">%s</td>' % (cssclass, body)

views.py

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

from django.shortcuts import render_to_response, get_object_or_404
from django.utils.safestring import mark_safe
from django.template import RequestContext
from django.utils import translation
 
from e_cidadania.apps.spaces.models import Meeting, Space
from e_cidadania.apps.cal.models import EventCalendar
from e_cidadania import settings
 
def calendar(request, space_name, year, month):
 
    # Avoid people writing wrong numbers or any program errors.
    if int(month) not in range(1, 13):
        return render_to_response('cal/error.html',
                                  context_instance=RequestContext(request))
 
    place = get_object_or_404(Space, url=space_name)
    next_month = int(month) + 1
    prev_month = int(month) - 1
 
    meetings = Meeting.objects.order_by('meeting_date') \
                              .filter(space = place,
                                      meeting_date__year = year,
                                      meeting_date__month = month)
 
    cur_lang = translation.get_language()
    cur_locale = translation.to_locale(cur_lang)+'.UTF-8' #default encoding with django
    cal = EventCalendar(meetings, settings.FIRST_WEEK_DAY, cur_locale).formatmonth(int(year), int(month))
 
    return render_to_response('cal/calendar.html',
                              {'calendar': mark_safe(cal),
                               'nextmonth': '%02d' % next_month,
                               'prevmonth': '%02d' % prev_month,
                               'get_place': place},
                               context_instance = RequestContext(request))

If you remember, a few months ago I started a project called “Supervise”. It was a project management tool based on django, aimed to be the next “Redmine”, but for some personal reasons it had to be stopped (I’ve got a job!). Some new releases of django were made, and with them, hot new features were added.

The Supervise project was in an early stage of devlopment, so I said myself: “Why not start from scratch? It’s not that difficult”. So here it is: Supervise 2, a new version made on top of django 1.3. The list of features will be much the same as Supervise:

• Multiple project management like Redmine.
• Various SCM support: GIT, Bazaar, SVN, Mercurial. There will be more supported SCMs but for now that covers most of the developers needs.
• Issue tracking
• Wiki
• News
• User profiles

That’s for version 0.1-stable (which will come in a couple months I think)

For now you can watch the development or fork the code in both Gitorious and GitHub

This is a reminder for myself, to know how the new class-based generic views work. The code belongs to the GPL project e-cidadania. I’ll be posting as I code.

DETAILVIEW

The function-based code:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

def view_space_index(request, space_name):
 
    """
    Show the index page for the requested space. This is a conglomerate of
    various modules.
    """
    place = get_object_or_404(Space, url=space_name)
 
    extra_context = {
        'entities': Entity.objects.filter(space=place.id),
        'documents': Document.objects.filter(space=place.id),
        'proposals': Proposal.objects.filter(belongs_to=place.id).order_by('-pub_date'),
        'publication': Post.objects.filter(post_space=place.id).order_by('-post_pubdate'),
    }
 
    return object_detail(request,
                         queryset = Space.objects.all(),
                         object_id = place.id,
                         template_name = 'spaces/space_index.html',
                         template_object_name = 'get_place',
                         extra_context = extra_context,
                        )

The class-based code:

class ViewSpaceIndex(DetailView):
 
    """
    Show the index page of a space. Get various extra contexts to get the
    information for that space.
 
    The get_object method searches in the user 'spaces' field if the current
    space is allowed, if not, he is redirected 
    """
    context_object_name = 'get_place'
    template_name = 'spaces/space_index.html'
 
    def get_object(self):
        space_name = self.kwargs['space_name']
 
        for i in self.request.user.profile.spaces.all():
            if i.url == space_name:
                return get_object_or_404(Space, url = space_name)
 
        self.template_name = 'not_allowed.html'
        return get_object_or_404(Space, url = space_name)
 
    def get_context_data(self, **kwargs):
        context = super(ViewSpaceIndex, self).get_context_data(**kwargs)
        place = get_object_or_404(Space, url=self.kwargs['space_name'])
        context['entities'] = Entity.objects.filter(space=place.id)
        context['documents'] = Document.objects.filter(space=place.id)
        context['proposals'] = Proposal.objects.filter(space=place.id).order_by('-pub_date')
        context['publication'] = Post.objects.filter(post_space=place.id).order_by('-post_pubdate')
        return context

DELETEVIEW

Function-based:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

@permission_required('spaces.delete_document')
def delete_doc(request, space_name, doc_id):
 
    """
    Delete an uploaded document
    """
    place = get_object_or_404(Space, url=space_name)
 
    return delete_object(request,
                         model = Document,
                         object_id = doc_id,
                         login_required = True,
                         template_name = 'spaces/document_delete.html',
                         template_object_name = 'doc',
                         post_delete_redirect = '/',
                         extra_context = {'get_place': place})

Class-based:

1
2
3
4
5
6
7
8
9
10

class DeleteDocument(DeleteView):
 
    """
    Delete an uploaded document.
    """
    model = Document
 
    def get_queryset(self):
        objects = Document.objects.all().filter(id=self.kwargs['doc_id'])
        return objects

LISTVIEW

Function-based:

1
2
3
4
5
6
7
8
9
10
11
12

def list_all_docs(request, space_name):
 
    """
    List all docuemnts stored within a space.
    """
    place = get_object_or_404(Space, url=space_name)
 
    return object_list(request,
                       queryset = Document.objects.all().filter(space=place.id).order_by('pub_date'),
                       template_name = 'spaces/document_list.html',
                       template_object_name = 'doc',
                       extra_context = {'get_place': place})

Class-based:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

class ListDocs(ListView):
 
    """
    List all documents stored whithin a space.
    """
    paginate_by = 25
    context_object_name = 'document_list'
 
    def get_queryset(self):
        place = get_object_or_404(Space, url=self.kwargs['space_name'])
        objects = Document.objects.all().filter(space=place.id).order_by('pub_date')
        return objects
 
    def get_context_data(self, **kwargs):
        context = super(ListDocs, self).get_context_data(**kwargs)
        context['get_place'] = get_object_or_404(Space, url=self.kwargs['space_name'])
        return context

REDIRECTVIEW

Function-based:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

def go_to_space(request):
 
    """
    This view redirects to the space selected in the dropdown list in the
    index page. It only uses a POST petition.
 
    The 'raise Http404' isn't necessary, since if a space doesn't exist, it
    doesn't show, but just for security we will leave it.
    """
 
    place = get_object_or_404(Space, name = request.POST['spaces'])
 
    if request.POST:
        return redirect('/spaces/' + place.url)
 
    raise Http404

Class-based:

NOTE: Class-based redirects in django 1.3 only accepts GET petitions, instead of the recommended POST in django 1.2. This issue is fixed in 1.3.x

1
2
3
4
5
6
7
8
9
10
11

class GoToSpace(RedirectView):
 
    """
    This class redirects the user to a spaces after getting a GET petition.
 
    A 'raise Http404' is not necessary since a the objects the user can access
    are only the ones that are in the DB.
    """
    def get_redirect_url(self, **kwargs):
        self.place = get_object_or_404(Space, name = self.request.GET['spaces'])
        return '/spaces/{0}'.format(self.place.url)

UPDATE: I’ve rewrited the post since the instructions are much easier than I did put here (and more secure)

You will probably say: “Hey!, what’s in your mind to need django 1.3, it has been released just a week ago!”. The answer is: long-term development. Currently I’m working on e-cidadania, an e-democracy tool designed for participative processes. The development will take some time, and we’ve just started. It’s better to update everything now than later, but to the point, installing django 1.3 in dreamhost.

NOTE: I assume you know how to install a django project in DH. If not, please see this wiki

Let’s start.

STEP 1: CREATE A VIRTUALENV

You can create a virtualenv with the system-wide python installation (currently 2.5.2) or install your own python interpreter if you need it. We will use the system-wide python for convenience, since it’s still supported by django 1.3.

This part is extracted from here

NOTE: These commands must be typed at your home directory

$ wget http://pypi.python.org/packages/source/v/virtualenv/virtualenv-1.5.2.tar.gz
$ tar xzf virtualenv-1.5.2.tar.gz
$ python virtualenv-1.5.2/virtualenv.py $HOME/env

This will create a directory called “env” with all the stuff you need to work. Let’s do this virtual environment our default.

STEP 2: STABLISH THE VIRTUALENV PYTHON AS DEFAULT

The following instructions about stablishing the virtual python as default only work at shell level. For passenger the system-wide python is the default.

On ~/.bash_profile write:

export PATH=$PATH:$HOME/env/bin

This will stablish the directory ~/env/bin (where our virtual python is) before the system-wide python. To work right now without logging in again, we can do:

$ source .bash_profile

That will make the executables in ~/env/bin available for using right now, and between them, it’s easy_install, properly configured to install any python module locally without problem.

STEP 3: INSTALL DJANGO 1.3 AND ANY OTHER PACKAGES

Let’s install django 1.3 and some common python packages needed in almost every project:

$ easy_install -U django

For the moment we will install just django, because the rest of python packages are provided by the DH server. If you need non standard modules (p.e. python-dateutil) or other version of an already provided package, you must install them after django.

If you try to execute some django project right now you’ll see that the system-wide installation of django is the preferred. Let’s change that.

STEP 4: CONFIGURING PASSENGER_WSGI.PY

In our passenger_wsgi.py we must modify the PYTHONPATH, let’s do it:

import sys, os
 
# Python has no prepend function, so we must do an insert. We insert the key directories
# BEFORE ANYTHING, so the python installation sees them before the system-wide libraries.
 
sys.path.insert(0,'/home/USERNAME/env/bin')
sys.path.insert(0,'/home/USERNAME/env/lib/python2.5/site-packages/Django-1.3-py2.5.egg')
sys.path.insert(0,'/home/USERNAME/env/lib/python2.5/site-packages')
 
os.environ['DJANGO_SETTINGS_MODULE'] = "YOURPROJECT.settings"
import django.core.handlers.wsgi
application = django.core.handlers.wsgi.WSGIHandler()

Remember that if any of your modules fails, python will not fall back to the system-wide installation. Once a module is loaded, python does not seek another. Also, if you installed another package apart from django, you must do an insert including it.

That’s it! Now when you receive a visitor, Passenger will load you desired django version and packages.

This is more a reminder than a proper post, but let’s see how do we do this. I’ve been weeks wondering how the heck I could change the language on the fly in a website with the django i18n system, and the worst is that IT’S DOCUMENTED, but it’s sparse across the documentation and the source code docs (hey guys, you could write down all the documentation from time to time).

Let’s start:

To have the translation system working you need to add this middleware to your settings:

1

    django.middleware.locale.LocaleMiddleware

After that, on your urls.py file you need to include the i18n view:

1

    (r'^i18n/', include('django.conf.urls.i18n')),

That view only accepts POST requests, so if for any reason you need a GET, you will better be prepared, because you have to modify a core view, or create your own one. Ok, that view will put the url “i18n/setlang” to our service. Let’s go to the interesting part, the HTML code:

1
2
3
4
5
6
7
8
9

    <form action="/i18n/setlang/" method="post">{% csrf_token %}
      <input name="next" type="hidden" value="/" />
      <select name="language">
        {% for lang in LANGUAGES %}
          <option value="{{ lang.0 }}">{{ lang.1 }}</option>
        {% endfor %}
      </select>
      <input type="submit" value="Go" />
    </form>

This code will make a dropdown list with ALL the languages supported by django (and there are quite a few) so if you want to restrict the languages you must set up the LANGUAGES variable in your settings.py with something like this:

1
2
3
4
5

LANGUAGES = (
    ('es', 'Spanish'),
    ('en', 'English'),
    ('gl', 'Galician'),
)

Be warned, this language list is NOT translated, that means that wherever you are, that list will be always in english. To translate it you must make use of gettext directly. Do not attempt to use the translation utils from django, that will cause a circular import.

UPDATE:

Aleck gave us another way of doing the i18n selector, using links instead of a dropdown (which is very useful :) In this use case, we use list items for selecting from English and Dutch languages. This HTML code assumes you have this code in your settings.py:

1
2
3
4

LANGUAGES = (
    ('en', 'English'),
    ('nl', 'Dutch'),
)

HTML code:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

<li>
    <form name="setLangEnglish" action="/i18n/setlang/" method="POST">{% csrf_token %}
        <input name="next" type="hidden" value="/" />
        <input type="hidden" name="language" value="en" />
        <a href="#" onclick="document.setLangEnglish.submit();return false;">English</a>
    </form>
</li>
<li>
    <form name="setLangDutch" action="/i18n/setlang/" method="POST">{% csrf_token %}
        <input name="next" type="hidden" value="/" />
        <input type="hidden" name="language" value="nl" />
        <a href="#" onclick="document.setLangDutch.submit();return false;">Dutch</a>
    </form>                
</li>

Thank you very much Aleck!

UPDATE 2:

These days I’m updating the CSS of e-cidadania to Bootstrap 2.0 so I thought it could be a nice moment to change the language selector for something more stylish. Here is the code to use a list selector with bootstrap 2.0:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

<ul class="nav pull-right">
    <li class="dropdown">
        <a href="#" class="dropdown-toggle" data-toggle="dropdown">{% trans "Language" %}<b class="caret"></b></a>
        <ul class="dropdown-menu">
	    {% for lang in LANGUAGES %}
	        <li>
                    <form name="setLang{{ lang.1}}" action="/i18n/setlang/" method="POST">{% csrf_token %}
                        <input name="next" type="hidden" value="/" />
                        <input type="hidden" name="language" value="{{ lang.0 }}" />
                        <a href="#" onclick="document.setLang{{ lang.1 }}.submit();return false;">{{ lang.1 }}</a>
                    </form>
                </li>
            {% endfor %}    
	</ul>
    </li>
</ul>

Dejo esto aquí a modo de nota informativa y recuerdo para mis futuros problemas. Como ejemplo voy a coger el código real con el que he trabajado, ya qu es GPL.

models.py

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

class Space(models.Model):
 
    name = models.CharField(_('Name'), max_length=100, unique=True,
                            help_text=_('All lowercase. Obligatory.'))
    description = models.TextField(_('Description'))
    date = models.DateTimeField(auto_now_add=True)
    author = models.ForeignKey(User, blank=True, null=True, verbose_name=_('Author'))
 
    logo = models.ImageField(upload_to='spaces/logos',
                             verbose_name=_('Logotype'),
                             help_text=_('100px width, 75px height'))
    banner = models.ImageField(upload_to='spaces/banners',
                               verbose_name=_('Banner'),
                               help_text=_('75px height'))
    authorized_groups = models.ManyToManyField(Group,
                                            verbose_name=_('Authorized groups'))
    #theme = models.CharField(_('Theme'), m)
 
    # Modules
    mod_debate = models.BooleanField(_('Debate module'))
    mod_proposals = models.BooleanField(_('Proposals module'))
    mod_news = models.BooleanField(_('News module'))
    mod_cal = models.BooleanField(_('Calendar module'))
    mod_docs = models.BooleanField(_('Documents module'))
 
    class Meta:
        ordering = ['name']
        verbose_name_plural = _('Spaces')
 
    def __unicode__(self):
        return self.name

Los campos que desees rellenar automáticamente en la vista, deben ir obligatoriamente marcados como que se pueden dejar en blanco, mediante blank=True o null=True o ambos si es necesario (este caso).

Para no rompernos demasiado la cabeza, haremos uso del ModelForms de django, y de esa forma dejaremos un código limpio.

forms.py

1
2
3
4
5

from e_cidadania.apps.spaces.models import Space
 
class SpaceForm(ModelForm):
    class Meta:
        model = Space

Una vez hecho esto toca generar la vista. Desconozco si se puede generar una vista manipulable mediante vistas genéricas, pero me da que no. No os preocupéis, el código necesario para la vista es bastante intuitivo y muy cortito.

views.py

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

@permission_required('Space.add_space')
def create_space(request):
 
    space = Space()
    form = SpaceForm(request.POST or None, request.FILES or None, instance=space)
 
    if request.POST:
        form_uncommited = form.save(commit=False)
        form_uncommited.author = request.user
        if form.is_valid():
            form_uncommited.save()
            space = form.name
            return render_to_response(space)
 
    return render_to_response('spaces/add.html',
                              {'form': form},
                              context_instance=RequestContext(request))

El código es bastante autoexplicativo, pero de todas formas lo expondré. Cuando se crea un espacio nuevo se genera una instancia de Space para indicarle al formulario a dónde pertenece, aunque no es necesario, ya que lo detecta automáticamente. Además se genera un formulario vacío o lleno dependiendo de la situación.

Cuando se ejecuta la vista, se envía un request, si no era de tipo POST (cuando enviamos un formulario) request.POST y request.FILES no existen por lo tanto les otorga el valor None y con ello un formulario vacío.

Si no hay POST, la cláusula “if” no se ejecuta y al usuario se le presenta un formulario vacío listo para rellenar. Ahora viene lo bueno, ¿qué ocurre cuando rellenamos el formulario y lo enviamos? Hay campos como el de “autor” o el de fecha que deberían ser automáticos. Acordaros de que aunque no deben ser campos visibles al usuario, sí que tienen que estar en la plantilla como {{ is_hidden }}. De lo contrario estaréis enviando un formulario incompleto cuando hagáis el POST (por mucho que lo rellenéis más tarde en la vista).

El campo de fecha se rellenará sólo, así que por eso no te preocupes, el inconveniente viene a la hora de guardar el autor. Debemos guardar el formulario pero sin enviarlo, esto lo hacemos con un save() enviando el parámetro “commit=False“, para luego añadirle o modificarle los valores que nosotros necesitemos. Tras ello, pasamos la validación (si hicésemos la validación antes no la pasaría) y guardamos definitivamente. Después de eso detectamos el nombre del espacio que hemos creado y renviamos al usuario al mismo.

The Supervise project is going very well right now, I’ve commited the templates for the issues application and part of the projects. The “web view” is quite simple right now, due to the fact of using the django development server, but it will be improved when everything is ready.

What I must implement right now is a documentation system for the projects (it would be awesome if the docs application supported textile) and finish some details around (like user profile pages). After that the first alpha will be ready and I can start developing the repositories interface, the first ones will be Git, Bzr and Subversion, then I’ll try to implement Mercurial, CVS and some more.

If anyone wants to contribute, here is the repo:

bzr://halcyon.zapto.org/supervise

Just do:

$ bzr co bzr://halcyon.zapto.org/supervise

And a copy will be yours!

Puede que parezca todo emocionado escribiendo tantas entradas acerca de este «pequeño» proyecto, y es que realmente lo estoy, es probablemente el proyecto más grande que he hecho hasta la fecha, y el que más probabilidades tiene de tener una versión final.

El caso, que ya he empezado a desarrollar las vistas de la aplicación, ahí os va una pequeña vista previa de la versión de prueba del bugtracker/sistema de tickets/como querais llamarlo.