Using Custom Django QuerySets

When I first started working with Django, I became frustrated that managers were not chainable. (If you don’t know what a Manager is you probably won’t find this post very useful. I recommend the DjangoBook for beginners.) Here’s a tutorial on a coding pattern our development team uses all the time on our PerformMatch product. It was developed with the help of some Django Snippets and Ara Anjargolian.

For our example, we’ll use Sneetches. Let’s start with a simple model:

from django.db import models

class Sneetch(models.Model):
    has_star = models.BooleanField(default=False)

Now Sneetches are bigots. Obviously having a star is either unequivocally better or worse than not having a star, but preference can change quickly. We want to to be able to change our code just as rapidly. To achieve this we can build a custom manager to keep our logic in one place. It will look something like this:

class SneetchManager(models.Manager):
    def get_query_set(self):
        return self.filter(has_star=True)

We’ll also need to assign this manager to the model, so here’s what we’ll end up with:

from django.db import models

class SuperiorSneetchManager(models.Manager):
    def get_query_set(self):
        return self.filter(has_star=True)

class Sneetch(models.Model):
    has_star = models.BooleanField(default=False)

    superior_sneetches = SuperiorSneetchManager()

This is what django recommends in the documentation for writing managers, and it’s pretty good. I can make this call wherever I want:

    Sneetch.superior_sneetches.all()

And because of the way we organized the code, we can change our mind about whether Sneetches with stars on their bellies are better than Sneetches without stars on thars.

Overriding the get_query_set method is great, but it’s static. We can’t pass in any arguments. Maybe we want to work with either the superior or the inferior Sneetches. To do this, we can make a method that is an attribute of a manager. So let’s rewrite the manager, and this time we’ll just assign it to the the objects attribute of our class. Since models, by default, have an attribute objects which is the class models.Manager, we can override this attribute and since we will be replacing it with a class that extends models.Manager, we won’t lose any functionality:

class SneetchManager(models.Manager):
    def segregate(self, superior=True):
        if superior:
            return self.filter(has_star=True)
        else:
            return self.filter(has_star=False)

class Sneetch(models.Model):
    has_star = models.BooleanField(default=False)

    objects = SneetchManager()

Now we have some choice. We can make either of these calls:

    Sneetch.objects.segregate(superior=True)
    Sneetch.objects.segregate(superior=False)

We almost have something really useful here, but we have a glaring limitation, segregate is not a chainable attribute. Only Sneetch.objects knows about segregate. Take a look at this code:

    Sneetch.objects.all().segregate(superior=True)

This raises an AtrributeError because all returns a QuerySet, which has no attribute segregate. We can overcome this by defining our own QuerySet that does have an attribute segregate.

    from django.db.models.query import QuerySet

    class SneetchQuerySet(QuerySet):
        def segregate(self, superior=True):
            if superior:
                return self.filter(has_star=True)
            else:
                return self.filter(has_star=False)

We know from the first example, that managers always call get_query_set, so let’s leverage that and write a manager that gets our SneetchQuerySet into the action.

class SneetchManager(models.Manager):
    def get_query_set(self):
        return SneetchQuerySet(self.model)

Now when Sneetch.objects calls get_query_set it will return a SneetchQuerySet which has an attribute segregate. We have on final step before we put it all together.

Sneetch.objects still doesn’t have an attribute segregate so our original call of Sneetch.objects.segregate(superior=False) will fail. If you read the Django source, you will see that models.Manager defines all the documented methods of QuerySet. Here is an example (copied directly from the source of v1.1):

    def filter(self, *args, **kwargs):
        return self.get_query_set().filter(*args, **kwargs)

We could mimic this behavior and allow our manager to call segregate by defining a segregate attribute for the manager. That is perfectly legal, but it would also force us to define any new methods we add to SneetchQuerySet in two places. Instead, we can use the magic python method __getattr__. I’ll save an explanation of __getattr__ for another post, but you can check the docs.

Here’s our final product:

    from django.db import models
    from django.db.models.query import QuerySet

    class SneetchQuerySet(QuerySet):
        def segregate(self, superior=True):
            if superior:
                return self.filter(has_star=True)
            else:
                return self.filter(has_star=False)

    class SneetchManager(models.Manager):
        def get_query_set(self):
            return SneetchQuerySet(self.model)
        def __getattr__(self, name):
            return getattr(self.get_query_set(), name)

    class Sneetch(models.Model):
        has_star = models.BooleanField(default=False)

        objects = SneetchManager()

    #Both These are Valid

    Sneetch.objects.segregate()
    Sneetch.objects.filter(**kwargs).segregate()

And that’s it. For any model that has repetitive/complex queries, I almost always write a custom QuerySet and Manager.

You can leave a response, or trackback from your own site.
  • http://twitter.com/MyNameIss Чибисов Геннадий

    Thanks for this.

  • Ben

    This is awesome!  Great explanation of the interplay between Managers, Querysets and Models, and the techniques described here fit the bill for what I’m trying to do.

  • Ben

    FYI, this code (the __getattr__ part) breaks in newer versions of Django.  See https://code.djangoproject.com/ticket/15062 for discussion. 

    Confirmed fix (copied from the above discussion):

    def __getattr__(self, attr, *args):
    if attr.startswith(“_”): # or at least “__”
    raise AttributeError
    return getattr(self.get_query_set(), attr, *args) 

  • http://twitter.com/semel Lee Semel

    Thanks, that was a helpful post. I put in a request to add this to Django itself, and it looks like it was accepted: https://code.djangoproject.com/ticket/16748

  • http://zmsmith.com Zach Smith

    Thanks for this. Hadn’t seen it.

  • http://zmsmith.com Zach Smith

    Awesome, thanks!

    I’m glad a lot more people will get to see this than would have just on this blog.

  • http://www.kiwixi.com/ Lotfi Adjeroud

    Thank you, this was very useful for me.

  • http://neoascetic.github.com/ Pavel Puchkin

    I’m not sure, but are we really need this `__gettattr__` in `SneetchManager` class?

    Since `SneetchManager` is child of `models.Manager`, it inherits all parent’s methods, right? So when we call `Sneetch.objects.filter`, `SneetchManager`’s `get_query_set` will be called (even though `filter` method is inherited, `self` variable passed to it is a `SneetchManager` instance)

  • http://www.facebook.com/vijay.shanker.39 Vijay Shanker

     hi , how can i do some computation on field such as for a field x filter all those database records where x+30 <10 ?
    say var1=30, var2=10
    i tried this return self.filter(x+var1<10) inside a custom queryset which defined similar to your segragate
    .thanks

blog comments powered by Disqus