Nuno Mariz Weblog

My goal was to archive and display my internet lifestream. My first approach was writing a client for each API of the social networks that I'm in.
This turned out to be a complete waste of time and effort. All that I needed after all was a FriendFeed account that would centralize all my feeds.

Archiving and displaying your entries with Django is quite simple.
First of all, you need to download the Python FriendFeed API client. Then start a new application in your project, lets call it lifestream:

./manage.py startapp lifestream

On the settings.py add the lifestream project to the INSTALLED_APPS and a variable to store your FriendFeed username:

FRIENDFEED_USERNAME = 'your_username'

In the models.py add a model named Entry:

from django.db import models

class Entry(models.Model):
    id = models.CharField(max_length=255, primary_key=True)
    service_id = models.CharField(max_length=50, null=True, blank=True)
    service_name = models.CharField(max_length=50, null=True, blank=True)
    service_icon = models.URLField(max_length=255, verify_exists=False, null=True, blank=True)
    service_profile = models.URLField(max_length=255, verify_exists=False, null=True, blank=True)
    title = models.CharField(max_length=255, null=True, blank=True)
    link = models.URLField(max_length=255, verify_exists=False, null=True, blank=True)
    updated = models.DateTimeField(null=True, blank=True)
    published = models.DateTimeField(null=True, blank=True)
    media_title = models.CharField(max_length=255, null=True, blank=True)
    media_link = models.URLField(max_length=255, verify_exists=False, null=True, blank=True)
    media_thumbnail = models.URLField(max_length=255, verify_exists=False, null=True, blank=True)
    created = models.DateTimeField(auto_now_add=True)

    def __unicode__(self):
        return self.title

    class Meta:
        ordering = ['-published']
        verbose_name = 'Entry'
        verbose_name_plural = 'Entries'

    class Admin:
        list_display = ['title', 'service_name', 'published']
        list_filter = ['service_name']
        date_hierarchy = 'published'

Create an url.py on the lifestream folder:

from django.conf.urls.defaults import *
from lifestream.models import Entry
 
entry_list_dict = {
    'queryset' : Entry.objects.all(),
    'paginate_by' : 30,
}

urlpatterns = patterns('',   
    (r'^$', 'django.views.generic.list_detail.object_list', entry_list_dict),
)

As you can see, I've used a generic view. You can also use the date based generic views and pagination to build an archive like mine.

Add to your project root urls.py:

(r'^lifestream/', include('lifestream.urls'))

Create a template lifestream/entry_list.html:

{% for entry in object_list %}
<div class="source">
  <a href="{{ entry.service_profile }}" title="{{ entry.service_name }}"><img src="{{ entry.service_icon }}" alt="{{ entry.service_name }}" alt="{{ entry.service_name }}" /></a>
</div>
<div class="details">
  <ul>
    <li><a href="{{ entry.link }}">{{ entry.title }}</a></li>
    <li>{{ entry.published|timesince }} ago</li>
    {% if entry.media_thumbnail %}<li><a href="{{ entry.media_link }}"><img src="{{ entry.media_thumbnail }}" alt="{{ entry.media_title }}" /></a></li>{% endif %}
  </ul>
</div>
{% endfor %}

Finally, create a script to synchronize your feeds:

#!/usr/bin/env python
# -*- coding: utf-8 -*-

import sys
import os

ROOT_PATH = os.path.realpath(os.path.dirname(__file__))
PROJECT_PATH, PROJECT_DIR = os.path.split(ROOT_PATH)

sys.path.insert(0, ROOT_PATH)
sys.path.insert(1, PROJECT_PATH)

os.environ['DJANGO_SETTINGS_MODULE'] = '%s.settings' % PROJECT_DIR

from friendfeed import FriendFeed
from django.conf import settings
from lifestream.models import Entry

ff = FriendFeed()
feed = ff.fetch_user_feed(settings.FRIENDFEED_USERNAME)

for e in feed.get('entries'):
    entry, created = Entry.objects.get_or_create(id=e.get('id'))
    if created:
        service = e.get('service')
        entry.service_id = service.get('id')
        entry.service_name = service.get('name')
        entry.service_icon = service.get('iconUrl')
        entry.service_profile = service.get('profileUrl')
        entry.title = e.get('title')
        entry.link = e.get('link')
        entry.updated = e.get('updated')
        entry.published = e.get('published')
        media = e.get('media')
        if media:
            entry.media_title = media[0].get('title')
            entry.media_link = media[0].get('player') or entry.link
            thumbnails = media[0].get('thumbnails')
            entry.media_thumbnail = thumbnails[0].get('url')
        entry.save()

If you want, you can add a job in your crontab:

# synchronize every 15 mins
*/15 * * * *   root   /path/to/your/application/lifestream_cron.py

See my lifestream as the working example.

UPDATE: Friendfeed sends the time in UTC, if you want to use your timezone you have do some hacking:

Install pytz:

easy_install pytz

Import and assign your timezone to a variable:

import pytz
tz = pytz.timezone(settings.TIME_ZONE)

And replace entry.updated and entry.published with:

updated = e.get('updated')
updated = updated.replace(tzinfo=pytz.utc).astimezone(tz)
published = e.get('published')
published = published.replace(tzinfo=pytz.utc).astimezone(tz)
if settings.DATABASE_ENGINE == 'mysql': # http://code.djangoproject.com/ticket/5304
    updated = updated.replace(tzinfo=None)
    published = published.replace(tzinfo=None)
entry.updated = updated
entry.published = published

Thanks to Chris Kelly that send me an email reporting this.

Comments

#1 Justin Lilly on April 4, 2008 1:39 a.m. (1 year, 3 months ago)

It would be pretty nifty if I could click the links in your twitter feed of your lifestream and be taken to the link instead of the status.

<plaintext>I met <link>@justinlilly!</link> at my blog. <link>tinyurl.com/393kd</link></plaintext> or something similar.

Back to top

#2 Nuno Mariz on April 4, 2008 1:46 a.m. (1 year, 3 months ago)

Hi Justin,
I've made it generic for all kind of sources(services), but I can easily write a templatetag that formats the title.
Thanks for the suggestion.

Back to top

#3 Peter Baumgartner on April 4, 2008 2:36 a.m. (1 year, 3 months ago)

Jacob built an app called Jellyroll that has similar functionality without using FriendFeed. Out of curiousity, why not split media and services into different models?

Back to top

#4 kevin on April 4, 2008 5:24 a.m. (1 year, 3 months ago)

i was going to exactly the same thing - but super psyched to see someone else doing it already. nice job.

Back to top

#5 Nuno Mariz on April 4, 2008 9:38 a.m. (1 year, 3 months ago)

@Peter My idea was to simplify all the process, so I've made a generic table for the job.
If I add a new feed to my FriendFeed account, I don't need to change a thing in my lifestream app.

Back to top

#6 André Luís on April 4, 2008 9:46 a.m. (1 year, 3 months ago)

Whoa! Very nice Nuno. I'm not a django guy myself, but I like the end result. Similar to what I wrote in php, only I didn't go with FriendFeed. I went straight to the sources.

I do think one very important aspect in lifestreams is the feed pace. Some feeds have 1 post per week, others (like last.fm) can achieve hundreds per day. It should be balanced to forbid one source to takeover the stream.

Also clicking on the item and showing the item description would be extra cool. I hate that friendfeed only tells I posted a post but not the contents of the post.

Still, I hope you see these as tips and not criticism. ;) Very well done dude. Rock on. :)

Back to top

#7 Nuno Mariz on April 4, 2008 10:41 a.m. (1 year, 3 months ago)

@André Thanks for the tips, I will review all of them.

Back to top

#8 Chris Kelly on April 8, 2008 12:49 p.m. (1 year, 2 months ago)

I'm in the same place as Kevin - I was actually sitting down to write this (even have the FriendFeed.py file open!) and you've gone and done it! Great Job!

Back to top

#9 Stu on April 9, 2008 2:14 p.m. (1 year, 2 months ago)

This is v.cool... kinda like mugshot

Back to top

#10 Rui Ferreira on April 17, 2008 12:36 p.m. (1 year, 2 months ago)

http://code.google.com/p/django-friendfeed/ :)

Back to top

#11 Nuno Mariz on April 17, 2008 4:51 p.m. (1 year, 2 months ago)

@Rui Ferreira: Cool. Would be nice to have the archive functionality on it.

Back to top

#12 Rui Ferreira on April 18, 2008 5:35 a.m. (1 year, 2 months ago)

@Nuno: Junta-te ao projecto no google code. Para dizer a verdade o código foi "descolado" à pressa da minha página e ainda precisa de algum trabalho.

Back to top

#13 Nuno Mariz on April 18, 2008 7:31 p.m. (1 year, 2 months ago)

@Rui Ferreira: Ok, assim que passe a próxima semana infernal junto-me.

Back to top

#14 ilteris kaplan on May 12, 2008 6:44 a.m. (1 year, 1 month ago)

Hey Nuno, Thanks a lot for this tutorial. About when you are starting the application : python startproject lifestream. this gave me an error on the terminal. Is there a typo in there? I could able to start the app by doing ./django-admin.py startapp lifestream though. Maybe it's the different OS.

My main question is different. I couldn't for the life of me make the synchronization py file able to call the FRIENDFEED_USER var from my settings file. Are you going to make the code available for us to peek through it?

Thanks a lot for your efforts.

Back to top

#15 Nuno Mariz on May 13, 2008 1:31 a.m. (1 year, 1 month ago)

@ilteris Thanks for the correction, it was a typo.
About the question, I don't understand what you mean. Could you be more explicit?
The code that I use is all in this entry.

Back to top

#16 André Gonçalves on May 17, 2008 2:39 a.m. (1 year, 1 month ago)

uber cool

I've been working hard on a PHP framework which draws some inspiration from what i've learned from learning a little python and reading some django ideas, by the way

anyway, the fact is that writing a framework abstracts me from thinkin about concrete use cases...

I'm so gonna port this to my framework's library sometime soon...

Back to top

#17 Peter on August 4, 2008 10:12 p.m. (11 months ago)

Create tutorial on creating a lifestream. I'm currently looking at what I want to use as a lifestream (or more lifecache), because being able to look back into your past to see what you did a year ago is the main reason I want a lifestream. Hopefully I have some time next weekend to play around with your code. For now this page goes into my bookmarks.
Thanks for the great post.

Back to top

#18 Idan Gazit on September 2, 2008 10:55 a.m. (10 months ago)

Hey there,

I'm working on a BSD-licensed aggregation library that doesn't use the friendfeed approach, but I do use the code in your "sync" script as a starting point, and I'd like your permission to redistribute it.

If you could contact me via email, I'd be happy to show you the code + your credit in the readme.

Thanks!

-Idan

Back to top