Le coin à Guij

FOSDEM 2012 – can beer freeze?

guijemont — Thu, 02 Feb 2012 16:38:08 +0000

Thanks to my awesome employer Igalia, I am flying tomorrow to the cold city of Brussels to attend FOSDEM 2012.

I will give a lightning talk, Saturday at 13:40, about my balloon adventures at Nowhere, with an emphasis on how Free Software made it possible. Hopefully, this will motivate other Free Software hackers to get out of their basement and hack in the “real” world :).
The talk should be streamed live at this url (in the Ferrer room).

Many other Igalians will come (I think there will be 14 of us), and will give talks.

Gstreamer and OpenCV for image stabilisation

guijemont — Thu, 10 Nov 2011 10:37:34 +0000

I am now back from Prague where I gave a talk on image stabilisation (and my holiday pictures). Hopefully a video of the talk will soon be online. In the meantime, I would like to explain a bit my efforts in written form, with some details slightly updated from the talk (the code progressed a bit since then).

UPDATE: The talk is now online.

I got interested in the issues of image stabilisation through a helium balloon photography project in which I participated. I want to make a nice time lapse video from the pictures I have taken, but they were taken from a camera that was moving, which would make the result very shaky without some kind of postprocessing.

Thankfully, I work at Igalia, which means that on top of my personal time, I could spend on this project some company time (what we call internally our hackfest time, of up to 5 hours per week).

Original problem statement

I have around 4h30 of pictures taken from a balloon 100 metres high. The pictures were taken at a rate of one per minute, which makes around 270 pictures. I want to make a nice time lapse out of it. Simply using the frames as is to build a video does not work well. Partly because I would probably be legally required to include a warning to epileptic people at the beginning of the video, but mostly because people actually watching it would wish they were epileptic to have a good excuse not to watch it.

This is due to the huge differences occurring between two consecutive frames.

Here is an example of two consecutive frames in that series:

As you can see, from one frame to the next, a lot of pixels would change. And that does not look pretty. It is also pretty obvious that they are both pictures of the same thing, and could be made to be pretty similar, mainly by rotating one of them, and maybe reprojecting it a bit so that things align properly even though the point of view changed a bit from one frame to the next.

Standing on the shoulders of giants

There was no question in my mind that I wanted to use GStreamer for the task, by writing an element or set of elements to do the stabilisation. The two big advantages of this approach are:

I can benefit from all the other elements of GStreamer, and I can easily do things like decode my pictures, turn them in a video, stabilise it and encode it in a format of my choice, all in one command.
Others could easily reuse my work, potentially in ways I could not think of. One idea would be to integrate that in PiTiVi in the future.

Then, after some research, I realised that OpenCV provides a lot of the tools needed for the task as well.

Since I am still in a prototyping/research stage, and I hate to write loads of boilerplate, I am using python for that project, though a later rewrite in C or C++ is not impossible.

First things first

I will not present things exactly in the order I researched them, but rather in the order I should have researched them: starting with a simpler problem, then getting into the complications of my balloon problem.

The simpler problem at hand is presented to you by Joe the Hippo:

Joe the shaky hippo (video)

As you can see, Joe almost looks like he’s on a boat. He isn’t, but the cameraman is, and the video was taken with a lot of zoom. The movement in that video stream has a particularity that can make things simpler: the position of a feature on the screen does not change much from one frame to the next, because a very short amount of time happens between them. We will see that some potentially very useful algorithms take advantage of that particularity.

The steps of image stabilisation

As I see it for the moment, there are two basic steps in image stabilisation:

Find the optical flow (i.e. the movement) between two frames
Apply a transformation that reverts that movement, on a global (frame) scale

Step 2. is made rather easy by OpenCV with the findHomography() and warpPerspective() functions, so we won’t talk much about it here.

Optical flow

For all that matters in this study, we can say that for each frame the optical flow is represented by two lists of point coordinates origins and destinations, such that the feature at the coordinate origins[i] in the previous frame is at the coordinate destinations[i] in the current frame.

Optical flow algorithms can be separated in two classes, depending on whether they provide the flow for all pixels (Dense optical flow algorithms) or only for selected pixels (Sparse optical flow algorithms). Both classes can theoretically provide us with the right data (origins and destinations point lists) to successfully compute the opposite transformation we want to apply using findHomography().

I tried one algorithm of each class, choosing the ones that seemed popular to me after reading a bit of [Bradski2008]. Here is what I managed to do with them.

Dense optical flow

I tried to use OpenCV’s implementation of the Horn-Schunck algorithm [Horn81]. I don’t know if I used it incorrectly, or if the algorithm simply cannot be applied to that situation, but this is all I could do to Joe with that:

Now Joe is shaky and flickery

As you can see, this basically added flickering. Since that, I did not find time to improve this case before I realised that this algorithm is considered as obsolete in OpenCV, and the new python bindings do not include it.

Note that this does not mean that dense optical flow sucks: David Jordan, a Google Summer of Code student, does awesome things with a dense algorithm by Proesmans et al. [Proesmans94].

Sparse optical flow

I played with the Lucas-Kanade algorithm [Lucas81], with the implementation provided by OpenCV. Once I managed to find a good set of parameters (which are now the default in the opticalflowfinder element), I got pretty good results:

Joe enjoys the stability of the river bank, undisturbed by the movements of the water (video)

And it is quite fast too. On my laptop (with an i5 processor), I can stabilise Joe the hippo in real time [1] (it is only a 640×480 video, though).

[1]	For those who attended my talk at the Gstreamer Conference 2011: yes, now it is proper real time, I optimised the code a bit.

The balloon problem

As we seen in the previous section, for a shaky hippo video, [Horn81] isn’t any help, but [Lucas81] is pretty efficient. But can they be of any use for my balloon problem?

Unsuccessful results

I won’t show any video here, because there is nothing much to see. Instead, an explanation in pictures that show how the algorithms rate for the balloon time lapse.

This is what Horn-Schunck can do:

The picture shows two consecutive frames in the time lapse (the older is on the left). Each of the coloured lines goes from a point on the first image to the corresponding point on the second one, according to the algorithm (click on the image to see a larger version where the lines are more visible). Since Horn-Schunck is a dense algorithm, the coloured lines are only displayed for a random subset of points to avoid clutter.

Obviously, these lines show that the algorithm is completely wrong, and could not follow the big rotation happening between the two frames.

Does Lucas-Kanade rate better? Let’s see:

This is the same kind of visualisation, except that there is no need to chose a subset, since the algorithm already does that.

As for the result, it might be slightly less wrong than Horn-Schunck, but Lucas-Kanade does not seem to be of any help to us either.

The issue here, as said earlier, is that these two algorithms, like most optical flow algorithms, are making the assumption that a given feature will not move more than a few pixels from one frame to the next (for some value of “a few pixels”). This assumption is very clever for typical video streams taken at 25 or 30 frames per second. Unfortunately, it is obviously wrong in the case of our stream, where the camera has the time to move a lot between two frames (which are captured one minute apart).

Is all hope lost? Of course not!

Feature recognition

I found salvation in feature recognition. OpenCV provides a lot of feature recognition algorithms. I have tried only one of them so far, but I hope to find the time to compare it with others in the future.

The one I tried is SURF (for “Speeded Up Robust Features”, [Bay06]). It finds “interesting” features in an image and descriptors associated with them. The descriptors it provides are invariant to rotation and scaling, which means that it is in theory possible to find the same descriptors from frame to frame.

To be able to efficiently compare the sets of frame descriptors I get for two consecutive frames, I use FLANN, which is well integrated in OpenCV.

Here is a visualisation of how this method performs:

As you can see, this is obviously much better! There might be a few outliers, but OpenCV’s findHomography() can handle them perfectly, and here’s a proof video (I am not including it in the article since it is quite high resolution).

Obviously, the result is not perfect yet (especially in the end), but it is quite promising, and I hope to be able to fix the remaining glitches sooner than later.

Show me the code!

The code as well as a quick introduction on how to use it is available on github. Bugs and patches should be posted here.

[Bradski2008]

G. Bradski and A. Kaehler , “Learning OpenCV”, ISBN 978-0-596-51613-0, 2008.

[Horn81]

B. K. P. Horn and B. G. Schunck, “Determining optical flow,” Artificial Intelligence 17 (1981): 185–203, 1981. PDF

[Proesmans94]

M. Proesmans, L. Van Gool, E. Pauwels, A. Oosterlinck, “Determination of optical flow and its discontinuities using non-linear diffusion”, J.-O. Eklundh (Ed.), Computer vision — ECCV ’94, Lecture Notes in Comp. Science, Vol. 801, Springer, Berlin, 295–2304, 1994. PDF

[Lucas81]

B. Lucas and T. Kanade, “An Iterative Image Registration Technique with an Application to Stereo Vision”, Proc. of 7th International Joint Conference on Artificial Intelligence (IJCAI), 674–279 PDF

[Bay06]

H. Bay, T. Tuytelaars and L. Van Gool, “SURF: Speeded Up Robust Features”, 9th European Conference on Computer Vision, 2006. PDF

Prague, we meet again!

guijemont — Sat, 22 Oct 2011 22:45:29 +0000

Karlův most (Charles bridge)

Tomorrow I will fly to Prague, going to the GStreamer Conference, then to LinuxCon Europe and ELCE.
I’m excited to go back there, after having first visited this beautiful city in my last Eurotrip.
I will give a talk at the GStreamer Conference about my work on image stabilisation, related to that balloon project we did with Ugo.
I will go there sponsored by Igalia, with my friends and colleagues Víctor, Philippe and Javi. Víctor will give a talk as well, about the integration of syslink and GStreamer.

Eurotrip starting soon: Desktop Summit, Berlin

guijemont — Wed, 03 Aug 2011 12:03:02 +0000

On Friday, I am going on a little Eurotrip that will lead me through Germany, France, Ireland and Czech Republic. The whole trip will last a month.

If I manage better than some american guys, I will get to Berlin on Friday afternoon for the desktop summit:

I am going there sponsored by the awesome company I work for, Igalia. Many other igalians will be there too.

Grilo will be well represented, with a short talk by Philippe on his integration work of Grilo with the Gnome Shell, and a hacking session with Juan and myself.

I will stay in Berlin until the 15th of August, to attend the GObject Introspection hackfest, where I will also be sponsored by Igalia. I should admit I don’t know much about the internals of GI yet, but I am excited to “learn by hacking” in the company of great hackers!

Then I will continue my Eurotrip, but that is another story.

Playing with balloons

guijemont — Mon, 09 May 2011 08:00:56 +0000

These days, I’m spending a big part of my free time on a nice project with a few friends. That’s a project involving a balloon. We have recently started a blog called Balloon Freaks to talk about this, you might want to check it out.

Blog back online

guijemont — Tue, 26 Apr 2011 22:02:13 +0000

I finally found the time to migrate my blog to the new server I have (the old server died a few weeks ago).

I hope I managed to do the transition correctly, and that people following me through a feed reader and/or a planet won’t be flooded with a truckload of old articles. If that still happens, I apologise. Also, I know the alert readership that you are noticed that I went back to the default wordpress theme: this is by design (and a bit by laziness). I got tired of the old design, which was ugly anyway (it was hacked together from various bits by a very amateurish designer: me). So, here’s something sober, just like I want it to be.

As for more generic news, I am still happy, enjoying life in Barcelona, loving my job at Igalia, hacking on cool Grilo stuff (and sometime GStreamer stuff as well). Hopefully I will post some things regarding that in the not so distant future.

This might be difficult because most of my spare time these days is taken by what I call the “balloon project”, recently started with a few gifted friends of mine. More on that soon, I promise, as I have plenty of things to tell about this project that involves developments in a lot of domains in which I’d love to know more but am always discovering things, such as physics, electronics or computer science.

Fosdem

guijemont — Sat, 22 Jan 2011 14:42:24 +0000

Like a few friends have stated, I am now saying it loud and clear:

And for that I should thank the awesome company for which I work. I am sure great time will be had, as well as great conversations about free software, multimedia, technology, life, the universe and everything.

See you all at FOSDEM!

Recent hacks: python with gdb to follow gstreamer

guijemont — Tue, 09 Nov 2010 15:58:39 +0000

I’ve recently discovered one too little-known feature of gdb: you can script it with python. It’s even in the documentation, and there are nice tutorials available which I invite you to read.

The two main cool things you can do in python, is define new commands, and new convenience functions. Unfortunately, as you can read in this article, defining a new command requires a lot of boilerplate (and it’s the same for convenience functions).

Since I wanted to write a few commands, I couldn’t be bothered to copy-paste that boilerplate, so I ended up writing a nice magical class that makes registering a function easier. You can find that here.

With this new weapon in hand, I wrote new commands to be able to follow what happens in a GstAdapter. Basically, I wanted to be able to track the position in the original file of each byte that was going out of a given adapter. So I wrote some code to be called by gdb commands when some operations on a GstAdapter are done, doing the necessary calculations and storing, and voilà: I can print, with gdb, the file position of the data obtained with each call to gst_adapter_peek().

All the code and the gdb script can be found over there.

As you can see, a gdb script is still needed. I haven’t yet really tried to play with breakpoints in python, but I have a feeling this is not totally trivial, if at all possible. The other great limitation is speed. It feels like the calls to python stuff from gdb are very costly, and, in the case of GStreamer, don’t expect smooth playing if you use some gdb command implemented in python for every buffer transmitted.

Conclusion: it’s got limits, but overall, this ability to enhance gdb with python allows you to do things you couldn’t do easily otherwise, like store data in complex structures for your debugging, and prototype easily some debugging actions you want to do, where the only alternative is to write it in C and include it in the application/library you are debugging.

New job, new flat: new life!

guijemont — Tue, 05 Oct 2010 14:46:13 +0000

You might have noticed one way or another, that I have just started a new job at Igalia. I’m very excited about joining a company that seems to try and do everything the right way™, and with really great people.

At least for starters, and to flex my hacking muscles a bit, I’m going to hack on various gstreamer stuff, which is always fun. Also, I think this will be my first post to appear on planet Igalia, so: HI PLANET READERS!

Oh yeah, as the title says, I’ve just changed flat, am now living in a great flat in a brand new building with cool flatmates, so that’s good news as well! And yes, I’m still in Barcelona, now in Poble Sec, which proves a cool area so far.

Camino de Santiago

guijemont — Wed, 04 Aug 2010 15:10:11 +0000

Tomorrow (Thursday 5th of August 2010), I am going to meet Paola and Ugo on the Camino de Santiago. They’ve been walking since last Friday, and with them, I should try to reach Santiago de Compostela, after about 3 weeks of walking, if we make it to there, that is.

Since I’ll probably have plenty of things to tell from the road, I’ve started another blog just for that, which you can find there: http://santiaguij.blogspot.com/.

UPDATE: made it there, and finally uploaded all the videos, and wrote many articles, on my santiago blog.