Navigation

Annotated Facial Landmarks in the Wild: A Large-scale, Real-world Database for Facial Landmark Localization*

Martin Koestinger, Paul Wohlhart, Peter M. Roth, Horst Bischof
Institute for Computer Graphics and Vision, Graz University of Technology

*The work was supported by the FFG projects MDL (818800) and SECRET (821690) under the Austrian Security Research Programme KIRAS.


FAQ

Which annotations can I trust most? Is there a way to determine which annotations which are not suitable for a specific task?

The face bounding box may be used as quality estimate. E.g. rather small faces in (terms of geometric resolution) might be suitable for evaluation of a face detector but not to train a landmark detector. The annotated landmarks in these faces are not that precise. You can exclude these to train a landmark detector, e.g. SELECT face_id FROM FaceRect WHERE w > 256 ORDER BY w

Furhter note, that the data was annotated by our interns without zoom functionality in the GUI. Zoom functionality was introduced later for high-res images that show rather small faces (in respect to the image size, not in resolution).

I just have a question concerning the table "FaceRect". Is it normal that the top-left coordinates of the rectangle differ between the ones computed in your Matlab "main.m" and the x,y-values of the sql-table ?

The computation of rectangles as done in the matlab code can be considered out-of-date. In the c++ code we had a last minute change, that was not backported to matlab.

The problem is, we didn't find any consistent convention of how to define rectangle bounding boxes for out-of-plane rotated faces,let alone how to compute it from landmark annotations so we defined the following:

We calculate the pose and scale of the face from the fit of the mean 3d face model to the annotated points. In the 3d model there is a point specified right between the eyes. We project this point into the image and take it as an anchor point for the bounding box, such that the eyes we stay at approximately at the same point in the cropped face.

Initially we just placed the rectangle such that the anchor-point is horizontally centered and at one third from the top. The problem with this definition is, that it doesn't work for in plane rotation.

So in the c++ binary we now incorporated a shift of the bounding box center along the negative up vector. This gives a better fit.

If you want a consistent experience with I personally propose to use the c++ binary to calculate rectangles and save in the database and from then on just take it from there.

Anyway, our approach is just one way to go. We wanted to experiment with different versions of bounding box annotations, keeping different things fixed (for instance you could also try to center on the tip of the nose, or keep the eyes at fixed locations) in order to find out which convention gives the best crops for training data for different detection algorithms.

Some of the JPEG files contain data errors

Some of the original image files are damaged. This is not limited to the archives which are online. It seems to be that it happend at the time when we downloaded the images or even they were corrupted before. So consider it as real-world feature of the database :-)

Some of the images have feature coordinates with the values -1,-1. What is the significance of these values?

-1,-1 is a convention of our facedbsql library that the landmark is not set or has been deleted. Just ignore these.

Some of the images have feature coordinates with the values x = 0 or y = 0. What is the significance of these values?

The 0 values for the x or y coordinates of landmark points are correctly annotated. So the points are located at the image boundary.

Some of the coordinates have digits after the decimal point, others don't. Is this something I should worry about?

This is nothing to worry about. The reason is that the data was annotated by our interns without zoom functionality in the gui. Zoom functionality was introduced later for high-res images that show rather small faces (in repect to the image size, not in resolution). Therefore, only some values are decimal valued - annotations that we corrected later.

Some coordinates are negative ( x,y in the FaceRect table)!

That is correct. The rectangles are calculated automatically based on the manual specified landmark locations. If a face is not fully visible in the image or near one of the image borders it can happen that the FaceRect coordinates are negative. If you want to ignore these faces just filter them out by a SQL query.

The rectangle does not cover only the face region but rather larger portions in order to have a square rather than a rectangle which would exactly match the face itself

Our approach is just one way to go, let me briefly explain how we arrive at the bounding boxes. We calculate the pose and scale of the face from the fit of the mean 3d face model to the annotated points. In the 3d model there is a point specified right between the eyes. We project this point into the image and take it as an anchor point for the bounding box, such that the eyes stay at approximately at the same point in the cropped face.

Anyway, I recommend you to adapt this procedure for your particular purpose. Check out FaceRectCalculator.cpp or the provided matlab code. Please note that the computation of rectangles as done in the matlab code can be considered out-of-date. In the c++ code we had a last minute change, that was not backported to matlab. But it is a good starting point for testing and visualization of the crops.

Copyright 2010 ICG

Valid XHTML 1.1 Valid CSS