CS 180 Project 3: Face Morphing

By Jathin Korrapati

Part 1: Defining Correspondences

For this part, in order to morph two faces together, you have to first define corresponding points in the same order for each image (1:1 mapping like both of the 1st points are to the nose of each respective image, 2nd to the left eye, etc.). I used the correspondence tool that was listed on the website to label an image of myself and George Clooney. Points that seemed to work best for me when I chose points more spread out across the faces rather than in a compact space as it allowed for better morphing. I used more points on the more variable features in a person, like the nose, eyes, hair, and other ones, and less on more uniform features like the forhead or cheeks.

Here are the base images for George Clooney and myself:

Here are each of our faces labeled with the points

Once we have the correspondences (the points labaeled on each face), we can take the mean correspondences and use this to compute the Delaunay triangulation on the mean set of key points.

Part 2: Computing the Midway Face

Now, we have to compute the midway face between myself and the image of George Clooney. To do this, we must first compute the average shape from the mean correspondences (the ones we used in the previous part for a Delaunay Triangulation), then warp both of the faces into the average shape, and then average the colors of the warped faces in order to cross-dissolve it.

For the warping, we warp each triangle from the triangulation of each image into the average shape's triangulation. We can do this using the polygon function in skimage, which generates the triangle and its three vertices allowing us to mask and warp all the points in between instead of looping over all the pixels. Then, we apply an inverse warp to the triangles to find the pixel value at that pixel point.

(750, 602, 3)
(750, 602, 3)

Part 3: The Morph Sequence

To generate the morph sequence, we extend what we did in part 2, but with more control in choosing the warp_fraction factor and dissolve_frac factor. The cross-dissolve and warp_frac factor lies in a range between [0, 1], and the warp_frac. My morph from myself to George Clooney and back is displayed below:

Part 4: The "Mean face" of a population

For computing the "mean face" of our population here, I utilized the FEI database. This database has 400 images of 200 people, with each person having a neutral expression and smiling expression. There are a 100 male subjects and 100 female subjects in the database to sample from. I utilize a subset with 50 males and 50 females for computation purposes.

The FEI database has 46 correspondence points, but in order to generate better results, I had to manually add the corner points in each of the images to create better alignments and triangulations, which led to using a total of 50 points.

To actually calculate the mean image, we continue with a similar approach from what we did in part 2, but instead we generalize to a set of mean-face images. To do this, we compute the average face shape, warp each of the images in our dataset to this shape, and then take the average color of all the warped images. Note, I use bilinear interpolation here in order to calculate the color when we perform the warping, as it provided me better results.

Here are the "mean" faces of the two sub-populations in the FEI dataset, which are the smiling and neutral categories. My results are displayed below:

Here are the example images I tested on, warping some smiling faces to the average smiling face shape:

Here are them smiling now

Now, here are their neutral faces normalized to the average neutral face

Now, here are their smiling faces normalized to the average smiling face

Continuing on, I also warp my face to the average geometry of the dataset I used, and also warp the average face onto my geometry. Note, I also had to re-label my image again to be correspondent to the geometry of the FEI database (all images in that were labeled consistently, so using any sample image there worked). Another thing is, I think the reason my morphing the average mean shape to my face didn't work out super well is because my hair gets in the way part of my face as its a little messy, and that kind of messes up the morph creating the blur that you see with it (as some correspondences of points would be my hair to around their eyebrow or part of the mean face's head or something).

Part 5: Caricature: Extrapolating from the Mean:

For this part, I use the FEI population mean once again, and using it we can provide caricatures by extrapolation. When we perform extrapolation, it means our factor is either greater than 1, or less than -1. For my case, I utilized a value of 1.5, as I thought it looked pretty derpy to look at. Note, I used the re-labeled image I mentioned in the above part.

Part 6: Bells and Whistles

Part 6.1: Auto-Correspondences

In this part, I utilize auto correspondences to automatically label and configure the images for me. I found it kind of annoying to label the points with the tool extract them and then parse the json file to label the points, so this Bells & Whistles part seemed pretty cool to use. I used a model from CV2, called the Landmark model. The model for any given image will try and identify the face first, and then label it correspondingly (assume images are already scaled to be the same size). I foud experimenting a little that the results tended to be slightly better here, as opposed to being malleable to user input, it was a little more consistent with its labeling.