3D face tracking (state-of-the-art)

I’ve spend my day testing several face tracking solutions. The idea is to generate data for Godot Engine and control the face of an avatar with mine, for a numediart project.

Here is what i’ve tested:

OpenPose

Obviously, the famous library. Tricky to install: you need cuda toolkit & cuDNN to start compiling. As it is the first one, i didn’t manage to run it again at the end of the day, so no screenshot… There is no direct example for 3D face tracking, only 2D is working out-of-the-box, and fps is not sexy at all (~15fps)

https://github.com/CMU-Perceptual-Computing-Lab/openpose

Dlib

Lightweight library, compared to openpose, and therefore much more friendly. Managed to have a decent framerate wit my webcam, but still no 3D here…

https://github.com/davisking/dlib

Something looking like a 3d head estimation is demonstrated here: https://www.learnopencv.com/head-pose-estimation-using-opencv-and-dlib/

But I didn’t managed to see any results for 3d with the code provided in his version of dlib: https://github.com/spmallick/dlib

4dface

The original repository of the company 4dface, who has gone closed-source after starting to make some money.

The software uses a lot of git submodules, but is quite easy to setup when you have recompiled opencv3 & cmake 3.14 from source.

https://github.com/patrikhuber/4dface

The fps is terrible, but everything is there for 3d: mesh reconstruction and texture mapping! There is also lots of forks, maybe up-to-date… Definitively the best option, only have to focus on the poor framerate.

update: the Release flag was not set in CMake… By enabling it, the fps is jumping to 5fps. Not incredible, but enough to have the sensation of motion.

Next step is to extract landmarks in 3d and broadcast them in OSC!

SentiMask

I even tested closed-source library today! Purely commercial, at 399$ as entry-fee, i was hoping for a super-fast solution, making all the other ones insignificant, and, no. SentiMask is clearly not doing better than 4dface out-of-the-box. The rendering is just better.

https://www.neurotechnology.com/sentimask_sdk.html

The huge restriction implied by the license does not worth the ethical cost of using it.