A functional question about face effects

[yang]

hello,guys.
I have a question about facial recognition and face effects
I want to implement a project.
like:
When I look in the mirror, the camera in the mirror automatically recognizes the position of the face and adds facial effects.

The key issue is that the location of the face recognized by the camera may not be the same as the position in the mirror.

Do you have any good ideas?

thanks!

[yang]

Maybe my question is not very detailed.
please tell me , thanks~:cowboy_hat_face:

[j.e.f.f]

@yang Your first hurdle will be improving recognition performance on the Pi, which is currently really slow, far too slow to do real-time things like Snapchat-like facial effects (I’m assuming that’s what you’re after…) If you were to just use the existing libraries and modules, It takes upwards of 30 - 60 seconds for the mirror to respond with your personalized profile – and that’s just to process a single frame!!!. I talked to a dev friend of mine who said this is because the CPU is doing all the work, which it is not optimized for. Performance would be a whole lot better if the recognition library used the GPU for processing, but that’s WELL beyond my skill set :). Since you seem to want to add facial effects, I assume that means you know a thing or two about this kind of thing. If you can find a way to rewrite the library to run on the GPU, you would make many people in this forum very happy!

The second tricky part is identifying where the viewer’s face is actually being reflected in the mirror. Since the mirror surface itself is not an electronic device, you might need to do some pretty fancy math based on camera angle, focal length, position of the recognized face in the captured frame, etc. to estimate this. I suppose I would start with a bunch of test captures with my face positioned in a bunch of different places in the reflection and extrapolate from there. It means your camera would need to be in a permanently fixed position relative to the mirror’s surface for this to always work.

You could also make some kind of a calibration routine (e.g.“Position yourself such that your face is reflected in the bottom-left corner of the mirror, and hit the capture button”) and this could influence your positioning algorithm.

Looking at the mirror off axis (i.e.: from an angle to the left or right of straight on) would be particularly tricky… You’d need to figure out the angle at which the user is standing relative to the mirror and how that influences reflection position.

This is a pretty ambitious project. I wish you well on it, and if you can manage to make it work it would be quite an achievement!

I hope this helps!

• Jeff

[yang]

@j-e-f-f thanks very much , i’m from china . This is why I replied late. Your opinion is very useful to me.
About hardware performance issues. I think it can be done by other GPU devices.

The key is the second question you mentioned. How to position the face in the mirror.I will follow your suggestion to test.
If I succeed, I will show it here.
Thanks again!