@Realcumber @doublevr Hi, hope you are doing well. The passthrough is an amazing feature… could you please hear a small suggestion I had.
Existing:
[1] The current algorithm adjusts global chroma settings. It removes girls dark hair and other stuff also…
Right now, chroma is needed so that only girl remains in video, while remaining background gets mixed into the real background of the user. BUT, this chroma setting is different for every user, depending on the type of background they have.
Right now what we are doing, is a simple GLOBAL thresholding llike older Otsus Algorithm . Deep learning far outperforms such simple techniques in Computer VIsion.
What i Suggest: Remove Chroma Altogether.
This will remove the hassle of separate production costs, i.e. green screen everytime you shoot. ALSO, you could run this on potentially every video on SLR.
[1] Take 3d VR video SLR already shoots.
[2] Pass through a state-of-the-art 3d segmentation model. Those are free
[3] Get the girl’s body separated out from the background [segmentation mask] as well as the Depth mask.
[4] Do this for each frame. You have a segmentation mask for whole video. It CONTAINS 0 where background is there and 1 where girl is present.
On SLR APP Backend.
[1] Whenever someone clicks passthrough feature, the video combines with the segmentation mask in the above background:
New_video = (OLD_VIDEO + BACKGROUND ) //2[YES THIS IS SIMPLE FRAME WISE ADDITION ON PIXEL LEVEL]
This should be fast since this is a simple addition. So no latency.
Hope this helps,
thank you for listening to me,
p.s. I know it’s not my place to speak, you guys are already doing amazing work. Just thought to put my thoughts out there…
regards,