I am upset by the state of the passthrough. So i decided to take matters into my own hands. This is the proof of concept that studios dont need any green screens or global alpha matting for implementing passthrough. This works on a standard RGBD video shot from a go pro camera. Following the philosophy of @Tempest , i am open sourcing the code for this. Knowledge deserves to be free for all, and anyone should be free to commercialize from it.
@doublevr @Realcumber sir, i admire your efforts towards this community. i hope this saves you guys some production costs. This thing took me < 30 mins to write, but for some reason noone seems to have done this yet.
Credits go to facebooks Maskformer2.
Detail:
[1] I show a pair of frames. First frame is a standard frame from a czechvr video
[2] Second frame is the instance segmentation mask of several objects in the scene. The neural network knows which of the segmentation mask belongs to which class. I am showing that as a different colour in the image. Note that the frame is downsampled to 1024, by 1024 so the aspect ratio is screwed. But, i am getting these results on a single 8 GB Pascal 1080 card.
[3]Also, i show that this works across variety of sexual positions and can segment BOTH male actor, and female actor separately even if their bodies touch each other.
For reproducing the code, go to maskformer2 github repo, and follow the demo jupyter notebook .
I thank the authors of this amazing paper. As someone correctly said, the best things in the life are free.
@doublevr @Realcumber , i hope this helps your noble passthrough efforts. The only thing remaining now
[1] take alpha masks i am giving. crop the girl out.
[2] take any slr video. combine mask+video as simple alpha-matting. a video has multiple resolutions in slr. use lowest one. and upsample it to 8k videos. The effect will be same.
[3] let the background of the video be the user’s own background. That already comes from the guardians feed in quest 2.
sincerely,
bashturtle
get_seg.txt (1.5 KB)
