Sounds like a plan! I’d definitely be keen to test it out! In the mean time i want to try working on 4D Gaussian
My ultimate goal is to really get to the next step of AR Porn, so to start I’d be using something like VAM to generate the input data to make into 4D Gaussians!
And then making something akin to heresphere in Unreal to display the 4D Gaussians; and if it goes well I might try hiring some professionals to produce some videos with the camera set-up for it as a proof-of-concept
Was experiencing some choppiness trying to playback an 8K passthrough that i had proccessed, and it turns out that the videowriter was writing to H264 instead of HEVC (H265) and since most VR videos are gonna be more than 2K anyway (maximum resolution of H264 at 60 fps) I decided to permanently set the codec to H265
Oh yeah I’m subbed to SLR! and I do like those AI passthrough videos!
They are pretty decent in a pinch but I find that a lot of fine detail such as hair gets lost when it comes to AI passthrough; and those fine details are really what makes it immersive (especially with 6 DOF that’s properly calibrated, it’s mindblowing)
I much prefer the SLR originals passthrough videos that were shot on greenscreens but even then the mattes don’t seem to have enough detail and can seem aliased or too feathered
Just a thought; it might be a cheaper way to do passthrough videos by taking clean plates for stationary shots and processing them with BackgroundMattingV2 and then using the AI Passthrough model to matte out the male actors or handle situations where the camera is moving
I imagine it would probably require less studio setup like in the green screen videos but you retain very similar level of detail
Or you could even offer it as a service to other studios to convert their catalogue into high-fidelity passthrough videos; if you could have access to their RAW source footages or cleanplates
So do you basically upload the video onto SLR drive, and then if it’s enabled to be passthrough; it will be processed offline on the SLR servers to be passthrough?
Or will the passthrough matting be performed in realtime as the video is streamed?
Your welcome man. My thinking was it could potentially be a lot faster then current methods and fully automatic. Also more precise for making sure the male is always removed properly (probably based on coordinates and labeling the male could be removable with a script). Also when/if Heresphere adds mask playback that is additional to the video (like playing a seperate alpha mask video) processing times should deminish much further still (no re-en coding). I bet the the segmentation could also be used by another model to refine the edges.
Would be cool to get to a point where we can just have a little app that processes evrything in your library without taking a lifetime, that leaves the source files in tact and where we can just toggle the mask on and off with a button in heresphere.
It failed most of the time because the first 2 steps would fail sometimes (either failing to identify the female due to VR distortion or failing to generate a good enough initial mask) but when it did not fail, the result was great!
Some ways to fix it would be to split and undistort both views of the VR video using ffmpeg; then processing on those clips which may give a better result! or to train a version of SAM that’s properly tuned to nude bodies or to porn frames
About using the mask directly without encoding, I’d suggested it to the heresphere creator before, not sure if it’s something in his backlog right now, but i hope it is!
Because we could potentially create a new market for passthrough masks (just like funscripts)
Not too clear about the legality of it though but technically it is not supplanting the original video
For the little app, I believe SLR is working on something like that that will allow you to upload your own videos, and it should process the video using AI for funscripts and passthrough through their server?
But wow doing that in an open-source way will be cool though!
Yeah, cool stuff, cool stuff. I bet it won’t take long before this gets practical and good enough for mass generation at home of full length videos.
About SLR, they have done a great job improving their AI passthrough (I love it!) and SLR drive seems very interesting (passthrough and funscript generation too). The dick in dick out toggle is also an awesome addition.
But being able to generate this with a script automatically at home too and just processing everything you add to your library folders at night automatically and pausing generation once you wanna use your pc again seems like the dream as I don’t think automating this for SLR drive for all your videos will be feasible. You’d probably have to cherry pick, due to storage, upload speed, cost and processing constraints.
And like you said maybe they could be shared similar to funscripts (I think the decoders of a quest can only handle somewhat downscaled two color video to run in parallel so the size shouldn’t be that big). At some point in the future VR companies will probably start releasing the masks for download too. Just don’t see that happening anytime soon.
Just read an update about SLR drive btw, doublevr said first version in October. Don’t know if that will have AI passthrough and/or AI funscripts yet though.