How to make your own Passthrough videos!

Sounds like a plan! I’d definitely be keen to test it out! :slight_smile: In the mean time i want to try working on 4D Gaussian

My ultimate goal is to really get to the next step of AR Porn, so to start I’d be using something like VAM to generate the input data to make into 4D Gaussians!

And then making something akin to heresphere in Unreal to display the 4D Gaussians; and if it goes well I might try hiring some professionals to produce some videos with the camera set-up for it as a proof-of-concept

1 Like

Updated Github repo with a new change!

Was experiencing some choppiness trying to playback an 8K passthrough that i had proccessed, and it turns out that the videowriter was writing to H264 instead of HEVC (H265) and since most VR videos are gonna be more than 2K anyway (maximum resolution of H264 at 60 fps) I decided to permanently set the codec to H265

1 Like

I only have an AMD7900XTX graphics card, can’t I use it?

hmm unfortunately not!

Most AI/Neural Network/ML solutions rely on CUDA (which relies on having an Nvidia CUDA-capable GPU) this one included :frowning:

thanks T.T

1 Like

This might be obvious but it looks like a couple tweaks can be made to SETUP to make the install go smoother for newbies.

2 Likes

Oh wow
That’s cool
How do you like SLR PT AI videos Passthrough AR VR Porn Videos | SexLikeReal?

That’s how we got there

We also did comparison of different methods back in the days

1 Like

Oh yeah that’s a good catch!

totally forgot that sometimes people with cuda-capable GPUs may not have the cuda toolkit installed :slight_smile:

Updated!

Oh yeah I’m subbed to SLR! and I do like those AI passthrough videos!

They are pretty decent in a pinch but I find that a lot of fine detail such as hair gets lost when it comes to AI passthrough; and those fine details are really what makes it immersive (especially with 6 DOF that’s properly calibrated, it’s mindblowing)

I much prefer the SLR originals passthrough videos that were shot on greenscreens but even then the mattes don’t seem to have enough detail and can seem aliased or too feathered :frowning:

Just a thought; it might be a cheaper way to do passthrough videos by taking clean plates for stationary shots and processing them with BackgroundMattingV2 and then using the AI Passthrough model to matte out the male actors or handle situations where the camera is moving

I imagine it would probably require less studio setup like in the green screen videos but you retain very similar level of detail

Or you could even offer it as a service to other studios to convert their catalogue into high-fidelity passthrough videos; if you could have access to their RAW source footages or cleanplates :exploding_head:

1 Like

You should be able to make every video PT soon SLR Drive development logic - SLR Forum: VR porn videos and interactive sex toys

Checking on BackgroundMattingV2

1 Like

That’s cool!

So do you basically upload the video onto SLR drive, and then if it’s enabled to be passthrough; it will be processed offline on the SLR servers to be passthrough?

Or will the passthrough matting be performed in realtime as the video is streamed? :exploding_head:

it’s more like cloud streaming service

1 Like

Did you see the new segement anything model yet? Meta Segment Anything Model 2 Seems interesting

1 Like

Oh very cool!

Definitely will want to try that out too!

I expect that it might not handle hair or fine details as well as RVM or Background Matting V2 but i’d like to be proven wrong!

Thanks for the share! :slight_smile:

Your welcome man. My thinking was it could potentially be a lot faster then current methods and fully automatic. Also more precise for making sure the male is always removed properly (probably based on coordinates and labeling the male could be removable with a script). Also when/if Heresphere adds mask playback that is additional to the video (like playing a seperate alpha mask video) processing times should deminish much further still (no re-en coding). I bet the the segmentation could also be used by another model to refine the edges.

Would be cool to get to a point where we can just have a little app that processes evrything in your library without taking a lifetime, that leaves the source files in tact and where we can just toggle the mask on and off with a button in heresphere.

1 Like

Oh yeah it will be a really amazing thing to be able to remove the male properly!

Funny that you’ve mentioned the refining of the edges because i tried just that with the first SAM :smiley:

So i basically:

  1. Use a fast and quick segmentation model to create bounds by identifying the female
  2. Then sending those bounds into SAM to segment her specifically
  3. Then i would send it into:
    Entity/High-Quality-Segmention/README.md at main · qqlu/Entity · GitHub
    to refine the mask!

It failed most of the time because the first 2 steps would fail sometimes (either failing to identify the female due to VR distortion or failing to generate a good enough initial mask) but when it did not fail, the result was great!

Some ways to fix it would be to split and undistort both views of the VR video using ffmpeg; then processing on those clips which may give a better result! or to train a version of SAM that’s properly tuned to nude bodies or to porn frames

About using the mask directly without encoding, I’d suggested it to the heresphere creator before, not sure if it’s something in his backlog right now, but i hope it is! :slight_smile:

Because we could potentially create a new market for passthrough masks (just like funscripts)
Not too clear about the legality of it though but technically it is not supplanting the original video

For the little app, I believe SLR is working on something like that that will allow you to upload your own videos, and it should process the video using AI for funscripts and passthrough through their server?

But wow doing that in an open-source way will be cool though!

Yeah, cool stuff, cool stuff. I bet it won’t take long before this gets practical and good enough for mass generation at home of full length videos.

About SLR, they have done a great job improving their AI passthrough (I love it!) and SLR drive seems very interesting (passthrough and funscript generation too). The dick in dick out toggle is also an awesome addition.

But being able to generate this with a script automatically at home too and just processing everything you add to your library folders at night automatically and pausing generation once you wanna use your pc again seems like the dream as I don’t think automating this for SLR drive for all your videos will be feasible. You’d probably have to cherry pick, due to storage, upload speed, cost and processing constraints.

And like you said maybe they could be shared similar to funscripts (I think the decoders of a quest can only handle somewhat downscaled two color video to run in parallel so the size shouldn’t be that big). At some point in the future VR companies will probably start releasing the masks for download too. Just don’t see that happening anytime soon.

Just read an update about SLR drive btw, doublevr said first version in October. Don’t know if that will have AI passthrough and/or AI funscripts yet though.