New Better Method for Making Passthrough Videos!

[throwflmfp]

I found this on a reddit post and then worked with claude to add extensive improvements as well as making it Windows friendly.

It utilizes SAM3 and MatAnyone2 machine learning models to identify the model shape and create a very good mask of the subject for incredibly good passthrough VR video creation. Videos can then be watched on Heresphere without issue with passthrough enabled. It also handles clothed subjects just fine! So cosplay enthusiasts rejoice!

To be 100% clear, the SAM3 and MatAnyone2 scripts DO NOT add any generative AI content, they are finetuned machine learning models created to detect and mask, in this case, human subjects. The SAM3 woman detection model here is pretty unreal in it’s ability to detect and mask the female model/subject, even clothed.

Requires CUDA enabled cards with CUDA 12.8, PyTorch 2.6 and Python 3.12.

Reddit Post - https://www.reddit.com/r/oculusnsfw/comments/1t5li6v/i_made_a_free_masking_tool_to_turn_vr_videos_into/

Github - GitHub - garrrrido/vr-masking-tools: Automated pipeline to generate alpha matte masks of a subject and pack them into Fisheye190 VR video. Uses MatAnyone2 for masking, and a finetuned SAM3 to automate person detection and first-frame mask generation · GitHub

My Changes (With Claudes Help)
Now supports failure recovery and error log creation.

• You can stop and restart really long jobs without having to redo every previous step.
  Supports dual eye frame processing for segments.
• Requires pretty beefy cards with a lot of VRAM.
• Best speeds with 5090, 4090 and 3090s with their 24+gb of VRAM.
  Automatic video frame identification for perfect syncing between video and mask.
  Smart transition detection so areas of black are not created if the model is visible between both frames of a fade or cut.
  Full Windows Support
• requires installation of triton through python
  One script does everything
• converts 180 VR video to fisheye, creates the masking, and puts the video back together.
  Fixed a lot of bugs in the previous version
  Batch files for simple drag and drop video conversion.

Context, I had tried making passthrough videos previously with the scripts made by @blewClue215 and while okay, could never really get to the quality of passthrough videos seen on SLR. SLR for a brief period of time allowed members to create their own passthrough videos but then killed the service shortly after, so I was looking for a better way to create passthrough videos from my existing library. I stumbled on the reddit post and started hacking my way through the scripts with Claudes help to add Windows support and I was able to do that while also improving the pipeline massively and making it easier to work with.

Directions - windows_setup.txt
WINDOWS_SETUP.txt (14.5 KB)

I would ignore the instructions in the original linked github as it’s setup for linux or wsl2 environments specifically. Please follow the instructions in this document. They should work as this is what I did to get it working on my machine.

My New Scripts - The following scripts were heavily edited by Claude to fix a lot of bugs and add features that I suggested to make the process way easier and not require a linux or WSL2 environment to run.

pipeline.txt (84.2 KB)
ffmpeg_utils.txt (22.3 KB)
matanyone_runner.txt (4.0 KB)
sam3_utils.txt (10.9 KB)
inference_matanyone2.txt (9.0 KB)

pipeline.txt - grab the file, drop it into your vr-masking-tools directory, delete the existing pipeline.py file and change the file extension of the txt file to .py

utils/ffmpeg_utils.txt - grab the file, drop it into your vr-masking-tools/utils directory, delete the existing ffmpeg_utils.py file and change the file extension of the txt file to .py

utls/matanyone_runner.txt - grab the file, drop it into your vr-masking-tools/utils directory, delete the existing matanyone_runner.py file and change the file extension of the txt file to .py

utils/sam3_utils.txt - grab the file, drop it into your vr-masking-tools/utils directory, delete the existing sam3_utils.py file and change the file extension of the txt file to .py

inference_matanyone2.txt - grab the file, drop it into your vr-masking-tools/MatAnyone2 directory, delete the existing inference_matanyone2.py file and change the file extension of the txt file to .py

Batch Files
run.txt (580 Bytes)
run_fast.txt (625 Bytes)
unlock_nvenc.txt (1007 Bytes)
run.txt - drop this into your vr-masking-tools folder and change the extension from .txt to .bat - this starts a normal conversion run on the video file you drop onto it.

runfast.txt - drop this into your vr-masking-tools folder and change the extension from .txt to .bat - this starts a super optimized conversion run on the video file you drop onto it. best for testing the pipeline with a much smaller 1 to 2 minute to clip to test that its working and that the output is acceptable.

unlock_nvenc.txt - drop this anywhere and change the .txt to .bat - this will likely require administrator privileges. it modifies the registry to remove the nvenc session limit that is hardcoded for all nvidia video cards.

Support Note - I will do my best to support folks in here, but I am not a programmer, so I likely won’t be able to fix actual programming issues if they pop up. Also remember that Claude worked to try and make this as optimal for my 3090 machine as possible. You’re likely to struggle if you have an old nvidia card or have less than 8gb of VRAM.

AMD video cards are not supported, this requires Nvidia cards from the 3xxx generation and forward afaik.

Good luck! I may add two small 1 minute clips later so folks can get a preview of what this pipeline is capable of! It legitimately shocked me how good it was! I was able to fully convert an older 4k VRCosplayX video overnight in about 4-5 hours and the quality is pretty good! The only issue with all of this is that it converts everything into fisheye190. But if you’re of the coder type and can figure out how to make this work without doing that, feel free to upgrade the script!

DO NOT SELL ANYTHING MADE WITH THIS! - This is using the MIT Matlab license which is free to use for personal use, reach out to them as instructed in the original github posted if you’re interested in commercial use.

I am not responsible for anything that could happen to your GPU when running this pipeline. It’s more intense than any sort of gaming experience for it’s continuous use. A more detailed disclaimer is included in the windows setup document.

[throwflmfp]

Small 10 second clip example
original

converted with alpha mask

trust me when I say this doesn’t do it justice, please try it out for yourself with a smaller 1-2 minute video clip and try it out in VR before you engage in a 4-8 hour overnight conversion journey. but notice how the mask is even grabbing the clothing?

[Caranthir87]

Nicely done! Had to do some tweaks to get it working on my end, and added a command to skip conversion if the file already was in Fisheye190, but all in all a job well done, sir!

[coping]

thanks for sharing, works great, I have created additional pipeline to create Heresphere external alpha file to not touch the original video, with initial tests the mask looks almost perfect if the high resolution mask used, just sometimes observing some object detection inconsistency between eyes (i.e. one object is visible on one eye and not on other), but thats minor

[throwflmfp]

oh interesting, do you still have to convert to fisheye? or are you able to keep the original resolution?

[coping]

no, I don’t have to convert, thats the benefit of the external mask, the video can stay original quality and format (i.e. also 180° equirectangular)

[Caranthir87]

Mind sharing a bit more? Would love to try this out

[stustustustu]

Not having much luck getting this working on my end. When I get to ‘Step 9 - Verify’ I get ‘ModuleNotFoundError: No module named ‘matanyone2’’.

I’ve looked in the ‘.venv312\lib\site-packages’ folder and confirmed matanyone2.pth was created.

Back on Step 2 when cloning the MatAnyone2 git it doesn’t want to run because there’s already a folder called MatAnyone2 (it’s in the vr-masking-tools git) I’ve tried cloning it to a different folder then copying the contents into the existing one (I’ve also tried not doing this and running with just the contents from the vr-masking-tools git).

If I ignore the failed check and try run it anyway it gets to SAM3 MASK GENERATION and gives an error it gives an error about my card (RTX 5080) not compatable with the current PyTorch version.

[TITAAN]

Already got it to work. it says I have 11 hours left. I did not test try like he suggested. I gave it a full harem JAV scene with 7+ woman. lets see if its as good as he says it is.

SPECS:

13600kf
RTX 4070
32 GB ram 6000mhz
2 SSDs 400+ gb
1 HDD western digital 7200RPM

I got a Lian li AIO gallahand II two so if that helps in the cooling. my case is for show its one of those fish tank styled lian li evo case.

7 fans.

if my rig can passthrough 7+ JAV hoes in 8K ~ 58 mins.

this is the real deal and this guy is a legend. lets see tmr if its accurate as I also used the @blewClue215 previously but that is shiit!

[stustustustu]

Weird, maybe you already had a newer version of torch installed?

Regardless I got mine working. If anyone else runs into the same problem as me give this a shot. Uninstall torch (make sure you are in the environment).

pip uninstall torch torchvision torchaudio -y

Then install a newer release with.

pip install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/cu132

Mine is working fine after doing that.

[TITAAN]

yeah most likely u didn’t have @blewClue215 yet. and when trying to improve his files with AI, it eventually led me to getting the latest models.

like I said I used @blewClue215 pipeline but was really disgusted by the results. despite days of efforts it still rendered out the ladies and killed immersion.

I will update when this is done based on my time line

[TITAAN]

also u have a RTX5080.

can u run ur pc normally without screen freezes? if u mind to look at my set up above. is it due to my gpu lacking the VRAM?

or u just have way more better specs than mine?

[stustustustu]

No screen freezes here while it’s running, but I’ve only tested on a few 1-2 minute clips so far. I’ve also only tested on scenes with only 1 girl. I wouldn’t try launching a game or something at the same time though…

My specs are higher than yours, could be the VRAM (12gb vs my 16gb), I’ve also got 64gb of system RAM. No idea how your CPU compares to my 9800X3D.

You’ve got it running on one of the SSD’s I assume, I doubt anything like this will run well off a HDD.

[TITAAN]

yes I’m running it off my SSD.

I can’t even watch the news or run twitter, essentially I can even web browse… I have a secondary 1080p tho so at least im not suffering too much

I am just more curious on the result, seems like I will be one of the firsts to complete a full:

8K | 55 mins+ | JAV Harem |
more than 7+ woman (MDVR388) (I used JAVDB and magnet to my PikPak storage, a cloud storage which has a pc and mobile app. (I pay about $ 4 USD for 1 TB cloud storage) u can essentially skip the download waiting time from utorrent or BitTorrent and view the content, I use this to pirate movies.)

ANYWAYS. I assume I am really limit testing my 4070 to his limits.

will update tmrw when its done

[stustustustu]

Too late now since you are well into this, but I would have clipped out a part of that to test with (can do that easily with lossless cut, there’s accurate cut included with this too but I already use lossless cut). Heck I’m not even planning on making passthroughs of full videos, just the ~10 minutes I actually want.

[TITAAN]

Usually these JAV vids are cut in 50 mins sections A B C D E.

the scene I got and is processing right now is the multiple woman VS 1 guy scene (a proper Harem for kings, I think there are like 10 or more of these maps). which was 55 mins+ at 8K

Yeah I want the full exp. if u never heard of s1mpcity, I got a thread there. you can prolly see the final output cause I doubt I can post gofile links or equal here. (1 = i)

Hope this Thread owner is the passthrough messiah so we don’t need to wait for vrporn or SLR to produce content!

find me at s1mpcity brother :)

[stustustustu]

Something I would like to try and work out is a batch file to run multiple files back to back.

[stustustustu]

@TITAAN

NVDA (Nvidia) also announced they are removing gaming gpus from their revenue streams. that only tells me they are pivoting to AI :(

They all but said this already when they announced the 5000 series that’s basically just the 4000 series with a bit of a performance bump and price rise, previous gens had much better jumps.

[TITAAN]

I’ll just gate keep. this worked. has some false processing which expect from my test of jabs, they always move around.

but I followed it up by a proffesioanla and stationary production in VRbangers.

search passthrough in s1mpcity. (1=i).

I don’t think I can post a gofile here and just name my progress without getting banned.

Once on the site and made an account just write “Passthrough VR”

LOOK FOR:

[coping]

still making changes, fixing problems, once it is stable I might create a repo fork