I don’t know what your goal is, but ShaggySusu’s stuff can generally be tracked and scripted in one go.
Hello,
Sorry for replying the form this late. Do you by chance have the add-on for the blender? I have been scripting with OFS, it have been super sloppy. Recently found this post and decided to give a try, but I am new to blender only was able to move the cylinder, just wondering if there is workflow of scripting with Blender.
Thanks
Actually, I had plans to make an add-on for Blender to hide all the necessary parameters and create a universal tool for using gamepads/joysticks/HTC Vive/and others. But somehow people aren’t interested, and local scripts are enough for me. Try the old script for the Xbox controller. A little later, I can post a universal script that I used with the PS5 Access controller.
1 Like
I am only scripting with xbox controller at the moment, but thank you for all the insightful information.
Couple question, I am a bit confused in the importing the x, y, and rotation part of the keyframes. Coming from OFS, you script them individually most of time. Therefore do I need to create an empty script such as sway, surge, pitch and roll from OFS for blender to able to import them as x, y, rotation keyframes or can you create them inside of the blender itself?
Additionally for your workflow, do u record keyframes frame by frame (move joystick to position and save), or do you script them real time (or 0.5x speed) when you move joystick while saving the keyframes?
First, create the main script with up-down movements, then import it into Blender using key points for the cylinder. Next, for each frame, you need to set the XY rotations, XY movements, and Z rotations. After that, you can export all the movement rotations as separate scripts for SR6.
1 Like
I had some free time and came across an interesting article (to be honest, I had read it several times before, but finally understood it).
After spending a week tinkering with math and Python with MediaPipe, I marked up and trained my neural network—it looks promising as an assistant for writing multi-axis scripts based on VR video.
I still need to think about the logic. I don’t know yet.
It will be interesting to run the scenes through it, then export them to Blender and manually finish the scenes that are difficult for AI.
In the worst case, if such a tool does 70-80% of the work for me, I will be very happy. It’s much easier and faster to do 10 minutes frame by frame than an hour.
Just sharing.
3 Likes
While I’m thinking about the following algorithm for working with the utility:
- We view the video (I have now added the ability to transcode from Fisheye and Half equirectangular projections with a choice of angles 180/190/200 degrees) with Fisheye 120 degrees output with angles -30/0/ +30 and flat 60x60 degrees with angles -30/-15/0/+15/+30 degrees from the current one, the re-editing is done as skips of 2-4-10 frames, i.e. we display every 1-3-5th or 11th frame.
- Select the video segment you need. There are text fields for entering the start and end of the segment and buttons for -10/-1/+1/+10 frames.
- Select the method (action that occurs) and click the Run Recognition button.
- Algorithms with neural networks tailored specifically for this method recognize what is happening in the video with the specified angles for projections and write the correct vector and quaternion relative to the center of coordinates to a file (I think it is best to immediately consider the center as the base of the penis).
- The raw data file can be imported into Blender for manual editing.
Suggestions and comments are welcome—while I still have the motivation, I think I’ll try to implement the blowjob method. Theoretically, any existing VR methods can be added in the future. So far, I’ve tried object recognition (MobileNetV2, trained my own - needs to be redone), hands (Mediapipe), faces (Mediapipe/YuNet), poses (Mediapipe/MoveNet/RTMPose) - I haven’t encountered any particular difficulties or obstacles. On an old test PC (i5 6500t), playing images with a resolution of 3840x1920 runs at a speed of about 30 fps, face recognition at 20 fps, pose/hand/object recognition at 10-15 fps, and when using a pair of neural networks simultaneously at 8-10 fps - I tried hands+face and poses+objects.
The very first test. Everything is crooked and crooked, but it works!
I need to seriously rethink the orientation—right now it’s complete crap. I need to add filtering/interpolation, etc. But damn, it works
4 Likes
The standard approach to object recognition (where the neural network draws bounding boxes) doesn’t work well for my use case - since we don’t get the length/thickness/orientation of the penis, we can’t calculate its 3D coordinates in space or its interaction with the surrounding world - what’s the point of face/body/hand coordinates if we don’t know what they’re doing?
I spent a lot of time studying segmentation; early screenshots are below - it looks good.
Sure, there’s some outright trash out there, but I can work with that and use it to improve the neural network, the math, and so on.
1 Like
I ran a test video through the neural network - and was disappointed - it was running at about 4 fps and had poor recognition (I trained it on about 300 images). I added more images and annotated them (about 600), then redesigned the neural network architecture using MobileNet + U-Net - the speed and recognition improved. I’ve attached the test video with a compilation from VRHush. I probably need a couple hundred more images for complex frames involving hands, and then I can apply geometric calculations.
So, here’s what I have:
- Calibration (specify a reference point for calculations). This point will serve as the origin for multi-axis scripts.
- Penis tracking (find the center using a mask). This point calculates the current position (angles of inclination). I also want to add a calculation for average length.
- Hand tracking (vector between the knuckles from the index finger to the little finger). Position and rotations of the stroker on the penis for masturbation scenes.
- Face tracking (several points and vectors; we’re interested in the vector from the center of the mouth, perpendicular to the eyes). Position and rotations of the stroker for blowjob scenes.
All that’s left is to write this data to a file, filter it, interpolate it, and export it to Blender. In principle, you could write a multi-axis script right away—as long as there aren’t any major errors in the movements (so nothing gets torn off).
4 Likes
Tried using this so far and it works well for me but I did find that I got this error when running the export code.
any fixes? im on blender 5.0 btw.
I’m not at my computer right now, so I can’t help you or check right away, but starting with version 5, they did make some changes to the Python API. If you’re familiar with Python, you could try making the changes based on the example in the post below.
Or you can install a copy of Blender 4.x alongside it (you can install many different versions side by side) and export from there.
1 Like
I recently dug out my VR goggles, watched a couple of videos, and ran a couple of scripts. I want to get back into the community ;) After all, writing scripts and tinkering with math and stuff is a fun process.
After experimenting with neural networks, I started annotating a neural network for anatomy (the first phase with a penis on MobileNet + UNet was pretty successful), and I finished annotating about 1,500 images, trained the neural network, and tweaked the app a bit.
Here’s what it looks like and how it works now (of course, it’s not perfect).
P.S. Actually, the video stuttering is caused by a problem with capturing the screen using standard tools in Ubuntu. On my old, underpowered laptop, with standard detection, I get about 10 fps with geometry recalculation.
1 Like
I might be a bit late to this, but if u wanna segmentation without fine tune the model. Current base benchmark is SAM, and also check it out SAM 3D Body by Meta Research I was working on it a while ago, but kinda give up cuz other life events, but u might be able to work out something…
1 Like
It looks interesting - it’s worth a try.
Here’s what I’ve figured out so far while exploring the theoretical potential of neural networks for scripting:
- All pose detectors are configured to assume that the image is flat and that the person occupies almost the entire frame.
In this context, for VR video, the person is positioned very close to the camera - between 30 and 200 cm away. The neural networks start to go haywire. - Pose-recognition neural networks are trained on 17 points. Based on these points, it’s impossible to calculate the geometry of the pelvis and thoracic region, or the relative positions of these parts in three-dimensional space.
- Neural networks for image recognition do not construct object boundaries, but only a frame around them. That’s why I switched from frames to boundaries using UNet.
The only projection that’s more or less suitable for poses is the stereographic projection with a 150-degree field of view. Some poses (sitting upright facing forward, sitting upright facing backward) actually work quite well.
My current plan is to fine-tune the pose recognition neural network on specific poses for scripting.
To recognize body regions, hands, and the face, we need accurate 2D crops - which is exactly why we have cropping and view angle selection.
That’s precisely why we need to add regions to UNet that will allow us to reconstruct the correct 3D position and rotations of the torso, pelvis, and penis.
Right now, using the neural networks we already have (my MobileNet + U-Net, MediaPipe Hands/Face), anyone can create single-axis scripts, and I plan to polish this feature and release a test app for everyone.
While tinkering around with the ability to build scripts directly from video, I ran into a few issues, as usual. I found a way to fine-tune Google’s neural networks (MediaPipe Face/MoveMet).
So far, I’ve outlined the following:
- Segmentation using a new, completely different logic for dividing regions - about 250 images.
- Poses in a 150-degree stereo projection - 650 images; I’ve already performed a test tuning of MoveNet on this dataset.
- Face - 500 images; I’ve performed a tuning of BlazeFace (turquoise is the standard network, orange is the tuned version)..
Overall, the process is underway, but I already much prefer the visual results for poses and faces after the tuning.
I discovered an interesting bug that can be turned into an advantage.
When having sex with breasts, the pose-detection neural network (MoveNet), due to its internal mechanism (estimating the human frame, finding the center of gravity, and reconstructing the pose based on it), places the hip joint points at the tips of the breasts - meaning you essentially have a ready-made recognition algorithm for titfucking.
I spent almost a week working on a solver for sex scenes in the cowgirl position. I’m finally happy with the result. With these 3D measurements, I can move forward with the project.
4 Likes
True craftsmen make something first and then rework it. It’s a process, basically.
On the left is the flat crop we worked on:
- Our own MobileNet + UNet neural network.
- A tuned MoveNet.
- A tuned HITNet.
On the right is a point cloud rendering and lots and lots of math in Blender.
3 Likes