I don’t know what your goal is, but ShaggySusu’s stuff can generally be tracked and scripted in one go.
Hello,
Sorry for replying the form this late. Do you by chance have the add-on for the blender? I have been scripting with OFS, it have been super sloppy. Recently found this post and decided to give a try, but I am new to blender only was able to move the cylinder, just wondering if there is workflow of scripting with Blender.
Thanks
Actually, I had plans to make an add-on for Blender to hide all the necessary parameters and create a universal tool for using gamepads/joysticks/HTC Vive/and others. But somehow people aren’t interested, and local scripts are enough for me. Try the old script for the Xbox controller. A little later, I can post a universal script that I used with the PS5 Access controller.
1 Like
I am only scripting with xbox controller at the moment, but thank you for all the insightful information.
Couple question, I am a bit confused in the importing the x, y, and rotation part of the keyframes. Coming from OFS, you script them individually most of time. Therefore do I need to create an empty script such as sway, surge, pitch and roll from OFS for blender to able to import them as x, y, rotation keyframes or can you create them inside of the blender itself?
Additionally for your workflow, do u record keyframes frame by frame (move joystick to position and save), or do you script them real time (or 0.5x speed) when you move joystick while saving the keyframes?
First, create the main script with up-down movements, then import it into Blender using key points for the cylinder. Next, for each frame, you need to set the XY rotations, XY movements, and Z rotations. After that, you can export all the movement rotations as separate scripts for SR6.
1 Like
I had some free time and came across an interesting article (to be honest, I had read it several times before, but finally understood it).
After spending a week tinkering with math and Python with MediaPipe, I marked up and trained my neural network—it looks promising as an assistant for writing multi-axis scripts based on VR video.
I still need to think about the logic. I don’t know yet.
It will be interesting to run the scenes through it, then export them to Blender and manually finish the scenes that are difficult for AI.
In the worst case, if such a tool does 70-80% of the work for me, I will be very happy. It’s much easier and faster to do 10 minutes frame by frame than an hour.
Just sharing.
2 Likes
While I’m thinking about the following algorithm for working with the utility:
- We view the video (I have now added the ability to transcode from Fisheye and Half equirectangular projections with a choice of angles 180/190/200 degrees) with Fisheye 120 degrees output with angles -30/0/ +30 and flat 60x60 degrees with angles -30/-15/0/+15/+30 degrees from the current one, the re-editing is done as skips of 2-4-10 frames, i.e. we display every 1-3-5th or 11th frame.
- Select the video segment you need. There are text fields for entering the start and end of the segment and buttons for -10/-1/+1/+10 frames.
- Select the method (action that occurs) and click the Run Recognition button.
- Algorithms with neural networks tailored specifically for this method recognize what is happening in the video with the specified angles for projections and write the correct vector and quaternion relative to the center of coordinates to a file (I think it is best to immediately consider the center as the base of the penis).
- The raw data file can be imported into Blender for manual editing.
Suggestions and comments are welcome—while I still have the motivation, I think I’ll try to implement the blowjob method. Theoretically, any existing VR methods can be added in the future. So far, I’ve tried object recognition (MobileNetV2, trained my own - needs to be redone), hands (Mediapipe), faces (Mediapipe/YuNet), poses (Mediapipe/MoveNet/RTMPose) - I haven’t encountered any particular difficulties or obstacles. On an old test PC (i5 6500t), playing images with a resolution of 3840x1920 runs at a speed of about 30 fps, face recognition at 20 fps, pose/hand/object recognition at 10-15 fps, and when using a pair of neural networks simultaneously at 8-10 fps - I tried hands+face and poses+objects.
The very first test. Everything is crooked and crooked, but it works!
I need to seriously rethink the orientation—right now it’s complete crap. I need to add filtering/interpolation, etc. But damn, it works
3 Likes
The standard approach to object recognition (where the neural network draws bounding boxes) doesn’t work well for my use case - since we don’t get the length/thickness/orientation of the penis, we can’t calculate its 3D coordinates in space or its interaction with the surrounding world - what’s the point of face/body/hand coordinates if we don’t know what they’re doing?
I spent a lot of time studying segmentation; early screenshots are below - it looks good.
Sure, there’s some outright trash out there, but I can work with that and use it to improve the neural network, the math, and so on.
1 Like
I ran a test video through the neural network - and was disappointed - it was running at about 4 fps and had poor recognition (I trained it on about 300 images). I added more images and annotated them (about 600), then redesigned the neural network architecture using MobileNet + U-Net - the speed and recognition improved. I’ve attached the test video with a compilation from VRHush. I probably need a couple hundred more images for complex frames involving hands, and then I can apply geometric calculations.
So, here’s what I have:
- Calibration (specify a reference point for calculations). This point will serve as the origin for multi-axis scripts.
- Penis tracking (find the center using a mask). This point calculates the current position (angles of inclination). I also want to add a calculation for average length.
- Hand tracking (vector between the knuckles from the index finger to the little finger). Position and rotations of the stroker on the penis for masturbation scenes.
- Face tracking (several points and vectors; we’re interested in the vector from the center of the mouth, perpendicular to the eyes). Position and rotations of the stroker for blowjob scenes.
All that’s left is to write this data to a file, filter it, interpolate it, and export it to Blender. In principle, you could write a multi-axis script right away—as long as there aren’t any major errors in the movements (so nothing gets torn off).
3 Likes