Using Blender as a multi-axis script editor

Preface.
After writing a few single-axis scripts and wanting to try making my own devices (I’m still gathering money for a 3D printer, as the one I got for free has only 100x100x100 mm print area) it was interesting to try writing multi-axis scripts for interactive masturbators. As for me, just knowing how these scripts play and work on devices can give me a better understanding of how to make them better. Although a great tool like OFS can create multi-axis scripts, I decided to do something alternative for several reasons:

  • I definitely don’t like the recording format of real multi-axis scripts (lots of motion and rotation scripts).
  • I doubt that it’s mathematically correct to use Euler angles (it’s hard to use interpolation, what order to use when rotating - the result will be different, the presence of the “hinge lock”).
  • I didn’t like the toolkit that OFS has (maybe I’m just clumsy).
  • There must have been other reasons - can’t remember why :slight_smile:

I’ve been using Blender as a program for parametric drawing (similar to AutoCAD, SolidWorks) and for modeling 3D models for printing. Luckily this program can also do a lot of things with video and of course I couldn’t help but try to use it as a script editor.

Introductions.
I’m making scripts only for VR video so far and prepare flat video for scripting with ffmpeg command line:
ffmpeg -i video.mp4 -vf "crop=in_w/2:in_h:in_w:in_h, v360=input=hequirect:output=flat:pitch=-40:h_fov=100:v_fov=100:w=1024:h=1024, fps=30" output.mp4
Where:

  • ffmpeg - the utility itself (the full path may look like “C:\ffmpeg\bin\ffmpeg.exe”)
  • video.mp4 - incoming video file in VR format, for example the full path is “D:\VR\VRHush_From_The_Vault_Dani_Daniels_Oculus_HQ_3D_LR_180.mp4”
  • crop=in_w/2:in_h:in_w:in_h - trim the video, leave only the left half.
  • v360=input=hequirect:output=flat:pitch=-40:h_fov=100:v_fov=100:w=1024:h=1024 - converting video from the half of equirectangular projection with rotation by -40 degrees, angles 100x100 degrees and output resolution 1024x1024 pixels
  • fps=30 - number of frames per second
  • output.mp4 - outputs the flat video to a file, for example the full path “C:\Temp\VRHush_From_The_Vault_Dani_Daniels_Flat_30.mp4”.

Usually it’s the flat file that I then work with in OFS and create a per-frame script for complex scenes and for simple ones I use MTFG with manual correction of each movement.

Create a multi-axis script.
In Blender you can import our flat video as a sequence of frames:

  • add a Empty image
  • in the Property area select Object Data Properties to open Image
  • in the file selection window select our video made for OFS
  • look through the duration of our video in the file properties and specify how many frames we will import using the formula (minutes*60 + seconds)*frame count
  • in the Timeline area we also specify the desired number of frames:
  • switch to the Y-axis view by clicking on the necessary icon or by pressing the “Numpad 1” key
  • rotate our set of pictures by an angle of 90 degrees on the X axis and press the “Space” key to watch the video:
  • choose the units of measure we will use for positioning (I’ve set cm)
  • let’s create our Fleshlight, which will be in 3D emulate the movements of a real masturbator
  • I put the length of 20 cm, the discrimeter of 5 cm (which roughly corresponds to a full-size) and 5 vertices at the base (to understand where its sides), immediately put his beginning was in the center of coordinates (Z 10)
  • set the point for calculating movements and rotations to the origin of coordinates (where the 3D cursor is):
  • to import our script (which is a JSON file), I made a simple script, for this you can switch one of the areas to the text editor
  • insert the python code and run it:
import json
import bpy

funscriptName = "C:\\Temp\\Dani Daniels\\VRHush_From_The_Vault_Dani_Daniels.funscript"
scene = bpy.context.scene
obj = bpy.data.objects["Cylinder"]
obj_loc = obj.location

with open(funscriptName, "r") as funscriptFile:
    funscriptJson = json.load(funscriptFile)
    for action in funscriptJson['actions']:
        obj.location = (0.0, 0.0, float(action['pos']/1000))
        obj.keyframe_insert(data_path="location", frame=round((action['at']/1000)*30))
  • and we get all the keyframes from our script in the Timeline area:

The resulting script in step 2 is only Z-axis motions. At a minimum, we need to add X and Y positions for each keyframe, and add rotations around the XYZ axes. And while adding motions and rotations around ready-made points is easier than writing scripts of 2 motions and 3 rotations from “0”, it’s time consuming and time-consuming to do 2-3 times as much as writing one motion axis as usual.
Very often rotations around Z do not correspond to keyframes (start earlier and end later), so such points need to be created separately.

Install python libraries in Blender.
To work correctly, you need to install some python libraries that Blender doesn’t have.
For example, installing the library for working with Xinput gamepads on Windows.

import subprocess
import sys
import os
 
# path to python.exe
python_exe = os.path.join(sys.prefix, 'bin', 'python.exe')
target = os.path.join(sys.prefix, 'lib', 'site-packages')
 
# upgrade pip
subprocess.call([python_exe, "-m", "ensurepip"])
subprocess.call([python_exe, "-m", "pip", "install", "--upgrade", "pip"])
 
# install required packages
subprocess.call([python_exe, '-m', 'pip', 'install', 'XInput-Python', '-t', target])

print("DONE")

Gamification.
For each keyframe with the Z-axis you need to set 5 different values.

After spending a few hours on the script, the obvious idea was to use the gamepad’s analog stick and trigger axes for movements and rotations. No sooner said than done, and here is the script:

import bpy
import XInput
import mathutils
import time
import random

obj = bpy.data.objects["fleshlight"]
reference = bpy.data.objects["reference"]
camera = bpy.data.objects["camera"]
area = next(area for area in bpy.context.screen.areas if area.type == 'VIEW_3D')

# button & axis mapping
prev_keyframe_btn = 'A'
next_keyframe_btn = 'B'
prev_frame_btn = 'X'
next_frame_bnt = 'Y'
anim_play_btn = 'START'
anim_cancel_btn = 'BACK'
multi_view_btn = 'RIGHT_THUMB'
Z_rotation_btn = 'LEFT_THUMB'
left_location_btn = 'DPAD_LEFT'
right_location_btn = 'DPAD_RIGHT'
forward_location_btn = 'DPAD_UP'
backward_location_btn = 'DPAD_DOWN'
save_rotation_btn = 'LEFT_SHOULDER'
save_location_btn = 'RIGHT_SHOULDER'
thumb_location = 0
thumb_rotation = 1

camera.hide_set(True)

prev_keyframe, next_keyframe, prev_frame, next_frame = False, False, False, False
save_location, save_rotation, anim_play, anim_cancel, enable_anim_play = False, False, False, False, False
forward_location, backward_location, left_location, right_location = False, False, False, False
multi_view, enable_multi_view, Z_rotation, enable_Z_rotation = False, False, False, False

class ModalTimerOperator(bpy.types.Operator):
    """Operator which runs itself from a timer"""
    bl_idname = "wm.modal_timer_operator"
    bl_label = "Modal Timer Operator"

    _timer = None
    
    def modal(self, context, event):
        global prev_keyframe, next_keyframe, prev_frame, next_frame
        global save_location, save_rotation, anim_play, anim_cancel, enable_anim_play
        global forward_location, backward_location, left_location, right_location
        global multi_view, enable_multi_view, Z_rotation, enable_Z_rotation
        if event.type in {'RIGHTMOUSE', 'ESC'}:
            self.cancel(context)
            return {'CANCELLED'}
        obj_pos_x, obj_pos_y, obj_rot_euler_x, obj_rot_euler_y, obj_rot_euler_z = 0.0, 0.0, 0.0, 0.0, 0.0
        if event.type == 'TIMER':
            state = XInput.get_state(0)
            if XInput.get_button_values(state)[anim_play_btn] and not anim_play and enable_anim_play:
                bpy.ops.screen.animation_cancel(restore_frame=False)
                enable_anim_play = False
                anim_play = True
            if XInput.get_button_values(state)[anim_play_btn] and not anim_play and not enable_anim_play:
                bpy.ops.screen.animation_play(reverse=False, sync=False)
                enable_anim_play = True
                anim_play = True
            if not XInput.get_button_values(state)[anim_play_btn] and anim_play: 
                anim_play = False
            if XInput.get_button_values(state)[anim_cancel_btn] and not anim_cancel and enable_anim_play:
                bpy.ops.screen.animation_cancel()
                enable_anim_play = False
                anim_cancel = True
            if not XInput.get_button_values(state)[anim_cancel_btn] and anim_cancel: 
                anim_cancel = False
            if XInput.get_button_values(state)[prev_keyframe_btn] and not prev_keyframe:
                bpy.ops.screen.keyframe_jump(next=False)
                prev_keyframe = True
            if not XInput.get_button_values(state)[prev_keyframe_btn] and prev_keyframe: 
                prev_keyframe = False
            if XInput.get_button_values(state)[next_keyframe_btn] and not next_keyframe:
                bpy.ops.screen.keyframe_jump(next=True)
                next_keyframe = True
            if not XInput.get_button_values(state)[next_keyframe_btn] and next_keyframe:
                next_keyframe = False
            if XInput.get_button_values(state)[prev_frame_btn] and not prev_frame:
                bpy.ops.screen.frame_offset(delta=-1)
                prev_frame = True
            if not XInput.get_button_values(state)[prev_frame_btn] and prev_frame:
                prev_frame = False
            if XInput.get_button_values(state)[next_frame_bnt] and not next_frame:
                bpy.ops.screen.frame_offset(delta=1)
                next_frame = True
            if not XInput.get_button_values(state)[next_frame_bnt] and next_frame:
                next_frame = False
            if XInput.get_button_values(state)[multi_view_btn] and not multi_view and not enable_multi_view:
                area.spaces[0].region_3d.view_perspective = 'CAMERA'
                reference.hide_set(True)
                camera.hide_set(False)
                enable_multi_view = True
                multi_view = True
            if XInput.get_button_values(state)[multi_view_btn] and not multi_view and enable_multi_view:
                area.spaces[0].region_3d.view_perspective = 'ORTHO'
                reference.hide_set(False)
                camera.hide_set(True)
                enable_multi_view = False 
                multi_view = True
            if not XInput.get_button_values(state)[multi_view_btn] and multi_view: 
                multi_view = False
            if XInput.get_button_values(state)[Z_rotation_btn] and not Z_rotation and enable_Z_rotation:
                enable_Z_rotation = False
                Z_rotation = True
            if XInput.get_button_values(state)[Z_rotation_btn] and not Z_rotation and not enable_Z_rotation:
                enable_Z_rotation = True
                Z_rotation = True
            if not XInput.get_button_values(state)[Z_rotation_btn] and Z_rotation: 
                Z_rotation = False
            if XInput.get_button_values(state)[forward_location_btn] and not forward_location:
                obj.location[1] += 0.005
                forward_location = True
            if not XInput.get_button_values(state)[forward_location_btn] and forward_location:
                forward_location = False
            if XInput.get_button_values(state)[backward_location_btn] and not backward_location:
                obj.location[1] -= 0.005
                backward_location = True
            if not XInput.get_button_values(state)[backward_location_btn] and backward_location:
                backward_location = False
            if XInput.get_button_values(state)[left_location_btn] and not left_location:
                obj.location[0] -= 0.005
                left_location = True
            if not XInput.get_button_values(state)[left_location_btn] and left_location:
                left_location = False
            if XInput.get_button_values(state)[right_location_btn] and not right_location:
                obj.location[0] += 0.005
                right_location = True
            if not XInput.get_button_values(state)[right_location_btn] and right_location:
                right_location = False
            if XInput.get_thumb_values(state)[thumb_location] and not enable_Z_rotation:
                obj_pos_x = XInput.get_thumb_values(state)[thumb_location][0] * obj.location[2]
                obj_pos_y = XInput.get_thumb_values(state)[thumb_location][1] * obj.location[2]
                if obj_pos_x != 0 or obj_pos_y != 0:
                    obj.location[0] = obj_pos_x
                    obj.location[1] = obj_pos_y
            if XInput.get_thumb_values(state)[thumb_location] and enable_Z_rotation:
                obj_rot_euler_z = XInput.get_thumb_values(state)[thumb_location][0] * 0.8
                #obj_rot_euler_y = XInput.get_thumb_values(state)[1][0] * 0.8
                if obj_rot_euler_z != 0:
                    obj.rotation_euler[2] = obj_rot_euler_z
            if XInput.get_thumb_values(state)[thumb_rotation]:
                obj_rot_euler_x = -XInput.get_thumb_values(state)[thumb_rotation][1] * 0.8
                obj_rot_euler_y = XInput.get_thumb_values(state)[thumb_rotation][0] * 0.8
                if obj_rot_euler_x != 0 or obj_rot_euler_y != 0:
                    obj.rotation_euler[0] = obj_rot_euler_x
                    obj.rotation_euler[1] = obj_rot_euler_y
            if XInput.get_button_values(state)[save_rotation_btn] and not save_rotation:
                if not enable_Z_rotation:
                    quat = obj.rotation_euler.to_quaternion()
                    vect = obj.location.copy()
                    vect[0], vect[1] = 0.0, 0.0
                    vect.rotate(quat)
                    obj.location[0] = vect[0] + random.uniform(-0.25, 0.25)*0.01
                    obj.location[1] = vect[1] + random.uniform(-0.25, 0.25)*0.01
                    obj.keyframe_insert(data_path="rotation_euler")
                    obj.keyframe_insert(data_path="location")
                    bpy.ops.screen.keyframe_jump(next=True)
                    save_rotation = True
                else: 
                    quat = obj.rotation_euler.to_quaternion()
                    vect = obj.location.copy()
                    vect[0], vect[1] = 0.0, 0.0
                    vect.rotate(quat)
                    obj.location[0] = vect[0] + random.uniform(-0.25, 0.25)*0.01
                    obj.location[1] = vect[1] + random.uniform(-0.25, 0.25)*0.01
                    obj.keyframe_insert(data_path="location")
                    bpy.ops.screen.keyframe_jump(next=True)
                    save_rotation = True
            if not XInput.get_button_values(state)[save_rotation_btn] and save_rotation:
                save_rotation = False
            if XInput.get_button_values(state)[save_location_btn] and not save_location:
                if not enable_Z_rotation:
                    obj.keyframe_insert(data_path="location")
                    bpy.ops.screen.keyframe_jump(next=True)
                    save_location = True
                else: 
                    obj.keyframe_insert(data_path="rotation_euler")
                    obj.keyframe_insert(data_path="location")
                    bpy.ops.screen.keyframe_jump(next=True)
                    save_location = True
            if not XInput.get_button_values(state)[save_location_btn] and save_location:
                save_location = False
        return {'PASS_THROUGH'}

    def execute(self, context):
        wm = context.window_manager
        self._timer = wm.event_timer_add(1/30, window=context.window)
        wm.modal_handler_add(self)
        return {'RUNNING_MODAL'}

    def cancel(self, context):
        wm = context.window_manager
        wm.event_timer_remove(self._timer)
        
def get_override(area_type, region_type):
    for area in bpy.context.screen.areas: 
        if area.type == area_type:             
            for region in area.regions:                 
                if region.type == region_type:                    
                    override = {'area': area, 'region': region} 
                    return override
    #error message if the area or region wasn't found
    raise RuntimeError("Wasn't able to find", region_type," in area ", area_type,
                        "\n Make sure it's open while executing script.")


def menu_func(self, context):
    self.layout.operator(ModalTimerOperator.bl_idname, text=ModalTimerOperator.bl_label)


def register():
    bpy.utils.register_class(ModalTimerOperator)
    bpy.types.VIEW3D_MT_view.append(menu_func)


def unregister():
    bpy.utils.unregister_class(ModalTimerOperator)
    bpy.types.VIEW3D_MT_view.remove(menu_func)


if __name__ == "__main__":
    register()
    bpy.ops.wm.modal_timer_operator()

(I’ll be finishing up)
I put moves on the left stick, rotations on the right stick, Z rotations on the trigger, keyframe navigation on the ABXY buttons, and a write script on the bumpers.
It looks interesting and the script creation speed has increased:
ezgif.com-gif-maker (6)

Script export.
The code itself

import json
import bpy
import math

funscriptName = "VRHush_From_The_Vault_Dani_Daniels"
folder = "C:\\Temp\\"

funscript_orig = folder + funscriptName + ".mono.funscript"

fs_pitch = folder + funscriptName + ".pitch" + ".funscript"
fs_roll = folder + funscriptName + ".roll" + ".funscript"
fs_yaw = folder + funscriptName + ".twist" + ".funscript"
fs_x = folder + funscriptName + ".sway" + ".funscript"
fs_y= folder + funscriptName + ".surge" + ".funscript"
fs_z= folder + funscriptName + ".funscript"

obj = bpy.data.objects["fleshlight"]

with open(funscript_orig, "r") as funscriptFile:
    funscriptJson = json.load(funscriptFile)
    fs_orig_inv = funscriptJson['inverted']
    fs_orig_metadata = funscriptJson['metadata']
    fs_orig_range = funscriptJson['range']
    fs_orig_version = funscriptJson['version']

obj_pitch = {"actions":[], "inverted": fs_orig_inv, "metadata": fs_orig_metadata, "range":fs_orig_range, "version":fs_orig_version}
obj_roll = {"actions":[], "inverted": fs_orig_inv, "metadata": fs_orig_metadata, "range":fs_orig_range, "version":fs_orig_version}
obj_yaw = {"actions":[], "inverted": fs_orig_inv, "metadata": fs_orig_metadata, "range":fs_orig_range, "version":fs_orig_version}
obj_x = {"actions":[], "inverted": fs_orig_inv, "metadata": fs_orig_metadata, "range":fs_orig_range, "version":fs_orig_version}
obj_y = {"actions":[], "inverted": fs_orig_inv, "metadata": fs_orig_metadata, "range":fs_orig_range, "version":fs_orig_version}
obj_z = {"actions":[], "inverted": fs_orig_inv, "metadata": fs_orig_metadata, "range":fs_orig_range, "version":fs_orig_version}
#obj_yaw = {"at": , "pos":}
#obj_x = {"at": , "pos":}
#obj_y = {"at": , "pos":}

cur_frame = 0
frames = obj.animation_data.action.fcurves[2]
for kf in frames.keyframe_points:
    frame = int(kf.co[0])
    print(frame)
    bpy.context.scene.frame_set(frame)
    obj_pitch['actions'].append({"at":int((frame/30)*1000), 
    "pos":round(math.degrees(obj.rotation_euler[0])*1.11)+50})
    obj_roll['actions'].append({"at":int((frame/30)*1000), 
    "pos":round(math.degrees(obj.rotation_euler[1])*1.11)+50})
    obj_yaw['actions'].append({"at":int((frame/30)*1000), 
    "pos":round(math.degrees(obj.rotation_euler[2])*1.11)+50})
    obj_x['actions'].append({"at":int((frame/30)*1000), "pos":round(obj.location[0]*1000)+50})
    obj_y['actions'].append({"at":int((frame/30)*1000), "pos":round(obj.location[1]*1000)+50})
    obj_z['actions'].append({"at":int((frame/30)*1000), "pos":round(obj.location[2]*1000)})
        
with open(fs_pitch, 'w') as fp:
    json.dump(obj_pitch, fp, separators=(',', ':'))

with open(fs_roll, 'w') as fp:
    json.dump(obj_roll, fp, separators=(',', ':'))
    
with open(fs_yaw, 'w') as fp:
    json.dump(obj_yaw, fp, separators=(',', ':'))
    
with open(fs_x, 'w') as fp:
    json.dump(obj_x, fp, separators=(',', ':'))
    
with open(fs_y, 'w') as fp:
    json.dump(obj_y, fp, separators=(',', ':'))
    
with open(fs_z, 'w') as fp:
    json.dump(obj_z, fp, separators=(',', ':'))

Then, when I have free time, I need to put everything in order and make an addon for Blebder.

To be continued…

17 Likes

Pretty awesome. I could see blender being useful for 2d as well. Simply because things tracked move in 3 dimensions when we’re trying to estimate movement in one axis… can be a bit difficult with no visual aid.

1 Like

It’s hard to appreciate multi-axis movements - in Blender it seems easier to me.

In theory, I’m thinking of adding the ability to generate a point cloud from 3D video in Blender. But a little later :wink:

3 Likes

Thanks for the infos. Don’t you think this fits more in the #howto category?

You may be right. Although it’s a manual now, I hope it transforms into something more.

1 Like

Updated the topic.

this is really interesting when multiaxis toys become the norm. its a while since i last used blender maybe i should dl it again

1 Like

The global problem with devices is that they do not make stationary, but mobile, and a 6-axis device would be very difficult to make mobile. So it turned out no devices, no scripts, and vice versa.

So I did everything.
The link has both the video file, the blender file, the python scripts that I used and the scripts that came out in the end.
I don’t know what to do with the resulting data - I don’t have anything to test it on yet.

2 Likes

Small update.

Already wrote several multi-axis scripts - sharing my experiences
Tried the following gamepads:

  1. Xbox 360 - a timeless classic (one of the most comfortable for gaming in my opinion).
    Problems:
  • low sticks - hard to position and maintain positions in exact +/- 5 degrees in terms of rotation and position.
  • short triggers - it’s hard to make Z-axis rotation with +/- 5 degrees accuracy.
  • no paddles - it’s convenient to hang on them to switch between frames.
    Advantages:
  • comfortable and after many hours your hands do not get tired, it was him who wrote the script with Danni Daniels completely.
  1. Xiaomi Gamepad - well, why not, especially here it is.
    Problems:
  • dancing with tambourine around Windows to get it to work correctly.
  • no paddles - they are convenient to switch between frames.
    Advantages:
  • the sticks are taller and the triggers are more authentic. than the xbox 360
  • handy though I haven’t written much - I tried it for a couple of hours and moved on to the next member.
  1. Steam Controller - seemed like a winner and I wrote two multi-axis scripts with it!
    Problems:
  • very tired hands because of the uncomfortable grip for me - after a couple of hours straight to the point of pain.
  • very short triggers - i couldn’t make turns by z, i had to do it manually.
  • the crackling of the touchpad is annoying, and there is no tactile contact when it is switched off.
    Advantages:
  • there are paddles - it’s handy to hang on them to switch between frames.
  • I can do a lot with Blender via Steam (it can be done for any gamepad :wink: )
  • there’s a gyroscope - I haven’t figured out how to use it.
  1. Xbox Elite Series 2 - thought it would be worse than the 360. but one script for 42 minutes is written by him and I’m writing the second one now.
    Problems:
  • didn’t want to be friends with my bluetooth.
  • heavier than the previous ones - but maybe that’s an advantage, it feels different in the hands
    Advantages:
  • there are paddles. and even 4 (!) - it’s convenient to hang on them to switch between frames and something else
  • high stick for rotation - accuracy now +/- 1 degree.
  • quite long trigger - accuracy +/- 3-5 degrees.
  1. Nintendo Wii Remote + Nunchack - can’t say anything yet, haven’t tried it

Added how to install python libraries, fixed the export script.

Added gamepad button bindings, updated the script (the script has 3D video output, I’ll add a description later).

I guess I found the perfect controller for multi-axis scripting - HTC Vive Controller. Two base stations and a controller connected to a PC using Steam Controller reciever.
Yes, I lost a week while I was setting this all up - the speed of scripting has already increased significantly. I’m still doing everything in 3 passes (XY rotations, XY moves, Z rotations) I’m getting used to it. I think the next script will be written in two passes (XY rotations + XY moves, Z rotation) and once I get used to it I will write it in one pass ;).
I am satisfied.

1 Like

Added a video of creating a starter project in Blender.

2 Likes

Finally, at least someone has started using my designs (besides me) :wink:
I was approached with a problem on writing multi-axis scripts in Blender, which gave me an idea. That it would be nice to see the amplitude of movements (like in OFS).
Said and done. In Blender there is Graph Editor - with which you can do it.
Blender_C__Users_CyberYou_Documents_eroscripts_ADayWithBellaRolland_Dezyred_A_Day_With_Bella_Rolland_Cinema_Night.blend2024-01-2901-53-12-ezgif.com-video-to-gif-converter

1 Like

Hey there!

Thanks for the scripts! I spent a while looking at them and trying to get them to work, but thought animating with a gamepad was inconvenient. I instead tried this openvr streamer plugin that allows sending data from Valve Index controllers (and others) to Blender, which is then recorded into the keyframed cylinder. If you slow down the video in the background, you can pretty accurately record in 3D space with your hand while the video is playing.

However, this isn’t quite a drop-in replacement for your gamepad solution as there seems to be things going wrong with the axis / coordinate spaces, and when I exported my data, nothing happened with my multiaxis player. Also, since this way you’d be overwriting the keyframes for every recorded frame, it won’t work with your existing solution of first importing the “mono” funscript and recording axis one-by-one.

I recorded this brief demo on how it might work: 2.52 MB file on MEGA

I think to get this to work, some kind of normalization would have to be done so the space where you move the controller is known and within the limits of the device, and the rotation axises would have to be compatible with what the SR6 is expecting.

1 Like

I’ve added the openvr streamer plugin here as BlenderArtists appears to be dead right now.

openvr-streamer.txt (13.9 KB)

1 Like

I’m well past the point of capturing data from the HTC Vive Controller and correctly transferring that data into Blender.
ezgif.com-video-to-gif (4)
There are a lot of nuances that have not been solved yet, but my script is below.

import bpy, bpy_extras, mathutils
import openvr
import time, random, math


obj = bpy.data.objects["fleshlight"]
obj.rotation_mode = 'ZXY'
controller_quaternion = mathutils.Quaternion((0.0,1.0,0.0,0.0))
correction_quaternion = mathutils.Quaternion((1.0,0.0,0.0,0.0))
correction_vector = mathutils.Vector((0.0,0.0,0.0))
default_euler = mathutils.Euler((0.0,0.0,0.0))
default_vector = mathutils.Vector((0.0,0.0,1.0))

#reference = bpy.data.objects["reference"]
#camera = bpy.data.objects["camera"]
#area = next(area for area in bpy.context.screen.areas if area.type == 'VIEW_3D')

openvr.init(openvr.VRApplication_Scene)
vrsystem = openvr.VRSystem()

poses = []

def controller_state(pControllerState):
    # docs: https://github.com/ValveSoftware/openvr/wiki/IVRSystem::GetControllerState
    state = {}
    state['unPacketNum'] = pControllerState.unPacketNum
    state['trigger'] = pControllerState.rAxis[1].x
    state['trackpad_x'] = pControllerState.rAxis[0].x
    state['trackpad_y'] = pControllerState.rAxis[0].y
    state['ulButtonPressed'] = pControllerState.ulButtonPressed
    state['ulButtonTouched'] = pControllerState.ulButtonTouched
    state['menu_button'] = bool(pControllerState.ulButtonPressed >> 1 & 1)
    state['trackpad_pressed'] = bool(pControllerState.ulButtonPressed >> 32 & 1)
    state['trackpad_touched'] = bool(pControllerState.ulButtonTouched >> 32 & 1)
    state['grip_button'] = bool(pControllerState.ulButtonPressed >> 2 & 1)
    return state

# button & axis mapping
prev_keyframe_btn = 'A'
next_keyframe_btn = 'B'
prev_frame_btn = 'X'
next_frame_bnt = 'Y'
anim_play_btn = 'START'
anim_cancel_btn = 'BACK'
multi_view_btn = 'RIGHT_THUMB'
Z_rotation_btn = 'LEFT_THUMB'
left_location_btn = 'DPAD_LEFT'
right_location_btn = 'DPAD_RIGHT'
forward_location_btn = 'DPAD_UP'
backward_location_btn = 'DPAD_DOWN'
save_rotation_btn = 'LEFT_SHOULDER'
save_location_btn = 'RIGHT_SHOULDER'
thumb_location = 0
thumb_rotation = 1
 
#camera.hide_set(True)

prev_keyframe, next_keyframe, prev_frame, next_frame = False, False, False, False
save_location, save_rotation, anim_play, anim_cancel, enable_anim_play = False, False, False, False, False
forward_location, backward_location, left_location, right_location = False, False, False, False
l_trigger_pressed, r_trigger_pressed = False, False
multi_view, enable_multi_view, Z_rotation, enable_Z_rotation = False, False, False, 0
grid_pressed, trackpad_pressed, menu_pressed, correction = False, False, False, 0

class ModalTimerOperator(bpy.types.Operator):
    """Operator which runs itself from a timer"""
    bl_idname = "wm.modal_timer_operator"
    bl_label = "Modal Timer Operator"

    _timer = None
    
    def modal(self, context, event):
        global prev_keyframe, next_keyframe, prev_frame, next_frame
        global save_location, save_rotation, anim_play, anim_cancel, enable_anim_play
        global forward_location, backward_location, left_location, right_location
        global l_trigger_pressed, r_trigger_pressed
        global multi_view, enable_multi_view, Z_rotation, enable_Z_rotation
        global grid_pressed, correction_vector, correction_quaternion, trackpad_pressed, menu_pressed
        global correction, default_vector, default_euler
        if event.alt:
            if event.type == 'NUMPAD_0':
                enable_Z_rotation = 0
            if event.type == 'NUMPAD_1':
                enable_Z_rotation = 1
            if event.type == 'NUMPAD_2':
                enable_Z_rotation = 2
            if event.type == 'NUMPAD_3':
                enable_Z_rotation = 3
            cur_frame = bpy.context.scene.frame_current
            obj.location[0] =  obj.animation_data.action.fcurves[0].evaluate(cur_frame)
            obj.location[1] =  obj.animation_data.action.fcurves[1].evaluate(cur_frame)
            obj.rotation_euler[0] =  obj.animation_data.action.fcurves[3].evaluate(cur_frame)
            obj.rotation_euler[1] =  obj.animation_data.action.fcurves[4].evaluate(cur_frame)
            obj.rotation_euler[2] =  obj.animation_data.action.fcurves[5].evaluate(cur_frame)
            return {'PASS_THROUGH'}
        if event.type == 'RIGHTMOUSE':
            openvr.shutdown()
            self.cancel(context)
            return {'CANCELLED'}
        #obj_pos_x, obj_pos_y, obj_rot_euler_x, obj_rot_euler_y, obj_rot_euler_z = 0.0, 0.0, 0.0, 0.0, 0.0
        if event.type == 'TIMER':
            scr = bpy.context.screen 
            is_playing = scr.is_animation_playing
            controller_vector = mathutils.Vector((0.0,0.014368,-0.173514))
            poses, pControllerState = vrsystem.getControllerState(1)
            state = controller_state(pControllerState)
            #print(state)
            poses, _ = openvr.VRCompositor().waitGetPoses(poses, None)
            controller_pos = poses[1]
            vive_matrix = controller_pos.mDeviceToAbsoluteTracking
            #print(state)
            #print(vive_matrix)
            matrix = mathutils.Matrix([list(vive_matrix[0]),list(vive_matrix[1]),list(vive_matrix[2])])
            controller_matrix = bpy_extras.io_utils.axis_conversion('Z','Y','Y','Z') @ matrix
            controller_matrix.resize_4x4()
            vector = controller_matrix.to_translation()
            quaternion = controller_matrix.to_quaternion().normalized()
            obj_quaternion = quaternion @ controller_quaternion.conjugated().normalized()
            obj_quaternion = obj_quaternion @ correction_quaternion
            controller_vector.rotate(quaternion)
            obj_vector = vector - controller_vector - correction_vector
            euler = obj_quaternion.to_euler('ZXY')
            euler[0] = euler[0]*2
            euler[1] = euler[1]*2
            euler[2] = euler[2]*2
            #euler.rotate(vector_euler)
            #obj.rotation_euler = euler
            #obj_vector.rotate(vector_euler)
            #obj.location = obj_vector
            #angle = default_vector.angle(controller_vector)
            #print(angle)
            if enable_Z_rotation == 0 and not is_playing:
                if euler[0] > 0.7853: euler[0] = 0.7853
                if euler[0] < -0.7853: euler[0] = -0.7853
                if euler[1] > 0.7853: euler[1] = 0.7853
                if euler[1] < -0.7853: euler[1] = -0.7853
                if obj.rotation_euler[0] != euler[0]:
                    obj.rotation_euler[0] = euler[0]
                if obj.rotation_euler[1] != euler[1]:
                    obj.rotation_euler[1] = euler[1]
                if obj.location[0] != obj_vector[0]:
                    obj.location[0] = obj_vector[0]
                if obj.location[1] != obj_vector[1]:
                    obj.location[1] = obj_vector[1]
            if enable_Z_rotation == 1 and not is_playing:
                if euler[0] > 0.7853: euler[0] = 0.7853
                if euler[0] < -0.7853: euler[0] = -0.7853
                if euler[1] > 0.7853: euler[1] = 0.7853
                if euler[1] < -0.7853: euler[1] = -0.7853
                if obj.rotation_euler[0] != euler[0]:
                    obj.rotation_euler[0] = euler[0]
                if obj.rotation_euler[1] != euler[1]:
                    obj.rotation_euler[1] = euler[1]
                #obj.location[0] = obj_vector[0]
                #gz+0.5
                #obj.location[1] = obj_vector[1]
                #obj.location[0] = obj_vector[0]
                #obj.location[1] = obj_vector[1]
            #if angle < 0.7854 and enable_Z_rotation == 1:
                #obj.rotation_euler[0] = euler[0]
                #obj.rotation_euler[1] = euler[1]
            #if abs(pow(obj_vector[0],2) + pow(obj_vector[1],2)) <= 0.1 and enable_Z_rotation:
                #obj.location[0] = obj_vector[0]
                #obj.location[1] = obj_vector[1]
            if enable_Z_rotation == 2 and not is_playing:
                if euler[0] > 0.7853: euler[0] = 0.7853
                if euler[0] < -0.7853: euler[0] = -0.7853
                if euler[1] > 0.7853: euler[1] = 0.7853
                if euler[1] < -0.7853: euler[1] = -0.7853
                if obj.rotation_euler[0] != euler[0]:
                    obj.rotation_euler[0] = euler[0]
                if obj.rotation_euler[1] != euler[1]:
                    obj.rotation_euler[1] = euler[1]
            if enable_Z_rotation == 3 and not is_playing:
                if euler[2] > 0.7853: euler[2] = 0.7853
                if euler[2] < -0.7853: euler[2] = -0.7853
                elif obj.rotation_euler[2] != euler[2]:
                    obj.rotation_euler[2] = euler[2]
            if state['grip_button'] and not grid_pressed and correction == 0:
                correction_vector = obj_vector
                correction_quaternion = obj_quaternion.conjugated().normalized()
                correction = 2
                grid_pressed = True
            if state['grip_button'] and not grid_pressed and correction == 1:
                state_vector = mathutils.Vector((0.005,0.0,0.0))
                vector_euler[2] = obj_vector.angle(state_vector)
                correction = 2
                grid_pressed = True
            if state['grip_button'] and not grid_pressed and correction == 2:
                correction_quaternion = mathutils.Quaternion((1.0,0.0,0.0,0.0))
                correction_vector = mathutils.Vector((0.0,0.0,0.0))
                vector_euler = mathutils.Euler((0.0,0.0,0.0))
                correction = 0
                grid_pressed = True
            if not state['grip_button']:
                grid_pressed = False
            if state['trigger'] > 0.5 and not save_rotation and enable_Z_rotation == 0:
                #print (math.sqrt(obj.location[0]**2), math.sqrt(obj.location[1]**2))
                vect = obj.location.copy()
                angle = obj.rotation_euler.copy()
                angle[0] = -math.radians(vect[1]*500)
                angle[1] = math.radians(vect[0]*500)
                #print(angle)
                vect[0], vect[1] = 0.0, 0.0
                vect.rotate(angle.to_quaternion())
                obj.location[0] = vect[0]*(1 + 1 - vect[2]*10)
                obj.location[1] = vect[1]*(1 + 1 - vect[2]*10)
                obj.keyframe_insert(data_path="rotation_euler")
                obj.keyframe_insert(data_path="location")
                bpy.ops.screen.keyframe_jump(next=True)
                save_rotation = True
            if state['trigger'] > 0.5 and not save_rotation and enable_Z_rotation == 3:
                obj.keyframe_insert(data_path="rotation_euler")
                bpy.ops.screen.keyframe_jump(next=True)
                save_rotation = True
            if state['trigger'] > 0.5 and not save_location and enable_Z_rotation == 1:
                vect = obj.location.copy() 
                vect[0], vect[1] = 0.0, 0.0
                vect.rotate(obj.rotation_euler.to_quaternion())
                #print(1 + 1 - vect[2]*10)
                obj.location[0] = vect[0]*(1 + 1 - vect[2]*10)
                obj.location[1] = vect[1]*(1 + 1 - vect[2]*10)
                obj.keyframe_insert(data_path="rotation_euler")
                obj.keyframe_insert(data_path="location")
                bpy.ops.screen.keyframe_jump(next=True)
                save_location = True
            if state['trigger'] > 0.5 and not save_location and enable_Z_rotation == 2:
                vect = obj.location.copy() 
                vect[0], vect[1] = 0.0, 0.0
                vect.rotate(obj.rotation_euler.to_quaternion())
                #print(1 + 1 - vect[2]*10)
                obj.location[0] = vect[0]*(1 + 1 - vect[2]*10)
                obj.location[1] = vect[1]*(1 + 1 - vect[2]*10)
                obj.keyframe_insert(data_path="location")
                bpy.ops.screen.keyframe_jump(next=True)
                save_location = True
            if state['trigger'] <= 0.5 and (save_rotation or save_location):
                save_rotation, save_location = False, False
            if state['menu_button'] and not menu_pressed and enable_Z_rotation == 0:
                menu_pressed = True
                enable_Z_rotation = 1
            if state['menu_button'] and not menu_pressed and enable_Z_rotation == 1:
                menu_pressed = True
                enable_Z_rotation = 2
            if state['menu_button'] and not menu_pressed and enable_Z_rotation == 2:
                menu_pressed = True
                enable_Z_rotation = 3
            if state['menu_button'] and not menu_pressed and enable_Z_rotation == 3:
                menu_pressed = True
                enable_Z_rotation = 0
            if not state['menu_button']:
                menu_pressed = False
            if state['trackpad_pressed'] and not trackpad_pressed:
                if state['trackpad_y'] >= 0.5 and not enable_Z_rotation == 0:
                    enable_Z_rotation = 0
                    trackpad_pressed = True
                if state['trackpad_y'] <= -0.5 and not enable_Z_rotation == 2:
                    enable_Z_rotation = 2
                    trackpad_pressed = True
                if state['trackpad_x'] >= 0.5 and not enable_Z_rotation == 1:
                    enable_Z_rotation = 1
                    trackpad_pressed = True
                if state['trackpad_x'] <= -0.5 and not enable_Z_rotation == 3:
                    enable_Z_rotation = 3
                    trackpad_pressed = True
                cur_frame = bpy.context.scene.frame_current
                obj.location[0] =  obj.animation_data.action.fcurves[0].evaluate(cur_frame)
                obj.location[1] =  obj.animation_data.action.fcurves[1].evaluate(cur_frame)
                obj.rotation_euler[0] =  obj.animation_data.action.fcurves[3].evaluate(cur_frame)
                obj.rotation_euler[1] =  obj.animation_data.action.fcurves[4].evaluate(cur_frame)
                obj.rotation_euler[2] =  obj.animation_data.action.fcurves[5].evaluate(cur_frame)
            if not state['trackpad_pressed'] and trackpad_pressed:
                trackpad_pressed = False
            else: pass
        return {'PASS_THROUGH'}

    def execute(self, context):
        wm = context.window_manager
        self._timer = wm.event_timer_add(1/50, window=context.window)
        wm.modal_handler_add(self)
        return {'RUNNING_MODAL'}

    def cancel(self, context):
        wm = context.window_manager
        wm.event_timer_remove(self._timer)
        
def get_override(area_type, region_type):
    for area in bpy.context.screen.areas: 
        if area.type == area_type:             
            for region in area.regions:                 
                if region.type == region_type:                    
                    override = {'area': area, 'region': region} 
                    return override
    #error message if the area or region wasn't found
    raise RuntimeError("Wasn't able to find", region_type," in area ", area_type,
                        "\n Make sure it's open while executing script.")


def menu_func(self, context):
    self.layout.operator(ModalTimerOperator.bl_idname, text=ModalTimerOperator.bl_label)


def register():
    bpy.utils.register_class(ModalTimerOperator)
    bpy.types.VIEW3D_MT_view.append(menu_func)


# Register and add to the "view" menu (required to also use F3 search "Modal Timer Operator" for quick access).
def unregister():
    bpy.utils.unregister_class(ModalTimerOperator)
    bpy.types.VIEW3D_MT_view.remove(menu_func)


if __name__ == "__main__":
    register()
    # test call
    bpy.ops.wm.modal_timer_operator()

image

  1. I use an A5 paper cutting mat to determine the onset of coodinates, the calibration counts down when the (8) buttons are pressed.
  2. We need to determine how to position the mat on the axes - so that the controller and cylinder tilts in Blender match. I’m lucky and comfortable on my desk, I haven’t done any further math on how to rotate the coordinate system correctly.
  3. The point on the controller (5) coincides with the cylinder point.
  4. I doubled the turns as it turned out to be awkward to twist +/- 45 degrees with the controller.
  5. Menu button (1) switches between modes as with gamepad + added a new mode where XY angles are transmitted directly, and XY position is calculated by rotating a vector where 1 cm of position on the mat is equal to 5 degrees.
  6. Pulling the trigger (7) writes data to the keyframe depending on the mode.
  7. Pressing the touchpad (2) up/down - switches keyframes.
  8. Pressing the touchpad (2) left/right - switches frames forward/backward.
2 Likes

Does this mean I can use the Quest3 controller to record multi axis scripts on Blender?

Creating multi axis scripts is very difficult, and your work has reduced the difficulty of production. Thank you very much!