I found this recent AI model (Sound2Hap) for mapping sounds to haptics (vibrations), which scores better than 4 competing methods (7.6 vs 5.6 human rating). This model is trained on sounds and not music, so while the results for sounds are indeed good, the results for music are usable, but not great in my testing. It doesn’t really understand the underlying structure of music well.
I couldn’t find a model that is trained on music and specifically maps music to haptics. Does anybody know something that could prove useful? There are models that transform music to MIDI and to rhythm game maps, perhaps that could be a good start? Or has somebody already made such a thing for funscripts?
Of course FunscriptDancer exists and gives good results based on traditional audio analysis, however I was wondering if nowadays we might get an even more human-like result with even less effort.
You can download my Sound2Fun experiment, which requires Python.
To install the required python packages, run this command in the folder (may require administrator privileges):
pip install -r requirements.txt
or run the included bat file.
To create a funscript just drag and drop an mp4 video or mp3 file onto the python script. Takes only a few seconds per file.
There are some variables in the script.
- model: You can choose between model 1 and 2, they both do the same thing but slightly differently.
- epsilon: How much to simplify the model output for the funscript, lower numbers mean more detail is retained.
- invert: Invert the output (usually True for strokers and False for vibrators)
- valley_multiplier: Pulls down valleys in the script. Set to 1 to leave everything untouched.
- valley_factor: Pulls higher valleys proportionally further down. Set to 0 to leave everything untouched.
- peak_multiplier: Scales peaks up.
- delete_model_audio: The AI model produces a temporary mp3 file that lets you listen to the result, you could keep this if you want.