A theory for music to funscript

dimnogro · May 27, 2022, 4:34am

what is MIDI? midi is a standard old as hell.
It has 2 parts.
On the one hand, there are the midi events that are a port used by dj consoles and instruments. this is used when you connect a piano or dj console to the computer.
in the other part are the mid files that are used to express music in the form of musical notes and instruments. no audio just notes and instruments, midi files are very very light, a few Kb.
Mid File Example: METALLICA.One.mid — Free MIDI — BitMidi
Theoretically, reading the mid files and their different channels (instruments) could retrieve the notes of each channel and its time in the song to generate a movement in the funscript at a certain speed according to the pitch or some other midi property.
since ancient times there are also tools that allow you to convert audio music files to midi files. both in the form of applications and open source libraries.
this technique is used long ago by rhythm games such as audiosurft, ddr and others or Shazam

I personally don’t have the desire or time to do this. but I leave this theory since it seems to me a viable idea if someone wants to try to implement it

reference links:

gist.github.com

https://gist.github.com/natowi/d26c7e97443ec97e8032fb7e7596f0b0

audio2midi.md

# Recurrent Neural Network for generating piano MIDI-files from audio (MP3, WAV, etc.) 
https://github.com/BShakhovsky/PolyphonicPianoTranscription

# A python program which performs an FFT on an audio file and produces a MIDI file from the results
https://github.com/NFJones/audio-to-midi

# Extract the melody from an audio file and export to MIDI
https://github.com/justinsalamon/audio_to_midi_melodia

# Performs pitch detection on a polyphonic audio source and outputs to MIDI

This file has been truncated. show original

noise · May 27, 2022, 3:02pm

There’s already MIDI plugins for Unity and Unreal Engine, so adding MIDI to software should be a fairly straightforward thing.

If this turns into a real project, I’d suggest using MIDI 2.0 as a the standard, since it has a lot more resolution and good support for advanced features.