How to create subtitles for a scene (even if you don't understand the language)

k00gar · January 7, 2023, 3:32pm

Wow, impressive !

Thank you for sharing !

Alderamin · January 7, 2023, 6:45pm

Thanks for sharing, that’s some work you did there!
I’m curious how long it takes you to create the subtitles for a 30min video?

Zalunda · January 8, 2023, 12:19am

When it’s going well, about 2 to 3 times the length of a video. It depends on the number of subtitles in the scene.

For JAV, the first part often have a lot of talking (ex. 20 min of the 30 min), so there is more subtitles to run through all steps (manual and AI). For other parts, it can be less then twice the time because it often have only 5 minutes of talking (100-150 subtitles).

For example, for the first part of “SIVR-130”, a 38 minutes video, it took me:

About 40 minutes to make the “perfect-vad”.
About 15 minutes running Whisper, doing the google translation, etc.
About 25 minutes to run all the subtitles (~300) through ChatGPT.
In theory, 25 minutes to clean up the translated srt (i.e. choosing which translation to use). I say, in theory, because this is the video where most of the subtitles were desynchronize by one spot. I stopped. I’ll fix the problem and retry.

So, around 105 min for a 38 min video.

Zalunda · January 11, 2023, 2:02am

I updated my tool (v1.1.1) and the guide (v1.1). It’s should be more ‘robust’ now.

Problem is, every time I update the guide, I need to make another subtitle to make sure it works…

Here are scenes that got subtitles made using this guide: CRVR-286, EXVR-149, SIVR-115, SIVR-130.

EroSennin · January 11, 2023, 5:58pm

Thanks for the guide !

Could you please tell me how to view the subtitiles in heresphere (oculus quest 2) ?

Zalunda · January 11, 2023, 11:54pm

I don’t know about Quest 2. I’m using HereSphere on Windows.
From what I can understand, HereSphere doesn’t support subtitles yet on Quest 2.
It could be a good idea to ask if it’s in their plan to support it soon: HereSphere VR Video Player for Quest 2. It should be noted that HereSphere is a paid app though, but, IMO, it’s really worth the price.
I don’t know if other applications available on Quest 2 support subtitles.

spacemanbob · January 12, 2023, 7:52am

Awesome work @Zalunda! Really appreciate your efforts documenting this for others to follow. Subtitles definitely add a lot in Heresphere. Looking forward to taking a crack at this.

lebeastmode · January 22, 2023, 2:27pm

getting stuck says "FunscriptToolbox.exe is not recognized as an internal or external command, operable program or batch file

Zalunda · January 22, 2023, 2:32pm

Did you modify this line in the batch file?
“C:\AwesomeTools\FunscriptToolbox” need to be changed to the path where you unzipped FunscriptToolbox.

lebeastmode · January 22, 2023, 2:43pm

Yea I changed it and I also tried to recreate the path you used to see if it would run and its the same message I reinstalled python 3.9.

Zalunda · January 22, 2023, 3:01pm

Python is not needed to start FunscriptToolbox (i.e. it’s a .Net application). So, no need to ‘debug’ that part yet.

In a command prompt, can you try to call the tool directly?
It should be “C:\AwesomeTools\FunscriptToolbox\FunscriptToolbox.exe”.

If it doesn’t work, I’m not sure, maybe check if the “.Net framework” is installed.

lebeastmode · January 22, 2023, 5:02pm

yes i can call it. the only error says that “No verb selected”

ERROR(S):
No verb selected.

audiosync.createaudiosignature, as.cas Create audio signature for videos.

audiosync.createfunscript, as.cfs Take an audio signature and funscript and try to generate a funscript
synchronized to a different videos.

audiosync.verifyfunscript, as.vfs Verify a funscript.

subtitles.video2vadsrt, sub.vid2vad Extract an dummy .srt from video, using Voice Activity Detection

subtitles.srt2vadwav, sub.srt2vadwav Create temporary VADWAV for whipser

subtitles.srt2wavchunks, sub.srt2wavs Extract .srt, using Voice Activity Detection

subtitles.wavchunks2srt, sub.wavs2srt Merge .srt file transcribed by whisper to a single srt

subtitles.vadwav2srt, sub.vadwavsrt Created .srt from the transcribed result from whisper

subtitles.gpt2srt, sub.gpt2srt Merge srts files (from whisper-translated .srt of a extracted wav from
SubtitleEdit)

subtitles.srt2gpt, sub.srt2gpt Create ChatGPT prompts for translations

help Display more information on a specific command.

version Display version information.

This is the current bat file

@echo off

REM set path=C:\AwesomeTools\FunscriptToolbox;%path%

echo — subtitles.video2vadsrt —
“FunscriptToolbox.exe” ^
subtitles.video2vadsrt ^
–suffix “.temp.vad” ^
“*.mp4”

REM … Create file .temp.perfect-vad.srt with SubtitleEdit …
REM … Rename .temp.vad.wav to .temp.perfect-vad.wav

echo.
echo — subtitles.srt2wavchunks —
“FunscriptToolbox.exe” ^
subtitles.srt2wavchunks ^
“*.perfect-vad.srt”

echo.
echo — subtitles.srt2singlewav —
“FunscriptToolbox.exe” ^
subtitles.srt2vadwav ^
–suffix “.whisper” ^
“*.perfect-vad.srt”

REM … Use whisper to transcribe the whisper.wav files with the following setting:
REM Model: large-v2
REM Language: Japanese (or other)
REM URL: empty
REM Upload Files: *.whisper.wav (you can add multiple files)
REM Task: Transcribe
REM VAD: none (it’s important to set to None)

REM … Rename .srt files produced by whisper to “.whisper.srt”

echo.
echo — subtitles.wavchunks2srt —
“FunscriptToolbox.exe” ^
subtitles.wavchunks2srt ^
–suffix “.whisper.chunks” ^
–verbose ^
“*.perfect-vad.srt”

echo.
echo — subtitles.gpt2srt —
“FunscriptToolbox.exe” ^
subtitles.gpt2srt ^
“*.gptresults”

echo.
echo — subtitles.srt2gpt —
“FunscriptToolbox.exe” ^
subtitles.srt2gpt ^
“*whisper.jp.srt”

echo.
echo — subtitles.singlewav2srt —
“FunscriptToolbox.exe” ^
subtitles.vadwav2srt ^
–suffix “.jp” ^
–verbose ^
“*.whisper.wav”

pause

Zalunda · January 22, 2023, 8:27pm

If the FunscriptToolbox can start but it returns an error while running. It might be python after all.

In a command line, if you write “python.exe”, does it starts?

Python 3.9.0 (tags/v3.9.0:9cf6752, Oct  5 2020, 15:34:40) [MSC v.1927 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>>

If not, you might have to add the folder of python in the path in the script like this:

 set path=C:\AwesomeTools\FunscriptToolbox;[path of python bin];%path%

In my case, it was not needed but it might have been an option in the python installation that I selected. Usually, it’s in C:\python39 or C:\Users.…username…\AppData\Local\Programs\Python\Python39

If it still doesn’t work, please create an issue on Issues · Zalunda/FunscriptToolbox · GitHub, and attach the log file found in “%appdata\FunscriptToolbox\Logs” (i.e.
“C:\Users.…username…\AppData\Roaming\FunscriptToolbox\Logs”).

lebeastmode · January 22, 2023, 9:10pm

ok will do thanks

spacemanbob · January 22, 2023, 10:59pm

@Zalunda I’m trying to perform “pip install pytorch torchaudio IPython” as the FunscriptToolboxSubtitles.bat directs, but I received an error.

Exception: You tried to install “pytorch”. The package named for PyTorch is “torch”

So I installed “pip install torch torchaudio IPython” instead. That installation was successful. But I’m still getting an exception while running the .bat file:

Make sure that 'Python' is installed (https://www.python.org/downloads/).
If python is installed, make sure to run the following command in a command prompt: pip install pytorch torchaudio IPython
Script output:
Using cache found in C:\Users\r/.cache\torch\hub\snakers4_silero-vad_master
C:\Users\r\AppData\Local\Programs\Python\Python39\lib\site-packages\torchaudio\backend\utils.py:62: UserWarning: No audio backend is available.
warnings.warn(“No audio backend is available.”)
Traceback (most recent call last):
File “F:\S1TBBackup\FunscriptToolbox\PythonSource\funscripttoolbox-extract-vad.py”, line 26, in
wav = read_audio(inputwav, sampling_rate=sampling_rate)
File “C:\Users\r/.cache\torch\hub\snakers4_silero-vad_master\utils_vad.py”, line 122, in read_audio
wav, sr = torchaudio.load(path)
File “C:\Users\r\AppData\Local\Programs\Python\Python39\lib\site-packages\torchaudio\backend\no_backend.py”, line 16, in load
raise RuntimeError(“No audio I/O backend is available.”)
RuntimeError: No audio I/O backend is available.

Should I create an issue as well?

spacemanbob · January 22, 2023, 11:02pm

I reinstalled python 3.9

When you reinstalled, did you check the box on the bottom of the installer labeled “add Python to PATH?” That was my issue. I also followed this video for pip install help.

I also noticed another issue:

This is the current bat file
REM set path=C:\AwesomeTools\FunscriptToolbox;%path%

Don’t you need to remove the “REM” here? It’s excluding that line of code.

doublevr · January 22, 2023, 11:42pm

Thx for a great tutorial. We are adding subtitles into the SLR app with upcoming updates.

Let me know if any suggestions

Zalunda · January 23, 2023, 12:16am

I’m not sure I would say it’s a ‘great’ tutorial… but it’s a start…

Good to know about SLR app, it will allow more people to be able to use subtitles. Right now, I think only people that use HereSphere on PC can see subtitles in VR.

Nice catch. Yes, you are right, REM needs to be removed. That would explain @lebeastmode problem. I’ll remove it in the ‘template’ batch file as well.

And for your “No audio backend is available” problem, yes, please open an issue. It’s easier to follow in a dedicated thread. You might need another pip install (i.e. chatgpt tell me that “torchaudio supports several different audio backends, including SoX, ffmpeg, and soundfile”). I installed a bunch of python modules before for other applications, so it was difficult for me to know what was the ‘bare minimum’ for silerio-vad to work. Once we figure out all the needed modules, I’ll update the message in the application.

Just to be clear to everyone, this is still a really ‘messy’ guide. I’m still hoping for a new version of OpenAI Whisper to be able to make it more ‘user friendly’.

lebeastmode · January 23, 2023, 1:09am

yep removing the REM was the trick thanks!

lebeastmode · January 23, 2023, 1:47am

yea Im getting the pip install torch bug
and I reinstalled with the path checked

  Make sure that 'Python' is installed (https://www.python.org/downloads/).
    If python is installed, make sure to run the following command in a command prompt: pip install pytorch torchaudio IPython

i’ll follow along with in the next ticket thanks!