F.A.P.S (Funscript - Audio Processing System) by CaptainHarlock

To those of you who asked to be beta testers, I have good news and “bad” news, but more good than bad, lol

The good news:

  • Thanks to the first beta testers who helped me identify and fix installation issues, which was my biggest concern since I hadn’t been able to test it on other computers, the new installation system now works better, is simpler, and is compatible with modern GPUs.
    That means I think I could now upload a beta version for all of you, not privately but in this thread, so that together we can finish ironing out any problems it may still have.

The bad news:

  • Due to the architecture change that was made (from Pytorch 2.4+CUDA 12.4 to PyTorch 2.7+CUDA 12.8) so that it could work with modern GPUs such as the 40xx and 50xx series, it may now cause problems or not work with older GPUs.

Right now:
Nvidia 10xx series > Not compatible / Doesn’t work (Although I’m waiting to finish testing with a beta tester who is trying it out). UPDATE > Works (verified)
Nvidia 20xx series > Probably works but hasn’t been tested.
Nvidia 30xx series > Works (verified)
Nvidia 40xx series > Works (verified)
Nvidia 50xx series > Works (verified).

Also, note that to take full advantage of all features, it’s recommended to have a GPU with >8GB of VRAM.

The application has fallbacks to use librosa with CPU, but right now I’ve focused on making sure everything works well on GPU, so I can’t guarantee that the fallbacks work well at this time. In summary: The application is currently designed for use on modern Nvidia GPUs with 8GB of VRAM or more, and to make use of the most advanced AI models (SOTA, State Of The Art).
I have a very old version that works entirely with booksa+CPU, but if I wanted to go further and make it better, the use of GPU+AI is essential.

Right now, I’m polishing up a few last things, but I’ll be able to publish it here soon. :sign_of_the_horns:

5 Likes

Great news!

1 Like

Great news! :partying_face:

It has been confirmed that the application works with Nvidia 10xx series GPUs!
Since users have confirmed that the 10xx, 30xx, 40xx, and 50xx series work, I assume that the 20xx series should also work, although this hasn’t been confirmed by any users yet.

More good news is that yesterday I discovered a couple of bugs that I was able to fix:

  • It didn’t work properly when selecting the option for the video to have only one song.
  • It didn’t correctly detect beats at the very end of the song.

Between today and tomorrow, you’ll have the version published here! :grin: :smiling_face_with_horns: :sign_of_the_horns:

6 Likes

F.A.P.S v0.7 will be released in a couple of hours for everyone! :sign_of_the_horns:

dafoe

3 Likes

Done! F.A.P.S v0.7 available to everyone! :partying_face:

Enjoy!

If you have any questions, don’t hesitate to ask here.
To report bugs, please be as specific and detailed as possible, attaching screenshots of the application or console, or the log file.

jungle-welcome

5 Likes

CONGRATS!! This program makes the world a better place! :face_savoring_food:

1 Like

Thanks!!
youre-right-breaking-bad

And thanks to you for being such a great beta tester! :wink:

Any other place to download it from?


Ahora toca probarlo! / Time yo try it!

1 Like

Me vas a odiar, pero ahora hazlo para tarjetas AMD :joy:

punisher-no-no-no-tanklorde

La verdad es que no sé si es posible tal como funciona ahora y con todo lo que tiene. No tengo experiencia con AMD, sé que hay el ROCm pero no he tenido que trabajar nunca con él, y no sé si todos los modelos AI que usa la aplicación serían compatibles con eso.

The truth is, I don’t know if it’s possible considering how it works now and everything it has. I don’t have any experience with AMD. I know there’s ROCm, but I’ve never had to work with it, and I don’t know if all the AI models used by the application would be compatible with it.

1 Like

I just checked it and it doesn’t seem feasible. It would take a huge amount of work just to test it, and it’s more than likely that some of the dependencies used wouldn’t work, breaking the current pipeline.
I’m sorry, but to get the most out of it and access certain features, you need an Nvidia GPU.
But the application already has a fallback to “librosa+CPU,” and anyone can use that, even if you don’t have a dedicated GPU. The problem is that you won’t be able to use the moods that require AI models (“Fap Mixer,” “SixthSense,” “Hypno,” “Edging,” and any JOIs…), the rest can work with a CPU. The problem is that in my tests I focused on how they worked using the GPU, so I’m not sure how the funscript will turn out using a CPU, but that can be reviewed and adjusted to make it as similar as possible.

1 Like

Couple of issues I found:

  • I saw in the code there is a cli mode, however run.bat will not send the batch file arguments to the python code
  • In the queue page after I click on the ‘Add Multiple’ button and select a couple of videos the mood selector window is not localized in English
  • The same window does not contain ‘SixthSense’ and ‘Fap Mixer’ moods
  • After adding items to the queue there is a message box which is not localized (both messages when selecting a mood for all files and when selecting moods for each file separately)
  • In the Queue page queued item’s statuses not localized
  • When pressing the ‘Clear Queue’ button in the Queue page, the message box not localized
  • After clearing the queue with ‘Clear Queue’ button the ‘Queue’ page selector button still has the number of items (i.e ‘Queue (2)’ instead of ‘Queue’ when clearing the queue containing 2 items)
  • After adding items to the queue the ‘Queue’ page selector button loses it’s color. After switching pages and trying to go back to the queue page the page is not loading. The exception in the console window:
Exception in Tkinter callback
Traceback (most recent call last):
  File "C:\Users\xxx\AppData\Local\Programs\Python\Python312\Lib\tkinter\__init__.py", line 1968, in __call__
    return self.func(*args)
           ^^^^^^^^^^^^^^^^
  File "d:\Development\FAPS\FAPS_v0.7\venv\Lib\site-packages\customtkinter\windows\widgets\ctk_button.py", line 554, in _clicked
    self._command()
  File "d:\Development\FAPS\FAPS_v0.7\venv\Lib\site-packages\customtkinter\windows\widgets\ctk_segmented_button.py", line 167, in <lambda>
    command=lambda v=value: self.set(v, from_button_callback=True),
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "d:\Development\FAPS\FAPS_v0.7\venv\Lib\site-packages\customtkinter\windows\widgets\ctk_segmented_button.py", line 381, in set
    self._command(self._current_value)
  File "d:\Development\FAPS\FAPS_v0.7\venv\Lib\site-packages\customtkinter\windows\widgets\ctk_tabview.py", line 106, in _segmented_button_callback
    self._set_grid_current_tab()
  File "d:\Development\FAPS\FAPS_v0.7\venv\Lib\site-packages\customtkinter\windows\widgets\ctk_tabview.py", line 186, in _set_grid_current_tab
    self._tab_dict[self._current_name].grid(row=3, column=0, sticky="nsew",
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
KeyError: 'Queue (2)'

BTW is there a way to run AI Analysis for queued videos?

1 Like

Yep, some tabs have been left behind while I was making some changes. Not doing “AI Analysis” in the queue is a serious problem; it’s a key point.

Thank you very much for reporting the errors! I’m working on fixing them.

As for the queue and AI Analysis, since the intention of the queue is that there be no user intervention, I’ve made it so that by default it interprets that there is more than one song (if there is only one, no problem, it will detect only that one), and that it uses the “Mix cut/mixed songs” type of analysis; it won’t give perfect boundaries because the user can’t fine-tune them, but I think it will do the job.

My use case would be exactly that, find an interesting mood and running it on my whole collection. I love the idea of the SixthSense mood to separate the audio into tracks and generate variety based on that.
I understand that fully automated processing will produce lower quality results. It’s more important to me to have a script for all my PMVs than to have quality scripts for a couple of them and the rest is unscripted.

An idea for the queue: Select a folder and process all videos in that folder including sub folders. Preferably with a check box to ignore videos with existing funscripts.

“An idea for the queue: Select a folder and process all videos in that folder including sub folders. Preferably with a check box to ignore videos with existing funscripts.”

This would be possible to do.
WARNING: Stem separation is saved in WAV format for the highest quality and to facilitate good detection. This takes up space, and doing it using “Queue” for dozens or hundreds of videos can take up a LOT of space. Keep this in mind.

Yep, SixthSense, I think that, without a doubt, for now, it’s probably one of the best, if not the best mood created… But that depends on the user’s taste. But in your case, with so much automation, I think it would be the right one, or maybe also the AUTO mood, as it adapts to the “energy” of the video at any given moment to use one mood or another (note: it uses several moods but not all of them). The “AUTO” mood uses the default parameters of the different moods, but if you make changes to the parameters before using it, it will use those changed parameters.
The “Fap Mixer” can also be great if configured properly.
Another option for you would be to use “Queue” only to run the “AI Analysis,” which is the part that takes the longest to do. Then you could load any video where you had done that analysis, it would load the analysis cache, and then you could generate the funscript in seconds, so you can try out which of the moods fits best in each video.
Creating funscripts is super fast! That is, once the analysis is done, you could create ALL the funscripts for a video in less than a couple of minutes (not including moods such as “Hypno,” “Edging,” or “JOIs…”; those require extra analysis of the voice stem).

Anyway, I should tell you that the “Queue” tab was created very early on, taking into account the first moods created (the ‘musical’ ones), such as “Relaxed,” “Normal,” “Slow,” “Simple,” “Crazy,” or “Crazy+Voice.”
These “musical” moods have certain parameters that can alter the result, but basically they will have the same speed and type of reaction.
On the contrary, more recent moods such as “Fap Mixer,” “SixthSense,” or others that detect voice/phrases such as “Hypno,” “Edging,” and the “JOIs…” are highly configurable, and their result can be quite different depending on how they are configured.

Mental notes for me:

  • “AUTO” requires the user to choose between 3 types of ‘zone’ detection. Using it in “Queue” right now is probably not possible, I have to check it out.
  • The “Fap Mixer” mood also requires the user to choose how its 4 blocks are assigned (automatic or default), check how it’s applied in “Queue.”
  • Add an option in “Queue” to load a directory, allowing subdirectories to be used automatically.
  • Add an option in “Queue” to only perform “AI Analysis” without generating funscripts.

Why is the program so HUGE ? Almost 6 GigaByte ?

Because it already includes the AI models and dependencies (wheels) needed for installation.

Previously, it took up less than 400MB, but the installation required downloading those things, so it still took up more space, and having to download everything necessary could lead to problems, or certain wheels were not easily available, I even had to compile some of them myself to fit the system/application requirements.
Now all of this is included, and the installation is simpler and more stable.
I’m sorry, but if you want to work with the best AI models, that requires space. An example of this is image/video AI models, which can take up between 6-20GB each, lol

I understand. Too bad one needs to pay to download it.

Huh, what? No! You don’t need to pay anything! The app is completely free, lol :joy:

The donation link is OPTIONAL, just in case someone likes the app and finds it useful, and wants to show their appreciation for the work done.