I’ve been continuing my spiral into becoming delusional through claude code and experimenting with AI funscript generation.
This is not the first time I’ve been dabbling with this but it’s first time it’s worth sharing.
Through experimentation I landed on V-JEPA 2.1 a model by meta and found it to be incredibly potent for funscript generation.
Basically I’m using the smallest most distilled frozen V-JEPA 2.1 80M parameters straight from meta no finetuning to create a latent cache on disk for high quality synchronized video/funscript pairs and then I train a small head ~3.5M parameters on that latent cache to extract motion and generate funscript actions. I’m glossing over some implementation details but that’s the gist of it.
What makes V-JEPA a game changer? V-JEPA takes care of turning the RGB pixel soup values from video frames into meaningful representations so that even a bozo like me can train a model that would otherwise require a super computer with hundreds of GPUs on a single 4090 (to be honest not even that is required). Another aspect is that this approach requires a lot less training data.
I see a lot of scaling potential that I’m personally unable to tap into on my computer.
Which is why I think it is worth raising awareness. I’m sure other people with more access to hardware/GPUs/brain can do a better job.
For reference I have a 4090+9800x3d and 64GB of RAM and a 4TB NVME SSD so these are my limits. I’m not trying to flex but this is not enough. My dataset is already at ~340GB which means I’m completely bottlenecked by SSD speeds. One training run being 38 epochs at 5 minutes per epoch so I have a little more room for scaling before training times become unbearable for me but It’s not much every additional GB of training data adds seconds to the epoch time. I’m still seeing performance improvements whenever I add more data to my training set.
Inference doesn’t require such expensive hardware but a GPU is needed unless you want to wait days.
Anyway this is the part where I could post some impressive cherry picked examples but that would be dishonest so if someone wants a sample output post a video and I’ll generate something.
There’s a lot of content that is unfit basically anything which a human scripter would struggle with this model will also struggle with. 2D/3D animation will be very hit or miss my training data is very limited when it comes to that.
For VR content I need to do a 2D re-projection and aim the camera at the “penetration zone”. In general you can improve the models output by controlling the video input by for example cropping videos removing unneeded background and such. V-JEPA processes frames at a 384x384 resolution so any black bars or wide shots waste signal and cropping helps increase signal in that case.
The model struggles with fast strokes likely because I’m only processing it at 15 hertz but increasing that also increases everything else in a very linear way (latent caches and amount of training) so it’s not an option for me.
Outputs are by no means perfect funscripts and even the best ouptuts still need some manual touch ups. Although I’m sure some people would find it “good enough”.
Ideally I’d like people to use it as a first draft for funscripts that they then still polish by hand instead of uploading raw slop funscripts.
Which leads to the last question: should I release this as a tool that people can use? I have no intention of turning it into a product and selling it. I also believe anybody can build what I’ve built here. So it’s not that special.
I already have it packaged as an executable that you can just run but I hesitate to publish it. want to to do a vibe check first.
@serendipitis I did see your post in that other thread but at that point I was already knee deep into this
I’ll share more of my findings once I’ve released it. I don’t plan open sourcing it because it’s just a bunch of claude code authored python scripts nobody needs that. And it’s kinda useless without a dataset which I also can’t really share.
The general methodology is more interesting.
Awesome. Please make this work open sourced. If the community can participate by training these models, with something similar to seti@home, even better. This space can use some healthy competition. The end goal should be to have democratized, decentralized and cheap access to generating funscripts for any video we want to sync. Then this niche may finally blow up and become widely adopted, and not under the control of any one entity.
I agree with the vision but seems hard to do given copyright of the training material and the volume of data. It’s just not very crowd-sourceable. I guess I can share the python scripts but I stand by it not being useful.
Currently all I need is high quality scripts and videos with excellent synchronization and minimal effects/filler. The rest this model learns by itself.
So if someone wants to source me with that I’m down.
Can someone please ask me to generate a script so we can all determine if it’s rubbish or not
Or if I ever get to it, I will release the training and inference code for you to bring your own scripts and videos. Sidesteps the copyright problem. These are relatively small models.
For a little something I’m working on myself - not another tracker/generator, it’s about finetune/post-processing tracked scripts - I’m on the lookout for exact the same.
I’d have 1,5 more - not released, scripted myself, drop me a DM if you’d like to get these, too.
I’d like to get (and could use) the result from your tool for all of these, will work with these scripts (handcrafted + the tracked ones) anyway, feedback and comparing would be no problem this way.
As far as I can understand the tech language… Sounds cool! Thanks for linking my script and the compliment. Feel free to use any of my scripts for training the model (op included).
I’m still waiting for someone to hit me with a video that I can generate a funscript for. To manage expectations and so that I don’t feel like a fraud. Just anything
Personally I think it’s cool but you all haven’t even seen a single output.
Sounds almost ridiculous to ask but you and @k00gar should talk. Not saying you should join forces which would be most beneficial for the community, but you burnt out with OFS as I read and K00gar is on the best way to it spending literally day and night programming Fungen. I am sure you can do so without revealing any secrets.
Really like the frozen-encoder + tiny-head design — that’s the right call for a small-data regime, and caching the latents so a 3.5M head trains in minutes is smart.
One thing that’ll help you more than any architecture change: build a real evaluation harness before you tune anything else. Right now you (and the rest of us reading) can’t tell a good run from a bad one, because cherry-picked clips always look fine. What worked for me:
k-fold on your corpus instead of a single train/test split — one held-out clip’s score is noise.
An “A/A floor.” Have two humans script the same video and measure how much they agree. They won’t hit 100% — so 100% was never the target. I express model quality as “% of the human ceiling” (the human-to-human agreement gap). It reframes everything: my model sits around half the human ceiling, which sounds humbling but is an honest, stable number I can actually optimize against.
Watch for leakage — if the same video appears in train and test (even a different clip from it), your score jumps and it’s a lie. That single mistake inflated one of my numbers by a lot before I caught it.
A few findings from my own testing that might save you GPU time:
Depth cues don’t help. I benchmarked adding monocular/axial-depth features expecting a lift for POV content — got essentially zero. Don’t spend a weekend on it.
Raw data quantity plateaus faster than you’d think. I kept adding clips and accuracy went flat well before I expected — the lever turned out to be feature/representation quality, not more hours. Worth knowing before you assume 340 GB → linearly better. (Your encoder swap is probably the real lever, which is why your approach is interesting.)
A “filler” / rhythm layer kills most of the manual touch-up. Instead of going flat during gaps where motion is ambiguous, continue the established stroke cadence. Made outputs feel far more finished with no accuracy cost.
Clean the training set by duration. Heavily-truncated or over-long clips poisoned my training — I cap clip length and drop the badly-truncated ones.
Two things I can confirm from your post: 15 Hz really does cap fast strokes (same wall I hit), and for POV/VR, cropping to the action zone beats feeding the whole frame — you’re already doing the re-projection version of that.
I have built my own AI environment for this and more. Below is a short description of my stack.
The Environment A self-hosted, multi-node AI cluster running entirely on owned hardware. Local-first by design: large-language model inference, image & video generation, voice cloning, computer vision, and fully automated content pipelines all run on-premises with no per-call cloud cost. 100% Self-Hosted Zero Cloud Lock-In 4 GPUs · ~66 GB VRAM 9 nodes COMPUTE FLEET Local LLM · 14 Models ~66 GB ROCm + CUDA + Vulkan ~290 GB AGGREGATE VRAM Compute Fleet NODE COMPUTE RAM CLUSTER-WIDE ACCELERATOR 70+ TB POOLED STORAGE
This isn’t about OFS or fungen. I don’t care what editor people use. I plan on releasing this as a standalone app. Any editing will have to be done in other software.
I take what I can get the Ariana Marie sample is a tough one because the camera man loves rule of thirds getting that christmas tree in frame all the time (wasting screen space). I definitely think hoping into davinci resolve and cropping this one a little would improve the output but here it is raw. The model also has penis bias I don’t think there’s cunnilingus in the training set.
The Clea Gaultier one is a much better fit took like 5 minutes to generate half of which was spent in ffmpeg normalizing the input. Clea.funscript (175.0 KB)
I’ll post the VR one later still trying to source a low resolution gearvr version because crunching this down to input resolution using ffmpeg at high resolution takes longer than any inference
More training data still seems to make the model better but certainly not linearly. I think there’s so much variety to capture those 340GB are just 277 videos ~100 hours so I do there’s still lots of room. The current set has a strong BJ bias.
I have never tried using the distilled 300M parameter V-JEPA 2.1 model so I can’t say if provides better results for this application. I need to do some cheap A/B screening runs to see if makes a meaningful difference.
But switching at this point at corpus scale would mean I would have to recompute the entire latent cache which would proably take like ~20 hours full blast with the larger model. Where as with the current 80M model it’s more like ~8 hours. So the larger encoder would really have to show much superiour performance.
I think there’s some untapped inference tricks to combat the 15 hertz limits because theoretically if you have a 60 FPS source you could dynamically time warp the footage making fast sections playback at half speed so the model still processes at 15 hertz assuming everything is real time and then in the end you’d correct for the time warping. I think a smart solution is required because just bumping the processing rate is just really expensive.
I see a lot of potential here. The timing is more or less okay and follows the movement. From the issues I noticed, you should add some kind of post-processing algorithm to stabilize the stroke length (reducing the impact of camera movement and noise).
Yes that should be pretty easy to do.
I already have an app where you run the inference and afterwards when it’s done there’s a small review stage where you can change some settings that affect the decoding. basically how the models output is translated into actions. And there you would be able to do small tweaks in real time. There’s already such things as intensity, speed limits so adding an optional setting which tries to keep stroke length consistent would fit there perfectly.
