TLDR at the bottom
First…
What is VFR: VFR, or variable frame rate, is essentially when a video has different FPS values across the length of the video; so one section is 25fps, another is 28.56fps, another is 100fps. This causes “speed” issues in playback because each frame has a position in time where it is supposed to be. When a CFR video is converted to VFR, all of the frame data for the video is shifted a microscopic amount for each inserted frame. That leads to an imperceptible to the eye drift in the timing of the video. Funscripts are based on their own independent clock-time and are not locked to the video frames. This means that over time the scripts will desync from the video and drift.
VFR video fix here: Variable Frame-Rate Detection + Re-Sync Tool + Duplicate Frame Fix (Python) [UPDATED, AGAIN]
To answer @roa’s question … first a caveat. I am not 100% certain about anything I write below. These are mostly my suspicions based on the information/evidence I have available to me. I’d need to do a lot of testing (and my computer isn’t really built for that sort of testing) to be certain. I am also limited in my access to upscaling software; I can’t for example, test the newest versions of Topaz AI or any other AI Upscaler. So anyways, time to state my opinion like it’s fact…
My current theory is that it has to do with early versions of Topaz / versions that have frame interpolation built in, bad workflows, or folks who don’t correct for VFR after upscaling and/or interpolation.
Versions of Topaz AI – Topaz used to consistently drop frames during upscaling which would lead to the production of VFR video (until version 6.0.2 or 6.0.3 … for reference, the last non-subscription based model that people can access, and what I use, and what is frequently recommended for folks to use, is v2.6.4) and if frame interpolation is used on VFR video some really terrible things can happen; one thing is that it potentially locks in the desync from VFR so there’s no easy way to realign w/ scripts. Essentially, the VFR fix I published above will no longer work after frame interpolation because the framerate will be set via duplicated frames, and you’ll have to either decimate the video or use a brute force method at that point (like JAMR.)
Bad Workflow: If you are upscaling a video and want frame interpolation (aka to go from 30fps to 60fps or 120fps) you really ought to do the interpolation FIRST and then upscale. Not only does that lead to better looking video, imo, it also doesn’t run the risk of you “locking in” VFR fragments in the video. I personally recommend Flowframes (which g90ak told me about, thank you g90ak) using RIFE since it does much cleaner interpolation than Topaz. Then do your upscale. If you are upscaling a CFR video (constant frame rate) then after the upscale you should either use my VFR script above or use this ffmpeg command to detect whether your upscale is VFR:
ffmpeg -i input.mp4 -vf vfrdet -an -f null -
If your video upscale became VFR after upscaling, then there is a chance it will be desynced with the script. Test it, and if there is desync I’d recommend my VFR Resync script above (since it will detect hardware acceleration and run the conversion to CFR much more quickly than a generic ffmpeg prompt) but if you really want to do it by hand:
ffmpeg -i input.mp4 -r 30 -vsync cfr -c:v libx264 -crf 0 -c:a copy output.mp4
In the above ffmpeg command, r is the desired framerate. You’ll need to estimate what the video ought to be (again, my python script does this automatically).
Now… if the video you are upscaling is VFR, it’s sort of just tough luck. Basically you’re fucked. If you convert to CFR, it will desync. If you do frame interpolation, it will probably desync. If you upscale, it will likely desync. The script original made for a VFR video will need the grace of god on its side to match with an upscaled, frame interpolated video. Not saying it’s impossible, in fact, I bet there are plenty of cases on this site of VFR upscales that sync okay. Just know it’s gambling. Never tell me the odds and shit.
There are, of course, a million other ways that changing a video can alter the syncing of a funscript. VFR and frame duplication + deletion are just the ones I have been in an autistically-honed focus for the weekend haha.
Good luck with your upscaling!
TLDR bullet points of what to do to upscale a video that already has a script:
- Check if the video you’re upscaling has a constant frame rate using my tool or some other method.
- If it is VFR, weep. If it is CFR, then first and foremost decide if you’re doing frame interpolation (raising or lowering the FPS of the video.) I recommend you use a seperate program to your upscaler (like Flowframes) and do your interpolation before upscaling.
- Run your upscaler.
- Run my tool again (or do whatever you did before) to check if the upscaler created a VFR video. If it did, check for desync. Then I highly recommend you just let my python script convert it to CFR.
- If the video is not VFR but still has desync you can try to use the frame decimation function of my tool. If that doesn’t work, weep!
- If all else fails you can try using JAMR to update the funscript to match your now horrible, buggy, awful upscale. Yay!