r/ffmpeg • u/Coollius • 7d ago

ffmpeg is missing S_TEXT/WEBVTT mkv embedded subtitle support - happy 10 year anniversary!

This ffmpeg issue has recently passed the 10 year open mark. The tldr is that ffmpeg doesn't know how to read the matroska way of mkv embedded webvtt subtitles.

This has major downstream implications, for example jellyfin cant stream these subtitles if they're embedded, because ffmpeg cant read them. Ive seen so so many tools have the default of converting webvtt subtitles into srt or even ass just because webvtt ends up not working with ffmpeg based projects when the subtitles are muxed into an mkv. I've looked at this issue so so many times each time thinking, somebody will get to it eventually. There are even implementations to add support for this out there, but for some reason none of them have been merged. I have thought about tackling the issue myself but am doubtful of my abilities of contributing quality code considering the sheer downstream implications of any changes made to ffmpeg.

Just had to vent about this somewhere after a webvtt to srt conversion lost some important formatting in a movie once again that annoyed me. I could of course do seperate subtitle files but I must say single mkv files are something I value quite a bit. If anyone got some motivation and some pull in the ffmpeg project, I'd be very grateful to see this implemented & merged. Also shoutout to all open source developers who do all this for free, I know I'm not owed anything.

10 Upvotes

92% Upvoted

7 comments sorted by

0

u/archiekane 6d ago

You could always submit a PR and write the patch, as long as it doesn't have any knock on effect.

-3

u/chocolateAbuser 7d ago

and it also lacks an input that works well with programming languages like a json instead a randomly escaped command line

1

u/Malsententia 6d ago

What sort of language are you using where escaping the arguments is such a problem? I've wrapped ffmpeg in various ways for a couple projects in a couple languages and haven't had to worry about such things. Just call it like any executable via spawn or exec or whatever and pass it the arguments as an array. Easy peasy.

-1

u/chocolateAbuser 6d ago

it's easy peasy if you have command line like "ffmpeg -i something -c:v bla something.else"
anything more complex like weird filenames, articulated complex_filter, and possibly other stuff is not covered at all because languages don't have a really well made escape system, i had to do it on my own
first because this could run on windows or linux, then because different shells could have differences in escaping (|, ", ', &, >,, and such), and then there is the escaping of ffmpeg itself (tell me if you remember how many \ are required to escape a : without looking)
the point is this should be entirely avoided by having an actual programmatic parameter passing system; it's quite an endeavor, i know, but for professional a command line is really not right

1

u/Malsententia 5d ago edited 5d ago

But what language are you using where special escaping arguments passed to a separate executable is even necessary? It's easy to pass filter_complex [0:v]crop=1920:960:0:60,libplacebo=tonemapping=bt.2390:colorspace=bt709:color_primaries=bt709:color_trc=bt709:format=yuv420p,scale=w=720:h=360[vid],[vid]fade=in:0:6,fade=t=out:st=295.83:d=0.2[vid2] without trouble.

ffmpeg used within another program (excluding like, idk, shell scripts): Escaping things is a non-issue, because everything's presumably being passed as an array of arguments. Be it execv in c, or spawn in node, or w/e

ffmpeg used from the command line directly: well yeah, because that's how using the command line is for anything ever. And that sort of usage by nature is presumably one-off instances. And tab-complete generally has your back if it's a weird filename.

As a real-world example, over the past couple weeks, in my spare time, I've been throwing together a node.js pipeline that processes arbitrary files in order to have them play on a specialized site I help run. When I say arbitrary, I mean, "can handle damn near anything thrown at it.". HDR color? auto detects it with ffprobe (with json output) and then applies the proper libplacebo filters to convert it to standard color. Various international credits, or netflix intros, etc? Auto detects that with blackframe and blend, and cuts those out, and adds a brief fade in/out and afade in/out. Scaling the video to a set of varied resolutions? does that too. Automatic surround sound detection and conversion using various downmixxing weights, better than simple -ac 2, yep, it does that too.

And if I really want, I can feed it a filename with emojis.

https://i.imgur.com/FsvtiC9.jpeg

It's effectively a spiffy wrapper intended to reprocess damn near any video downloaded off the internet/torrents/etc, for playback on a specialized site.

And to actually test my own claims, I just copied the whole thing over to my windows vm. Honest to god, this was my first time even trying it on windows. To see if I would have to change or worry about anything different. It just worked.

https://i.imgur.com/QTKeMQh.png

The one thing I didn't have to give any extra thought for: escaping characters 🤷‍♂️. At the end of the day, it's just assembling an array of [-parameter, value, -parameter, value], and you're good. Maybe a little extra string ops for joining the varied filters with ",", and some [chaining].

I can't think of any particular way to feed ffmpeg that would be more efficient.

1

u/chocolateAbuser 5d ago

we are using mainly c#, other people in the company use also python and go; we used to call ffmpeg also by php and vbs
there are a group of issues that bother me about this that affect each other, command line is annoying not only for escaping but also for the order/structure of the arguments when you have to build a pipeline, and a clearly (to an extent) structured object is the usual way people adopt to send commands built from various parts
filter_complex is not that much of an issue when when it's simple, on windows often i don't even need to escape it at all to set the usual encoding parameters, and files should be sanitized before getting there too
but there are more difficult cases (and obv. once it's resolved it should be good and work at least most of the times, but getting there is really annoying), i don't have the will to go dig up what i found over the years sorry but one easy example: if you have to draw text you set the filter, cool, then you need to set the font, on linux it's /bla/bla/somefont, should work without that many issue, on windows there will be something like a c:\folder\font or some other type of path, network, unc, whatever, and you have to escape all that (ps i always keep the file: prefix, i always try to be as explicit as i can)
the issue with text wasn't really about having weird things like emojis, as i said it's more about the characters that the various shells may interpret with different priorities, like % & |, because if you make a bat it works in one way, if you run it in cmd it's another, if you start the process with some settings it behaves one way, with another setting it could be different, etc
also there could be very long arguments, thousands of characters or more, yes, there is the possibility to put filter in a file, it doesn't solve all situations afair and also i would like to not use a file i have to clean
another thing is using other inputs rather than just files, we use ffmpeg also for other stuff rather than encoding, sometimes you have to use urls, sure you look at what char is pestering you this time, you find a way to escape it, all clear, but if there was a way to avoid this and to add useless logic it would have been better
so other thing would be having a structure, especially as i said before when you have to build a pipeline and you don't remember if for example putting an hardware encoder switch in a certain position it gets instantiated before or after other parts of the internal pipeline; maybe it's more gstreamer territory but and some coworkers use gstreamer too, i feel sometimes ffmpeg still would be more appropriate, but man
other unpleasant part could be connecting multiple inputs and multiple outputs, again positioning all the switches, building the graph... you do what you have to do, it's not simple to begin with, and you get the habit, but once you stop using it for a while and restarting again it's headaches