Audio/video desync as time passes

[Matgoeth]

Hello, vbence! First of all: thanks for this project, even though it's prototype it certainly works wonders!

However, once I start streaming after some time audio and video does desync: I believe audio gets behind the video, but I may need to test this a bit more. As for now I'd like to highlight this issue with somewhat vague info, but I am happy to help if there's need for any specific info (I glanced through the code myself but it seemed like "pieces" of videos [MovieFragments in your class nomenclature] do contain all the data and they are not separated streams but I didn't spend too much time on this).

Anyway:
Receiving end were PCs/laptops with Google Chrome v 96.0.4664.110 (yeah, I didn't check other browsers, just me and my "testers/watchers") - for most of the time refreshing the stream resolved problem for them for next 10 minutes or so.I verified this is also the case for me and refreshing did seem to help.

I was streaming using OBS using rtmp protocol, CBR 5000kbps 1080p with 30 frames (keyframe interval was set to 2, PC is rather hi-end with i9-9900k, so fully capable of streaming such content with very rare frame drops). I was streaming to localhost (so basically there's no latency between OBS sender and stream-m receiver) and I was having 3 people watching which shouldn't be an issue having 20mbps upload (saying "shouldn't", I didn't do any fancy stats.

Of course I am happy to help/log anything more if it's needed: I do have quite good java knowledge, just almost zero knowledge when it comes to encoding/streaming (I mostly do around other types of apps)

Once again, thanks a lot for your app!

[mbattista]

1. how did you watch the stream? Did you call the via the player.html or did you call the stream directly via the /consume/[...] link?
2. "refreshing the stream" means reloading the page on the receiver or stopping and starting OBS?
3. Which version of stream-m are you using?

[Matgoeth]

Hiya, @mbattista !

1. Stream was watched via .../consume/... link, didn't know about the player TBH (I've rushed through readme :D) I can try if problem persisth through player if it's in the README.
2. reloading .../consume/... page, I didn't need to reload OBS nor did I restart stream-m (I could, but reloading page just resolved it for next couple of minutes)
3. I used 1.0.1-SNAPSHOT (commit from 31/12/20) I quickly glanced through other forks (yours included) but I declined them since 5000kbps should fit into max buffer size of stream-m in this repo (I believe yours had it slightly tweaked, wasn't sure if BufferedInput/Output would create any difference)

Best,
Mati

[mbattista]

1. I also use the consume directly. But you can try the player. It is in the server.properties.sample example as player-demo. If you serve the zip file unchanged the player will try to play a stream named first.
2. Reloading should not influence anything on the server side. Have all viewers of your stream the same out of sync problem? I use stream-m regularly with OBS and nvenc and have never experienced out of sync problems.
3. Very good. My buffer size was the wrong solution for the Problem. There was a problem with rtmp chunkes. While I tried to increase the Buffer to stop the chunking vbence managed to fix the error. So my fork is way worse then vbence master!

[Matgoeth]

Okay I briefly tested this but it doesn't seem player is better at this (furthermore, it does seem to stop sometimes for no reason but it may be due to my poor configuration).
Generally right now I do have this situation: after couple of minutes watching the stream video is behind by a few seconds. I was thinking if making any video would be helpful but it'll look just like I desynced them in some video edit soft.

Theoretically I can duplicate my tabs to hear the desync even clearer but then let's say one tab would get behind it would blur the results, so I won't make assumptions. However video is clearly "behind" the audio after a few minutes.

Of course I am happy to test/send anything to help.

Best,
Mati

[mbattista]

Improving the demo player is something I wanted to do. But since calling the Video stream works so flawlessly I never prioritised it.

Can you call the stream (/consume/[...] URL) with Firefox or VLC and check if it desyncs there too?

I opened a Bug Report for chromium 86 last year because the ffmpeg which was shipped with this version threw a demuxer error. ( https://bugs.chromium.org/p/chromium/issues/detail?id=1147209#c6 ).

So perhaps chromium 96 has a out of sync bug this time?
You can also try to look at the media pane in the chrome dev-tools if you see anything that could explain the out of sync.

[Matgoeth]

Sure, this may be the case also, I can also try other browsers. But I was thinking you guys can first verify it's not only for me? (cause it may be my misconfigured stream still, then stream-m is okay and it's my fault)

[vbence]

Hi, thanks for all the info you guys collected, I'm trying to reproduce this issue and have one candidate. In my case OBS encodes a 48k audio stream, but in the metadata is reports 44k. This can cause timing issues because I re-calculate the timestamps (instead of trusting the client - which in this case would be better).

I found this issue by looking at a Wireshark dump (using 1935 as rtmp.port will help Wireshark guess the right protocol).

Now I want to parse sample rate out of the AudioDecoderConfig section.

I'm not saying this is the same issue manifesting in your system, but cleaning up the timing code could not hurt.

[vbence]

A quick update: timing looks OK, I tried two different methods, both have the same effect. Although I am experiencing a different audio-issue from what @Matgoeth describes (audio starts stuttering for me after some time, maybe the same issue just my slightly older i5-4590 manifests it differently).

Now I'm looking into mp4 generation, there are a lot of combination of mp4 features you can use or not use. MPEG-DASH defines some of these, but as I see nobody truly implements it. - So now I'll fire up ISOViewer and check out a few chunks youtube is sending, and maybe I can come up with an mp4 that Chrome likes.

[Matgoeth]

Thanks @vbence ! Do you think it's chrome that generates an issue then?
As said, I strongly recommend having two or more viewers, I believe it can be even just two different tabs/windows of same browser.

Ofc I am happy to help if there's something not too deep with streaming/encoding (therefore my knowledge will be sufficient :) )

[vbence]

Hi I created a new branch, audio-retiming where I added exact audio timestamp calculation, this is an improvement by itself, so it can't hurt. Also added two new debug messages.

The first, diffMs will give you the time difference (milliseconds) between the new keyframe and the last audio "frame". Values smaller than 1024 audio samples are hidden. which is normal. This is 23ms for 44100 Hz audio and 21ms for 48kHz.

I added a second one, AAC timing diff you can see it if the timestamps sent by OBS drift away from the recalculated timestamps (based on the actual audio data). This message is only displayed if it is ever greater than 1 ms.

I would ask you to help with two experiments; first you can do everything as you did in the original report, the two questions:

• Is it fixed? (It is possible that the recalculated timestamps fixed the issue).
• Do the new messages show any meaningful information?

And in the second scenario I'd like you to try to reproduce the issue but using a "Media Source" in OBS (instead of capturing audio/video). As media file you can use the same one I am testing with: https://download.blender.org/demo/movies/BBB/bbb_sunflower_1080p_30fps_normal.mp4

• Does the issue manifest in this schenario?

Edit: small diffMs values are now hidden, only suspicious ones show up.

[Matgoeth]

Hi @vbence , good work here!

Sorry for not replying for some time, it's not easy to find some time between years end and another beginning right now. I didn't have the chance to gather more people so I was not able to test it out properly in the same manner but in order not to stop issue I've recompiled branch you've mentioned and simply streamed for about 20 mins to my other device (android smartphone) and...

• issue didn't seem to happen during this time period - however, there was only one recipient of this stream with chrome for android
• I've gathered some evidence ( pastebin here ) during last couple of seconds of my stream but I doubt those will help you. I've managed to spent around half an hour on that and I've tidied up your stats gathering mechanism a bit (see pull request), this should be less spammy (ofc you can tinker with it any way you like in case you want more data)

As for now I am looking for a chance to test it out a little more (and hopefully with wider audience). Anyone else willing to try, please do so.

[Matgoeth]

I've managed to gather someone else on stream too and... here are logs

Generally:

• I didn't encounter any problems with stream but I was watching it from another device like last time (chrome on Android)
• For others however stream was heavily desynchronized audio/video-wise, they also had problems with buffering (there was one other person beside me, bitrate was 5k with 1080p30, Chrome on Windows was used as well as Edge (still, Chromium underneath) which shouldn't be a problem at 30mbps upload of mine (all wired)

Basically I don't know what the problem is unfortunately :(