So the audio recording for iOS was practically finished today, FWSoundMixer was modified to deal with some stuff breaking the recording in HungryHero(and other games/apps in the future as well). I’ve got the HH gameplay video with audio saved in camera roll, was almost ready to publish the video to YouTube (and here), when I noticed the audio getting slightly delayed / out of sync. I wasn’t too sure if this delay was “fixed” (that would be less serious) or growing. Unfortunately, it was growing – started to be apparent when I made the recording longer.
I realized the method I’m using for recording audio on iOS currently might not be ideal. The ANE is recording all the audio into temporary WAV file – this is composed by AVFoundation with the video track after the recording is finished. The obvious problem is, that this raw WAV file doesn’t contain any info about time when the audio packets are coming in. I’m still not exactly sure why the video tends to be a bit shorter than the audio after recording(apparently losing video frames maybe because of lags), in any case because there’s no way for AVFoundation to sync audio/video, it just slaps them together and the longer audio track causes it to go out of sync with the video slowly.
Rather than trying to even the tracks I decided it’s not good to rely on ideal recording situation anyway.
Long story short I’ll need to try recording all the audio with AVFoundation & encode it on the fly (like the video) so that its composed as its coming in. I hope AVFoundation takes care of the interleaving & mixing, just as FFmpeg does in the desktop versions of FlashyWrappers.