Month: November 2014

FlashyWrappers 2.21 was released! Grab it here: http://rainbowcreatures.com/product_flashywrappers.php

All of the changes and fixes included are exclusively for the iOS version, which needed some stabilizing before starting on Android or iterating on the other platforms. From changelog:

- iOS: Fixed an issue generating internal OpenGL error which would demonstrate usually as errors in Stage3D when creating vertex buffers during capture
- iOS: Fixed a bug which caused the screen go black for various seemingly random reasons, such as texture upload (only FW logo would be visible in the free version)
- iOS: Fixed the wrong video orientation outside of iOS
- iOS: Fixed buggy Retina implementation causing the screen to possibly scretch while capturing
- iOS: Added different app / movie fps handling for captureFullscreen (you don't need to worry about your target fps now - only on iOS so far!)
- iOS: Audio is being captured directly into mp4 stream. Capturing to temp WAV stream and merging later with video is still supported if needed to save few fps while recording as well
- iOS: No longer needed depthAndStencil enabled for captureFullscreen to work (this was related to a bug which has now been fixed)
- iOS: You can now call captureFullscreen() all the time on ENTER_FRAME, it will capture only after myEncoder.init() and stop after myEncoder.encodeIt()
- iOS: Added platform default for testing outside of iOS

Extra for iOS: Hungry Hero (starling game) integration example

Extra for iOS: Latest version of FWSoundMixer for iOS inside the HH integration example

There will still be work going on iOS – hopefully to get rid of needing FWSoundMixer for audio recording & eliminating the video post processing time & fix any new bugs that appear.  As you might have known there was also work going on FWSoundMixer for iOS. This is also close to finish, but the microphone issue needs to be stabilized / more understood first to make a more “stable” release. Nevertheless, the latest ANE of FWSoundMixer is included in FW 2.21 (as that case doesn’t need microphone), in the HH Starlig iOS example.

Also in the plans is further simplifications of the API for all platforms.

 

Althought the iOS issues in the encoder were solved, there is one more thing – in the related FWSoundMixer for iOS, which is going to be released as well in the next batch. Everything is working fine, except for when recording with microphone. As soon as sound mixer starts recording microphone, the rest of the audio playback goes laggy. This results in more strange things as a result when playing back the video(mic being my favorite thing – out of sync 🙂 ).

I’m not entirely sure, but it seems to me when using SampleDataEvent for both microphone recording and also as callback for making sound mixer perform “audio step” creates some kind of collision. In other words, SampleDataEvent with microphone capture and SampleDataEvent for playing PCM sound (you can do both), when used together on iOS seem to cause lags. On desktop I cannot either notice it or the CPU is so powerful it’s just not apparent. Also, I might be just doing something wrong.

Right now trying to find a workaround either by eliminating the SampleDataEvent for audio recording or trying to record microphone natively, even that seems to be problematic as expected (fight with AIR over microphone ensues probably). Resolving microphone is important for integration into several apps that I’m about to do.

Managed to fix the bug I was talking about yesterday. Long story short, we are sometimes our worst enemies – and this bug was entirely of my own making(as most bugs are). I had to analyze OpenGL pipeline on iOS and compare several cases to finally spot the obvious thing (and before that I had to comment out lines from Starling source for hours to isolate the problematic command in AS3 :). I was passing the wrong argument to OpenGL function. This caused the ANE working only under certain conditions, but not working under other conditions. As soon as texture has been uploaded before init, it would break the “needed” conditions. Internally, the texture ID had to equal FBO ID, because I was using one instead of another.

Anyway, this fixes the seemingly random “black screen” issue! So we’re back in “release mode” with Hunger Hero Starling demo and the next FW. No more examples / features will make it into the next release.

The last part for me was to implement the recording only after a level is started (and end after level is lost / hero dies). If this works (and after this fix, it should, because it won’t matter when “init” is called anymore) – I’ll start preparing the new release package, with the new Hungry Hero sample included.

I’m sorry this release is taking longer than expected(and sorry to everyone waiting for some kind of demo / help we discussed over e-mail) but I really needed to nail down these big iOS issues. As a result 3 major issues killed on iOS (this upload texture interference, earlier vertex buffer upload interference, problems with audio sync / timing). I won’t get everything in that I wanted(killing the video post-processing will be more tricky than I thought for example), but gaining stability is #1 goal for this release. Everything else can be only improved from then on 🙂

In case you’re wondering whats currently going on: I’ve been hunting down a strange bug which demonstrated with my Starling game sample on iOS. It demonstrated only when FW was initialized at frame greater than zero (0). If late initialization happened, black screen was displayed while capturing (with FlashyWrappers logo). This is last *known* serious bug needed to be fixed on iOS. Of course I can’t rule out more Stage3D related bugs as the OpenGL might be intricately interacting with the ANE’s changes, but hopefully not.

But even from this bug it seems, if I move internal initialization of the texture cache and other OpenGL related things before anything else happens, it is fine. Which makes sense, at startup there’s little OpenGL code coming from Starling/Stage3D to collide with the initialization inside the ANE. I still want to see the cause of the collisions (in my ideal world I should be able to call the init late and it should avoid all issues / collisions).

But if I can’t prevent  / figure out the collisions quickly, I’ll just force / move the custom OpenGL inits into early code which will be called either automatically or manually by some method, as “workaround” to prevent any similar collision related bugs on init.

Many / slashes / in / today’s / post 🙂

It seems the most serious issues with audio sync on iOS have been fixed. By capturing video frames on ENTER_FRAME, it was necessary to insert some kind of time “gateway” as when to write a frame or not given the desired target video fps. In general, you want the library to write every other frame if you’re targeting lower fps than your game is running at (which you should). As soon as this was done, the sync issues went away. This gateway is in the native code, taking advantage of the precise measurements of time elapsed on iOS.

But majority of work was spent on rewriting the “temporary” audio recording into WAV. Now both the audio recording and encoding are happening realtime, the audio is being encoded into AAC and inserted into the MP4.

So to sum up, now the audio seems to be in sync all the time, for any video fps. For example your gameplay is 60fps, your video is 24 fps, your audio should still be in sync within the 24 fps video, even originally it was in sync at 60fps. The only exception would be if you tried to record at fps higher than the game is capable to render. That might mess things up. For example if you recorded the video at 60fps but your game rendered only at 50fps, you would be “missing” video frames and got the out of sync with audio again.

This is a situation you don’t want to get into in the first place. You might just play it safe and set the video fps to something like 20fps. Or even better, you can do a little test recording, measure the fps and set your “real recording” target fps 15%-25% lower to leave a buffer for any “bumps” in the fps (or perhaps if the target is under acceptable number such as 20, resolution of video would need to be decreased as well).

Which is a good idea for a new library feature. One method to determine the ideal fps for your videos so you’re not stuck with one fps, but instead people with more powerful iPads can record at better quality 🙂

Anyway, coming up before the next version release are still 1 or 2 fixes. Right now I’m trying to completely get rid off post processing of the video on iOS. It can take quite a long time(think ~10 seconds for 1 minute of video recording…on one of the latest iPads). It’s async, so it doesn’ t freeze the game or anything, but it’s still annoying to wait after the recording was finished. This is mostly only to flip the video upside down, because of OpenGL. In the next iOS release, flipping will very likely be done straight on the texture by OpenGL to avoid the costly post processing.

 

 

So the audio recording for iOS was practically finished today, FWSoundMixer was modified to deal with some stuff breaking the recording in HungryHero(and other games/apps in the future as well). I’ve got the HH gameplay video with audio saved in camera roll, was almost ready to publish the video to YouTube (and here), when I noticed the audio getting slightly delayed / out of sync. I wasn’t too sure if this delay was “fixed” (that would be less serious) or growing. Unfortunately, it was growing – started to be apparent when I made the recording longer.

I realized the method I’m using for recording audio on iOS currently might not be ideal. The ANE is recording all the audio into temporary WAV file – this is composed by AVFoundation with the video track after the recording is finished. The obvious problem is, that this raw WAV file doesn’t contain any info about time when the audio packets are coming in. I’m still not exactly sure why the video tends to be a bit shorter than the audio after recording(apparently losing video frames maybe because of lags), in any case because there’s no way for AVFoundation to sync audio/video, it just slaps them together and the longer audio track causes it to go out of sync with the video slowly.

Rather than trying to even the tracks I decided it’s not good to rely on ideal recording situation anyway.

Long story short I’ll need to try recording all the audio with AVFoundation & encode it on the fly (like the video) so that its composed as its coming in. I hope AVFoundation takes care of the interleaving & mixing, just as FFmpeg does in the desktop versions of FlashyWrappers.

Just added new Twitter account for FlashyWrappers which you can follow to get updates or new FlashyWrappers releases!

 

Follow FlashyWrappers here: https://twitter.com/flashywrappers/

UPDATE: Managed to resolve the vertex buffer issues! The ANE was calling nonsense argument at one point which triggered OpenGL error, which demonstrated only when creating vertex / index buffer arrays in Stage3D. I won’t bother with technical details(thanks to Apple Instruments for OpenGL ES though!). Suffice to say no more Stage3D errors when recording Hungry Hero, hurray :). One step closer to the next release.

---

 

Part of resolving the issues I mentioned in the last post is testing the libraries on iPad1 as well.

I’ve tested AIR video recording with the Starling Hungry Hero game on this poor old device today and the results are (note, the first fps is the video recording fps, the second fps is the observed / measured gameplay fps while recording):

Realtime audio recording on (with sound mixer recording all the game sounds / music – this guy is eating a bit of CPU himself)

iPad1 1024 x 768 @60fps: ~30fps

iPad1 1024 x 768 @24fps: ~35fps

Realtime audio recording off (without sound mixer – you might add background music to video still)

iPad1 1024 x 768 @ 60fps: ~34fps

iPad1 1024 x 768 @24fps: ~40fps

 No recording at all (audio / video)

iPad 1: ~60fps

As you can see even the video encoding takes pretty drastically like 20-30 fps off your game, the fact that it works on such a slow device as iPad1 is to me mindblowing, especially remembering the earlier results without any AIR OpenGL capturing. Pretty much it had issues even on iPad Retina with half resolution recording at 20 fps.

Also, even in the worst case it’s keeping the game pretty playable at 30fps on iPad1.

…so now “just” to get rid of these issues I discovered yesterday.

But to end on positive note – you saw I’m mentioning “realtime audio recording”? 🙂 Yes, it ALMOST works now. The only remaining issue is that the audio doesn’t want to merge with the video when the video is flipped upside down (this is a needed thing as we’re capturing from OpenGL where everything is flipped..so we need to flip back). I laughed at this one a bit, maybe the audio has nausea from all the flipping. Anyway, I think / hope this should be pretty easy to resolve.  Much easier than the vertex / index buffer issue which I’m working on right now. I’m about to analyze iPad’s OpenGL hoping to see what’s going on under the hood in terms of what AIR is calling, to open that “black box” and figure out where I might be messing it up.

 

In case someone is already testing the beta iOS accelerated recording (captureFullscreen) mode (likely you come from Starling forums), please be aware of several issues I’ve discovered in the recent days, while integrating the Hungry Hero game:

• When recording from Starling (and possibly Stage-3D only) project, calling myEncoder.init later then after creating Context3D will produce black screen with FlashyWrappers logo when recording. Ie. if you’re calling init in the middle of rendering (as you normally would) this will happen. I know how to fix it “temporarily” with a simple workaround, I’m not sure exactly why it happens at this point however so I can’t bring “proper” solution quickly. Part of the problem is AIR being essentially a black box doing stuff with OpenGL which I must anticipate / figure out.
• Vertex & index buffers creation is failing during & after capturing video (not before). This was not apparent right away, as the game appears to be working normally but something in the ANE is breaking something else in OpenGL that AIR expects. I’m investigating what this could be. The ANE is creating another set of internal vertex/index buffers in the ANE for rendering the quad with AIR framebuffer to the screen,  I’m not sure how that could interfere however as I made sure to get currently bound buffers / rebind them after using my own.

Unfortunately things like these are more expected with Stage3D as Stage3D is pretty much a wrapper for OpenGL ES and therefore people might manipulate more things in the “AIR blackbox” that might get interference from my ANE somehow. With MovieClip projects there is pretty much a neverchanging render loop so everything should be stable in those projects. I’m already planning on releasing a new FlashyWrappers version addressing these issues (they delayed me from audio recording as they are high priority at the moment).