FlashyWrappers SDK blog

This is a prerelease of 2.4.

flashywrappers.com/download/?file=FW24

Why “prerelease”?

The most important reasons are Android, 2 new ANE’s and examples.
The new automatic test example app is not 100% ready yet to make it into the release, though it helped us to reveal many bugs which would otherwise likely slip through. But its still mostly “internal” code, not ready for presentation. Secondly, the amount of new code is so high that we first want to get this package through some test cases(there is enough people interested to try it out), which we assume will reveal issues that our tests didn’t.

Android – thats a different story. This platform is pretty complex, and solving issue once doesn’t mean it won’t return again(apparently). Which is precisely what happened after adding audio – it was working fine at one point, just before release it stopped working. We’ll be fixing this as literally the first thing for the next release, unless something more serious appears in the established platforms. The Android in general might and might not work from device to device as well.

Lots of new code – there is one more new ANE than what was promised. Apart from the new OS X ANE, there’s also a new Windows ANE (completely rewritten), which wasn’t supposed to be a part of 2.4. It uses MediaFoundation(Windows own H.264/AAC encoders) so MP4’s should now be free & fully legal on all platforms for commercial use, except Flash. This is of course great from MP4 support standpoint, but creates increased risk for more new bugs than usual. Warning, this new Windows ANE will work only for Windows 7 & up.

The 2.4 prerelase comes with a new API which is mostly not compatible with the 2.3 API (another risk) – but don’t worry, it is again easier to use. Just take a look at the documentation pdf to find out how to use it. It’s one of the main reasons (apart from the 2 new ANE’s) why even this prerelease took so long. Various things have been automated / simplified / made internal so you don’t have to worry about them.

In general, various issues in 2.3 still were / are coming in so we were fixing 2.3 issues while working on new 2.4 issues. But of course, by only fixing nothing would ever get released. So we just had to say stop and make this release, as often the best response we had when someone asked how to fix some kind of 2.3 issue – “wait for 2.4”. And we don’t want to keep anyone waiting any longer – if only for the sake to test the new API.

Here’s the changelog of the 2.4 PR1 (prerelease 1):


FlashyWrappers 2.4 Prerelease 1 (unofficial)
--------------------------------------------
- All platforms: All the SWF's / SWC's now assume Flash Player 15.0 as minimum target to work 100% correctly, instead of 11.5. With 11.5 it will still work, except grabbing Stage3D using FW's capture method(you can still supply BitmapData captured from Stage3D).
- !! NEW PLATFORM ADDED: Mac - The iOS ANE has been ported to Mac. The FFmpeg based Mac ANE was removed, so OGV videos are no longer supported on Mac. The Mac ANE is powered by AVFoundation so the MP4's produced have no licensing strings attached.
- !! NEW PLATFORM ADDED: Windows - Brand new Windows ANE for encoding mp4's based on MediaFoundation (Windows native library). The FFmpeg based OGV Windows ANE was removed, so OGV videos are no longer supported on Windows. The new ANE works on Windows 7 and up. The MP4's have no licensing strings attached.
- iOS: Rebuilt against iOS SDK 8.3
- Android(HW): Audio encoding & muxing added - you can record audio now, using FWSoundMixer or by sending raw audio packets to the encoder, same as on other platforms. This has negative effect on FPS currently, we'll be working on fixing this in the future releases for Android.
NOTE: In Prerelase 1 this was working on and off(originally it was, now it isn't) with video / webcam. We'll be looking at it for Prerelease 2 - there was no time to fix this properly till the release.
- All platforms: FW is now working in two basic modes - "realtime" and "non-realtime". Couple of important things work differently, so make sure to read the manual to understand when to use which mode (it should be pretty intuitive though - realtime is for webcam, games etc., non-realtime for anything you can post-process or record at low fps).
- All platforms: Important API changes(simplifications and some paradigm changes):
- All platforms: microphone recording is now automatic / internal if AUDIO_MICROPHONE is used! You do not need to supply audio frames from microphone manually anymore. If you do that, erase the code when using 2.4, instead keep using AUDIO_MICROPHONE. If you insist, you can still do this manually in AUDIO_MONO mode. When using microphone, you can ask for microphone permissions sooner with myEncoder.askMicrophonePermission().
- All platforms: in "realtime" mode, the frames are automatically dropped in case your stage fps is greater than the video fps (for example stage fps = 60, video fps = 20, only every 3nd frame is saved). This was only working for iOS before, so *make sure to remove any framedrop code you might have in your app* - otherwise you'll be dropping too many frames.
- All platforms: in "realtime" mode, the audio and video timestamps(pts) are now based on system time. In "non-realtime" mode, they are still based on constant counters. This fixes audio / video syncing as it could cause "drifting" of video and audio tracks. Especially in cases where your app fps was < target video fps(even having slight drops under that) would cause broken sync and other strange effects (video "speeding up" and being shorter than the audio track for example).
- All platforms: Number of extra checks added to avoid crashes, for example FW won't attemp OpenGL on Android / iOS in case you've published your app in CPU renderMode.
- All platforms if captureStage3D was used: fixed a small memory leak in init, where temp bitmap for context3D was not disposed on each init.
- API change: "encodeIt" removed(!) / replaced by "finish".
- API change: "init" renamed to "start".
- API change: FW Captures fullscreen by default now - in case you want to capture fullscreen, you basically call start(), capture() on ENTER_FRAME and finish(), with no arguments on all platforms.
- API change: Start sets width & height based on stage dimensions by default. If you override width & height and try to capture Stage3D or DisplayObject (including the whole stage) larger than the video width & height, FW will try to scale the captured frames down to fit to video.
- API change: The dimensions can be set separately also with myEncoder.setDimensions(w, h) - this improves the code readability avoiding long list of parameters in start.
- API change: As a logical consequence of the above, "frameScale" was removed from init as the scaling is now handled automatically (so if your captured movieclip is larger than the target video dimensions, it is scaled down).
- API change: "XXX_captureFullscreen", "captureMC" and "captureStage3D" are all beign deprecated. They are still working but FW is moving towards a single method called "capture". With no arguments specified, this tries to capture the whole stage(using OpenGL acceleration where possible, recommended on mobile). If DisplayObject is used as parameter, it works like former captureMC, where available. If Context3D is used as parameter, it works like former captureStage3D.
- API change: "audio" was removed from init, it was replaced by audio recording modes. The modes are called FWVideoEncoder.AUDIO_MICROPHONE, FWVideoEncoder.AUDIO_MONO, FWVideoEncoder.AUDIO_STEREO, FWVideoEncoder.AUDIO_OFF.
- API change: "multithreaded" was removed from init and basically replaced by "realtime" argument. This doesn't explicitely set multithreading on or off but tries to optimize FW for recording in realtime(or non-realtime), but you don't care anymore whenever MT is used or not, its up to the library.
- API change: "tryFinish" was removed. You don't need to check anymore on every frame if all threads finished - this is done internally now. Just call "finish" and then wait for "encoded" status event (see manual for updated code example).
- Flash: Compiled with the latest Crossbridge community. Improves performance, for example reduces the total lag at "start" by about 1/3.
- Flash: Fixed a bug with ENTER_FRAME stage listener not being removed (this was added only at extension constructor so at maximum there was 1 extra ENTER_FRAME listener in the app)
- Flash: Fixed a non-fatal bug in Encoder (null error) displaying in trace / Flash log after end of encoding when using multithreading
- Flash: Added worker communication for sending traces / messages out into main thread if needed (for debugging)
- Flash: Moved the thread (Worker) initialization into "start" instead of starting it at ~1 second into the recording. This moved the slight lag connected with thread initialization at better place and doesn't interfere with the recording at around 1 second.
- Flash: Fixed a bug in FFmpeg based FlashyWrappers - when changing the resolution the frames would keep the previous resolution, causing in either green areas and only part of the frame being saved or crashes.


So where’s FW 2.4? There have been couple of last minute fixes and API changes which make the API even easier to use. Because of the API changes, new cases popped up which might have crashed the app and as these cases are found out, “cleaner” ways to handle them are being implemented (AS3 errors or warnings at least).

The new API, in its latest incarnation, got rid of canFinish(), added internal microphone recording (just by setting audio recording flag to AUDIO_MICROPHONE – no other code needed) and other changes. Its shaping up to be possible to pretty much use FW only like this:

myEncoder.start(fps, audio_recording_mode);

myEncoder.capture();

myEncoder.finish();

(Plus, react to status events). So the API is finally getting closer to where it should be.

This  will definitely be the biggest FW API update yet and because of the number of changes, it might spawn more new bugs than usual – so more testing is needed before releasing it.

Timestamp fix

The most important fix which was added at the last minute, and delayed the release the most: audio / video timestamp fix.

Until now, FlashyWrappers used simple counters as base for pts (timestamps) of audio / video tracks. This could cause issues with audio / video syncing on all platforms, especially in case where your app fps would drop under the target video fps. One drop just for a moment and the audio / video synchronization was damaged.

To fix that(and other things), FlashyWrappers 2.4 introduces two modes: “realtime”, optimized for realtime capturing and “non-realtime”, which in many ways works like 2.3. This was already mentioned, but newly also the way timestamps are calculated depend on setting(or not setting) the appropriate mode.

Realtime mode

• Timestamps for audio / video tracks are calculated based on system time. Doesn’t suffer from sync issues when framerate drops, but “captures” all the lags you experience when capturing live – its live capture, after all!
• Frames are dropped in case too many frames are being sent to the video encoder (game fps = 60 but video fps = 20, FW will drop frames to compensate)

Non-realtime mode

• Timestamps for audio / video tracks are based on internal counters (like in FW 2.3) – ideal for post processing where you are rendering something at 5fps or in the background but need smooth 20fps video out of it.
• No frames are dropped, it doesn’t care about fps and just sends every frame into the encoder.

New example & automatic tests in 2.4

FW 2.4 libraries are almost ready, but one more important thing will happen before it can be released. We’re working on a brand new “example”, which replaces all of the “old” examples. This example will include automatic tests, which will greatly enhance the stability of the platform going forward. Simply put, it will go through all the basic settings (audio on, audio off, record webcam, record realtime, non-realtime, Stage3D, MovieClip, fullscreen…you get the idea) and record a short test video. It will then log all the results, allowing to see if anything unexpected happened.

This way, it won’t be possible to “forget” testing a case, and ensures all the cases which worked in the previous release will work in the new release. In case some new issue appears, it will be added as a new test case in case it’s considered a good idea.

Having a single example of course also simplifies the testing and building on our end. The API is getting so simple(at least to the end user) that one example inside the manual should be enough to get you started.

There’s still loads of work remaining everywhere, but 2.4 will push FW one step further towards the goal of simple, reliable yet powerful video recording SDK for AIR(and possibly other platforms as well in the future).

Apart from the brand new OS X ANE, also coming up in the 2.4 release will be the new (and improved) Android ANE. The new Android ANE was already introduced in the previous release, taking advantage of Android hardware acceleration for video encoding. But as promised – this new release is going to add audio recording & muxing AAC stream into mp4.

As of now, it is possible to capture fullscreen OpenGL and microphone from AIR, the synchronization appears to be working fine as well – exampleCamera runs reasonably smoothly on our test ASUS tablet, at around 35-40fps while capturing video & audio (vs 60fps when not). There is still room for improvement by adding more multithreading, so the fps should improve in the future. Nevertheless, this release is still in alpha and it is not recommended to use it in production.

Windows mp4 ANE

Whats that with all the new ANE’s? Well, as stated earlier, using FFmpeg and mp4 related codecs(H.264/AAC) presents always licensing issues. Even when using OpenH264 for H.264, the AAC part still needs licensing. The core of the problem is as follows: By embedding an encoder(software encoding raw frames into one of those streams) somebody should pay for licensing those encoders – most likely you – the person releasing end-user software.

That’s why we decided to instead use encoders embedded in the OS (ie. Android, iOS, Windows, OS X). It means more versions of source code, which is not great internally, but unfortunately there’s no other way if we want to support mp4 without you having to deal with licensing.

The new Windows mp4 ANE is going to use MediaFoundation on Windows (instead of FFmpeg / OpenH264 which the current one uses). MediaFoundation includes its own H.264 and AAC encoders which FW will use. This will work only on Windows 7+, but we believe that is reasonable considering Windows 10 being around the corner and W7 being the most favorite Windows around.

In fact, the new Windows ANE is already working in the first tests 😉 We’re just waiting for the dust to settle a bit around the new OS X release coming in 2.4.

Few last words

We’ve been looking at why people want to use FW. Don’t think that the form you fill in when downloading FlashyWrappers SDK is used just for some kind of marketing. It can always give us idea where to steer FW if we see some recurring theme.

We’ve noticed quite many people assume FW can capture videos running in Flash. While that to some extent is true (it can capture the picture frames if the video is inside MovieClip), there’s no way for it to capture audio currently. There might not be a way ever in Flash, and on other platforms only after implementing app-wide audio capture where possible (this is one of the next big goals).

But so far FW is not yet that magical box which just captures everything regardless of what it is – though thats the ideal goal, of course 🙂

FlashyWrappers v2.4 is coming up and here are some more details!

As you might have noticed, the versioning has accelerated a bit from 2.3 straight to 2.4. That’s because we’ve finally got a chance to get back to the general API and it is being simplified & cleared up.  Also some long delayed features have been  added to 2.4, so we felt it deserved that .4 mark.

The biggest news is the new Mac ANE, which comes just after the new Android ANE introduced in the previous release. This ANE is not “alpha” because it’s 90% identical to iOS ANE, which was already tested by our customers. This ANE outputs mp4 videos natively  & uses AVFoundation instead of FFmpeg, so licensing issues are gone for Mac (regarding FFmpeg and mp4’s).

As for the API simplifications, one sign it needed work was the growing number of capture methods. We’ve had captureMC, captureStage3D, iOS_captureFullscreen and recently also android_captureFullscreen. Another issue were some slight differences in how .init should be used, in regards to OpenGL capture on iOS(you had to set width & height to -1, very hacky), which resulted in if‘s to use different init configs depending on the platform. There were other issues with init as well (long param list even if not needed), and the names didn’t really express what the methods were doing as the meaning shifted for some platforms. For example,  encodeIt in the stone ages of FW really encoded all frames into the movie, but that is no longer true.

So here are the upcoming API changes you should prepare for:

• init was renamed to start
• in a typical case of fullscreen realtime capturing on any platform, you just call myEncoder.start(), with no arguments. This will automatically get stage dimensions for fullscreen capture.
• if you need to capture downscaled video (typically in Flash to increase encoding speed), you specify your desired video dimensions similar to what you’ve done before and the captured frames will downscale automatically. You can use a new method myEncoder.setDimensions(w, h), or you could also set dimensions using myEncoder.start(fps, w, h, ….). In the past you had to use width / height and frameScale combo to achieve downscaling.
• you can set your target fps either with myEncoder.setFps(fps), or inside start as mentioned above. If you don’t specify fps, it is automatically set to equal the fps of your Flash stage. An important new feature is, that in “realtime” mode frames are automatically skipped if needed to keep the video fps and stage fps in sync. For example, if your game is running at 60fps but your video target is 20fps, FlashyWrappers saves roughly every 3rd frame to compensate for the differences using some timer math. This worked only on iOS until now, while in 2.4+ it works on all platforms. So, you wont’t need to concern yourself about fps differences between video and Flash stage anymore.
• all the XX_captureXX variants are being deprecated (they will still work but a compiler warning displays when using them) in favor of a single capture method. With no arguments, this method tries to capture fullscreen. Otherwise, you can give it DisplayObject or Context3D to capture. Hint: FW will try to accelerate fullscreen capture in realtime mode whenever possible, so use that for maximum speed (on iOS this takes advantage of OpenGL capturing).
• formalized “realtime / non-realtime” modes – FW introduces a new switch in start called realtime. It’s a concept that FW has been working with all the time anyway(even mentioned in the docs), but until now it was up to the user to understand how to enable some kind of realtime capturing in the best way, usually by turning multithreading on (on some platforms), or by using iOS_captureFullscreen on iOS etc. From now on, this decision proccess is shifted towards FlashyWrappers. You just tell FW whenever you wish to capture in realtime or not and it switches multithreading or does other decisions which favor speed whenever needed. It also turns on/off fps syncing (frame skipping), which might not be desirable in non-realtime (post-processing) mode.
• encodeIt renamed to finish.
• all those changes and more which are not mentioned here are trying to make FlashyWrappers even simpler to use and more understandable & readable in the code. This is how exampleCamera FlashyWrappers core source code looks for all platforms in 2.4:


if (platform == 'FLASH') myEncoder.setDimensions(400, 300);
myEncoder.start();
...
myEncoder.capture();
...
myEncoder.finish();


Of course the events & saving video are left out, but it is hopefully clear how much simpler the code now looks. In 2.3, this is your exampleCamera init (now start) platform dependent code:

if (myEncoder.platform == "FLASH" || myEncoder.platform == "ANDROID") {
myEncoder.init(w, h, 20, 1000000, true, true, 44100, 1, 64000, 0.5);
} else {
if (myEncoder.platform == "IOS") {
myEncoder.init(-1, -1, 20, 1000000, true, true, 44100, 1, 64000);
} else {
myEncoder.init(w, h, 20, 1000000, true, true, 44100, 1, 64000);
}
}

 

Of course you’ll still be able to set some of the former parameters, but some of them are gone (like audio, frameScale, multithreading).

The release of the Android HW encoder preview is almost ready (whats basically left is implementing  FW logo rendering in the demo version). Included in FlashyWrappers 2.3 will be also slightly updated iOS ANE. With that being said, here are some more details about what to expect and whats missing in the first “real” Android version.

Significantly increased speed – of course the biggest advantage of the new Android version is a major speed increase, which should allow realtime recording of AIR content in high resolution (webcam, games and so on). The tablet being used for all development tests has the following specs:

ASUS K01A – Intel Atom Z2520, CPU: 2×1,2 GHz, GPU: PowerVR SGX544MP2,

You can get one for < $100. In general, when not recording the fps stays at  constant 60fps, when recording 1024×768 @20 fps video from AIR, the apps fps fluctuates anywhere between 30-45, which leaves comfortable room to record 20fps videos. There is still room for improving the fps and it should get better in the future releases.

MP4 file output – by utilizing the device encoders you’ll get mp4 output natively, no ogv’s. This will alow saving to gallery later on(not yet implemented), or simply replaying the video on the phone, which was not possible with the ogv. There are no mp4 licensing issues (similar to iOS) thanks to using the device encoders.

NO audio support yet – the audio will be implemented in the next release. This release shouldn’t be used in production anyway – we’re curious on which devices the ANE “explodes”, as the video / GLES part is probably the most sensitive and device dependent part.

The release is probably 1-2 weeks away, depending on when the new FlashyWrappers website can be up and running.

And finally, what was that about desktop MP4’s? 

As you probably know, the whole MP4 business on desktop is one big mess currently, when it comes to licensing. Dealing wtih H.264 and AAC licensing +  OpenH264 isn’t exactly saving the day as that’s a partial solution just for the video (and it has its own gotchas). In Flash Player, that’s how it will stay for the time being unfortunately.

However, on AIR Mac and AIR Windows, change is coming. When using  encoders of the host operating system (OS X and Windows), the licensing is not an issue because similar to iOS and now Android, FlashyWrappers is utilizing the OS encoders, which are being licensed by Apple or Microsoft, not you.

On OS X the new Mac encoder is coming sooner than you think, as it’s quite simple to port the iOS encoder and utilize OS X AVFoundation H.264 and AAC encoders. In fact, we have a working Mac version right now and we’re testing it 🙂

Windows will need to be implemented from scratch, so the ETA on that is not known yet. Adding Windows platform and getting rid of ffmpeg in general will mean more overhead unfortunately, but we feel that getting rid of the licensing block is one of the most important thing we must do for FlashyWrappers to be really useful. And useful very much means supporting the most popular video format today (mp4), on all platforms with no strings attached (especially no licensing strings). Well, on all platforms except Flash. There’s no solution in sight there unless Adobe opens up their embedded H264 & AAC Flash encoder to developers. Because again, Adobe has licensed those encoders.

In the neverending quest for MP4 without licensing woes, there appears to be an (obvious) way to make this happen for 2 more platforms: AIR Windows & Mac, still no way for Flash Player. But dealing with those 2 platforms would mean Flash Player becoming  the last platform with mp4 licensing issues.

The answer is – using the native OS Windows / Mac encoders. This would mean leaving ffmpeg for mp4 (goodbye to “crossplatform” internal code for those 2 platforms). But obviously, getting this to work would mean AIR mp4 native encoding, completely free of any royalties or licensing for H.264 / AAC – which is a higher priority than keeping the code comfortable.

So this will become pretty high priority after stabilizing Android.

 

New FlashyWrappers release (codename 2.3) is taking shape right now. This release will come much sooner compared to the last one, addressing some issues, adding several new features and adding the new Android encoder demo.

There are 2 bigger reasons for this release: The new Android ANE and also brand new FlashyWrappers dedicated website.

As FlashyWrappers is growing as a project and there’s a steady demand for this library, it’s time to give it proper separate space, domain and in time hopefully more stuff (such as support forums). It will also contain a little showcase with the most creative apps FlashyWrappers users are working on / were finished.

If you plan on using local video recording and develop using AIR, you’ll hopefully  continue to find FlashyWrappers useful. Here’s the upcoming changelog for 2.3, keep in mind that it might still change as the release is not finalized yet.

FlashyWrappers 2.3
-------------------
- !! NEW PLATFORM ADDED: Android(HW) alpha - preview "first look" Android ANE with mp4 hardware accelerated encoding and OpenGL fullscreen capture. This works only on Android 4.3+.
- iOS: Can specify filename for saving the mp4 to the application storage. You will be responsible to erase these files if needed (instead of the default temp filename which is being reused).
- iOS: Fixed mp4 metatags from QuickTime to MPEG4 for better compatibility, it's possible to play the videos on Android now.
- iOS: Changed internal "hijacking" of AIR's OpenGL FBO - switching texture / renderbuffer inside AIR's FBO instead of switching to secondary FBO increases performance and reliability of the whole solution. 
- iOS: Added support for recording from GPU mode in standard / high resolution in addition to Direct mode.
- iOS: Added the option to more precisely influence H264 encoding quality and level.

Great news everyone! Adobe just fixed the Flash Player bug affecting FlashyWrappers since FP 16, which caused multithreading to not work(which broke realtime recording pretty much, and it also broke the online demo).  This fix comes also as a result of your votes, so thanks to that FlashyWrappers will work normally again since version 17.0.134!

This version is still not officially “out”, but it shouldn’t take long now. The most important thing is we know it’s coming. Time to return to Flash Player issues & requests!