FlashyWrappers SDK blog – Page 4 – Create cool AIR / Flash apps with video recording

FlashyWrappers 2.25 demo is out (full version coming soon)

Finally, FW 2.25 demo is out! Instead of releasing both demo and full versions at the same time, I’ve decided to rather wait for a bit seeing what the feedback would be to have a chance fixing any major issue possibly popping up.

It should only take few days now though in case nothing major appears. The links for new versions will be sent out to current FlashyWrappers customers. If we’re working closely together though and you’re being given updated FW there shouldn’t be much new for you.

So what’s new since the last release? This is the biggest update since FlashyWrappers 2 was released. Not only does it contain many iOS fixes (some of them major, ie. fixing app crashes), but also some new iOS features / optimizations. It’s not all about the iOS though. All desktop AIR platforms(Windows / Mac) and Flash Player are getting the promised OpenH264 codec, which means one important thing: MP4 support.

For Windows and Mac OpenH264 was linked in dynamically, which means you can download it from CISCO as well & avoid having to pay royalties for H.264. For Flash Player downloading is not possible as CISCO doesn’t provide any kind of OpenH264 “SWC”, so you will still fall into the licensing trap (for small apps it doesn’t mean automatically royalties payment though, be sure to reserach the H.264 subject if interested).

For all platforms, the API has changed and simplified slightly as well, wiping out the need to use some methods completely (see changelog), renaming others.

There’s still intensive work going on, primarily working on iOS as there are still some issues needing to be solved before it can be considered really stable. At the same time, my attention was turning towards Flash Player & dealing with video replays, BUT, then the Flash Player 16 bug happened and all of that lost priority until Adobe fixes its multithreading (very sad). If you didn’t vote yet for the bug mentioned in the previous post, please do! It’s very important to get this fix done.

So until Adobe fixes that, the next thing to do will be trying to recompile FFmpeg for Android with hardware acceleration support. This might be kind of a “shortcut” before introducing fully accelerated Android encoding. I’d say it could be at about 40-50% the encoding speed of iOS (compared to like 5-10% of what it is now), so it would still be a nice step.

Anyway, getting HW accelerated encoding is important for Android, also because this will produce native MP4 videos! Which will close MP4 support on all platforms.

Work is going on FlashyWrappers practically every day, it’s mostly bug fixing so not many updates are given on this blog, but be assured that the library is improved all the time. Also the website backend was improved (had to deploy totally new payment backend because of new EU VAT rules). That sort of work is not really visible but needs to happen to keep the FlashyWrappers project going.

Get FW 2.25 demo at the product page (use the blue download button):

http://rainbowcreatures.com/product_flashywrappers.php

Here’s a detailed changelog of FW 2.25:

FlashyWrappers 2.25 ------------------- - !! NEW CODECS SUPPORT ADDED: Flash Player, Windows, Mac - added support for: MP4 container, OpenH264 video, AAC audio. This means you can record MP4 files in AIR Windows, Mac and Flash Player as well (not Android, HW encoders which will come will output MP4's natively). Note, those are all BSD or LGPL codec implementations, however they still involve patents. - New known issue: Flash Player 16 and above multithreading broken. There's a known bug breaking multithreading support in Flash Player 16+ for all Crossbridge apps/libs, including FlashyWrappers. Please force Adobe to fix this by voting here https://bugbase.adobe.com/index.cfm?event=bug&id=3938269 - All platforms: Removed getVideoFramesSent(), added getEncodingProgress() which returns Number 0-1 to report on encoding progress. - All platforms: setFrames() renamed to setTotalFrames(), and it's now needed only when in non-multithreaded mode to set in advance how many frames are expected. Multithreaded mode doesn't need this method anymore. iOS doesn't need this as well, like in previous versions - All platforms: new method naming rules - methods exclusively available on platforms are named "platform_method()". For example "captureFullscreen()" becomes "iOS_captureFullscreen()", "saveToCameraRoll()" becomes "iOS_saveToCameraRoll()" and so on. This is part of the general API cleanup. - All platforms(except iOS as there's iOS_captureFullscreen available): captureStage3D now has a new 2nd argument called "flushToEncoder", set to true by default. This simply means that captureStage3D() now captures frame as expected, without the need to call flushFrame() afterwards. If you still need to capture stage3D and MovieClip layer on top, in one frame, do it like this: captureStage3D(context3D, false); captureMC(YourMCLayer);. If you're only capturing Stage3D use captureStage3D(context3D) or captureStage3D(context3D, true). - All platforms: In the wake of the change above with captureStage3D there's no need to expose flushFrame() anymore so it was switched from public to private. If you're using this with stage3D capturing simply erase this call - Mac: Recompiled with the latest FFmpeg 2.5.4 & YASM support (everything should be faster due to optimizations) - Mac: Fixed long issue when developing on Mac in Flash CSX IDE(and probably other IDE's). When IDE's were unzipping the ANE, its symlinks were being destroyed, causing undefined _ctx.call errors on launch. In standalone published builds this was fine. Now FlashyWrappers ANE automatically fixes the symlinks after launching from IDE on Mac. The only disadvantage is, the error still appears, but only once. If you rebuild and launch again everything should work just fine and the ANE will stay fixed (unless the IDE's cache is erased). - iOS: Added 64-bit support, also added 64-bit FWSoundMixer beta inside this package for Hungry Hero iOS demo - iOS: Added native method to mix audio tracks instead of addSoundtrack's current "Flash" method to avoid "extract()" method of Flash Sound class, which creates delay/freezeup on iOS. The method is still called addSoundTrack, just does the mixing differently internally on iOS now - iOS: The addSoundtrack method also "knows" when you are calling it. If you call it at some point *after* starting video recording, it will insert itself at that time in the final video. You can stop it with stopSoundtrack() at any point while recording video, and it will cut the soundtrack in the resulting video accordingly. In other words, addSoundtrack on iOS is capable to start for example 20 seconds after starting video recording and end at 22 seconds, just based on the time you called them(addSoundtrack at 20secs, stopSoundtrack at 22secs after starting to record) - iOS: Added cleaner handling of the final video encoding(export) interruption - after encodeIt is called and before video is exported. Before it would just throw error, now it detects suspension, cancels the encoding cleanly and throws 'encoding_cancel' event. You should deal with the event accordingly. - iOS: Added more detailed logging in multiple sensitive places for easier debugging - iOS: Fixed a bug when on second recording the fps of the video decreased in case the video fps was different from the stage fps - iOS: Fixed some recoverable errors crashing the app anyway(throwing exception from completion block basically, which terminated the extension and the app as well) - iOS: Fixed crashing when app was suspended while recording. FlashyWrappers stops and cleans up all recorders, and dispatches "stopped" event when the app is reactivated so that your app can properly deal with the unexpected stop instead of trying to resume recording - iOS: Fixed crashing on saving videos when user didn't allow saving to camera roll. FlashyWrappers class now throws 'cameraroll_failed' if the permission wasn't given but you're still trying to save the video - iOS: Fixed audio tracks possibly overflowing video track slightly, causing the final video export freeze or crash - iOS: Fixed incorrectly set durations for audio tracks and video composition layer track, which might have caused random issues - iOS: Added progress report method for the "encodeIt()" which can take a while (call myEncoder.getEncodingProgress() after encodeIt() until getting 'encoded' event) - iOS: Added the plug for native microphone recording by the upcoming FWSoundMixer for iOS. FlashyWrappers is now looking for microphone m4a file to mix into the final video - Internal: Merged Flash Player wrapper with iOS wrapper, will allow smoother & faster framework updates

Please vote on Adobe bug to fix FlashyWrappers for FP

I’ve been communicating both on Crossbridge forums and with my Adobe contact, who was nice enough to speak with the Adobe internal team about the issue crippling multithreading in FW. I sent him link to FW online demo so the team has something to test this issue against. While they said to check the issue no clear date or timeline was given yet. This bug was a deliberate change to tighten security, but as a side effect it broke all C++ -> Flash crosscompiled apps using multithreading, like FlashyWrappers for Flash Player.

Everyone who can spare 5 minutes please sign up at Adobe(as Flash / AIR developer you probably are signed up already) and vote for this bug:

https://bugbase.adobe.com/index.cfm?event=bug&id=3938269

Thanks!

New issue in latest Flash Player affecting FlashyWrappers

In recent days and weeks there have been strange issues randomly reported regarding FlashyWrappers. Even for people with working FlashyWrappers, they started to report recording not working after few frames. Unfortunately, this turned out to be a problem in Flash Player 16+ breaking the technology being used to build FlashyWrappers. This technology was abandoned by Adobe almost right after it was released. It is being maintained by the community now and workarounds are being created.

The issue shows up only when using FlashyWrappers in multithreaded mode in Flash Player, because that’s where the issue originates. All native builds for AIR are using native threading and therefore shouldn’t be affected.

See this thread for details & progress: https://github.com/crossbridge-community/crossbridge/issues/77

The issue seems solvable even there might be some speed tradeoffs. It might still take some time to come up with the final solution and then also to rebuild FlashyWrappers for Flash Player with patched Crossbridge.

iOS: 64-bit universal builds now available

The “universal” FlashyWrappers iOS builds containing both armv7 and arm64 are available as of now. The ANE was successfully compiled with AIR 16 and tested on iOS using exampleBasic and exampleCamera. In both the free & full package, the new ANE is called “FWEncoderANE_64.ane”.

As you surely know, Apple will only accept apps built in this way, which means you’ll need to use AIR 16. And AIR 16 can only work with universally built ANE’s.

For the current customers, this ANE is also available for download separately (without the watermark of course :). There are also some code changes in general from the previous version, so make sure to test your app before submitting new versions to Appstore.

All the current customers will be e-mailed until end of this week with the 64-bit universal ANE download link. If you need the file really fast, just use the contact form http://rainbowcreatures.com/contact.php to ask for it.

In other news, openh264 is working just fine, just need to get everything together for the new release. OGVPlayer still needs some work, so I’m looking at whenever to release early without OGVPlayer or later with OGVPlayer included. In general OGVPlayer and the Android acceleration are the tasks needing most of the time at the moment.

New year, new codec

Today and yesterday, FlashyWrappers were successfully compiled and tested with the OpenH264 codec on AIR Windows and Flash Player platforms. Compiling for Flash Player was especially questionable (as the codec obviously wasn’t built for such “weird” platform like Alchemy 2 aka Crossbridge), but luckily by modifying the makefiles slightly everything is working just fine.

The result is FlashyWrappers will newly support H.264 codec and output MP4 files in the next release for AIR Windows, Mac (to be tested yet but assumed to work) and Flash Player. Theora / Vorbis (OGV) builds will still be supplied.

NOTE (again): By using OpenH264 especially in this first release where it is embedded in both the Flash Player library and the AIR library you, as the H.264 encoder distributor to the end user will not be protected from potentially paying royalties to MPEG LA. We’ll offer (optional) mechanism for the AIR binaries to be automatically downloaded from CISCO to avoid this alltogether (in that case CISCO pays any royalties as the dll’s will be downloaded ie. distributed from their URL’s), but there’s no option like that for the Flash Player version. What this means for you is making sure to understand MPEG LA license terms. You might have to become a licensee. Royalties for 0 – 100,000 units / year seem to be zero though (http://www.mpegla.com/main/programs/avc/Documents/AVC_TermsSummary.pdf). In case of Flash Player, it would probably mean 100K downloads / year of FlashyWrappers, so you might want to delay downloading the actual FW SWF until your app is just about to use the encoder.

Merry Christmas & H264 / MP4 for desktop news!

Merry Christmas to everyone!

These days before Christmas have been hectic for various reasons – largely supporting numerous FlashyWrappers customers (or people interested in FW) in regards to video replays. I recognize this is a pretty big issue and as far as desktop is concerned especially(where .ogv videos are currently supported). As you probably know, OGV videos can’t be played in Flash Player. There are several alternative plans to deal with this for current customers, including creating OGV video player.

However, by far the biggest demand there is when it comes to encoding to MP4 files. And understandably so, as MP4 is the only viable format to be played natively on any mobile device these days. Up until recently, there has been issues with doing that as the x264 encoder (used together in ffmpeg, on desktop FlashyWrappers) would effectively mean recompiling FFmpeg in GPL mode. This would not only mean forced GPL license on the whole of FlashyWrappers, but also forced GPL license for your software (as a customer of FlashyWrappers). This has always been the biggest issue for me really. I don’t really like to force any licenses on my customers and I’m sure many customers would not expect that in the first place.

The situation on mobile devices with iOS or Android is different – hardware H264 encoders can be utilized on those devices and nobody has to worry about licensing or royalty payments.

A change is coming to FW on desktops though. I’ve been researching quite alot around the OpenH264 lib in recent months. I came to the conclusion that it is pretty safe to include the library into FlashyWrappers (or rather, create another branch of FlashyWrappers desktop with official H264 support). So what are the catches? I’m not a lawyer and this is only based on my personal research, but:

1) First let me say that the good news is OpenH264 licenses both for their source code and binaries is BSD. This is absolutely non-restrictive license regardless of commercial or non-commercial use. That means, no license forcing you to make your software open sourced. That’s the most important thing

2) CISCO paid licensing fees for the binaries they are building from their sources. From what I understood, the party distributing the encoder must pay fees to MPEG LA. To make sure CISCO is the distributor all the time, you are required to make your app download their binaries before using them. That way CISCO is the distributor, you are not. This would be applicable only for AIR of course, where AIR could call the externally downloaded EXE file.

3) I will however go the other way first – including CISCO source code into one FlashyWrappers branch. Which will mean SWC / ANE files with no binaries to download. This is of course the only way for Flash Player based apps which can not download or execute any binaries, and as a side effect it will also work in desktop AIR, which is why this way is the priority.

So what about the royalty payments?

From what I personally understood, the royalty payments are zero for 0 – 100,000 units / installs per year of the encoder. I’m sure that many if not most of my customers (and me) won’t go above that number. This number is included in the MPEG LA license which you can also check, and I found confirmations on several forums that this is true (since 2005).

In case you get over 100K units then the fees are really not as horrible as you might have imagined – something like $0.2 / $0.1 per unit depending on how many units(installs) you sell.

Having said that, I feel it is safe to provide FlashyWrappers for desktop with OpenH264. I’ll need to find the time along the Android version (and there might still be issues with compiling this using FlasCC / Alchemy 2) but I’ll at least start the attempt very soon.

The only unclear part for me so far was whenever or not you’d need to obtain a license still from MPEG LA, but I’m assuming that BSD license of CISCO applies in this case (and CISCO dealt their license with MPEG LA). I can’t imagine this being a huge issue though.

Coming back to video replays – having H264 support should mean the ability to introduce video replays natively. Two big issues squished in one go 🙂

Android version progress

Work on Android library has been started recently, so far Android’s HW accelerated video encoder has been tested successfully to generate a small video from inside FlashyWrappers ANE. The next steps will involve connecting AIR’s OpenGL layer (similar to iOS) to video output and writing those frames. Audio will be mixed together with video as well.

The first (alpha) version of the HW accelerated functions for Android will be hopefully available pretty soon (probably weeks), and it will work on Android 4.3+. Due to the enormous variations among the Android versions and devices FW won’t be able to support hardware acceleration in every possible device at first, but support will be gradually added to support at least popular devices (those running on 4.3+ will come first).

The library will return a flag indicating HW support availability and you will be able to decide how to handle the situation (disabling recording completely, fallback to software OGV encoding). With hardware encoder you’ll be able to record mp4 files natively on Android in your AIR apps.

FW 2.21 released

FlashyWrappers 2.21 was released! Grab it here: http://rainbowcreatures.com/product_flashywrappers.php

All of the changes and fixes included are exclusively for the iOS version, which needed some stabilizing before starting on Android or iterating on the other platforms. From changelog:

- iOS: Fixed an issue generating internal OpenGL error which would demonstrate usually as errors in Stage3D when creating vertex buffers during capture
- iOS: Fixed a bug which caused the screen go black for various seemingly random reasons, such as texture upload (only FW logo would be visible in the free version)
- iOS: Fixed the wrong video orientation outside of iOS
- iOS: Fixed buggy Retina implementation causing the screen to possibly scretch while capturing
- iOS: Added different app / movie fps handling for captureFullscreen (you don't need to worry about your target fps now - only on iOS so far!)
- iOS: Audio is being captured directly into mp4 stream. Capturing to temp WAV stream and merging later with video is still supported if needed to save few fps while recording as well
- iOS: No longer needed depthAndStencil enabled for captureFullscreen to work (this was related to a bug which has now been fixed)
- iOS: You can now call captureFullscreen() all the time on ENTER_FRAME, it will capture only after myEncoder.init() and stop after myEncoder.encodeIt()
- iOS: Added platform default for testing outside of iOS

Extra for iOS: Hungry Hero (starling game) integration example

Extra for iOS: Latest version of FWSoundMixer for iOS inside the HH integration example

There will still be work going on iOS – hopefully to get rid of needing FWSoundMixer for audio recording & eliminating the video post processing time & fix any new bugs that appear. As you might have known there was also work going on FWSoundMixer for iOS. This is also close to finish, but the microphone issue needs to be stabilized / more understood first to make a more “stable” release. Nevertheless, the latest ANE of FWSoundMixer is included in FW 2.21 (as that case doesn’t need microphone), in the HH Starlig iOS example.

Also in the plans is further simplifications of the API for all platforms.

Microphone trouble

Althought the iOS issues in the encoder were solved, there is one more thing – in the related FWSoundMixer for iOS, which is going to be released as well in the next batch. Everything is working fine, except for when recording with microphone. As soon as sound mixer starts recording microphone, the rest of the audio playback goes laggy. This results in more strange things as a result when playing back the video(mic being my favorite thing – out of sync 🙂 ).

I’m not entirely sure, but it seems to me when using SampleDataEvent for both microphone recording and also as callback for making sound mixer perform “audio step” creates some kind of collision. In other words, SampleDataEvent with microphone capture and SampleDataEvent for playing PCM sound (you can do both), when used together on iOS seem to cause lags. On desktop I cannot either notice it or the CPU is so powerful it’s just not apparent. Also, I might be just doing something wrong.

Right now trying to find a workaround either by eliminating the SampleDataEvent for audio recording or trying to record microphone natively, even that seems to be problematic as expected (fight with AIR over microphone ensues probably). Resolving microphone is important for integration into several apps that I’m about to do.

iOS last known bug fixed!

Managed to fix the bug I was talking about yesterday. Long story short, we are sometimes our worst enemies – and this bug was entirely of my own making(as most bugs are). I had to analyze OpenGL pipeline on iOS and compare several cases to finally spot the obvious thing (and before that I had to comment out lines from Starling source for hours to isolate the problematic command in AS3 :). I was passing the wrong argument to OpenGL function. This caused the ANE working only under certain conditions, but not working under other conditions. As soon as texture has been uploaded before init, it would break the “needed” conditions. Internally, the texture ID had to equal FBO ID, because I was using one instead of another.

Anyway, this fixes the seemingly random “black screen” issue! So we’re back in “release mode” with Hunger Hero Starling demo and the next FW. No more examples / features will make it into the next release.

The last part for me was to implement the recording only after a level is started (and end after level is lost / hero dies). If this works (and after this fix, it should, because it won’t matter when “init” is called anymore) – I’ll start preparing the new release package, with the new Hungry Hero sample included.

I’m sorry this release is taking longer than expected(and sorry to everyone waiting for some kind of demo / help we discussed over e-mail) but I really needed to nail down these big iOS issues. As a result 3 major issues killed on iOS (this upload texture interference, earlier vertex buffer upload interference, problems with audio sync / timing). I won’t get everything in that I wanted(killing the video post-processing will be more tricky than I thought for example), but gaining stability is #1 goal for this release. Everything else can be only improved from then on 🙂