Right, I was going to follow up and say that ImageToBuffer in
MovieEncoderDataStream.write creates an int array out of the image data. So
using any other format that is not raw RGB will break the QuickTimeMux.

I presume this is the reason why you can only play output videos on a Mac.
QuickTime is the only player that can understand the raw RGB format. 

Doing the following in MovieEncoderDataStream.write, converts the Image to a
byte[]

        ByteArrayOutputStream stream = new ByteArrayOutputStream();
        Buffer b = new Buffer();
        try
        {
            ImageIO.write((BufferedImage) i,"JPG",stream);
            stream.flush();
            b.setData(stream.toByteArray());
            b.setLength(((byte[]) b.getData()).length);
        } catch (IOException e)
        {
            e.printStackTrace();
        }

This is very specific to converting to using a byte[] only, but I guess some
logic could be added to figure out what encodeFormat is using (
Format.byteArray or Format.intArray ).

Does that make sense?