Android MediaCodec图片合成视频

陨石坠灭

发布于 2020-01-21 16:02:29

4.1K1

发布于 2020-01-21 16:02:29

利用MediaCodec可以录制视频，可是可以将图片合成视频吗？之前使用ffmpeg来实现。但是，ffmpeg却是c++写的，而且非常占用内存，虽然它是非常棒的音视频处理库，但是杀鸡焉用牛刀，所以今天就讲一下：如何利用Android API中的MediaCodec来实现图片合成视频

YUV是为了解决彩色电视与黑白电视的兼容性。黑白视频只有Y值，也就是灰度。而彩色电视则有YUV3个分量，如果只读取Y值，就只能显示黑白画面了。YUV最大的优点在于只需占用极少的带宽。

参考

Android MediaCodec 硬编码器封装 - https://blog.csdn.net/devil__lee/article/details/49508773

图文详解YUV420数据格式 - https://www.cnblogs.com/Sharley/p/5595768.html

如何正确使用ImageReader与YUV_420_888和MediaCodec将视频编码为h264格式？- https://stackoverrun.com/cn/q/12725625

AVFrame 与 yuv420那些事 - https://blog.csdn.net/lanxiaziyi/article/details/74139729?utm_source=blogxgwz6

YUV420P和YUV420有什么区别? - https://bbs.csdn.net/topics/80129347

Java实现的RGB转YUV420方法 - https://blog.csdn.net/u012149399/article/details/78799990

YUV简介

YUV是一种也是编码方法。比如我们常用的RGB，是指红(red)、绿(green)，蓝(blue)，而YUV则是指：

Y：亮度(Luminance、Luma)；
U：色度(Chrominance )；
V：浓度(Chroma)。

rgb与yuv互转：

rgb转yuv

Y = 0.299 R + 0.587 G + 0.114 B
U = - 0.1687 R - 0.3313 G + 0.5 B + 128
V = 0.5 R - 0.4187 G - 0.0813 B + 128

yuv转rgb

R = Y + 1.402 (V-128)
G = Y - 0.34414 (U-128) - 0.71414 (V-128)
B = Y + 1.772 (U-128)

颜色取样

将图片编码为YUV格式的数据时，将对图片上的点进行采样存储。

以黑点表示采样该像素点的Y分量，以空心圆圈表示采用该像素点的UV分量

通常采用的方式有

YUV 4:4:4采样(全采样)，每一个Y对应一组UV分量；
YUV 4:2:2采样，每两个Y共用一组UV分量；
YUV 4:2:0采样，每四个Y共用一组UV分量。

存储方式

planar：平面格式，先存储所有的Y，然后存储所有的U，然后存储V
packed：打包格式，YUV交替存储

这里举例YUV420p与YUV420sp的区别：

YUV420p：YYYYYYYY VV UU
YUV420sp：YYYYYYYY UVUV

由VU顺序的不同YUV420p可分为I420和YV12，上诉例子是YV12；YUV420sp可分为 NV12与NV21，上诉例子是NV12；

难点

1. 获取设备可渲染的颜色空间模式

由于不同手机生产商对颜色空间的渲染模式不尽相同，所以需要区别对待。不过大多是手机都是支持YUV420p、YUV420sp其中的一种。

public int[] getMediaCodecList() {
        //获取解码器列表
        int numCodecs = MediaCodecList.getCodecCount();
        MediaCodecInfo codecInfo = null;
        for (int i = 0; i < numCodecs && codecInfo == null; i++) {
            MediaCodecInfo info = MediaCodecList.getCodecInfoAt(i);
            if (!info.isEncoder()) {
                continue;
            }
            String[] types = info.getSupportedTypes();
            boolean found = false;
            //轮训所要的解码器
            for (int j = 0; j < types.length && !found; j++) {
                if (types[j].equals("video/avc")) {
                    found = true;
                }
            }
            if (!found) {
                continue;
            }
            codecInfo = info;
        }
        Log.d(TAG, "found" + codecInfo.getName() + "supporting" + " video/avc");
        MediaCodecInfo.CodecCapabilities capabilities = codecInfo.getCapabilitiesForType("video/avc");
        return capabilities.colorFormats;
}

得到颜色空间模式后，可以判断选择其中一种来对图片进行编码：

public int getColorFormat(){
      int colorFormat;
      int[] formats = this.getMediaCodecList();

        lab:
        for (int format : formats) {
            switch (format) {
                case MediaCodecInfo.CodecCapabilities.COLOR_FormatYUV420SemiPlanar: // yuv420sp
                    colorFormat = format;
                    break lab;
                case MediaCodecInfo.CodecCapabilities.COLOR_FormatYUV420Planar: // yuv420p
                    colorFormat = format;
                    break lab;
                case MediaCodecInfo.CodecCapabilities.COLOR_FormatYUV420PackedSemiPlanar: // yuv420psp
                    colorFormat = format;
                    break lab;
                case MediaCodecInfo.CodecCapabilities.COLOR_FormatYUV420PackedPlanar: // yuv420pp
                    colorFormat = format;
                    break lab;
            }
        }

        if (colorFormat <= 0) {
            colorFormat = MediaCodecInfo.CodecCapabilities.COLOR_FormatYUV420SemiPlanar;
        }
     return colorFormat;
}

然后就是设置MediaFormat:

MediaFormat mediaFormat = MediaFormat.createVideoFormat(MediaFormat.MIMETYPE_VIDEO_AVC, width, height);
mediaFormat.setInteger(MediaFormat.KEY_COLOR_FORMAT, colorFormat);
mediaFormat.setInteger(MediaFormat.KEY_BIT_RATE, width, * height);
mediaFormat.setInteger(MediaFormat.KEY_FRAME_RATE, 16);
mediaFormat.setInteger(MediaFormat.KEY_I_FRAME_INTERVAL, 10);

2. rgb转YUV420p、YUV420sp、YUV420pp、YUV420psp

这里只贴出rgb转YUV420p、YUV420sp，rgb转YUV420pp和YUV420psp的代码并没有找到，只能自己写，虽然也写了，但是还没有验证过，就不贴出来了。

由于YUV420不是全采样，U和V的数据都是with*height*(1⁄4)，所以数据长度为：1(Y)+1⁄4(U)+1⁄4(V) = 3/2。具体代码如下：

encodeYUV420SP：

    private void encodeYUV420SP(byte[] yuv420sp, int[] argb, int width, int height) {
        final int frameSize = width * height;

        int yIndex = 0;
        int uvIndex = frameSize;

        int a, R, G, B, Y, U, V;
        int index = 0;
        for (int j = 0; j < height; j++) {
            for (int i = 0; i < width; i++) {

                a = (argb[index] & 0xff000000) >> 24; // a is not used obviously
                R = (argb[index] & 0xff0000) >> 16;
                G = (argb[index] & 0xff00) >> 8;
                B = (argb[index] & 0xff) >> 0;

                // well known RGB to YUV algorithm
                Y = ((66 * R + 129 * G + 25 * B + 128) >> 8) + 16;
                V = ((-38 * R - 74 * G + 112 * B + 128) >> 8) + 128; // Previously U
                U = ((112 * R - 94 * G - 18 * B + 128) >> 8) + 128; // Previously V

                yuv420sp[yIndex++] = (byte) ((Y < 0) ? 0 : ((Y > 255) ? 255 : Y));
                if (j % 2 == 0 && index % 2 == 0) {
                    yuv420sp[uvIndex++] = (byte) ((V < 0) ? 0 : ((V > 255) ? 255 : V));
                    yuv420sp[uvIndex++] = (byte) ((U < 0) ? 0 : ((U > 255) ? 255 : U));
                }

                index++;
            }
        }
    }

encodeYUV420P:

    private void encodeYUV420P(byte[] yuv420sp, int[] argb, int width, int height) {
        final int frameSize = width * height;

        int yIndex = 0;
        int uIndex = frameSize;
        int vIndex = frameSize + width * height / 4;

        int a, R, G, B, Y, U, V;
        int index = 0;
        for (int j = 0; j < height; j++) {
            for (int i = 0; i < width; i++) {

                a = (argb[index] & 0xff000000) >> 24; // a is not used obviously
                R = (argb[index] & 0xff0000) >> 16;
                G = (argb[index] & 0xff00) >> 8;
                B = (argb[index] & 0xff) >> 0;

                // well known RGB to YUV algorithm
                Y = ((66 * R + 129 * G + 25 * B + 128) >> 8) + 16;
                V = ((-38 * R - 74 * G + 112 * B + 128) >> 8) + 128; // Previously U
                U = ((112 * R - 94 * G - 18 * B + 128) >> 8) + 128; // Previously V

                yuv420sp[yIndex++] = (byte) ((Y < 0) ? 0 : ((Y > 255) ? 255 : Y));
                if (j % 2 == 0 && index % 2 == 0) {
                    yuv420sp[vIndex++] = (byte) ((U < 0) ? 0 : ((U > 255) ? 255 : U));
                    yuv420sp[uIndex++] = (byte) ((V < 0) ? 0 : ((V > 255) ? 255 : V));
                }

                index++;
            }
        }
    }

然后统一处理：

   private byte[] getNV12(int inputWidth, int inputHeight, Bitmap scaled) {

        int[] argb = new int[inputWidth * inputHeight];

        //Log.i(TAG, "scaled : " + scaled);
        scaled.getPixels(argb, 0, inputWidth, 0, 0, inputWidth, inputHeight);

        byte[] yuv = new byte[inputWidth * inputHeight * 3 / 2];

        switch (colorFormat) {
            case MediaCodecInfo.CodecCapabilities.COLOR_FormatYUV420SemiPlanar: // yuv420sp
                encodeYUV420SP(yuv, argb, inputWidth, inputHeight);
                break;
            case MediaCodecInfo.CodecCapabilities.COLOR_FormatYUV420Planar: // yuv420p
                encodeYUV420P(yuv, argb, inputWidth, inputHeight);
                break;
            case MediaCodecInfo.CodecCapabilities.COLOR_FormatYUV420PackedSemiPlanar: // yuv420psp
                encodeYUV420PSP(yuv, argb, inputWidth, inputHeight);
                break;
            case MediaCodecInfo.CodecCapabilities.COLOR_FormatYUV420PackedPlanar: // yuv420pp
                encodeYUV420PP(yuv, argb, inputWidth, inputHeight);
                break;
        }
//        scaled.recycle();

        return yuv;
    }

3. 保存为mp4格式的视频

视频处理需要用到MediaMuxer：

 mediaMuxer = new MediaMuxer(out.getAbsolutePath(), MediaMuxer.OutputFormat.MUXER_OUTPUT_MPEG_4);

其中out为视频输出文件。

生成MediaCodec对象：

try {
     mediaCodec = MediaCodec.createEncoderByType(MediaFormat.MIMETYPE_VIDEO_AVC);
     //创建生成MP4初始化对象
     mediaMuxer = new MediaMuxer(out.getAbsolutePath(), MediaMuxer.OutputFormat.MUXER_OUTPUT_MPEG_4);
} catch (IOException e) {
     e.printStackTrace();
}

mediaCodec.configure(mediaFormat, null, null, MediaCodec.CONFIGURE_FLAG_ENCODE);
mediaCodec.start();

导出视频文件后的处理：

public void finish() {
        isRunning = false;
        if (mediaCodec != null) {
            mediaCodec.stop();
            mediaCodec.release();
        }
        if (mediaMuxer != null) {
            try {
                if (mMuxerStarted) {
                    mediaMuxer.stop();
                    mediaMuxer.release();
                }
            } catch (Exception e) {
                e.printStackTrace();
            }
        }
}

核心处理逻辑：

public void encode(Bitmap bitmap) {
        final int TIMEOUT_USEC = 10000;
        isRunning = true;
        long generateIndex = 0;
        MediaCodec.BufferInfo info = new MediaCodec.BufferInfo();

        ByteBuffer[] buffers = null;
        if (Build.VERSION.SDK_INT <= Build.VERSION_CODES.LOLLIPOP) {
            buffers = mediaCodec.getInputBuffers();
        }

        while (isRunning) {

            int inputBufferIndex = mediaCodec.dequeueInputBuffer(TIMEOUT_USEC);
            if (inputBufferIndex >= 0) {
                long ptsUsec = computePresentationTime(generateIndex);
                if (generateIndex >= mProvider.size()) {
                    mediaCodec.queueInputBuffer(inputBufferIndex, 0, 0, ptsUsec,
                            MediaCodec.BUFFER_FLAG_END_OF_STREAM);
                    isRunning = false;
                    drainEncoder(true, info);

                } else {
                    if (bitmap == null) {
                        bitmap = mProvider.next();
                    }
                    byte[] input = getNV12(getSize(bitmap.getWidth()), getSize(bitmap.getHeight()), bitmap);
                    bitmap = null;
                    //有效的空的缓存区
                    ByteBuffer inputBuffer = null;
                    if (Build.VERSION.SDK_INT <= Build.VERSION_CODES.LOLLIPOP) {
                        inputBuffer = buffers[inputBufferIndex];
                    } else {
                        inputBuffer = mediaCodec.getInputBuffer(inputBufferIndex);//inputBuffers[inputBufferIndex];
                    }
                    inputBuffer.clear();
                    inputBuffer.put(input);
                    //将数据放到编码队列
                    mediaCodec.queueInputBuffer(inputBufferIndex, 0, input.length, ptsUsec, 0);
                    drainEncoder(false, info);
                }

                generateIndex++;
            } else {
                Log.i(TAG, "input buffer not available");
                try {
                    Thread.sleep(50);
                } catch (InterruptedException e) {
                    e.printStackTrace();
                }
            }
}

其中对InputBuffer的获取做了兼容处理，mProvider是一个接口，用来获取位图。computePresentationTime的代码：

private long computePresentationTime(long frameIndex) {
     return 132 + frameIndex * 1000000 / mFrameRate;
}

然后就是buffer输出：

private void drainEncoder(boolean endOfStream, MediaCodec.BufferInfo bufferInfo) {
        final int TIMEOUT_USEC = 10000;

        ByteBuffer[] buffers = null;
        if (Build.VERSION.SDK_INT <= Build.VERSION_CODES.LOLLIPOP) {
            buffers = mediaCodec.getOutputBuffers();
        }

        if (endOfStream) {
            try {
                mediaCodec.signalEndOfInputStream();
            } catch (Exception e) {
            }
        }

        while (true) {
            int encoderStatus = mediaCodec.dequeueOutputBuffer(bufferInfo, TIMEOUT_USEC);
            if (encoderStatus == MediaCodec.INFO_TRY_AGAIN_LATER) {
                if (!endOfStream) {
                    break; // out of while
                } else {
                    Log.i(TAG, "no output available, spinning to await EOS");
                }
            } else if (encoderStatus == MediaCodec.INFO_OUTPUT_FORMAT_CHANGED) {
                if (mMuxerStarted) {
                    throw new RuntimeException("format changed twice");
                }

                MediaFormat mediaFormat = mediaCodec.getOutputFormat();
                mTrackIndex = mediaMuxer.addTrack(mediaFormat);
                mediaMuxer.start();
                mMuxerStarted = true;
            } else if (encoderStatus < 0) {
                Log.i(TAG, "unexpected result from encoder.dequeueOutputBuffer: " + encoderStatus);
            } else {
                ByteBuffer outputBuffer = null;
                if (Build.VERSION.SDK_INT <= Build.VERSION_CODES.LOLLIPOP) {
                    outputBuffer = buffers[encoderStatus];
                } else {
                    outputBuffer = mediaCodec.getOutputBuffer(encoderStatus);
                }

                if (outputBuffer == null) {
                    throw new RuntimeException("encoderOutputBuffer "
                            + encoderStatus + " was null");
                }

                if ((bufferInfo.flags & MediaCodec.BUFFER_FLAG_CODEC_CONFIG) != 0) {
                    Log.d(TAG, "ignoring BUFFER_FLAG_CODEC_CONFIG");
                    bufferInfo.size = 0;
                }

                if (bufferInfo.size != 0) {
                    if (!mMuxerStarted) {
                        throw new RuntimeException("muxer hasn't started");
                    }

                    // adjust the ByteBuffer values to match BufferInfo
                    outputBuffer.position(bufferInfo.offset);
                    outputBuffer.limit(bufferInfo.offset + bufferInfo.size);

                    Log.d(TAG, "BufferInfo: " + bufferInfo.offset + ","
                            + bufferInfo.size + ","
                            + bufferInfo.presentationTimeUs);

                    try {
                        mediaMuxer.writeSampleData(mTrackIndex, outputBuffer, bufferInfo);
                    } catch (Exception e) {
                        Log.i(TAG, "Too many frames");
                    }

                }

                mediaCodec.releaseOutputBuffer(encoderStatus, false);

                if ((bufferInfo.flags & MediaCodec.BUFFER_FLAG_END_OF_STREAM) != 0) {
                    if (!endOfStream) {
                        Log.i(TAG, "reached end of stream unexpectedly");
                    } else {
                        Log.i(TAG, "end of stream reached");
                    }
                    break; // out of while
                }
            }
    }
}

如果看过其他类似博客的代码，就会发现代码其实差不多，但是却不是我想要的，所以只能自己改。其中一些博客还是写的非常棒，从中学到了很多。

这篇文章讲的是利用纯Android API实现的图片合成视频文件，其中我有查询到利用ffmpeg的，利用opencv/javacv的，但是这边文章介绍的方式没有引用第三方库，因此打包出来的apk文件肯定是很小的。

为了解决这个问题查了不少资料，花了不少时间，不过还好，终于搞定了。

本文参与腾讯云自媒体分享计划，分享自作者个人站点/博客。

原始发表：2018/12/11，如有侵权请联系 cloudcommunity@tencent.com 删除

https