Android平台上OpenCV 深度网络实现对象检测

OpenCV学堂

发布于 2018-04-13 15:35:30

1.7K0

发布于 2018-04-13 15:35:30

Android平台上OpenCV 深度网络实现对象检测

自OpenCV3.3发布包含深度神经网络(DNN)模块的SDK以后，OpenCV4Android SDK就开始支持Android客户端使用深度学习实现对象检测，特别是基于SSD的mobilenet网络模型，可以在移动端达到较高的帧率，实时视频对象检测，SSD mobilenet支持20种对象检测。模型的下载可以到本人GIHUB:

https://github.com/gloomyfish1998/opencv_tutorial

在data/model目录里面下载即可。

下面就说说如何在Android Studio中开发一个基于深度学习-SSD网络的对象检测演示程序。

一：下载与导入网络模型

从上面的GITHUB目录下载模型文件与描述文件之后，在Android Studio中新建一个空项目，导入OpenCV4Android 3.4的SDK支持，如果不知道怎么配置Android Studio与OpenCV4Android SDK，就请看下面这篇文章即可：

OpenCV3.2集成Android Studio2.2开发配置

配置好之后，新建res/raw目录，然后copy下载好的模型文件与描述文件到raw目录里面，修改文件名称，最终整个目录结构如下：

然后通过如下代码加载SSD网络的权重数据与描述

加载数据文件

private String initBinaryData()throws IOException {
    InputStream input = getResources().openRawResource(R.raw.mobilenetssd_binary);
    File cascadeDir = this.getDir("ssd", Context.MODE_PRIVATE);
    File file = new File(cascadeDir.getAbsolutePath() + "mobilenetssd_binary.caffemodel");
    FileOutputStream out = new FileOutputStream(file);
    byte[] buff = new byte[1024];
    int len = 0;
    while((len = input.read(buff)) != -1) {
        out.write(buff, 0, len);
    }
    input.close();
    out.close();
    return file.getAbsolutePath();
}

加载描述文件

private String initDescData()throws IOException {
    InputStream input = getResources().openRawResource(R.raw.mobilenetssd_desc);
    File cascadeDir = this.getDir("ssd", Context.MODE_PRIVATE);
    File file = new File(cascadeDir.getAbsolutePath() + "mobilenetssd_desc.prototxt");
    FileOutputStream out = new FileOutputStream(file);
    byte[] buff = new byte[1024];
    int len = 0;
    while((len = input.read(buff)) != -1) {
        out.write(buff, 0, len);
    }
    input.close();
    out.close();
    return file.getAbsolutePath();
}

二：使用JavaCameraView打开Android手机摄像头

在Android平台上使用摄像头，需要调用OpenCV4Android SDK的JavaCameraView来打开摄像头实现预览。通过CvCameraViewListener2接口类加载监听摄像对预览帧数据的接收、处理、与返回显示，该类有三个方法需要重载，分别是：

相机启动时候调用

public void onCameraViewStarted(int width, int height);

相机停止时候调用

public void onCameraViewStopped();

相机工作时候，预览帧接收与处理

public Mat onCameraFrame(CvCameraViewFrame inputFrame);

首先看一下JavaCamerview的布局定义：

<?xml version="1.0" encoding="utf-8"?>
<RelativeLayout xmlns:android="http://schemas.android.com/apk/res/android"
    xmlns:tools="http://schemas.android.com/tools"
    android:layout_width="match_parent"
    android:layout_height="match_parent"
    android:paddingTop="3dp"
    android:paddingBottom="3dp"
    android:paddingLeft="3dp"
    android:paddingRight="3dp"
    tools:context="com.example.zhigang.ssddemo.MainActivity">
    <org.opencv.android.JavaCameraView
        android:layout_width="match_parent"
        android:layout_height="match_parent"
        android:visibility="gone"
        android:id="@+id/camera_view_id"/>
</RelativeLayout>

然后通过下面的一段代码打开相机，设置监听

getWindow().setFlags(WindowManager.LayoutParams.FLAG_FULLSCREEN, WindowManager.LayoutParams.FLAG_FULLSCREEN);
getWindow().addFlags(WindowManager.LayoutParams.FLAG_KEEP_SCREEN_ON);
mcameraView = findViewById(R.id.camera_view_id);
mcameraView.setVisibility(SurfaceView.VISIBLE);
mcameraView.setCvCameraViewListener(MainActivity.this); // setup frame listener
mcameraView.setCameraIndex(0);
mcameraView.enableFpsMeter();
mcameraView.enableView();

在重载的相机开始方法中实现SSD网络初始化：

@Override
public void onCameraViewStarted(int width, int height) {
    try {
        net = Dnn.readNetFromCaffe(initDescData(), initBinaryData());
    } catch (IOException ioe) {
        ioe.printStackTrace();
    }
}

在MainActiviy的onCreate方法最后调用下面代码，加载OpenCV库支持

//OpenCV库静态加载并初始化
private void staticLoadCVLibraries(){
    boolean load = OpenCVLoader.initDebug();
    if(load) {
        Log.i("CV", "Open CV Libraries loaded...");
    }
}

最后千万别忘记在manifest.xml中添加相机权限支持:

<uses-permission android:name="android.permission.READ_EXTERNAL_STORAGE" />
<uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE" />
<uses-permission android:name="android.permission.CAMERA" />

<uses-feature android:name="android.hardware.camera" />
<uses-feature android:name="android.hardware.camera.autofocus" />
<uses-feature android:name="android.hardware.camera.flash" />

三：在预览帧中检测对象

在重载的预览帧处理方法onCameraFrame中实现SSD网络对每帧图像的实时对象检测，代码实现如下：

// Get a new frame
Imgproc.cvtColor(frame, frame, Imgproc.COLOR_RGBA2RGB);
// Forward image through network.
Mat blob = Dnn.blobFromImage(frame, IN_SCALE_FACTOR,
        new Size(IN_WIDTH, IN_HEIGHT),
        new Scalar(MEAN_VAL, MEAN_VAL, MEAN_VAL), false, false);
net.setInput(blob);
Mat detections = net.forward();
int cols = frame.cols();
int rows = frame.rows();
Size cropSize;
if ((float)cols / rows > WH_RATIO) {
    cropSize = new Size(rows * WH_RATIO, rows);
} else {
    cropSize = new Size(cols, cols / WH_RATIO);
}
int y1 = (int)(rows - cropSize.height) / 2;
int y2 = (int)(y1 + cropSize.height);
int x1 = (int)(cols - cropSize.width) / 2;
int x2 = (int)(x1 + cropSize.width);
Mat subFrame = frame.submat(y1, y2, x1, x2);
cols = subFrame.cols();
rows = subFrame.rows();
detections = detections.reshape(1, (int)detections.total() / 7);
for (int i = 0; i < detections.rows(); ++i) {
    double confidence = detections.get(i, 2)[0];
    if (confidence > THRESHOLD) {
        int classId = (int)detections.get(i, 1)[0];
        int xLeftBottom = (int)(detections.get(i, 3)[0] * cols);
        int yLeftBottom = (int)(detections.get(i, 4)[0] * rows);
        int xRightTop   = (int)(detections.get(i, 5)[0] * cols);
        int yRightTop   = (int)(detections.get(i, 6)[0] * rows);
        // Draw rectangle around detected object.
        Imgproc.rectangle(subFrame, new Point(xLeftBottom, yLeftBottom),
                new Point(xRightTop, yRightTop),
                new Scalar(0, 255, 0));
        String label = classNames[classId] + ": " + confidence;
        int[] baseLine = new int[1];
        Size labelSize = Imgproc.getTextSize(label, Core.FONT_HERSHEY_SIMPLEX, 0.5, 1, baseLine);
        // Draw background for label.
        Imgproc.rectangle(subFrame, new Point(xLeftBottom, yLeftBottom - labelSize.height),
                new Point(xLeftBottom + labelSize.width, yLeftBottom + baseLine[0]),
                new Scalar(255, 255, 255), Core.FILLED);
        // Write class name and confidence.
        Imgproc.putText(subFrame, label, new Point(xLeftBottom, yLeftBottom),
                Core.FONT_HERSHEY_SIMPLEX, 0.5, new Scalar(0, 0, 0));
    }
}

运行显示截图如下：