源码地址:https://github.com/deepsadness/AppRemote
上一章中,我们简单实现了PC的投屏功能。 但是还是存在这一些缺陷。
所以这边文章。我们需要根据上面的需求。来对我们的代码进行优化。
其实在上一章中,我们已经获取了屏幕信息。只是没有发送给client端。这边文章中,我们进行发送。
private static void sendScreenInfo(Size size, ByteBuffer buffer, FileDescriptor fileDescriptor) throws IOException {
//将尺寸数据先发送过去
int width = size.getWidth();
int height = size.getHeight();
byte wHigh = (byte) (width >> 8);
byte wLow = (byte) (width & 0xff);
byte hHigh = (byte) (height >> 8);
byte hLow = (byte) (height & 0xff);
buffer.put(wHigh);
buffer.put(wLow);
buffer.put(hHigh);
buffer.put(hLow);
// System.out.println("发送尺寸 size result = " + write);
// int write = Os.write(fileDescriptor, buffer);
byte[] buffer_size = new byte[4];
buffer_size[0] = (byte) (width >> 8);
buffer_size[1] = (byte) (width & 0xff);
buffer_size[2] = (byte) (height >> 8);
buffer_size[3] = (byte) (height & 0xff);
writeFully(fileDescriptor, buffer_size, 0, buffer_size.length);
System.out.println("发送尺寸 size result ");
buffer.clear();
}
//从客户端接受屏幕数据
uint8_t size[4];
socketConnection->recv_from_(reinterpret_cast<uint8_t *>(size), 4);
//这里先写死,后面从客户端内接受
int width = (size[0] << 8) | (size[1]);
int height = (size[2] << 8) | (size[3]);
printf("width = %d , height = %d \n", width, height);
这样就可以获得屏幕的尺寸信息,保证不同手机分辨率也能正常使用了。
有点胖.png
尽管我们通过这样获取了正确的屏幕信息,但是SDL显示的画面,还是有些奇怪。比我们预期的胖了一点。
通过下面的方式,来重新计算窗口的尺寸。这样才能显示正常。
//这里是给四周留空隙。
#define DISPLAY_MARGINS 96
struct size {
int width;
int height;
};
// get the preferred display bounds (i.e. the screen bounds with some margins)
static SDL_bool get_preferred_display_bounds(struct size *bounds) {
SDL_Rect rect;
#if SDL_VERSION_ATLEAST(2, 0, 5)
# define GET_DISPLAY_BOUNDS(i, r) SDL_GetDisplayUsableBounds((i), (r))
#else
# define GET_DISPLAY_BOUNDS(i, r) SDL_GetDisplayBounds((i), (r))
#endif
//获取显示的大小
if (GET_DISPLAY_BOUNDS(0, &rect)) {
// LOGW("Could not get display usable bounds: %s", SDL_GetError());
printf("Could not get display usable bounds: %s\n", SDL_GetError());
return SDL_FALSE;
}
//设置大小
bounds->width = MAX(0, rect.w - DISPLAY_MARGINS);
bounds->height = MAX(0, rect.h - DISPLAY_MARGINS);
return SDL_TRUE;
}
// return the optimal size of the window, with the following constraints:
// - it attempts to keep at least one dimension of the current_size (i.e. it crops the black borders)
// - it keeps the aspect ratio
// - it scales down to make it fit in the display_size
static struct size get_optimal_size(struct size current_size, struct size frame_size) {
if (frame_size.width == 0 || frame_size.height == 0) {
// avoid division by 0
return current_size;
}
struct size display_size;
// 32 bits because we need to multiply two 16 bits values
int w;
int h;
if (!get_preferred_display_bounds(&display_size)) {
// cannot get display bounds, do not constraint the size
w = current_size.width;
h = current_size.height;
} else {
w = MIN(current_size.width, display_size.width);
h = MIN(current_size.height, display_size.height);
}
SDL_bool keep_width = static_cast<SDL_bool>(frame_size.width * h > frame_size.height * w);
//缩放之后,保持长宽比
if (keep_width) {
// remove black borders on top and bottom
h = frame_size.height * w / frame_size.width;
} else {
// remove black borders on left and right (or none at all if it already fits)
w = frame_size.width * h / frame_size.height;
}
// w and h must fit into 16 bits
SDL_assert_release(w < 0x10000 && h < 0x10000);
return (struct size) {w, h};
}
//调用
void set(){
struct size frame_size = {
.height=screen_h,
.width=screen_w
};
struct size window_size = get_optimal_size(frame_size, frame_size);
//创建window
sdl_window = SDL_CreateWindow(
name,
SDL_WINDOWPOS_UNDEFINED, SDL_WINDOWPOS_UNDEFINED,
window_size.width, window_size.height,
SDL_WINDOW_RESIZABLE);
}
这样才能显示正常的窗口了。
正常的比例.png
我们知道在Android中有几种方式可以对手机的Android发起模拟按键。
Usage: input [<source>] <command> [<arg>...]
The sources are:
dpad
keyboard
mouse
touchpad
gamepad
touchnavigation
joystick
touchscreen
stylus
trackball
The commands and default sources are:
text <string> (Default: touchscreen)
keyevent [--longpress] <key code number or name> ... (Default: keyboard)
tap <x> <y> (Default: touchscreen)
swipe <x1> <y1> <x2> <y2> [duration(ms)] (Default: touchscreen)
draganddrop <x1> <y1> <x2> <y2> [duration(ms)] (Default: touchscreen)
press (Default: trackball)
roll <dx> <dy> (Default: trackball)
就可以对屏幕上(100,100)的位置,进行模拟点击。
当API 15之后,我们使用InputManager。
public InputManager getInputManager() {
if (inputManager == null) {
IInterface service = getService(Context.INPUT_SERVICE, "android.hardware.input.IInputManager");
inputManager = new InputManager(service);
}
return inputManager;
}
我们知道Android中的按键事件对应的是KeyEvent
,而手势事件对应的是MotionEvent
。
public class KeyEventFactory {
/*
创建一个KeyEvent
*/
public static KeyEvent keyEvent(int action, int keyCode, int repeat, int metaState) {
long now = SystemClock.uptimeMillis();
/**
* 1. 点击的时间 The time (in {@link android.os.SystemClock#uptimeMillis}) at which this key code originally went down.
* 2. 事件发生的时间 The time (in {@link android.os.SystemClock#uptimeMillis}) at which this event happened.
* 3. UP DOWN MULTIPLE 中的一个: either {@link #ACTION_DOWN},{@link #ACTION_UP}, or {@link #ACTION_MULTIPLE}.
* 4. code The key code. 输入的键盘事件
* 5. 重复的事件次数。点出次数? A repeat count for down events (> 0 if this is after the initial down) or event count for multiple events.
* 6. metaState Flags indicating which meta keys are currently pressed. 暂时不知道什么意思
* 7. The device ID that generated the key event.
* 8. Raw device scan code of the event. 暂时不知道什么意思
* 9. The flags for this key event 暂时不知道什么意思
* 10. The input source such as {@link InputDevice#SOURCE_KEYBOARD}.
*/
KeyEvent event = new KeyEvent(now, now, action, keyCode, repeat, metaState,
KeyCharacterMap.VIRTUAL_KEYBOARD,
0,
0,
InputDevice.SOURCE_KEYBOARD);
return event;
}
/*
通过送入一个ACTION_DOWN 和ACTION_UP 来模拟一次点击的事件
*/
public static KeyEvent[] clickEvent(int keyCode) {
return new KeyEvent[]{keyEvent(KeyEvent.ACTION_DOWN, keyCode, 0, 0)
, keyEvent(KeyEvent.ACTION_UP, keyCode, 0, 0)};
}
}
private static long lastMouseDown;
private static final MotionEvent.PointerCoords[] pointerCoords = {new MotionEvent.PointerCoords()};
private static final MotionEvent.PointerProperties[] pointerProperties = {new MotionEvent
.PointerProperties()};
public static MotionEvent createMotionEvent(int type, int x, int y) {
long now = SystemClock.uptimeMillis();
int action;
if (type == 1) {
lastMouseDown = now;
action = MotionEvent.ACTION_DOWN;
} else {
action = MotionEvent.ACTION_UP;
}
MotionEvent.PointerCoords[] pointerCoords = {new MotionEvent.PointerCoords()};
MotionEvent.PointerCoords coords = pointerCoords[0];
coords.x = 2 * x;
coords.y = 2 * y;
MotionEvent.PointerProperties[] pointerProperties = {new MotionEvent
.PointerProperties()};
MotionEvent.PointerProperties props = pointerProperties[0];
props.id = 0;
props.toolType = MotionEvent.TOOL_TYPE_FINGER;
coords = pointerCoords[0];
coords.orientation = 0;
coords.pressure = 1;
coords.size = 1;
return MotionEvent.obtain(
lastMouseDown, now,
action,
1, pointerProperties, pointerCoords,
0, 1,
1f, 1f,
0, 0,
InputDevice.SOURCE_TOUCHSCREEN, 0);
}
public static MotionEvent createScrollEvent(int x, int y, int hScroll, int vScroll) {
long now = SystemClock.uptimeMillis();
MotionEvent.PointerCoords[] pointerCoords = {new MotionEvent.PointerCoords()};
MotionEvent.PointerCoords coords = pointerCoords[0];
coords.x = 2 * x;
coords.y = 2 * y;
MotionEvent.PointerProperties[] pointerProperties = {new MotionEvent
.PointerProperties()};
MotionEvent.PointerProperties props = pointerProperties[0];
props.id = 0;
props.toolType = MotionEvent.TOOL_TYPE_FINGER;
coords = pointerCoords[0];
coords.orientation = 0;
coords.pressure = 1;
coords.size = 1;
coords.setAxisValue(MotionEvent.AXIS_HSCROLL, hScroll);
coords.setAxisValue(MotionEvent.AXIS_VSCROLL, vScroll);
return MotionEvent.obtain(lastMouseDown, now, MotionEvent.ACTION_SCROLL, 1, pointerProperties, pointerCoords, 0, 0, 1f, 1f, 0,
0, InputDevice.SOURCE_MOUSE, 0);
}
public boolean injectInputEvent(InputEvent inputEvent, int mode) {
try {
return (Boolean) injectInputEventMethod.invoke(service, inputEvent, mode);
} catch (InvocationTargetException | IllegalAccessException e) {
e.printStackTrace();
throw new AssertionError(e);
}
}
值得注意的是:一次点击事件是由一个DOWN 和UP事件组成的。
通过SDL2的事件循环来监听,对输入的事件进行相应
需要注意的是:
开启事件循环前
开启事件循环后出现窗口上的按钮.png
开启事件循环代码 :
//开启Event Loop
for (;;) {
SDL_WaitEvent(&event);
//这里我们主要相应了
if (event.type == SDL_MOUSEBUTTONDOWN) { //点击事件的DOWN
handleButtonEvent(sc, &event.button);
} else if (event.type == SDL_MOUSEBUTTONUP) { //点击事件的UP
handleButtonEvent(sc, &event.button);
} else if (event.type == SDL_KEYDOWN) { //按键事件DOWN
handleSDLKeyEvent(sc, &event.key);
} else if (event.type == SDL_KEYUP) { //按键事件UP
handleSDLKeyEvent(sc, &event.key);
} else if (event.type == SDL_MOUSEWHEEL) { // 滚轮事件
//处理滑动事件
handleScrollEvent(sc, &event.wheel);
} else if (event.type == SDL_QUIT) { // 点击窗口上的关闭按钮
printf("rev event type=SDL_QUIT\n");
sc->destroy();
break;
}
事件处理代码 : 其实就是将这些事件解析成坐标,然后通过socket发送
//对应点击事件
void handleButtonEvent(SDL_Screen *screen, SDL_MouseButtonEvent *event) {
int width = screen->screen_w;
int height = screen->screen_h;
int x = event->x;
int y = event->y;
//是否超过来边界
bool outside_device_screen = x < 0 || x >= width ||
y < 0 || y >= height;
if (event->type == SDL_MOUSEBUTTONDOWN) {
}
printf("outside_device_screen =%d\n", outside_device_screen);
if (outside_device_screen) {
// ignore
return;
}
char buf[6];
memset(buf, 0, sizeof(buf));
printf("event x =%d\n", event->x);
printf("event y =%d\n", event->y);
printf("event char size =%zu\n", sizeof(char));
buf[0] = 0;
if (event->type == SDL_MOUSEBUTTONDOWN) {
//发送down 事件
buf[1] = 1;
} else {
// 发送UP事件
buf[1] = 0;
}
//高8位
buf[2] = event->x >> 8;
//低8位
buf[3] = event->x & 0xff;
//高8位
buf[4] = event->y >> 8;
//低8位
buf[5] = event->y & 0xff;
int result = send(client_event, buf, 6, 0);
printf("send result = %d\n", result);
}
// 对应滑动事件
// Convert window coordinates (as provided by SDL_GetMouseState() to renderer coordinates (as provided in SDL mouse events)
//
// See my question:
// <https://stackoverflow.com/questions/49111054/how-to-get-mouse-position-on-mouse-wheel-event>
void handleScrollEvent(SDL_Screen *sc, SDL_MouseWheelEvent *event) {
//处理滑动事件
int x_c;
int y_c;
int *x = &x_c;
int *y = &y_c;
SDL_GetMouseState(x, y);
SDL_Rect viewport;
float scale_x, scale_y;
SDL_RenderGetViewport(sc->sdl_renderer, &viewport);
SDL_RenderGetScale(sc->sdl_renderer, &scale_x, &scale_y);
*x = (int) (*x / scale_x) - viewport.x;
*y = (int) (*y / scale_y) - viewport.y;
int width = sc->screen_w;
int height = sc->screen_h;
//是否超过来边界
bool outside_device_screen = x_c < 0 || x_c >= width ||
y_c < 0 || y_c >= height;
printf("outside_device_screen =%d\n", outside_device_screen);
if (outside_device_screen) {
// ignore
return;
}
SDL_assert_release(x_c >= 0 && x_c < 0x10000 && y_c >= 0 && y_c < 0x10000);
//使用这个来记录滑动的方向
// SDL behavior seems inconsistent between horizontal and vertical scrolling
// so reverse the horizontal
// <https://wiki.libsdl.org/SDL_MouseWheelEvent#Remarks>
// SDL 的滑动情况,两个方向不一致
int mul = event->direction == SDL_MOUSEWHEEL_NORMAL ? 1 : -1;
int hs = -mul * event->x;
int vs = mul * event->y;
char buf[14];
memset(buf, 0, sizeof(buf));
printf(" x_c =%d\n", x_c);
printf(" y_c =%d\n", y_c);
printf(" hs =%d\n", hs);
printf(" vs =%d\n", vs);
buf[0] = 0;
//滚动事件
buf[1] = 2;
//高8位
buf[2] = x_c >> 8;
//低8位
buf[3] = x_c & 0xff;
//高8位
buf[4] = y_c >> 8;
//低8位
buf[5] = y_c & 0xff;
//继续滚动距离
buf[6] = hs >> 24;
//低8位
buf[7] = hs >> 16;
buf[8] = hs >> 8;
buf[9] = hs;
//高8位
buf[10] = vs >> 24;
//低8位
buf[11] = vs >> 16;
buf[12] = vs >> 8;
buf[13] = vs;
int result = send(client_event, buf, 14, 0);
printf("send result = %d\n", result);
}
//对应键盘上的按钮事件。
void handleSDLKeyEvent(SDL_Screen *sc, SDL_KeyboardEvent *event) {
//分别对应 mac 上的 control option command
int ctrl = event->keysym.mod & (KMOD_LCTRL | KMOD_RCTRL);
int alt = event->keysym.mod & (KMOD_LALT | KMOD_RALT);
int meta = event->keysym.mod & (KMOD_LGUI | KMOD_RGUI);
printf("ctrl = %d,", ctrl);
printf("meta = %d,", meta);
printf("alt = %d,\n", alt);
////因为我是mac键盘,期望control+ H = home键 control+b = back键
//再去取keycode
SDL_Keycode keycode = event->keysym.sym;
printf("keycode = %d, action type = %d\n", keycode, event->type);
printf("b = %d, action type = %d\n", SDLK_b, event->type);
if (event->type == SDL_KEYDOWN && ctrl != 0) {
//这个时候发送的是按下的状态
if (keycode == SDLK_h) {
char buf[4];
memset(buf, 0, sizeof(buf));
buf[0] = 0;
//自定义的案件事件
buf[1] = 3;
//1 是 down
buf[2] = 1;
//key code home 键对应的是 3
buf[3] = 3;
int result = send(client_event, buf, 4, 0);
printf("send result = %d\n", result);
} else if (keycode == SDLK_b) {
char buf[4];
memset(buf, 0, sizeof(buf));
buf[0] = 0;
//自定义的案件事件
buf[1] = 3;
//1 是 down
buf[2] = 1;
//key code back 键对应的是 4
buf[3] = 4;
int result = send(client_event, buf, 4, 0);
printf("send result = %d\n", result);
}
}
if (event->type == SDL_KEYUP && keycode != 0) {
if (keycode == SDLK_h) {
char buf[4];
memset(buf, 0, sizeof(buf));
buf[0] = 0;
//自定义的案件事件
buf[1] = 3;
//1 是 up
buf[2] = 0;
//key code home 键对应的是 3
buf[3] = 3;
int result = send(client_event, buf, 4, 0);
printf("send result = %d\n", result);
} else if (keycode == SDLK_b) {
char buf[4];
memset(buf, 0, sizeof(buf));
buf[0] = 0;
//自定义的案件事件
buf[1] = 3;
//1 是 up
buf[2] = 0;
//key code back 键对应的是 4
buf[3] = 4;
int result = send(client_event, buf, 4, 0);
printf("send result = %d\n", result);
}
}
}
这里可以看到,根据每一种事件,都定义了对应的方式进行发送。那Android端,可以通过对应的方式进行接收就可以了~
do {
//读到数据
int read = Os.read(fileDescriptor, buffer);
System.out.println("read=" + read + ",position=" + buffer.position() + "," +
"limit=" + buffer.limit() + ",remaining " + buffer.remaining());
//当读到的长度为0,就结束了。
if (read == -1 || read == 0) {
//如果这个时候read 0 的话。就结束
break;
} else {
buffer.flip();
//上面定义的,如果是按钮事件,第一个必须是0
byte b = buffer.get(0);
//进入对应的事件
if (b == 0 && read > 1) { //如果是0 的话,就当作是Action
//第2个是判断事件的类型
byte type = buffer.get(1);
//按键事件。它发送时定义的长度是6
if (type < 2 && read == 6) {//action down 1 down 0 up
System.out.println("enter key event");
buffer.position(1);
int x = buffer.get(2) << 8 | buffer.get(3) & 0xff;
int y = buffer.get(4) << 8 | buffer.get(5) & 0xff;
//接受到事件进行处理
boolean key = createKey(serviceManager, type, x, y);
buffer.clear();
} else if (type == 2 && read == 14) { //滚动事件.定义的长度是14
buffer.position(1);
//x,y是接触的点,hs是水平的滑动,vs 是上下的滑动
int x = buffer.get(2) << 8 | buffer.get(3) & 0xff;
int y = buffer.get(4) << 8 | buffer.get(5) & 0xff;
int hs = buffer.get(6) << 24 | buffer.get(7) << 16 | buffer.get(8) <<
8 | buffer.get(9);
int vs = buffer.get(10) << 24 | buffer.get(11) << 16 | buffer.get(12) <<
8 | buffer.get(13);
//接受到事件进行处理
boolean b1 = injectScroll(serviceManager, x, y, hs, vs);
// 处理完,记得清楚buffer
buffer.clear();
} else if (type == 3 && read == 4) { //接受按键事件,长度是4
System.out.println("enter key code event");
int action = buffer.get(2) == 1 ? KeyEvent.ACTION_DOWN : KeyEvent.ACTION_UP;
int keyCode = buffer.get(3);
boolean key = injectKeyEvent(serviceManager, action, keyCode);
// 处理完,记得清楚buffer
buffer.clear();
}
}
}
} while (!eof);
这样就可以进行事件的相应了。
优化后的线程模型如下:
- client端(PC)
- event_loop
SDL的EventLoop。复制渲染上屏和分发事件
- event_sender(Socket send)
接受SDL分发的事件。并把对应的事件通过Socket分发给Android手机。
- screen_receiver(Socket recv)
通过Socket接受的 H264 Naul,使用FFmpeg进行解码。
- server端(Android)
- screen record (Socket InputStream)
使用SurfaceControl和MediaCodec进行屏幕录制,录制的结果通过Socket发送
- event_loop (Socket OutputStream)
接受Socket发送过来的事件。并调用对应的API进行事件的注入(InputManager)
### 线程通信
- frames
两块缓存区域。
- decode_frame
解码放置的frame
- render_frame
渲染需要的frame.使用该frame 进行render
数据流动
- 生产的过程
screen_receiver 负责生产。
- 消费的过程
event_loop 负责消费。将两块缓存区域进行交换,并把render_frame上屏
- event
一个event_queue队列来接受。可以使用链表
数据流动
- 生产的过程
event_loop 负责生产。并把数据送入队列当中
- 消费的过程
event_sender 负责消费。如果队列不为空,则进行发送
这里就不详细说明了。具体可以看代码就明白了。
最后的结果.gif
就和Vysor
和scrcpy
一样,我们可以通过投屏PC ,并操作手机了。而且在很低的延迟下。
源码地址:https://github.com/deepsadness/AppProcessDemo
还有更多的细节处理,可以参考scrcpy
Android PC投屏简单尝试 这一系列文章,终于到了尾声。总共横跨了大半年的事件。 最后分成下面几个方面来进行一下总结
SurfaceControl
的方法,来完成录制屏幕数据的获取。(参考adb screenrecord 命令)就是适合简单的发送Bitmap。只要接受端能够解析这个bitmap数据,就可以完成数据的展示。
可以通过在服务端建立RTMP协议,然后通过这个协议进行。使用RTMP协议发送的好处在于,需要播放的端只要支持该协议,就可以轻松的进行拉流播放。
这个仅仅适合于PC能够直接用ADB和手机连接的场景。 但是在这个场景下,投屏的效果清晰,流畅,延迟很低。 暂时部分,因为直接发送H264数据,只要进行解码后,就可以进行播放了。(文章使用了SDL2的方式进行了方便的播放。)
整个过程中 我们对Media Codec和ImageReader/RTMP协议/FFmpeg/SDL2/Gradle进行了知识点的串联。 其实还是挺好玩的。
如果是需要改成手机和手机连接。我们要怎么实现呢? 其实从上面不难看出。如果是手机和手机连接。 在近距离,我们可以简单的使用蓝牙进行Socket(类似ADB和USB的通信方式)。 如果是远距离,就可以通过RMTP的方式,来进行推流和拉流。
最后,完结撒花?~~