演讲稿: 如何使用增强现实技术提高应用的用户体验

With the accellerated evolution of mobile hardware, application developers tend to reconsider the importance of the Augmented Reality technology ( called AR for short in this article subsequently ) in regard to satisfy their customers better with enhanced user experience.

What is AR? See one explanation from Wikipedia:

Augmented reality (AR) is an interactive experience of a real-world environment where the objects that reside in the real world are enhanced by computer-generated perceptual information. AR can be defined as a system that fulfills three basic features: a combination of real and virtual worlds, real-time interaction, and accurate 3D registration of virtual and real objects.

This blog will not explore the possible scenario where AR can play its important role. Instead, Jerry will introduce a kind of development approach which enables developers to generate applications leveraging AR without deep knowledge on Computer Graphics.

This approach consists of pure JavaScript technical stack：the combination of React-Native plus ViroReact.

Jerry has introduced how to use Cordova to package a hybrid mobile application in the past:

• Step by step to create a custom Cordova plugin for Android and consume it in your UI5 application
• Step by step to create a custom Cordova plugin for Android and consume it in your UI5 application
• How to debug UI5 application packaged into a mobile device via Cordova with a custom plugin

And compared with Cordova, React-Native has quite different design idea: while Cordova can easily allow existing web application to run under different mobile platform at the cost of some performance loss, the application built by React-Native can behave more like a native mobile application.

For the concrete comparison between these two technical stacks, please refer to this medium article.

ViroReact，a platform for developers to rapidly build native cross platform AR applications using React Native. Its website provides many step by step guide by following which, you can soon get a running Hello World application with AR enablement.

Although just a single declaration code to introduce the dependency of ViroReact to your React-Native application as below, it's just the tip of iceberg.

When we enter the root folder of application with react-viro declared as dependency in its package.json and perform npm install, lots of artifacts are automatically downloaded into the node_modules folder:

Expand folder react-viro and we can find two separate folder, android and ios, which contains corresponding library codes for each platform. This mechanism implies how ViroReact achieves the cross platform enablement of AR feature. And what's the meaning of arcore_client under android folder above?

Before we go deep into the source code, let's acquaint ourselves with some prerequisite knowledge. In ViroReact website, it is documented that both ARKit from iOS and ARCore from Android are supported.

In this blog I choose Android platform which means I will concentrate on ARCore. Still remember the arcore_client folder mentioned previously?

Let's first see some real examples, developed by my colleague Wang Leo:

https://youtu.be/05A0bIiVouE

https://youtu.be/Rt163cQ_fbg

Amazing, isn't it?

Below are steps how to develop an application like that and make it running in your Android phone.

As the first step, ARCore should be installed on the Android phone.

ARCore is Google’s platform for building augmented reality experiences. Using different APIs, ARCore enables your phone to sense its environment, understand the world and interact with information.

Check the supported device list from Google website and install ARCore into your device according to corresponding instructions:

In ViroReact website there is a nice tutorial for beginners( just as our SAP UI5 tutorials):

Just follow the steps there and finally you will get a Hello World application with some computer generated virtual objects positioned within your real world:

In fact the AR car demo by my colleague Leo is also based on this tutorial, so I will introduce the delta part Leo has added on top of the Hello World tutorial.

In React-Native package.json file, name the project as "ViroSample" and declare the dependency of React-Native and ViroReact as below:

Execute the following two command lines：

• react-native start
• react-native run-android

And you can find the corresponding generated Java source code for Android platform under android folder. The most important one is MainActivity.java, where the project name defined in package.json is injected:

And in MainApplication.java, the entry point of React-Native JavaScript module is injected in Android callback getJSMainModuleName:

In my opinion the common process of ViroReact development consists of three major steps: Match, Replace and Augment.

Match step

As developers mix computer generated virtual objects with real world, they must decide which places in real environment could be used to hold the virtual objects and tell AR application the exact position. In ViroReact based AR application this is achieved by the tag <ViroARImageMarker>. See one example below.

As its name hints, this tag declares a marker which guides ViroReact with exact position where the virtual object should be put on. As long as end user opens device Camera and detects a real object in the real world which is exactly the same as the object represented by the marker, we call this process as "matched" and ViroReact will automatically replace marker with another prepared virtual object ( will introduce soon ). As long as the marker is matched by some object in the real world, the replace process is automatically done without any additional coding from developer.

A marker is a placeholder in the application and could have various types, and of course ViroARImageMarker has Image as marker type, the image marker is indicated as name "logo" defined in line 44 above.

In the above example, the marker named "logo" is created by the following code:

The input of createTargets API is an image file stored in project folder:

This is reason in the youtube demo video that first Leo displays this image in his screen, then scan it via phone camera, and the image marker is replaced by the prepared car 3D model.

Replace step

As mentioned earlier, there is nothing to do from developer in this step except providing a 3D model to be replaced by image marker.

ViroReact provides the tag <Viro3DObject> which represents the 3D car model consisting of a pure model ( .obj file ) and a series of material textures.

Below is the obj file for this car demo, opened by 3D Design software:

In Paint 3D software, we can drag it and have a 360 degree view on it. You can see the dynamic effect from this video:

You might think that the car model has no color on its surface and doesn't look not fancy at all. So do I. That's why we need to prepare texture as well.

In the project folder six different image files are stored there as raw material for model texture.

Colorful materials are created using those texture images files, and finally generate the glossy car model in the youtube video.

Augment step

Finally it's time for we developers to play with those generated virtual car model by JavaScript code. Since now we have six colorful textures at hand, it follows logically that it could be nice if the car in the AR application could change color every time it's tapped in the phone.

A piece of cake: register a function to onClick event handler:

And change the binding field for texture property accordingly:

Check this video below about how the Tesla car model in the application can change color once pressed:

Hope this blog can give you a basic understanding how to develop AR application using React-Native + ViroReact, thanks for reading.