都说语音是人机交互的重要手段,虽然个人觉得在大庭广众之下,对着手机发号施令会显得有些尴尬。但是在资源受限的物联网应用场景下(无法外接鼠标键盘显示器),如果能够通过语音来控制设备,与设备进行交互,那还是很实用的。继上一篇《Windows 10 IoT Serials 4 - 如何在树莓派上使用Cortana语音助手》之后,本文将详细讲述如何为运行Windows 10 IoT Core系统的树莓派添加语音识别和语音交互功能。
注意,这里音频输出设备和显示设备是可选的,并不是必须的。
这里将LED连接到树莓派的GPIO5和GPIO6两个引脚,同时,把麦克风设备插入到树莓派的USB接口。如果准备了音频输出设备(如耳机或音响)和显示设备(显示器),请连接到树莓派的3.5mm音频接口和HDMI接口。
本应用程序使用的开发环境是Windows 10+Visual Studio 2015 Community,注意,Visual Studio需要包含Universal Windows App Development Tools组件。
新建工程时,选用Universal模板,工程命名为RPiVoiceControl,如下图所示。
因为要用到GPIO引脚控制LED,所以需要为工程添加Windows IoT Extension for UWP引用,如下图所示。
由于需要使用Microphone,所以需要在工程的Package.appxmanifest文件中,勾选Microphone,如下图所示。
另外,由于需要使用到语音识别、LED和UI控件等资源,需要为应用程序引入命名空间,如下:
using System; using System.Diagnostics; 此处省略若干…
using Windows.Devices.Gpio; //LED using Windows.Media.SpeechRecognition;//语音识别 using Windows.Media.SpeechSynthesis; using Windows.Storage; using Windows.ApplicationModel;
为项目添加新的xml文件,命名为Grammar.xml,用于定义语音指令。项目中用到的语音指令符合Speech Recognition Grammar Specification Version 1.0 (SRGS)标准,其具体协议可以参考MSDN上的这个文档:Create Grammars Using SRGS XML (Microsoft.Speech)。
之后,打开该文件,为其添加如下语音指令。
<?xml version="1.0" encoding="utf-8" ?> <grammar version="1.0" xml:lang="en-US" root="automationCommands" xmlns="http://www.w3.org/2001/06/grammar" tag-format="semantics/1.0">
<rule id="root"> <item> <ruleref uri="#automationCommands"/> <tag>out.command=rules.latest();</tag> </item> </rule>
此处省略代码,具体请参考Github上项目的完整代码。
<rule id="deviceActions"> <one-of> <item> light <tag> out="LIGHT"; </tag> </item> <item> led <tag> out="LED"; </tag> </item> </one-of> </rule>
</grammar>
如果不准备给树莓派接显示器的可以直接忽略这一步,如果需要在程序运行过程中查看状态的,可以加入一些简单的控件,这里只是加入了两个指示LED灯状态的Ellipse 控件、两个指示程序运行状态的TextBlock 控件和一个MediaElement 控件,代码如下。
<Grid Background="{ThemeResource ApplicationPageBackgroundThemeBrush}"> <StackPanel HorizontalAlignment="Center" VerticalAlignment="Center"> <Ellipse x:Name="bedroomLED" Fill="LightGray" Stroke="White" Width="100" Height="100" Margin="10"/> <Ellipse x:Name="kitchenroomLED" Fill="LightGray" Stroke="White" Width="100" Height="100" Margin="10"/> <TextBlock x:Name="GpioStatus" Text="Waiting to initialize GPIO..." Margin="10,50,10,10" TextAlignment="Center" FontSize="26.667" /> <TextBlock x:Name="VoiceStatus" Text="Waiting to initialize Microphone" Margin="10,50,10,10" TextAlignment="Center" TextWrapping="Wrap" /> <MediaElement x:Name="mediaElement"></MediaElement> </StackPanel> </Grid>
后台代码中,首先需要定义应用程序使用的资源对象,如GPIO、画刷、定时器、部分代码如下,
private const int BedRoomLED_PINNumber = 5; private GpioPin BedRoomLED_GpioPin; private GpioPinValue BedRoomLED_GpioPinValue; private DispatcherTimer bedRoomTimer;
private const int kITCHENLED_PINNumber = 6; private GpioPin kITCHENLED_GpioPin; private GpioPinValue kITCHENLED_GpioPinValue; private DispatcherTimer kITCHENTimer;
private SolidColorBrush redBrush = new SolidColorBrush(Windows.UI.Colors.Red); private SolidColorBrush grayBrush = new SolidColorBrush(Windows.UI.Colors.LightGray);
然后,在MainPage的构造函数中,添加资源的初始化,部分代码如下:
public MainPage() { this.InitializeComponent(); Unloaded += MainPage_Unloaded;
// Initialize Recognizer initializeSpeechRecognizer();
InitBedRoomGPIO(); InitKITCHENGPIO();
bedRoomTimer = new DispatcherTimer(); bedRoomTimer.Interval = TimeSpan.FromMilliseconds(500); bedRoomTimer.Tick += BedRoomTimer_Tick;
kITCHENTimer = new DispatcherTimer(); kITCHENTimer.Interval = TimeSpan.FromMilliseconds(500); kITCHENTimer.Tick += KITCHENTimer_Tick; }
在initializeSpeechRecognizer函数中,完成语音识别状态改变事件的添加、语音指令文件的加载,部分代码如下:
private async void initializeSpeechRecognizer() { // Initialize recognizer recognizer = new SpeechRecognizer(); // Set event handlers recognizer.StateChanged += RecognizerStateChanged; recognizer.ContinuousRecognitionSession.ResultGenerated += RecognizerResultGenerated; // Load Grammer file constraint string fileName = String.Format(SRGS_FILE); StorageFile grammarContentFile = await Package.Current.InstalledLocation.GetFileAsync(fileName); SpeechRecognitionGrammarFileConstraint grammarConstraint = new SpeechRecognitionGrammarFileConstraint(grammarContentFile);
// Add to grammer constraint recognizer.Constraints.Add(grammarConstraint);
SpeechRecognitionCompilationResult compilationResult = await recognizer.CompileConstraintsAsync(); Debug.WriteLine("Status: " + compilationResult.Status.ToString());
// If successful, display the recognition result. if (compilationResult.Status == SpeechRecognitionResultStatus.Success) { Debug.WriteLine("Result: " + compilationResult.ToString());
await recognizer.ContinuousRecognitionSession.StartAsync(); } else { Debug.WriteLine("Status: " + compilationResult.Status); } }
之后,添加RecognizerResultGenerated和RecognizerStateChanged两个事件的处理,主要用于语音识别结果和状态发生变化的处理。部分代码如下: private async void RecognizerResultGenerated(SpeechContinuousRecognitionSession session, SpeechContinuousRecognitionResultGeneratedEventArgs args) { // Check for different tags and initialize the variables String location = args.Result.SemanticInterpretation.Properties.ContainsKey(TAG_TARGET) ? args.Result.SemanticInterpretation.Properties[TAG_TARGET][0].ToString() : "";
String cmd = args.Result.SemanticInterpretation.Properties.ContainsKey(TAG_CMD) ? args.Result.SemanticInterpretation.Properties[TAG_CMD][0].ToString() : "";
String device = args.Result.SemanticInterpretation.Properties.ContainsKey(TAG_DEVICE) ? args.Result.SemanticInterpretation.Properties[TAG_DEVICE][0].ToString() : "";
Windows.ApplicationModel.Core.CoreApplication.MainView.CoreWindow.Dispatcher.RunAsync(Windows.UI.Core.CoreDispatcherPriority.Normal, () => { VoiceStatus.Text= "Target: " + location + ", Command: " + cmd + ", Device: " + device; });
switch (device) { case "hiActivationCMD"://Activate device SaySomthing("hiActivationCMD", "On"); break;
case "LIGHT": LightControl(cmd, location); break;
default: break; } }
// Recognizer state changed private async void RecognizerStateChanged(SpeechRecognizer sender, SpeechRecognizerStateChangedEventArgs args) { Windows.ApplicationModel.Core.CoreApplication.MainView.CoreWindow.Dispatcher.RunAsync(Windows.UI.Core.CoreDispatcherPriority.Normal, () => { VoiceStatus.Text = "Speech recognizer state: " + args.State.ToString(); }); }
定义函数SaySomthing,用于反馈的语音生成,这样,用户就可以听到树莓派的语音反馈了。部分代码如下:
private async void SaySomthing(string myDevice, string State, int speechCharacterVoice = 0) { if (myDevice == "hiActivationCMD") PlayVoice($"Hi Jack What can i do for you"); else PlayVoice($"OK Jack {myDevice} {State}", speechCharacterVoice); await Windows.ApplicationModel.Core.CoreApplication.MainView.CoreWindow.Dispatcher.RunAsync(Windows.UI.Core.CoreDispatcherPriority.Normal, () => { VoiceStatus.Text = $"OK -> ===== {myDevice} --- {State} ======="; }); } 最后,在两个定时器的溢出事件处理中,加入对LED灯的处理,部分代码如下:
private void BedRoomTimer_Tick(object sender, object e) { if (BedRoomLED_GpioPinValue == GpioPinValue.High) { BedRoomLED_GpioPinValue = GpioPinValue.Low; BedRoomLED_GpioPin.Write(BedRoomLED_GpioPinValue); bedroomLED.Fill = redBrush; } else { BedRoomLED_GpioPinValue = GpioPinValue.High; BedRoomLED_GpioPin.Write(BedRoomLED_GpioPinValue); bedroomLED.Fill = grayBrush; } }
在Visual Studio中设置编译的平台为ARM,调试设备为Remote Machine,在Debug选项卡中,设置树莓派的IP地址,点击调试。如下图所示。
程序运行以后,用户可以通过语音指令与树莓派进行交互。
首先,用户可以使用“Hi Jack”与设备交互,可以听到设备有回复,用于确认应用程序是否正确运行。
其次,用户可以使用“Turn On/Off Bedroom Light”和“Turn On/Off kitchen Light ”来控制两个LED灯,同时,在应用程序的界面上还可以看到灯的状态和语音识别的状态,如下图所示。
应用程序运行的实物图如下:
本项目的代码已经发布到Github上,链接如下:https://github.com/shijiong/RPiVoiceControl,欢迎下载使用。