Bridging Google Assistant and Belphanior

Having acquired a Google Home not too long ago, I decided to delve into building a bridge between it and Belphanior, my home automation framework. While there are a few sharp edges, I was surprised to discover it wasn't as challenging as I expected! The following is an overview of the process and some things I ran into.

Architecture Overview

Google Assistant endpoints (Home, Allo, and the Assistant on Android phones) can be extended with additional functionality via the Actions on Google framework. This is a set of technologies that basically let you define a conversational flow and an endpoint that will receive and act on information from the ongoing conversation.

I based my bridge very heavily on the "Google Facts" demo that serves as the getting started process for Actions on Google.

The technologies I used include 

1. Actions on Google, which manages the action that handles requests from Google Assistant.
2. api.ai, which hosts the conversational agent. Assistant interacts with the user via rules in this agent, and the interactions become requests to an HTTP endpoint I specify.
3. Cloud Functions, a Google Cloud service that runs a bit of script in response to an HTTP request (like a smaller version of App Engine). My agent sends requests to this endpoint, which is interpreted and sent to my home automation service via Pub/Sub.
4. Cloud Pub/Sub, another Google Cloud service that supports posting messages to be picked up asynchronously. I used this so that the home automation service (which runs on a Raspberry Pi in my house) can receive requests without needing to expose a world-visible endpoint that could be vulnerable to hacking.

Wiring up these tools took some time, but once completed I had a full end-to-end solution for voice and text control of the house.

The Process

Setting up the action

Actions on Google are basically small sets of rules for extending the things Google Assistant knows how to do. They're triggered via a command phrase such as "Okay, Google, tell <your project's name> <command sent to the project>", for example, "Okay, Google, please tell Belphanior 'Lights on.'" 

Starting with a new project is simple enough: you create one from the Actions console and follow the flow provided to set up the project. In my case, I ended up changing the name to "The house ghost," because (a) 'Belphanior' is a bit hard to pronounce and (b) Google generally forbids one- or two-word names for assistant apps.

App configuration, including the name for assistant purposes.

A bit about this process: Google very much designs this flow to get you to a published assistant app that could be used by anyone, but for my purposes I only intend to allow my own house to be controlled (I didn't even add multiple account support). Getting as far as testing the action was enough.

Setting up the agent

Actions allow for a couple of options in setting up the conversation flow, but the easiest one to use was api.ai. It provides a convenient interface for setting up a conversational agent (for multiple systems, including Actions on Google and Slack). The api.ai framework is pretty rich, including neat capabilities such as training conversation components with multiple variants (i.e. "Let me buy X", "I want to buy X", "How can I buy X", etc.) and creating flows from one intent to another.

For my purposes, however, the functionality I want is very simple: I want api.ai to forward the message directly to my cloud function. So I only need the default intent, and I indicate that it should handle the request ("fulfillment") via the webhook I will specify.

"Use webhook" is basically the interesting part here.

Once the default fallback intent is set up, I go to the "Fulfillment" tab and specify where the agent should send requests. The webhook is just a world-visible URL; I specify my cloud-functions endpoint here. Note: what's missing from this screenshot is that I also specified an HTTP Basic Auth username and password. Those will be sent to the Cloud endpoint in the "Authorization" header as the Base64 conversion of "username:password" (no newline). It's a wise idea to do this at least, since a Cloud Functions HTTP endpoint will accept any request that comes into it.

Don't forget to set up basic auth

The only remaining step in api.ai is to enable the integration to Actions on Google, which is under the "Integrations" tab. Just hit the toggle button to enable the integration, then click "Settings" to get the detail dialog. At this point, I can click "UPDATE" to publish the configuration to Actions on Google, and then "TEST to use the Actions on Google simulator to try and control my app.

I can attempt to send commands such as "please tell the house ghost Lights On", but the result will be that nothing happens, because the Cloud Function webhook doesn't exist yet.

Setting up the Cloud Function handler

Now that we have the API set up, we can set up our Cloud Function handler to catch the requests. I have an existing Google Cloud Project that I use for various things; since I don't use any Cloud Functions yet, I just added one there named "speech-list-options".

When you create a Cloud Function, you have the option to configure what triggers it (HTTP in this case) and provide code that it executes. The code for my function is here; essential steps:

1. Verify auth header
2. Initialize an API.AI app from the 'actions-on-google' library
3. Grab the raw speech input to the agent
4. Bundle up the raw speech as a command published to the 'belphanior-commands' topic (which is provided by the @google-cloud/pubsub library)

The libraries are pulled in by the rules specified in the package.json tab of the Cloud Functions editor:

{
  "name": "speech-execute",
  "version": "0.0.1",
  "dependencies": {
    "actions-on-google": "^1.0.0",
    "@google-cloud/pubsub": "~0.10.0"
  }
}

After that, the only remaining configuration is to specify a bucket to serve as "staging" for Cloud Functions and the entrypoint function that should be run when the HTTP trigger is hit (listOptions).

Setting up the Cloud Function

At this point, every command sent is published to the 'belphanior-commands' topic in my project. Using the Cloud Console and the "Pub/Sub" tab, I then created the topic and a pull subscription (projects/<project name>/subscriptions/belphanior-butler).

Receiving the commands

Receipt of the commands is done by a small service running on the same Raspberry Pi that hosts Belphanior's home automation core (which holds an App Engine default service account credential). The overall flow of the service is:

1. Establish connection to the Belphanior butler
2. Establish connection to Pub/Sub in the project on the subscription previously set up
3. Poll for new messages 
1. New message received: treat the message body as a Belphanior command and send it to the butler.

All the pieces are now in place. The test version of house ghost runs under my Google account so, it can be accessed via the simulator, my smartphone, or a Google Home registered to my account.

Issues

Overall, this process works; I've only encountered a couple of issues.

• It appears that running an app in test mode "wears off" eventually, and the test has to be restarted. It'd be convenient if there were a way to build an app intended to only be used by one user, but I don't see a mechanism to do that.
• Because of the need to use Pub/Sub to serve as a pull-request target (so my Raspberry Pi doesn't itself need a world-viewable IP address tied to a domain name), all communication from the Assistant to Belphanior is one-way. To improve this, I'd either have to run Belphanior itself in the cloud or have a protocol for shipping some of the house's state to the cloud (possibly also via Pub/Sub) so that the Cloud Function could answer questions about the state of the house from locally-cached data. At this time, it's not a need I have, but if I ever need to ask questions like "Are the lights on?" it's a problem I'd have to solve.
• After a period of inactivity (a few minutes), the next Pub/Sub message can take over thirty seconds to get picked up by my Raspberry Pi's polling. I don't know precisely what the issue is, but it seems that Pub/Sub may not be optimized for rapid delivery of sparse, low-volume messages. I've heard rumor that it is possible to force a "buffer flush" by increasing the number of messages I send at one time (buffering the "payload" message with a half-dozen "no-op" messages before and after it), but I haven't yet experimented with that option.

Once the pipeline is set up, I'm very impressed with the reliability of Home as a voice control solution; it's extremely good at dissecting my intended message without operating off of a dictionary of possible inputs I provide. Voice recognition tech keeps getting better!

Dipping a toe into Z-Wave: OpenZWave in Go with the Ninja Blocks wrapper library

Having installed OpenZWave and successfully set up my UZB key as a controller, the next part of the process was to bridge the OpenZWave library to my Belphanior home automation core. Belphanior communicates with its servants via an HTTP-based RPC protocol; easiest route seemed to be to set up a web server to handle requests from Belphanior and translate them into Z-Wave commands.

OpenZWave is a C++ library; while I can work in C++, for web servers and the like I prefer a less detailed abstraction. Fortunately for me, an Australian firm called Ninja Blocks put together an excellent Go-language wrapper available on their github repo, which was exactly the tool for the job.

Here, I document what I liked about go-openzwave and my experience getting it working with Belphanior.

Dipping a toe into Z-Wave with the UZB controller hardware

Quite some time ago, I laid down some things I'm looking for in a home automation solution. I've recently come to the conclusion that Z-wave is probably mature enough to begin experimenting with. It's a closed protocol owned by a single company, but the company has been around long enough and has affiliated with enough consumer partners that it's unlikely to vanish tomorrow. It may not be secure (it looks like there are security features in the protocol, but I don't know yet if all devices support them), but it should support the debugging and status features I'm looking for that I was never able to get working with INSTEON. To that end, I purchased a UZB stick from Z-Wave.Me and a GE Dimmable Lamp Module and started experimenting. This will be an ongoing trip; we'll see where we end up.

Reflections on Home Automation

It's been a little while since I've put more time into my home automation program. I've been feeling a little slowed down by the hardware available to me. My current solution for automation is an INSTEON network---a few wall switches and X10 relays with a central controller, itself driven by Belphanior. Recently, the communication between the controller and the wall switches seems to have gone out, and diagnosing the issue has been difficult.

Having worked with INSTEON for awhile now, I know a couple of features I'd be looking for in the next iteration of the hardware I used:

Diagnostic Tools: The biggest challenge with working with INSTEON is that when two devices cannot see each other, the root cause can be hard to determine. Ideally, I'd like to have a way to monitor the airwaves for the INSTEON signal to verify it is being transmitted and received correctly, but I'm unaware of any dedicated diagnostic monitoring devices for this purpose.

Status Feedback: While in principle INSTEON includes status updates to and from the devices, in practice I've found it to be spotty and not sufficiently reliable to base any sense of the house's state upon.

Security: INSTEON is protected by a longer ID than X10, but there's nothing really to prevent someone equipped with the right technology from radio-signalling the controllers, listening for their addresses in the radio chatter, etc. While I'm sure I can rely on security through obscurity in the short run, it's probably wise to lock down the signals longer-term.

I'll have to do some research; I'm not sure what other devices are available on the market to satisfy these goals (especially ones that might fit into a wall-socket form factor).

Belphanior Home Automation System

I've been playing around with home automation as a hobby for quite a long time now. One thing I found frustrating was the lack of flexibility in the central controllers I've worked with. To that end, I'm pleased to introduce Belphanior, my own automation-oriented remote signalling and scripting system.

Belphanior is intended for a standard household setup (multiple computers and automaed systems behind a local intranet). Key to its design is a multi-process approach; the core of Belphanior is a communications protocol built atop HTTP and JSON, which allows a single "butler" process to coordinate and control multiple "servant" processes. This allows you to run the servants locally on the machines with the hardware that needs to be controlled. It's also designed to be self-documenting; "roles" (specific types of tasks servants can perform) are first and foremost human-readable, with the details of implementation left up to the creators of the servants. So an "output" role can be handled by a text-to-speech servant, a logger, a marquee-display controller, etc.

Belphanior's jumping-off point is at belphanior.net. There, you'll find a "getting started" guide and a list of the currently-implemented servants. Servants and the butler are available as open source under the MIT license, so if you wish to create your own, you are free to do so (and if you'd like to provide me a link to them, I'd be happy to post them on belphanior.net!).

A short list of things I've used Belphanior to do:

• Turn the main lights off in the house at the end of the day (I'm terrible about leaving basement lights on when I go to bed).
• Control Christmas tree lights and other seasonal home decorations to save power.
• Drive the train under the Christmas tree every two hours (see "Making the Train Run On Time").

I'll be continuing to extend Belphanior and the home automation system as I go, and I'll be updating this blog with any fun developments.

Have fun, and happy hacking!

Making The Train Run On Time

I have a few INSTEON modules and a 2412N controller. I've been experimenting with home automation for awhile, and this year I thought I'd do something fun.

I unpacked the model train that goes under our family tree and connected the power box to an INSTEON lamp module. Using Belphanior, my not-quite-yet-fully-released home automation core, I wrote a short script that drives the train for a few seconds (timing is a little sloppy, as it takes several seconds for the 2412N to respond to HTTP requests from the 2412n servant, but it does the job).

"cycle train" script

With the script written, all I had to do was fire it every two hours. Fortunately, there is a Belphanior servant that can monitor an iCal-formatted calendar and notify the butler to run scripts in response to calendar events. I pulled up a Google Calendar that I keep for this purpose and populated it with a cycle of the train every two hours (except at night, to save power).

Belphanior calendar, showing "cycle train" command

The result is shown below. I'm pretty satisfied with it; simple, but effective!