Getting Started: Creating a Reaction



Getting Started

Welcome to Asteroid! We're here to help you build powerful AR applications.

Let's get started in Asteroid by creating our first Reaction. Reactions are functional responses to something your program observes in your environment, through a sensor input from your device - like a camera, microphone, or other sensor. A simple example of a Reaction might be informational: your app sees an object through your camera and displays the object's Wikipedia page next to it. You might also want to learn the name of an unfamiliar yoga pose, or identify your geographical surroundings by looking at local architecture.

To make a working Reaction, we'll need to do a few things:

Declare our inputs, or state what exactly we want our app to react to Define our model, or decide how our app will identify and parse that input Determine where to send the outputs of our app’s observation, or what we should do with our observations, we’re going to connect your computer’s camera to a machine learning model, which will send a text string to a 3D text object.

Getting our Assets

When you load up your client, you’ll start with a series of assets from our Store already linked within it. Open the bottom menu and scroll down to the “Store” section. Scroll through there to find the asset called “AgeNet Model” – that’s the model we’ll be using to process our inputs. Click the “+” button on that to load it into your client.

The AgeNet model takes an image as an input and predicts the age of the person in it. Asteroid is modular, which means we could have chosen any other type of model – like a frog classifier or a deep filter, for example.

If everything worked correctly, a “Model” element should appear in your client’s UI like this: MLModelNode (You can rename the Object by double clicking on the text)

If you click on the “Model” row, it should expand to show more detail and an option to run. Click on the box below the “Run Model” button to select the AgeNet model. selectMLModel This will load the model to be ready for use. This is all we’ll do with the ML Model for now. We’ll return to it soon.

Adding our Inputs and Outputs

Since we’re dealing with AR, our primary input is going to be our camera. Navigate to the bottom of the menu to find "Camera Input" and click the “+” next to it. It should appear in the Storyboard (the left panel of the client).

Head back to the menu again to add a 3D Text object – this is what’s going to receive the output of the AgeNet model (your age!). Your Storyboard should now look like this:

ReactionElements

Making the Reaction

Here’s where Asteroidreally helps make things simple: we’re going to connect these three nodes together, which will create our new Reaction and the functionality for our app. Each Asteroid node has an input and output, and connecting them is the (easy) way you can move data around your application.

To make things easiest, arrange your nodes from left to right, in this order: Camera Object, ML Model, 3D Text, since that’s the order we want them to execute in. FromLeftToRight

Now that we have all three of our basic building blocks in place, we can wire them up (connect them together) to give our app the logic it needs to work.

On the Camera Input Object, click on the right-hand dot on the “Output Frame” row, and drag it onto the “data” row of the ML Model object. That tells Asteroid to send the output frame from your camera as data to the AgeNet model that we loaded beforehand. It should look like this:

DataConnection

The output of sending the output frame through the AgeNet model is a string of text – the model’s guess of your age – that you can use for anything that you'd like. Here, we're just going to use it to render some 3D text.

We’ll start by sending the model output – or the “classLabel” object in our ML Model node – into the 3D Text object to display the model’s results. Click the down-facing chevron on the ML Model object to reveal its other properties. Click and drag from the “classLabel” row into the “Text Value” row of the 3D Text patch. This will send the results of the ML Model into the 3D Text object.

Your final version should look like this:

AllConnected

Ignition

To make it all run, click on “Start Camera” in the Camera node, and your Scene panel on the right will fill with your computer’s video. The 3D Text will then show you how old the ML Model thinks you are. (We aren’t responsible for inaccurate guesses, but we do apologize for its potentially offensive guesses 😛).

Here’s our final product:

FinalProduct (Note: I am offended by AgeNet’s guess of my age.)

Where to go from here

We've designed Asteroid to be a modular creation experience. You can save this Reaction as a ".reaction" file and share it with others, or use it in other apps. You can swap out the assets, the 3D models, and the ML models individually to make a different Reaction super easily. Asteroid is also designed to be a platform between developers. If you create a Reaction that you'd like to sell or share, you can do so through our Store. Just email us and we'll help you get it hosted.

Please enjoy Asteroid, and reach out to us with any feedback or problems :)

Follow us on twitter Join us on Spectrum Jump into our Slack!