BiX Workshop 1: Connecting a VR headset and Microcontroller

Example tutorial for connecting an ESP32 Board with a Meta Quest 2 Headset and setting up basic Controller -> VR and VR -> Controller interactions.

What is needed

  • A VR-capable computer
  • Meta Quest 2
  • Adafruit Feather ESP32 V2 Board
  • Unity with Android (API 32 or lower) installed

Part 1: Connecting Arduino to Quest using Singularity

Set up ESP32

  • Open “setup_esp32_simple_return.ino” in the Arduino IDE
  • Change the board name on line 7, in the Setup Function

BTSerial.begin("Tyler_ESP32");

  • Plug the ESP32 into your computer using USB
  • In the Arduino IDE, select Board and Port
    • Board: Tools->Board->esp32 or ESP32 Arduino-> Adafruit Feather ESP32 V2
    • Port: Tools->Port->COM#
    • Note: Port may be titled something different if not using the Alienware laptops
  • Click on the upload arrow to load this program to the ESP32

Pair Quest to ESP32

  • In the Quest headset, go to Quick Settings->Bluetooth->Pair New Device, and choose the name you gave to your ESP32.

Enable Hand Tracking

  • In Unity go to Quick Settings -> Hands -> Enable Hands and Body Tracking
  • Activate by tapping your controllers together twice, then place down on your nearest surface

Set up the Unity Project

  • Open Unity Hub, and open or add the project “BiX_Quickstart”
  • Click on the “BluetoothCommunicator” in the Hierarchy pane
  • Change the “Board Name” to match the name you put on the ESP32, exactly.
  • Save the project.
  • Open the build panel in File->Build Settings (you can drag this tab next to the “Inspector” tab)
  • With your Quest plugged into your computer, choose the Quest headset in the “Run Device” menu. Click refresh if it is not there.
  • Click “Build and Run” and give the Build any name you like.

Test the project

  • Once the build finishes, put on your headset and see if the microcontroller has connected. You will see the name in the menu, and “Connected to the Microcontroller” above.
  • If the microcontroller is not connecting, try quitting the app, and reopening it (Library -> Applications ->Unknown Sources (dropdown menu) -> Build name).
  • If issues persist, try forgetting the microcontroller and reconnect it in the Bluetooth settings.

Part 2: Sending Button Presses to the Quest

Have the ESP32 send button state

Inside “setup_esp32_simple_return.ino” in the Arduino IDE

  • Create a new line at line 3:

// pin 38 correlates to the ESP32’s SW38 built-in button
const int BUTTON_PIN = 38;

  • Inside the setup() function, add a line at the end (after line 11):

// initialise SW38 as a input
pinMode(BUTTON_PIN, INPUT);

  • At the end of the loop() function (after line 29, outside the if block), add these lines:
// monitor button via a boolean variable
// buttonState returns true when NOT pressed
// buttonState returns false when pressed
bool buttonState = digitalRead(BUTTON_PIN);
if(buttonState == true){
// if button NOT pressed, send message “on”
   BTSerial.println("on");
   }else{
// if button pressed, send message “off”
   BTSerial.println("off");
   } 
// .5 second delay, helpful for not overloading the system
delay(500);

Upload this code to your ESP32

  • When the upload is complete, in your Quest headset, restart the app. When the app starts you should start seeing “Message Received: on” and when you press and hold the SW38 button on the ESP32, it will change to  “Message Received: off”

Receive the message in Unity and turn a sphere on and off

In Unity we will add a sphere that disappears when you press the SW38 Button.

  • Click the plus button in the top left corner of the hierarchy frame, choose 3D Object -> Sphere
  • In the Inspector tab, change the scale of the sphere to 0.3 for x, y, and z (this is equivalent to 0.3 meters in vr space).
  • Move the sphere closer to the player.

  • Open the “ExampleCommunicator” by double clicking the script in the BluetoothCommunicator Object or in Assets -> BiXSpecific -> Singularity -> ExampleCommunicator
  • Add a new line before the Start function (line 14):
// public variables are assigned a value in the inspector
public GameObject objectToHide;

At the end of the “onMessageReceived” function (line 67):

// if message received is “on”, reactivate objectToHide
if("on".Equals(message)){
   objectToHide.SetActive(true);
// if message received is “off”, deactivate objectToHide
   } else if ("off".Equals(message)) {
   objectToHide.SetActive(false);
   }

Save this script in the code editor.

  • In the Unity editor, drag the sphere onto the “Object To Hide” game object space.
  • Save the Unity project and Build and Run the project to the Quest Headset.

Test that the button controls the sphere’s visibility

  • Test that the sphere disappears when the button is pressed.
  • You may need to adjust the position of the sphere based on what you observe.

Part 3: Send grab message to ESP32

Have Unity send message when the sphere is grabbed and released

In the code editor for Unity, add the following functions to “ExampleCommunicator.cs” by creating a new line after 81, before the closing bracket:

// new function that, when triggered, will send a message of “grab” to the connected bluetooth microcontroller
public void sendGrabMessage(){
mySingularityManager.sendMessage("grab", myDevice);
}
// new function that, when triggered, will send a message of “grab” to the connected bluetooth microcontroller
public void sendReleaseMessage(){
mySingularityManager.sendMessage("release", myDevice);
}

  • Save this script.
  • In Unity, select the sphere object in the hierarchy
  • In the inspector tab, select Add Component, and search for “XR Grab Interactable”
  • In the “XR Grab Interactable” component, open the “Interactable Events” drop down 
  • Under “Select Entered” add a new event by clicking the + on the lower right.
  • Drag the BluetoothCommunicator object into the “None (object)” space.
  • In the “No Function” menu, select ExampleCommunicator -> SendGrabMessage()
  • Do the same thing for the “Select Exited” event, but this time use the sendReleaseMessage() function.
  • Locate the “Rigid Body” component and uncheck “Use Gravity”, change drag to 1000.
  • Save the Unity project, and build it to your Quest.

Receive the message on the ESP32 and turn on and off LED

In the Arduino IDE, add the following lines of code in the “if(BTSerial.available()>0)” condition (after line 27):

// “data” reads incoming messages
if (data == "grab"){
// if data = “grab”, turn built-in LED on
   digitalWrite(13, HIGH);
   } else if (data == "release"){
// if data = “release”, turn built-in LED off
  digitalWrite(13, LOW);
}
  • Upload this code to the ESP32.
  • Test that you can grab “pinch” the sphere, and the ESP32 LED13 red light turns on, and off when the sphere is released.

Example tutorial by Tyler Beatty

Bodies in X (eXtended realities)