The best way to learn to teleoperate a remote robot is through Virtual Reality
Many regard applications for Motor BCI as a zero-sum game: Either robotized devices let Users interact with the physical world, however slowly and imprecisely, or VR makes them move, feel, and appear to sip water like any ordinary person but without satisfying their thirst.
In reality, these applications reinforce each other and should be regarded as two sides of the same coin. Take teleoperation for instance:
To operate robots, the Users need to train. The most realistic, safe, and cost-effective training is via VR simulation. Furthermore, when the inputs and outputs of these robots mimic humanoid functions, embodiment via responsive HMD can be even more advantageous.
Meet Reachy, the pinnacle of teleoperation by Pollen Robotics:
Video 15; Pollen Robotics, 2021: Teleoperating the Reachy robot with VR in everyday environments and performing daily tasks.
In November 2022, the 12 robot finalists competed for a whopping $10M, solving many advanced mobility, haptics, manipulation, and interaction tasks at the global ANA Avatar XPRIZE.
These were not just ordinary robots but robotic avatars teleoperated by a human in real-time. Using these systems, anyone can be physically present in a remote location, bringing their desires and senses with them.
Reachy snatched the $2M second-place prize, and that is not the only achievement setting it apart. Unlike its competition-optimized peers, Reachy was designed for general everyday use and is already sold across the globe for approximately €22–40k (Ackerman, 2023; The Connexion, 2023). Lucky for us, much of what makes Reachy up is open-source.
🤖Coupled with VR and Motor BCIs, mission-critical robotic avatar systems like Reachy can grant Users the freedom to once again interact with the physical world.
Other Electronic Applications
Beyond robotic avatars, Users may combine their Motor BCI with electronic orthoses, exoskeletons, or stimulators to help them perform their intended movement using their own body. Examples per each category include:
(To play, see footnote link) Video 16; Serruya et al., 2022: Video of the participant using MyoPro, an FDA-cleared electronic arm orthosis, to grab bottles with their own hand in a proof of concept Motor BCI study by Serruya et al., (2022) — Plain language summary: “In our trial, a sensor was implanted into the surface of the brain, near the site of the stroke, and was connected to a computer that generated a command to open and close the hand with a motorized brace worn on the hand. This person was able to use their own brain activity to trigger the brace and pick up and move objects.”
Video 17; Voice of America, 2020: Video of the participant using an exoskeleton developed by Clinatec to move their whole body in a proof of concept Motor BCI study by Benabid et al. (2019) — Interpretation: “These results showed long-term (24-month) activation of a four-limb neuroprosthetic exoskeleton by a complete brain–machine interface system using continuous, online epidural ECoG to decode brain activity in a tetraplegic patient. Up to eight degrees of freedom could be simultaneously controlled using a unique model, which was reusable without recalibration for up to about 7 weeks.”
Video 18; Global News, 2023: Video of the participant using Medtronic’s epidural electrical stimulator to walk again using their own legs in a proof of concept Motor BCI study by Lorach et al. (2023) — Associated clinical briefings: “Spinal-cord injury interrupts communication between the brain and spinal cord, leading to paralysis. An implant that decodes the brain signals that control movements and drives electrical stimulation of the spinal cord re-establishes this communication, enabling an individual with spinal-cord injury to walk naturally.”
In addition to restoring walking with epidural electrical stimulation, the same group also restored a participant’s hand and finger function in a first-in-human study (ONWARD, 2023).
Beyond the depicted, arguably less mature ‘Neural Bypass’ applications also include other forms of spinal cord stimulation, applications based on FES — activating muscles directly — and applications of peripheral nerve stimulation (see Fig. 32, for a review).
Figure 32; Losanno et al., 2023: Neurotechnologies to restore hand functions — Hand functions can be restored by electrically stimulating different regions of the neuromuscular system using different interfaces. a, Dexterity versus potential for deployment and comfort of use for different strategies to restore hand functions. b, Functional electrical stimulation (FES) is performed using transcutaneous or implanted (epymisial or intramuscular) electrodes targeting the extrinsic and intrinsic hand muscles. c, Peripheral nerve stimulation (PNS) is applied through epineural electrodes, such as the cuff electrode and the flat interface nerve electrode (FINE), or intrafascicular electrodes, such as the transverse intrafascicular multichannel electrode (TIME) and the Utah slanted electrode array (USEA)149, targeting the median, radial and ulnar nerves above their bifurcations. d, Spinal cord stimulation (SCS) is implemented using transcutaneous, epidural or intraspinal leads targeting the C5–T1 spinal nerves.” Reprinted from Figure 4.
Unfortunately, aside from aiding rehabilitation, neither of these Motor BCI applications is ripe for the clinical market of daily living and instead should be regarded as experiments validated in the lab (TRL 4).
For illustration, even Ipsihand — the non-invasive rehabilitative device alternative to Serruya et al. (2022) depicted in the Orhtoses section — struggles to receive HCPCS II reimbursement coverage from CMS (CMS, 2023) despite receiving FDA premarket approval in 2021.
Robotic Arms
The final category of robotics is electronic prostheses–specifically, robotic arms mounted on a wheelchair. Despite nearly a decade since Barack Obama shook Nathan Copeland’s Motor BCI-controlled hand (Photo 3) and numerous academic demonstrations, many commercial ventures do not appear dedicated to integrating robotic arms with Motor BCIs.
Photo 3; NBC News, 2016: US President Obama shakes hands with Nathan Copeland.
Perhaps the closest relative, Phantom Neuro, is a company developing a minimally invasive muscle-machine interface that can communicate with devices over wifi (Phantom Neuro, 2023a). Considering that one of their latest investors and R&D partners is none other but the leader of the Motor BCI pack, Blackrock Neurotech (Blackrock Neurotech, 2022) and their advisory board includes the founder and CEO of Synchron (Phantom Neuro, 2023b) who are a Motor BCI manufacturer furthest ahead with their pivotal clinical trials (Fultinavičiūtė, 2023), Phantom Neuro will likely lead the application of robotics to Motor BCI.
However, Phantom’s go-to-market strategy is aimed at the amputee market, soaking up much of their R&D spend (Brodwin, 2022 (validated by Phantom Neuro)). Starting with the arm and progressively evolving to other body parts may sound reasonable for amputees. However, Users with severe paralysis would need to wait years for other or whole-body robotics applications despite already having access to these movement intentions under the same Motor BCI (see Integrated Decoding section for more).
Phantom Neuro appears to be an exception to the otherwise academically dominated frontier in implantable prostheses ([Fernandez et al., 2023](Improving Model-Based Neural Control of Upper Extremity Prostheses with Machine Learning); Ortiz-Catalan et al., 2023).
The Motor BCI research community particularly favors the Luke/DEKA Arm and the Modular Prosthetic Limb, both used extensively across groups (Hochberg et al., 2006; Collinger et al., 2012; Hochberg et al., 2012; Resnik et al., 2012; Greenspoon et al., 2023; Herring et al., 2023). After all, these two state-of-the-art hands gobbled up nearly $50M in Defense Advanced Research Projects Agency (”DARPA”) funding to push the frontier (Pope, n.a). DARPA, not to be glanced over, has funded BCI research and development more than any other organization, significantly steering the field’s commercialization agenda (Fig. 33).
Figure 33; Pooja Rao, 2020: Darpa Funding for BCI over the decades
Figure 34; Pooja Rao, 2020: Involvement of DARPA in BCI companies.
Synchron, Kernel, Neuralink, Paradromics, Deepmind (MuJoCo), along with the previously mentioned prosthetic efforts and half of all invasive neural interfaces in the US owe some of their Motor BCI accomplishments to DARPA’s funding (Fig. 34).
Other biomorphic robotic arms with myoelectric controls, such as i-Limb, Ability Hand, Adam Hand, and Utah Arm, among others (for review, see Chapter 4, Jette et al., 2017), are already penetrating the amputee and humanoid robotics market and, if repurposed, are further candidates for clinical Motor BCI applications (BionIT Labs, 2020). Similarly, robotic orthoses and exoskeletons are improving patient outcomes despite their lack of Motor BCI integration. For instance, Wandercraft recently revealed the Atalante X as the first self-balancing, walking exoskeleton one can control by a handheld remote. While the company’s previous exoskeleton was used in rehabilitation clinics since 2019 and received FDA approval in 2023 (FDA News, 2023), the new model (available for pre-order) is intended for everyday home use.
Coming Back Full Circle
Each of these prosthetic arms, exoskeletons, and other robotic applications takes time to master (Espinosa et al., 2019), requiring much training before the User can accept the device in their everyday life. If they are unsatisfied with the functionality and struggle to use it, they will quickly reject it in favor of a simpler solution (Yamamoto et al., 2019).
VR offers the User a safe, reliable, untiring, and measurable way to gain expertise and confidence using their “new body” while the robotic applications mature. It is imperative, however, to design the VR system to accurately represent the biomechanics of movement so that a) the progression to control robotic systems is seamless and b) Users remain immersed in the virtual reflection of reality.
Part 16 of a series of unedited excerpts from uCat: Transcend the Limits of Body, Time, and Space by Sam Hosovsky*, Oliver Shetler, Luke Turner, and Cai Kinnaird. First published on Feb 29th, 2024, and licensed under CC BY-NC-SA 4.0.
uCat is a community of entrepreneurs, transhumanists, techno-optimists, and many others who recognize the alignment of the technological frontiers described in this work. Join us!
*Sam was the primary author of this excerpt.
Comments