Backyard Brains Logo

Neuroscience for Everyone!

+1 (855) GET-SPIKES (855-438-7745)

items ()

Houston, the Eagle Has Landed: The Hacker Hand Doth Work (After All)!

the hacker hand doth work

— Written by Fredi Mino —

Welcome to the final update on my TinyML Robot Hand project! After collecting sEMG (surface electromyography) data, feeding them into a neural network, and producing a machine learning model that can accurately classify different hand gestures, I can proudly say that my eagle has landed!

Deploying and integrating the model proved to be a lot more challenging than I anticipated. The offline model reached a high accuracy (~ 90%), but as soon as I tried to deal with real data, the classifier performed at worse than chance accuracy. No matter how similar I tried to make my real-time processing pipeline compared to my offline pipeline, it seemed like nothing could fix it.

But just when everything seemed lost, we had a breakthrough at the last minute. I’ll explain what we did further down, but TL;DR, it worked!

In the end, it all came back to a topic I have been discussing throughout the blog updates. Offline models are great for data analysis and provide great insight into the nature of our signals and the features that can be extracted from them. Nevertheless, the good performance of an offline model is not guaranteed to be translated to online models.

In my case, the root of the problem was the difference in magnitude between the data recorded with the Spike RecorderTM, vs the data recorded with the Arduino Nano. Since the waveforms of the data remained approximately the same, you would think that the neural network would focus on them, and ignore the differences in amplitude, but that wasn’t the case.

To solve this issue, I ended up creating a new dataset using the data recorder from the Arduino Nano, and I was finally able to get back to 72% classification accuracy on the testing dataset. This accuracy is not excellent, but at the very least, it helped me succeed in controlling the hacker hand most of the time.

As the fellowship comes to an end, I wanted to take a moment to look back at what we did:

We hypothesized that a 5-channel signal should allow us to discriminate between 5 finger gestures, which is a hard problem because the muscles controlling finger movements are located deep in the forearm, so writing a classical computer program was out of the question.

diagram of all muscles in the forearm
Diagram of all muscles in the forearm

Then, we collected data to test our hypothesis and concluded that there were enough qualitative differences in the data for different gestures, implying that a neural network had a chance at succeeding at this task. We then trained the neural network with data that was processed in Python (an offline model) and succeeded with good classification accuracy.

waveform profiles of each channel for each gesture
Waveform profiles of each channel for each gesture

Next, we tried to recreate this success in the real-time system (read: Arduino/C++), but we failed because the real-time data was different than the offline training data. Finally, we fixed this issue by training the network with data captured in the real-time system.

training dataset
Feature space of the training dataset collected in real-time
the hacker hand doth work - confusion matrix
Confusion matrix of the testing dataset generated by Edge Impulse

In general, it seems like we have succeeded as a proof of concept, and as always, there are certain aspects of this project that I would like to revisit in the future:

  • I need an experiment to determine if the classification accuracy of the real-time system matches the testing accuracy reported by Edge Impulse.
  • I need to explore whether or not all channels contribute useful information across different gestures, and if they don’t, I need to determine how many channels do I actually need to control the system.
  • Finally, as a computational neuroscientist, I would like to explore if there are different neural network architectures better suited for this kind of problem.

Overall, I had fun building working on this project, and I hope you had fun following along too. Although there is some work left to be done, I think this project is ready to help you get started in your journey in Neuroscience and Neurotechnology.  I genuinely believe that this is a great way to get you comfortable with the basics of neural interfaces,  digital signal processing, and all the fun stuff you need to deal with when working with electrophysiological signals and real-time systems. I invite you to reproduce this project, and then go beyond; the applications are only limited by your creativity!

No Comments

No comments yet.

RSS feed for comments on this post.

Sorry, the comment form is closed at this time.