Hey everyone! I’m Pablo, a junior from Nido de Aguilas High School in Santiago, Chile. In my free time, I like to doodle and run.
My project is a multi-channel version of the experiment that my colleague and friend Cristian developed: it consists of using the SpikerShield Pro’s ability to get data from multiple channels to create a musical instrument. In this instrument, flexing a muscle is analogous to playing a key in a keyboard. Obviously, the amount of channels limits this keyboard to six notes, but according to my limited musical knowledge, this is enough to create a coherent melody. In fact, the Arduino program currently has four settings which can be accessed using the red button: Mary Had a Little Lamb, Frere Jacques, major pentatonic scale and the minor blues scale. All the notes are in arrays with six elements, each corresponding to a channel. To add more possibilities, holding the white button in the board makes all the notes in the current setting one octave higher. You can download my code here.
The “loop” part of the code works by reading the red button, white button, and all six channels. First, it decides which set of notes to use for that iteration of the loop, which is controlled by the red button, then it checks if the white button has been clicked, which affects the pitch of the final note it plays. The last step is to decide which tone to actually play, which the code does by selecting the largest reading of all the muscles. Now, you might be thinking that playing music with two vastly different muscles, say your forehead and your forearm, will never work because a signal from the forearm will always be bigger than the signal of even the strongest forehead flex. However, the SpikerShield Pro can control the gain from each individual channel (the little white knobs) which can make a channel more or less sensitive to a signal, so every muscle has a fair chance of being played.
One challenge I faced when I developed this project is the lack of documentation of this particular product for novice programmers. Most of the times I’ve played around with an Arduino, I’ve relied extensively on the built-in tutorials and online resources, but this time I only had the board’s schematic, which at first glance bears a closer resemblance to black spaghetti than a discernible circuit and the default program which sends the signals from the board to Spike Recorder. Running the aforementioned program was not a challenge, but reading the code, not being fully aware of what it was, proved to be confusing. I only started making progress once Tim Marzullo showed me an outdated sketch meant for this shield. However, with this project in the open, I doubt this is a problem other users will face; the heart of the code — presenting the sensor’s readings as an array and mapping those raw values to a usable scale — can be used for most projects.
The second biggest challenge was and still is, my absolute ignorance about music theory. I never learned to play an instrument, and the most complicated song I managed to play is “Hot Crossed Buns”, though that is probably a skill I’ve lost. I’ve always enjoyed music, but much like hot dogs, I preferred to enjoy the finished product rather than learning how it is made. After adding the melody of Mary Had a Little Lamb and Frere Jacques, I did not know what other songs to add. After a fair amount of research, I came upon pentatonic scales, which are comprised of five notes.
Though the musical aspect is worth examining, what attracted me more is its role in many musical traditions, ranging from the ancient Greeks to the Andes. During the 19th century, composers like Debussy used the simplicity of the scale to create a folksy in their composition, resulting in music like La fille aux cheveux de lin. Later on, rock, blues, and jazz artists adopted the scale as a tool for their respective styles of improvisation. I think this is the area where my particular instrument shows the most potential because it is only capable of playing one note at a time, and also because flexing muscles to create sound is very intuitive. However, this is a hypothesis I will let the reader confirm.
“A new generation of students has access to 3D printers and other DIY technology…”
I was speaking recently to one of our colleagues at Temple University about a project several of his students were working on, and he said something that really struck me, “these students are the first generation to have grown up with 3D printers in their schools, some even in their basements. They know how to use these and other maker tools, and it’s changing education,”
He’s right – every year now, we see more and more schools with Maker Spaces, 3D printers, DIY Electronics, and time set aside for creative, scientific projects. These students are already outpacing the old standards and this phenomenon certainly heralds a bright future!
The project we were discussing was a multi-channel neuroprosthetic that his students were developing with 3D modeling software, a 3D printer, and Backyard Brains tools! Check it out below:
A new trend? Students develop Neuroprosthetics, start Student Organization
This past summer, rising sophomore Morgan R., of Temple University pursued a summer project: with the help of one of her professors, she began the process of developing an affordable, 3D printed Neuroprosthetic powered by the Backyard Brains SpikerShield. What started as a fun summer project grew when she started bringing her friends and classmates on board. Thanks to their interdisciplinary connections, they realized they had the opportunity to make something out of the project and started a Neuroprosthetics Organization at the university, with the aim to develop and donate affordable prosthetics to those of different ability who could benefit from assistive technology.
We reached out directly to Morgan, and her classmate Gabby to learn more about the project, and after some Q&A they provided us with a lot of great details about their project! Below are their words and photos as they describe the process involved, from idea, to prototyping, to student org!
Morgan and Gabby: Our team launched this project by examining the question: How can we make prosthetics more accessible to the general public? After doing research on the industry and current methods, we concluded that they were so expensive because many of the companies who make them are mostly research and prototype oriented and not thinking about accessibility to end users who could use them today.
For our design, we concluded that a myoelectric (or EMG) prosthetic hand could be developed to be both affordable and versatile. We constructed the hand through an engineering program called AutoCAD. This software allows the user to create three-dimensional models that can be printed using a 3D Printer. We drafted two separate original designs for the hand and deliberated the pros and cons of both versions.
We decided upon our preferred design, then we printed and assembled our first hand.
In the end, we decided to print the fingers in three separate pieces, the palm in two separate pieces, and a forearm structure. There were channels running through the palm into the fingers to allow strings of fishing line to run up the forearm structure and loop through at the tips of the finger digits. Each finger needed three strings, so there were about 15 strings total, so after the loop at the top there were about thirty strings running through the tunnel opening.
We came up with a strategy to organize the assortment of strings running through the bottom by two different color coding systems (one upon which finger it was and the other was whether the string received constant tension or an occasional burst. From there, we attached the strings from the opening of the tunnel to servos installed in the forearm attachment. The servos were then connected to an Arduino breadboard and the muscle backyard brains SpikerShield pro. We then used their code to experiment with the mechanism.
We are excited to continue moving forward with this project this project and to continue research in this field. During the next phase of the project, we would like to use a more precise 3D printer, to reduce the amount of variation from the 3D model to the print. Furthermore, we would like to find a better material to use instead of fishing line, as the degree of motion is not where we would like it to be. The fishing line could only handle so much tension, furthermore, the natural tendons found in the body have more elasticity than the fishing wire. We need to be able to apply a similar degree of tension to the line to create natural movements without the assistance of an external wire.
We also face issues on the software side. It is challenging to identify individual finger movements through EMG signals. We came up with a few short-term solutions for this: for example, with the single channel SpikerShield, we set finger servomotor activation at different thresholds, so depending on how strong the user’s flex was would change what fingers were activated. The 6-channel SpikerShield Pro has a lot of opportunities to offer individual finger control, but it is challenging to differentiate so many different signals from the forearm.
We are proud of what we’ve accomplished in such a short amount of time and see this as a strong foundation for our future work. We don’t doubt that we will be able to overcome these obstacles. We hope to have the prosthetic working well enough in the next year to give it to a living patient.
From an academic perspective, this project allowed us to put the skills that we gained through our engineering classes to practical use. Furthermore, it allowed us to experience working on a team with individuals who were not all pursuing the same field of study. Teamwork and reliability were key to the success of the project, it took the knowledge and skill of each discipline of the team to succeed.
This team’s work is a great example of a growing trend: there is a new generation of students who have grown up with access to 3D printers, Arduinos, and other DIY tools. They are also one of the first student groups to begin with the single channel Muscle SpikerShield Bundle and then upgrade their design to allow for multi-channel control by implementing the Muscle SpikerShield Pro!
As you can see above, the students’ development involves a lot of trial and error as they work on functionally mimicking the movement of their prosthetics’ fingers. They are making great progress, and we are excited to share updates from them as their work continues over the school year!
Not just University Students…
It’s not just university students developing Neuroprosthetics and assistive neuro technologies! Here are a few examples of MS and HS students who have developed their own devices using Backyard Brains kits.
This prosthetic grabber was made with a simple servo motor and is strong enough to grip and lift a can of sparkling mineral water! Now nothing will stop anyone from enjoying their bubbles.
This is a great example of a functional prosthetic model, or Biomimicry – by combining our kits with Lego Mindstorms, the students created a doll that would mimic another students kick by recording from their leg.
This student combined his VEX robotics kit with a Muscle SpikerShield to create his NeuroClaw!
Planning an 8th Grade DIY Neuroprosthetics Lab
Ms. Farkas has big plans for her 8th graders this year: continuing their experience with DIY neuroscience from last year, she is branching into the world of prosthetics! Following a successful Donors Choose, she is now planning a unit where groups of students will all be responsible to design and create devices which will be controlled by their nervous systems!
She describes it best:
This year, we want to continue my students’ Neuroscience journey! With the help of the Backyard Brains Muscle SpikerShield Kits, we plan to conceptualize, research, design, build and control our own Neuroprosthetics. Through collaboration with the team at Backyard Brains, we are piloting a project aimed at middle school students!
We’re excited to update you on the results of her class projects!
One of the core tenets of Backyard Brains is our slogan, neuroscience for everyone! We constantly work to drive the world around us into the neurorevolution, and when we hear about projects like Peter Buczkowski’s master’s thesis, we know we’re doing something right.
Peter Buczkowski graduated in 2013 with a Bachelor of Arts and in 2017 with a Master of Arts in the Digital Media from the University of the Arts in Bremen. His idea for his masters thesis was born out of a TENS unit, after seeing our Human-Human Interface TED Talk. “I especially liked the receiving part of possessed hand experiment and the idea to use the human as an interface. This inspired me to do my own experiments in that field,” Peter told us. “I chose three topics and build three projects to cover a wide spectrum so one can see the possibilities of this technology in different areas.”
Peter started out with the most basic of scientific endeavors: solving a problem. Doing any sort of human neuroscience or biofeedback research is made a little more difficult the fact that most types of patch electrodes are sticky and a hassle to use, not to mention not very aesthetically pleasing. So he set out to fix that, and now, his projects center around the idea of “stationary” electrodes: not necessarily something attached to the body, but something that a person can just hop onto and start learning. His three projects deal with photography, video game skill, and muscle memory, using the paradigms apparent in our Human-Human Interface experiments to create his designs.
The Prosthetic Photographer
His first project is called the Prosthetic Photographer. “The Prosthetic Photographer is a modular camera attachment that forces you with electric impulses to take beautiful pictures,” Peter wrote. Typical advice for a budding photographer is just to go out and take thousands of photos, and you will learn the difference between and okay shot and a beautiful one. The Prosthetic Photographer aims to shorten that process through machine learning. Using machine learning to distinguish between high and low quality photos and neural networking to connect the computer, the camera, and the user, the ProstheticPhotographer is an example of machine learning and human learning coming together.
The device is a modular one that can be added to any compatible camera, utilizing a TENS unit to render the user as a conduit for its learning, controlling the photographs being taken and teaching its concept of aesthetics to the user. Electrodes on the camera’s handle transmits a shock signal to the user causing an involuntary press of a button, and a subsequent shutter click. A camera with its own eye for beauty! Photography will never be the same.
Building upon the machine learning aspect of his work, Peter moved on to his second project, utilizing Twitch to condition people to play video games perfectly. Twitch.tv is an online streaming platform that lets gamers both showcase their play and observe others in order to beat a particularly hard section of a game. This unconventional style of video game play gave Peter an idea: what if a computer were to tell you what your next move is?
In a version of the classic computer game “Snake,” a computer calculates whether the next move should be left, right, up, or down, dividing the buttons between two electrode arrays (one for each hand). The computer then transmits its decision to the corresponding button and stimulates the finger to press that button, and the snake moves in the decided-upon direction. Sure, it takes the human guesswork out of the game, but without a human, it would not be possible!
Finally, Peter built the Medium Machine, the most speculative of his projects. According to his website, “The Medium Machine enables [a computer] to transfer data and information in the form of muscle contractions into the unconscious mind of a human.” The inspiration for this project arose from a short story called “Johnny Mnemonic” by William Gibson, in which a man’s brain is turned into a sort of repository for information that he transports from client to recipient. With the Medium Machine, Peter hoped to effect a similar repository–albeit without removing the user’s memory to make room for it. Again, the muscles are connected to stimulation, this time encoded by the computer in a certain pattern or message. The contractions force the finger to push a button in a cadence that could mean anything–until it is decoded by the right person.
“The human becomes a medium and a messenger between systems,” Peter wrote. Just like in the story! The possibilities for discovery and the applications of the science are endless.
We are very intrigued to follow along with Peter as he pursues these projects and starts more. He is currently working with other innovators to create business plans for their projects. Learn more about Peter’s work on his website, http://peterbuczkowski.com
Do you have an application of our products, or a story to share about your own work? Send us a message at email@example.com!