Researchers at Northwestern University have created the first intelligent wearable technology that continuously monitors how much people use their voices, alerting them to misuse before vocal fatigue or injury occurs.
The first-of-its-kind, battery-powered, wireless device, and accompanying algorithms might be a game-changer for professional singers, teachers, lawmakers, call-center workers, coaches, and anybody else who relies on their voices for successful communication and an income. It could also let clinicians monitor patients with vocal issues remotely and continuously during their treatment.
The research behind the new technology, which was developed by an interdisciplinary team of materials scientists, biomedical engineers, opera singers, and a speech-language pathologist, will be published in the Proceedings of the National Academy of Sciences during the week of February 20.
The soft, flexible, postage-stamp-sized device clings snugly to the upper chest in order to detect the small vibrations caused by speaking and singing. From there, the gathered data is instantly sent over Bluetooth to the user’s smartphone or tablet, allowing them to monitor their vocal activity throughout the day in real time and measure their cumulative total vocal usage. Proprietary machine-learning algorithms differentiate between speech and singing, allowing singers to track each activity individually.
The app allows users to select their individual vocal thresholds. When they approach this threshold, their smartphone, smartwatch, or an additional wrist-mounted gadget alerts them with real-time haptic stimulation. Then, they can rest their voices prior to overusing them.
“The device perfectly monitors the amplitude and frequency of speaking and singing,” said bioelectronics pioneer John A. Rogers of Northwestern University, who led the development of the device. “These two variables are the most significant in determining the overall stress on the vocal folds. Awareness of these characteristics, both in the present moment and over time, is vital for managing healthy vocalization patterns.”
“It’s easy for individuals to forget how much they use their voice,” said Theresa Brancaccio, a voice expert at Northwestern University and co-author of the study. “Because they have lived and learned, classical singers with experience tend to be more cognizant of their vocal usage. However, some people, especially those with less training or who must talk frequently for their work (e.g., teachers, politicians, and sports coaches), frequently do not understand how far they are pushing themselves. We want to increase their awareness to aid in accident prevention.”
Rogers is the Louis Simpson and Kimberly Querrey Professor of Materials Science and Engineering, Biomedical Engineering, and Neurological Surgery at Northwestern University’s Feinberg School of Medicine and McCormick School of Engineering. Moreover, he serves as the director of the Querrey Simpson Institute for Bioelectronics. Mezzo-soprano Brancaccio, a renowned operatic performer, is a senior lecturer at Northwestern’s Bienen School of Music, where she teaches voice and vocal pedagogy.
For the millions of Americans whose livelihoods depend on speaking or singing, vocal fatigue is a continual, looming danger. The frequent condition arises when the vocal folds enlarge from overuse, resulting in a harsh voice and diminished endurance. Vocal fatigue has a negative impact on vocalists, reducing their ability to sing clearly and hit the same notes as a healthy voice. A singer’s plans can be quickly interrupted at most by a single brief bout of vocal exhaustion. In the worst case, it can cause sufficient damage to derail a career.
Lack of awareness is the root cause of the issue. Individuals rarely make the association between vocal activities and how they influence their voices. Although one in thirteen U.S. adults has experienced vocal fatigue, the majority of individuals do not realize they are overusing their voices until hoarseness develops.
“What gets people into trouble is a series of unfortunate circumstances,” said Brancaccio. “They may have rehearsals, teach classes, and participate in class debates before attending a noisy party where they must shout above the background noise. Then, add a sickness or cold to the mix. Many are unaware of how frequently they cough or clear their throats. When these occurrences accumulate over a period of days, this can cause significant strain on the voice.
As an advocate for vocal health, Brancaccio has spent decades investigating strategies to keep her pupils cognizant of how frequently they use their voices. In 2009, she encouraged her kids to keep a paper budget, recording, among other things, every time they spoke, sang, and drank water. A decade later, she transformed the concept into Singer Savvy, an app that provides each user with a tailored vocal budget and assists them in staying within that budget.
Separately, Rogers, in partnership with Shirley Ryan AbilityLab researchers, had created a wireless wearable gadget to monitor eating and speaking in stroke patients. The sensor resembling a bandage assesses swallowing abilities and speech patterns to track the recovery of stroke victims. In the first few weeks of the COVID-19 pandemic, Rogers’s team developed technology to monitor coughing, a major indication of the disease.
“I wanted to collect more information so that our tracking system would be more precise and accurate,” Brancaccio explained. “So, I contacted John to determine if his sensors could assist us in gathering more information.” Rogers stated, “I believed it was a tremendous chance for us to extend our technology beyond their extremely important but narrowly targeted applications in health care to something that may attract a larger community of users.” “Anybody who frequently uses their voice could profit.”
Also, they collaborated with speech pathologist and voice expert Aaron M. Johnson to investigate how the devices could be used to assess and track treatment for patients with vocal abnormalities. Johnson, who co-directs the Voice Center at NYU Langone, stated that the compact, wireless gadget might help track patients’ voices outside of clinical settings.
Johnson, co-author of the study and associate professor in the department of otolaryngology at NYU Grossman School of Medicine, said, “Changing how and how often patients use their voice is a vital component of voice treatment.” “This technology will allow patients and their therapists to learn voice use patterns and alter vocal demand in order to alleviate vocal fatigue and accelerate recovery from voice disorders.” The process of transferring vocal skills and exercises from therapy sessions to everyday life is one of the most difficult components of voice therapy, and this gadget could significantly facilitate that process.”
To precisely assess vocal load over time, the researchers upgraded Rogers’ existing instruments. This covers the frequency, the volume, the amplitude, the duration, and the time of day. Like with Rogers’s earlier devices for COVID-19 and stroke victims, the new device detects vibrations rather than recording sound. This enables the device to precisely detect the user’s voice action, rather than the ambient noise around them.
The greatest difficulty was developing algorithms that can discriminate between speech and singing. To address this obstacle, Brancaccio enlisted voice and opera students to train machine-learning algorithms through a range of singing activities. A group of classical singers with diverse vocal ranges, from bass to soprano, wore the devices while humming, singing staccato scales and songs, reading, and other activities. Each singer produced 2,500 one-second-long singing windows and 2,500 one-second-long speaking windows.
The resulting algorithm can distinguish singing from speech with an accuracy of greater than 95%. And when utilized in a choral situation, the device records only the wearer’s data and not surrounding noise.
“Prolonged conversation is one of the most exhausting activities for singers-in-training,” remarked Brancaccio. “By separating singing and speaking, it can help individuals become more conscious of how much they speak. There is evidence that occasional 15- to 20-minute periods of complete stillness scattered throughout the day can aid in the recovery and repair of vocal fold tissues.”
The user merely sticks the device to the sternum, below the neck, and synchronizes it with the accompanying software. Rogers’s team is now developing a mechanism to tailor each user’s vocal budget. Users will hit a button within the app whenever they suffer vocal discomfort at any moment throughout the day, effectively capturing the instantaneous and cumulative vocal load at the time. These data can serve as a personalized vocal fatigue threshold. A haptic gadget will vibrate as an alarm when the user approaches or surpasses their customized threshold.
This haptic device is comparable in size and shape to a wristwatch and features numerous motors that can activate in various patterns and intensities to transmit different messages. Inside the application, users can also view a graphical display that separates information into speaking and singing categories.
“It uses Bluetooth, so it can communicate with any gadget that has an inbuilt haptic motor,” explained Rogers. “Thus, you need not utilize our bracelet. You could use a normal smartwatch to provide haptic feedback.” Other vocal monitoring systems exist, but they require large collars, tethering lines, and cumbersome apparatus. Some also employ implanted microphones to record audible verbal data, which raises privacy concerns.
“They do not function for continuous monitoring in a real-world setting,” stated Brancaccio. “Instead of wearing bulky, connected equipment, I can adhere this soft, wearable device to my body.” Once it’s turned on, I barely notice it. It’s extremely simple and light.” Since Rogers’ earlier devices measured body temperature, heart rate, and respiratory activity, the researchers incorporated these features into the vocal monitoring device.
They anticipate that these additional data will aid in investigating fundamental scientific problems regarding vocal fatigue. Rogers stated, “This is more hypothetical, but it could be intriguing to see how physical exercise impacts vocal tiredness.” Is singing while dancing more taxing on the vocal folds than singing when not physically exerting oneself? These are the types of questions we can pose and answer quantitatively.”
In the meanwhile, Brancaccio is thrilled that her pupils will have access to a tool that can aid in harm prevention. She hopes that others, particularly non-singers, would recognize the importance of maintaining healthy voice chords.
“Your voice is a part of your identity, regardless of whether or not you are a singer,” she said. It is essential to daily living and should be protected. The Querrey Simpson Institute for Bioelectronics at Northwestern University financed the project titled “Closed-loop network of skin-interfaced wireless devices for assessing vocal fatigue and delivering user feedback.”
