Computers Math

During even the most routine visits, physicians listen to sounds inside their patients’ bodies — air moving in and out of the lungs, heart beats and even digested food progressing through the long gastrointestinal tract. These sounds provide valuable information about a person’s health. And when these sounds subtly change or downright stop, it can signal a serious problem that warrants time-sensitive intervention.
Now, Northwestern University researchers are introducing new soft, miniaturized wearable devices that go well beyond episodic measurements obtained during occasional doctor exams. Softly adhered to the skin, the devices continuously track these subtle sounds simultaneously and wirelessly at multiple locations across nearly any region of the body.
The new study will be published on Thursday (Nov. 16) in the journal Nature Medicine.
In pilot studies, researchers tested the devices on 15 premature babies with respiratory and intestinal motility disorders and 55 adults, including 20 with chronic lung diseases. Not only did the devices perform with clinical-grade accuracy, they also offered new functionalities that have not been developed nor introduced into research or clinical care.
“Currently, there are no existing methods for continuously monitoring and spatially mapping body sounds at home or in hospital settings,” said Northwestern’s John A. Rogers, a bioelectronics pioneer who led the device development. “Physicians have to put a conventional, or a digital, stethoscope on different parts of the chest and back to listen to the lungs in a point-by-point fashion. In close collaborations with our clinical teams, we set out to develop a new strategy for monitoring patients in real-time on a continuous basis and without encumbrances associated with rigid, wired, bulky technology.”
“The idea behind these devices is to provide highly accurate, continuous evaluation of patient health and then make clinical decisions in the clinics or when patients are admitted to the hospital or attached to ventilators,”said Dr. Ankit Bharat, a thoracic surgeon at Northwestern Medicine, who led the clinical research in the adult subjects. “A key advantage of this device is to be able to simultaneously listen and compare different regions of the lungs. Simply put, it’s like up to 13 highly trained doctors listening to different regions of the lungs simultaneously with their stethoscopes, and their minds are synced to create a continuous and a dynamic assessment of the lung health that is translated into a movie on a real-life computer screen.”
Rogers is the Louis Simpson and Kimberly Querrey Professor of Materials Science and Engineering, Biomedical Engineering and Neurological Surgery at Northwestern’s McCormick School of Engineering and Northwestern University Feinberg School of Medicine. He also directs the Querrey Simpson Institute for Bioelectronics. Bharat is the chief of thoracic surgery and the Harold L. and Margaret N. Method Professor of Surgery at Feinberg. As the director of the Northwestern Medicine Canning Thoracic Institute, Bharat performed the first double-lung transplants on COVID-19 patients in the U.S. and started a first-of-its-kind lung transplant program for certain patients with stage 4 lung cancers.

Comprehensive, non-invasive sensing network
Containing pairs of high-performance, digital microphones and accelerometers, the small, lightweight devices gently adhere to the skin to create a comprehensive non-invasive sensing network. By simultaneously capturing sounds and correlating those sounds to body processes, the devices spatially map how air flows into, through and out of the lungs as well as how cardiac rhythm changes in varied resting and active states, and how food, gas and fluids move through the intestines.
Encapsulated in soft silicone, each device measures 40 millimeters long, 20 millimeters wide and 8 millimeters thick. Within that small footprint, the device contains a flash memory drive, tiny battery, electronic components, Bluetooth capabilities and two tiny microphones — one facing inward toward the body and another facing outward toward the exterior. By capturing sounds in both directions, an algorithm can separate external (ambient or neighboring organ) sounds and internal body sounds.
“Lungs don’t produce enough sound for a normal person to hear,” Bharat said. “They just aren’t loud enough, and hospitals can be noisy places. When there are people talking nearby or machines beeping, it can be incredibly difficult. An important aspect of our technology is that it can correct for those ambient sounds.”
Not only does capturing ambient noise enable noise canceling, it also provides contextual information about the patients’ surrounding environments, which is particularly important when treating premature babies.
“Irrespective of device location, the continuous recording of the sound environment provides objective data on the noise levels to which babies are exposed,” said Dr. Wissam Shalish, a neonatologist at the Montreal Children’s Hospital and co-first author of the paper. “It also offers immediate opportunities to address any sources of stressful or potentially compromising auditory stimuli.”
Non-obtrusively monitoring babies’ breathing

When developing the new devices, the researchers had two vulnerable communities in mind: premature babies in the neonatal intensive care unit (NICU) and post-surgery adults. In the third trimester during pregnancy, babies’ respiratory systems mature so babies can breathe outside the womb. Babies born either before or in the earliest stages of the third trimester, therefore, are more likely to develop lung issues and disordered breathing complications.
Particularly common in premature babies, apneas are a leading cause of prolonged hospitalization and potentially death. When apneas occur, infants either do not take a breath (due to immature breathing centers in the brain) or have an obstruction in their airway that restricts airflow. Some babies might even have a combination of the two. Yet, there are no current methods to continuously monitor airflow at the bedside and to accurately distinguish apnea subtypes, especially in these most vulnerable infants in the clinical NICU.
“Many of these babies are smaller than a stethoscope, so they are already technically challenging to monitor,” said Dr. Debra E. Weese-Mayer, a study co-author, chief of autonomic medicine at Ann & Robert H. Lurie Children’s Hospital of Chicago and the Beatrice Cummings Mayer Professor of Autonomic Medicine at Feinberg. “The beauty of these new acoustic devices is they can non-invasively monitor a baby continuously — during wakefulness and sleep — without disturbing them. These acoustic wearables provide the opportunity to safely and non-obtrusively determine each infant’s ‘signature’ pertinent to their air movement (in and out of airway and lungs), heart sounds and intestinal motility day and night, with attention to circadian rhythmicity. And these wearables simultaneously monitor ambient noise that might affect the internal acoustic ‘signature’ and/or introduce other stimuli that might affect healthy growth and development.”
In collaborative studies conducted at the Montreal Children’s Hospital in Canada, health care workers placed the acoustic devices on babies just below the suprasternal notch at the base of the throat. Devices successfully detected the presence of airflow and chest movements and could estimate the degree of airflow obstruction with high reliability, therefore allowing identification and classification of all apnea subtypes.
“When placed on the suprasternal notch, the enhanced ability to detect and classify apneas could lead to more targeted and personalized care, improved outcomes and reduced length of hospitalization and costs,” Shalish said. “When placed on the right and left chest of critically ill babies, the real-time feedback transmitted whenever the air entry is diminished on one side relative to the other could promptly alert clinicians of a possible pathology necessitating immediate intervention.”
Tracking infant digestion
In children and infants, cardiorespiratory and gastrointestinal problems are major causes of death during the first five years of life. Gastrointestinal issues, in particular, are accompanied by reduced bowels sounds, which could be used as an early warning sign of digestion issues, intestinal dysmotility and potential obstructions. So, as part of the pilot study in the NICU, the researchers used the devices to monitor these sounds.
In the study, premature babies wore sensors at four locations across their abdomen. Early results aligned with measurements of adult intestinal motility using wire-based systems, which is the current standard of care.
“When placed on the abdomen, the automatic detection of reduced bowel sounds could alert the clinician of an impending (sometimes life-threatening) gastrointestinal complication,” Shalish said. “While improved bowel sounds could indicate signs of bowel recovery, especially after a gastrointestinal surgery.”
“Intestinal motility has its own acoustic patterns and tonal qualities,” Weese-Mayer said. “Once an individual patient’s acoustic ‘signature’ is characterized, deviations from that personalized signature have potential to alert the individual and health care team to impending ill health, while there is still time for intervention to restore health.”
In addition to offering continuous monitoring, the devices also untethered NICU babies from the variety of sensors, wires and cables connected to bedside monitors.
Mapping a single breath
Accompanying the NICU study, researchers tested the devices on adult patients, which included 35 adults with chronic lung diseases and 20 healthy controls. In all subjects, the devices captured the distribution of lung sounds and body motions at various locations simultaneously, enabling researchers to analyze a single breath across a range of regions throughout the lungs.
“As physicians, we often don’t understand how a specific region of the lungs is functioning,” Bharat said. “With these wireless sensors, we can capture different regions of the lungs and assess their specific performance and each region’s performance relative to one another.”
In 2020, cardiovascular and respiratory diseases claimed nearly 800,000 lives in the U.S., making them the first and third leading causes of death in adults, according to the Centers for Disease Control and Prevention. With the goal of helping guide clinical decisions and improve outcomes, the researchers hope their new devices can slash these numbers to save lives.
“Lungs can make all sorts of sounds, including crackling, wheezing, rippling and howling,” Bharat said. “It’s a fascinating microenvironment. By continuously monitoring these sounds in real time, we can determine if lung health is getting better or worse and evaluate how well a patient is responding to a particular medication or treatment. Then we can personalize treatments to individual patients.”
The study, “Wireless broadband acousto-mechanical sensors as body area networks for continuous physiological monitoring,” was supported by the Querrey-Simpson Institute for Bioelectronics at Northwestern University. The paper’s co-first authors are Jae-Young Yoo of Northwestern, Seyong Oh of Hanyang University in Korea and Wissam Shalish of the McGill University Health Centre. More

Investigators from the UCLA Health Jonsson Comprehensive Cancer Center have developed an artificial intelligence (AI) model based on epigenetic factors that is able to predict patient outcomes successfully across multiple cancer types.
The researchers found that by examining the gene expression patterns of epigenetic factors — factors that influence how genes are turned on or off — in tumors, they could categorize them into distinct groups to predict patient outcomes across various cancer types better than traditional measures like cancer grade and stage.
These findings, described in Communications Biology, also lay the groundwork for developing targeted therapies aimed at regulating epigenetic factors in cancer therapy, such as histone acetyltransferases and SWI/SNF chromatin remodelers.
“Traditionally, cancer has been viewed as primarily a result of genetic mutations within oncogenes or tumor suppressors,” said co-senior author Hilary Coller, professor of molecular, cell, and developmental biology and a member of the UCLA Health Jonsson Comprehensive Cancer Center and the Eli and Edythe Broad Center of Regenerative Medicine and Stem Cell Research at UCLA. “However, the emergence of advanced next-generation sequencing technologies has made more people realize that the state of the chromatin and the levels of epigenetic factors that maintain this state are important for cancer and cancer progression. There are different aspects of the state of the chromatin — like whether the histone proteins are modified, or whether the nucleic acid bases of the DNA contain extra methyl groups — that can affect cancer outcomes. Understanding these differences between tumors could help us learn more about why some patients respond differently to treatments and why their outcomes vary.”
While previous studies have shown that mutations in the genes that encode epigenetic factors can affect an individual’s cancer susceptibility, little is known about how the levels of these factors impact cancer progression. This knowledge gap is crucial in fully understanding how epigenetics affects patient outcomes, noted Coller.
To see if there was a relationship between epigenetic patterns and clinical outcomes, the researchers analyzed the expression patterns of 720 epigenetic factors to classify tumors from 24 different cancer types into distinct clusters.
Out of the 24 adult cancer types, the team found that for 10 of the cancers, the clusters were associated with significant differences in patient outcomes, including progression-free survival, disease-specific survival, and overall survival.

This was especially true for adrenocortical carcinoma, kidney renal clear cell carcinoma, brain lower grade glioma, liver hepatocellular carcinoma and lung adenocarcinoma, where the differences were significant for all the survival measurements.
The clusters with poor outcomes tended to have higher cancer stage, larger tumor size, or more severe spread indicators.
“We saw that the prognostic efficacy of an epigenetic factor was dependent on the tissue-of-origin of the cancer type,” said Mithun Mitra, co-senior author of the study and an associate project scientist in the Coller laboratory. “We even saw this link in the few pediatric cancer types we analyzed. This may be helpful in deciding the cancer-specific relevance of therapeutically targeting these factors.”
The team then used epigenetic factor gene expression levels to train and test an AI model to predict patient outcomes. This model was specifically designed to predict what might happen for the five cancer types that had significant differences in survival measurements.
The scientists found the model could successfully divide patients with these five cancer types into two groups: one with a significantly higher chance of better outcomes and another with a higher chance of poorer outcomes.
They also saw that the genes that were most crucial for the AI model had a significant overlap with the cluster-defining signature genes.
“The pan-cancer AI model is trained and tested on the adult patients from the TCGA cohort and it would be good to test this on other independent datasets to explore its broad applicability,” said Mitra. “Similar epigenetic factor-based models could be generated for pediatric cancers to see what factors influence the decision-making process compared to the models built on adult cancers.”
“Our research helps provide a roadmap for similar AI models that can be generated through publicly-available lists of prognostic epigenetic factors,” said the study’s first author, Michael Cheng, a graduate student in the Bioinformatics Interdepartmental Program at UCLA. “The roadmap demonstrates how to identify certain influential factors in different types of cancer and contains exciting potential for predicting specific targets for cancer treatment.”
The study was funded in part by grants from the National Cancer Institute, Cancer Research Institute, Melanoma Research Alliance, Melanoma Research Foundation, National Institutes of Health and the UCLA Spore in Prostate Cancer. More

The widespread adoption of nuclear power was predicted by computer simulations more than four decades ago but the continued reliance on fossil fuels for energy shows these simulations need improvement, a new study has shown.
In order to assess the efficacy of energy policies implemented today, a team of researchers looked back at the influential 1980s model that predicted nuclear power would expand dramatically. Energy policies shapes how we produce and use energy, impacting jobs, costs, climate, and security. These policies are generated using simulations (also known as mathematical models) which forecast things like electricity demand and technology costs. But forecasts may miss the point altogether.
Results published today (Wednesday, 15 November) in the journal Risk Analysis showed the team found simulations that inform energy policy had unreliable assumptions built into them and that they need more transparency about their limitations. To amend this, they recommend new ways to test simulations and be upfront about their uncertainties. This includes methods like ‘sensitivity auditing’, which evaluates model assumptions. The goal is to improve modelling and open up decision-making.
Lead researcher Dr Samuele Lo Piano, of the University of Reading, said: “Energy policy affects everybody, so it’s worrying when decisions rely on just a few models without questioning their limits. By questioning assumptions and exploring what we don’t know, we can get better decision making. We have to acknowledge that no model can perfectly predict the future. But by being upfront about model limitations, democratic debate on energy policy will improve.”
Modelling politics
A chapter of a new book, The politics of modelling(to be published on November 20), written by lead author Dr Lo Piano, highlights why the research matters for all the fields where mathematical models are used to inform decision and policy-making. The chapter considers the inherent complexities and uncertainties posed by human-caused socio-economic and environmental changes.
Entitled ‘Sensitivity auditing — A practical checklist for auditing decision-relevant models’, the chapter presents four real-world applications of sensitivity auditing in public health, education, human-water systems, and food provision systems. More

A team of researchers from Nanyang Technological University, Singapore (NTU Singapore) has developed a computer program that creates realistic videos that reflect the facial expressions and head movements of the person speaking, only requiring an audio clip and a face photo.
DIverse yet Realistic Facial Animations, or DIRFA, is an artificial intelligence-based program that takes audio and a photo and produces a 3D video showing the person demonstrating realistic and consistent facial animations synchronised with the spoken audio (see videos).
The NTU-developed program improves on existing approaches, which struggle with pose variations and emotional control.
To accomplish this, the team trained DIRFA on over one million audiovisual clips from over 6,000 people derived from an open-source database called The VoxCeleb2 Dataset to predict cues from speech and associate them with facial expressions and head movements.
The researchers said DIRFA could lead to new applications across various industries and domains, including healthcare, as it could enable more sophisticated and realistic virtual assistants and chatbots, improving user experiences. It could also serve as a powerful tool for individuals with speech or facial disabilities, helping them to convey their thoughts and emotions through expressive avatars or digital representations, enhancing their ability to communicate.
Corresponding author Associate Professor Lu Shijian, from the School of Computer Science and Engineering (SCSE) at NTU Singapore, who led the study, said: “The impact of our study could be profound and far-reaching, as it revolutionises the realm of multimedia communication by enabling the creation of highly realistic videos of individuals speaking, combining techniques such as AI and machine learning. Our program also builds on previous studies and represents an advancement in the technology, as videos created with our program are complete with accurate lip movements, vivid facial expressions and natural head poses, using only their audio recordings and static images.”
First author Dr Wu Rongliang, a PhD graduate from NTU’s SCSE, said: “Speech exhibits a multitude of variations. Individuals pronounce the same words differently in diverse contexts, encompassing variations in duration, amplitude, tone, and more. Furthermore, beyond its linguistic content, speech conveys rich information about the speaker’s emotional state and identity factors such as gender, age, ethnicity, and even personality traits. Our approach represents a pioneering effort in enhancing performance from the perspective of audio representation learning in AI and machine learning.” Dr Wu is a Research Scientist at the Institute for Infocomm Research, Agency for Science, Technology and Research (A*STAR), Singapore.

The findings were published in the scientific journal Pattern Recognition in August.
Speaking volumes: Turning audio into action with animated accuracy
The researchers say that creating lifelike facial expressions driven by audio poses a complex challenge. For a given audio signal, there can be numerous possible facial expressions that would make sense, and these possibilities can multiply when dealing with a sequence of audio signals over time.
Since audio typically has strong associations with lip movements but weaker connections with facial expressions and head positions, the team aimed to create talking faces that exhibit precise lip synchronisation, rich facial expressions, and natural head movements corresponding to the provided audio.
To address this, the team first designed their AI model, DIRFA, to capture the intricate relationships between audio signals and facial animations. The team trained their model on more than one million audio and video clips of over 6,000 people, derived from a publicly available database.
Assoc Prof Lu added: “Specifically, DIRFA modelled the likelihood of a facial animation, such as a raised eyebrow or wrinkled nose, based on the input audio. This modelling enabled the program to transform the audio input into diverse yet highly lifelike sequences of facial animations to guide the generation of talking faces.”
Dr Wu added: “Extensive experiments show that DIRFA can generate talking faces with accurate lip movements, vivid facial expressions and natural head poses. However, we are working to improve the program’s interface, allowing certain outputs to be controlled. For example, DIRFA does not allow users to adjust a certain expression, such as changing a frown to a smile.”
Besides adding more options and improvements to DIRFA’s interface, the NTU researchers will be finetuning its facial expressions with a wider range of datasets that include more varied facial expressions and voice audio clips. More

Stroke is a leading cause of long-term disability worldwide. Each year more than 15 million people worldwide have strokes, and three-quarters of stroke survivors will experience impairment, weakness and paralysis in their arms and hands.
Many stroke survivors rely on their stronger arm to complete daily tasks, from carrying groceries to combing their hair, even when the weaker arm has the potential to improve. Breaking this habit, known as “arm nonuse” or “learned nonuse,” can improve strength and prevent injury.
But, determining how much a patient is using their weaker arm outside of the clinic is challenging. In a classic case of observer’s paradox, the measurement has to be covert for the patient to behave spontaneously.
Now, USC researchers have developed a novel robotic system for collecting precise data on how people recovering from stroke use their arms spontaneously. The first-of-its-kind method is outlined in a paper published in the November 15 issue of Science Robotics.
Using a robotic arm to track 3D spatial information, and machine learning techniques to process the data, the method generates an “arm nonuse” metric, which could help clinicians accurately assess a patient’s rehabilitation progress. A socially assistive robot (SAR) provides instructions and encouragement throughout the challenge.
“Ultimately, we are trying to assess how much someone’s performance in physical therapy transfers into real life,” said Nathan Dennler, the paper’s lead author and a computer science doctoral student.
The research involved combined efforts from researchers in USC’s Thomas Lord Department of Computer Science and the Division of Biokinesiology and Physical Therapy. “This work brings together quantitative user-performance data collected using a robot arm, while also motivating the user to provide a representative performance thanks to a socially assistive robot,” said Maja Matari?, study co-author and Chan Soon-Shiong Chair and Distinguished Professor of Computer Science, Neuroscience, and Pediatrics. “This novel combination can serve as a more accurate and more motivating process for stroke patient assessment.”
Additional authors are Stefanos Nikolaidis, an assistant professor of computer science; Amelia Cain, an assistant professor of clinical physical therapy, Carolee J. Winstein, a professor emeritus and an adjunct professor in the Neuroscience Graduate Program, and computer science students Erica De Guzmann and Claudia Chiu.

Mirroring everyday use
For the study, the research team recruited 14 participants who were right-hand dominant before the stroke. The participant placed their hands on the device’s home position — a 3D-printed box with touch sensors.
A socially assistive robot (SAR) described the system’s mechanics and provided positive feedback, while the robot arm moved a button to different target locations in front of the participant (100 locations in total). The “reaching trial” begins when the button lights up, and the SAR cues the participant to move.
In the first phase, the participants were directed to reach for the button using whichever hand came naturally, mirroring everyday use. In the second phase, they were instructed to use the stroke-affected arm only, mirroring performance in physiotherapy or other clinical settings.
Using machine learning, the team analyzed three measurements to determine a metric for arm nonuse: arm use probability, time to reach, and successful reach. A noticeable difference in performance between the phases would suggest nonuse of the affected arm.
“The participants have a time limit to reach the button, so even though they know they’re being tested, they still have to react quickly,” said Dennler. “This way, we’re measuring gut reaction to the light turning on — which hand will you use on the spot?”
Safe and easy to use

In chronic stroke survivors, the researchers observed high variability in hand choice and in the time to reach targets in the workspace. The method was reliable across repeated sessions, and participants rated it as simple to use, with above-average user experience scores. All participants found the interaction to be safe and easy to use.
Crucially, the researchers found differences in arm use between participants, which could be used by healthcare professionals to more accurately track a patient’s stroke recovery.
“For example, one participant whose right side was more affected by their stroke exhibited lower use of their right arm specifically in areas higher on their right side, but maintained a high probability of using their right arm for lower areas on the same side,” said Dennler.
“Another participant exhibited more symmetric use but also compensated with their less-affected side slightly more often for higher-up points that were close to the mid-line.”
Participants felt that the system could be improved through personalization, which the team hopes to explore in future studies, in addition to incorporating other behavioral data such as facial expressions and different types of tasks.
As a physiotherapist, Cain said the technology addresses many issues encountered with traditional methods of assessment, which “require the patient not to know they’re being tested, and are based on the tester’s observation which can leave more room for error.”
“This type of technology could provide rich, objective information about a stroke survivor’s arm use to their rehabilitation therapist,” said Cain. “The therapist could then integrate this information into their clinical decision-making process and better tailor their interventions to address the patient’s areas of weakness and build upon areas of strength.” More

3D printing is advancing rapidly, and the range of materials that can be used has expanded considerably. While the technology was previously limited to fast-curing plastics, it has now been made suitable for slow-curing plastics as well. These have decisive advantages as they have enhanced elastic properties and are more durable and robust.
The use of such polymers is made possible by a new technology developed by researchers at ETH Zurich and a US start-up. As a result, researchers can now 3D print complex, more durable robots from a variety of high-quality materials in one go. This new technology also makes it easy to combine soft, elastic, and rigid materials. The researchers can also use it to create delicate structures and parts with cavities as desired.
Materials that return to their original state
Using the new technology, researchers at ETH Zurich have succeeded for the first time in printing a robotic hand with bones, ligaments and tendons made of different polymers in one go. “We wouldn’t have been able to make this hand with the fast-curing polyacrylates we’ve been using in 3D printing so far,” explains Thomas Buchner, a doctoral student in the group of ETH Zurich robotics professor Robert Katzschmann and first author of the study. “We’re now using slow-curing thiolene polymers. These have very good elastic properties and return to their original state much faster after bending than polyacrylates.” This makes thiolene polymers ideal for producing the elastic ligaments of the robotic hand.
In addition, the stiffness of thiolenes can be fine-tuned very well to meet the requirements of soft robots. “Robots made of soft materials, such as the hand we developed, have advantages over conventional robots made of metal. Because they’re soft, there is less risk of injury when they work with humans, and they are better suited to handling fragile goods,” Katzschmann explains.
Scanning instead of scraping
3D printers typically produce objects layer by layer: nozzles deposit a given material in viscous form at each point; a UV lamp then cures each layer immediately. Previous methods involved a device that scraped off surface irregularities after each curing step. This works only with fast-curing polyacrylates. Slow-curing polymers such as thiolenes and epoxies would gum up the scraper.
To accommodate the use of slow-curing polymers, the researchers developed 3D printing further by adding a 3D laser scanner that immediately checks each printed layer for any surface irregularities. “A feedback mechanism compensates for these irregularities when printing the next layer by calculating any necessary adjustments to the amount of material to be printed in real time and with pinpoint accuracy,” explains Wojciech Matusik, a professor at the Massachusetts Institute of Technology (MIT) in the US and co-author of the study. This means that instead of smoothing out uneven layers, the new technology simply takes the unevenness into account when printing the next layer.
Inkbit, an MIT spin-off, was responsible for developing the new printing technology. The ETH Zurich researchers developed several robotic applications and helped optimise the printing technology for use with slow-curing polymers. The researchers from Switzerland and the US have now jointly published the technology and their sample applications in the journal Nature.
At ETH Zurich, Katzschmann’s group will use the technology to explore further possibilities and to design even more sophisticated structures and develop additional applications. Inkbit is planning to use the new technology to offer a 3D printing service to its customers and to sell the new printers. More

With 3D inkjet printing systems, engineers can fabricate hybrid structures that have soft and rigid components, like robotic grippers that are strong enough to grasp heavy objects but soft enough to interact safely with humans.
These multimaterial 3D printing systems utilize thousands of nozzles to deposit tiny droplets of resin, which are smoothed with a scraper or roller and cured with UV light. But the smoothing process could squish or smear resins that cure slowly, limiting the types of materials that can be used.
Researchers from MIT, the MIT spinout Inkbit, and ETH Zurich have developed a new 3D inkjet printing system that works with a much wider range of materials. Their printer utilizes computer vision to automatically scan the 3D printing surface and adjust the amount of resin each nozzle deposits in real time to ensure no areas have too much or too little material.
Since it does not require mechanical parts to smooth the resin, this contactless system works with materials that cure more slowly than the acrylates which are traditionally used in 3D printing. Some slower-curing material chemistries can offer improved performance over acrylates, such as greater elasticity, durability, or longevity.
In addition, the automatic system makes adjustments without stopping or slowing the printing process, making this production-grade printer about 660 times faster than a comparable 3D inkjet printing system.
The researchers used this printer to create complex, robotic devices that combine soft and rigid materials. For example, they made a completely 3D-printed robotic gripper shaped like a human hand and controlled by a set of reinforced, yet flexible, tendons.
“Our key insight here was to develop a machine vision system and completely active feedback loop. This is almost like endowing a printer with a set of eyes and a brain, where the eyes observe what is being printed, and then the brain of the machine directs it as to what should be printed next,” says co-corresponding author Wojciech Matusik, a professor of electrical engineering and computer science at MIT who leads the Computational Design and Fabrication Group within the MIT Computer Science and Artificial Intelligence Laboratory (CSAIL).

He is joined on the paper by lead author Thomas Buchner, a doctoral student at ETH Zurich, co-corresponding author Robert Katzschmann, PhD ’18, assistant professor of robotics who leads the Soft Robotics Laboratory at ETH Zurich; as well as others at ETH Zurich and Inkbit. The research will appear in Nature.
Contact free
This paper builds off a low-cost, multimaterial 3D printer known as MultiFab that the researchers introduced in 2015. By utilizing thousands of nozzles to deposit tiny droplets of resin that are UV-cured, MultiFab enabled high-resolution 3D printing with up to 10 materials at once.
With this new project, the researchers sought a contactless process that would expand the range of materials they could use to fabricate more complex devices.
They developed a technique, known as vision-controlled jetting, which utilizes four high-frame-rate cameras and two lasers that rapidly and continuously scan the print surface. The cameras capture images as thousands of nozzles deposit tiny droplets of resin.
The computer vision system converts the image into a high-resolution depth map, a computation that takes less than a second to perform. It compares the depth map to the CAD (computer-aided design) model of the part being fabricated, and adjusts the amount of resin being deposited to keep the object on target with the final structure.

The automated system can make adjustments to any individual nozzle. Since the printer has 16,000 nozzles, the system can control fine details of the device being fabricated.
“Geometrically, it can print almost anything you want made of multiple materials. There are almost no limitations in terms of what you can send to the printer, and what you get is truly functional and long-lasting,” says Katzschmann.
The level of control afforded by the system enables it to print very precisely with wax, which is used as a support material to create cavities or intricate networks of channels inside an object. The wax is printed below the structure as the device is fabricated. After it is complete, the object is heated so the wax melts and drains out, leaving open channels throughout the object.
Because it can automatically and rapidly adjust the amount of material being deposited by each of the nozzles in real time, the system doesn’t need to drag a mechanical part across the print surface to keep it level. This enables the printer to use materials that cure more gradually, and would be smeared by a scraper.
Superior materials
The researchers used the system to print with thiol-based materials, which are slower-curing than the traditional acrylic materials used in 3D printing. However, thiol-based materials are more elastic and don’t break as easily as acrylates. They also tend to be more stable over a wider range of temperatures and don’t degrade as quickly when exposed to sunlight.
“These are very important properties when you want to fabricate robots or systems that need to interact with a real-world environment,” says Katzschmann.
The researchers used thiol-based materials and wax to fabricate several complex devices that would otherwise be nearly impossible to make with existing 3D printing systems. For one, they produced a functional, tendon-driven robotic hand that has 19 independently actuatable tendons, soft fingers with sensor pads, and rigid, load-bearing bones.
“We also produced a six-legged walking robot that can sense objects and grasp them, which was possible due to the system’s ability to create airtight interfaces of soft and rigid materials, as well as complex channels inside the structure,” says Buchner.
The team also showcased the technology through a heart-like pump with integrated ventricles and artificial heart valves, as well as metamaterials that can be programmed to have non-linear material properties.
“This is just the start. There is an amazing number of new types of materials you can add to this technology. This allows us to bring in whole new material families that couldn’t be used in 3D printing before,” Matusik says.
The researchers are now looking at using the system to print with hydrogels, which are used in tissue-engineering applications, as well as silicon materials, epoxies, and special types of durable polymers.
They also want to explore new application areas, such as printing customizable medical devices, semiconductor polishing pads, and even more complex robots.
This research was funded, in part, by Credit Suisse, the Swiss National Science Foundation, the Defense Advanced Research Projects Agency (DARPA), and the National Science Foundation (NSF). More

Researchers have developed a new deep learning AI tool that generates life-like birdsongs to train bird identification tools, helping ecologists to monitor rare species in the wild. The findings are presented in the British Ecological Society journal, Methods in Ecology and Evolution.
Identifying common bird species through their song has never been easier, with numerous phone apps and software available to both ecologists and the public. But what if the identification software has never heard a particular bird before, or only has a small sample of recordings to reference? This is a problem facing ecologists and conservationists monitoring some of the world’s rarest birds.
To overcome this problem, researchers at the University of Moncton, Canada, have developed ECOGEN, a first of its kind deep learning tool, that can generate lifelike bird sounds to enhance the samples of underrepresented species. These can then be used to train audio identification tools used in ecological monitoring, which often have disproportionately more information on common species.
The researchers found that adding artificial birdsong samples generated by ECOGEN to a birdsong identifier improved the bird song classification accuracy by 12% on average.
Dr Nicolas Lecomte, one of the lead researchers, said: “Due to significant global changes in animal populations, there is an urgent need for automated tools, such acoustic monitoring, to track shifts in biodiversity. However, the AI models used to identify species in acoustic monitoring lack comprehensive reference libraries.
“With ECOGEN, you can address this gap by creating new instances of bird sounds to support AI models. Essentially, for species with limited wild recordings, such as those that are rare, elusive, or sensitive, you can expand your sound library without further disrupting the animals or conducting additional fieldwork.”
The researchers say that creating synthetic bird songs in this way can contribute to the conservation of endangered bird species and also provide valuable insight into their vocalisations, behaviours and habitat preferences.

The ECOGEN tool has other potential applications. For instance, it could be used to help conserve extremely rare species, like the critically endangered regent honeyeaters, where young individuals are unable to learn their species’ songs because there aren’t enough adult birds to learn from.
The tool could benefit other types of animal as well. Dr Lecomte added: “While ECOGEN was developed for birds, we’re confident that it could be applied to mammals, fish (yes they can produce sounds!), insects and amphibians.”
As well as its versatility, a key advantage of the ECOGEN tool is its accessibility, due to it being open source and able to used on even basic computers.
ECOGEN works by converting real recordings of bird songs into spectrograms (visual representations of sounds) and then generating new AI images from these to increase the dataset for rare species with few recordings. These spectrograms are then converted back into audio to train bird sound identifiers. In this study the researchers used a dataset of 23,784 wild bird recordings from around the world, covering 264 species. More