Computers Math

Modern robots know how to sense their environment and respond to language, but what they don’t know is often more important than what they do know. Teaching robots to ask for help is key to making them safer and more efficient.
Engineers at Princeton University and Google have come up with a new way to teach robots to know when they don’t know. The technique involves quantifying the fuzziness of human language and using that measurement to tell robots when to ask for further directions. Telling a robot to pick up a bowl from a table with only one bowl is fairly clear. But telling a robot to pick up a bowl when there are five bowls on the table generates a much higher degree of uncertainty — and triggers the robot to ask for clarification.
Because tasks are typically more complex than a simple “pick up a bowl” command, the engineers use large language models (LLMs) — the technology behind tools such as ChatGPT — to gauge uncertainty in complex environments. LLMs are bringing robots powerful capabilities to follow human language, but LLM outputs are still frequently unreliable, said Anirudha Majumdar, an assistant professor of mechanical and aerospace engineering at Princeton and the senior author of a study outlining the new method.
“Blindly following plans generated by an LLM could cause robots to act in an unsafe or untrustworthy manner, and so we need our LLM-based robots to know when they don’t know,” said Majumdar.
The system also allows a robot’s user to set a target degree of success, which is tied to a particular uncertainty threshold that will lead a robot to ask for help. For example, a user would set a surgical robot to have a much lower error tolerance than a robot that’s cleaning up a living room.
“We want the robot to ask for enough help such that we reach the level of success that the user wants. But meanwhile, we want to minimize the overall amount of help that the robot needs,” said Allen Ren, a graduate student in mechanical and aerospace engineering at Princeton and the study’s lead author. Ren received a best student paper award for his Nov. 8 presentation at the Conference on Robot Learning in Atlanta. The new method produces high accuracy while reducing the amount of help required by a robot compared to other methods of tackling this issue.
The researchers tested their method on a simulated robotic arm and on two types of robots at Google facilities in New York City and Mountain View, California, where Ren was working as a student research intern. One set of hardware experiments used a tabletop robotic arm tasked with sorting a set of toy food items into two different categories; a setup with a left and right arm added an additional layer of ambiguity.

The most complex experiments involved a robotic arm mounted on a wheeled platform and placed in an office kitchen with a microwave and a set of recycling, compost and trash bins. In one example, a human asks the robot to “place the bowl in the microwave,” but there are two bowls on the counter — a metal one and a plastic one.
The robot’s LLM-based planner generates four possible actions to carry out based on this instruction, like multiple-choice answers, and each option is assigned a probability. Using a statistical approach called conformal prediction and a user-specified guaranteed success rate, the researchers designed their algorithm to trigger a request for human help when the options meet a certain probability threshold. In this case, the top two options — place the plastic bowl in the microwave or place the metal bowl in the microwave — meet this threshold, and the robot asks the human which bowl to place in the microwave.
In another example, a person tells the robot, “There is an apple and a dirty sponge … It is rotten. Can you dispose of it?” This does not trigger a question from the robot, since the action “put the apple in the compost” has a sufficiently higher probability of being correct than any other option.
“Using the technique of conformal prediction, which quantifies the language model’s uncertainty in a more rigorous way than prior methods, allows us to get to a higher level of success” while minimizing the frequency of triggering help, said the study’s senior author Anirudha Majumdar, an assistant professor of mechanical and aerospace engineering at Princeton.
Robots’ physical limitations often give designers insights not readily available from abstract systems. Large language models “might talk their way out of a conversation, but they can’t skip gravity,” said coauthor Andy Zeng, a research scientist at Google DeepMind. “I’m always keen on seeing what we can do on robots first, because it often sheds light on the core challenges behind building generally intelligent machines.”
Ren and Majumdar began collaborating with Zeng after he gave a talk as part of the Princeton Robotics Seminar series, said Majumdar. Zeng, who earned a computer science Ph.D. from Princeton in 2019, outlined Google’s efforts in using LLMs for robotics, and brought up some open challenges. Ren’s enthusiasm for the problem of calibrating the level of help a robot should ask for led to his internship and the creation of the new method.
“We enjoyed being able to leverage the scale that Google has” in terms of access to large language models and different hardware platforms, said Majumdar.
Ren is now extending this work to problems of active perception for robots: For instance, a robot may need to use predictions to determine the location of a television, table or chair within a house, when the robot itself is in a different part of the house. This requires a planner based on a model that combines vision and language information, bringing up a new set of challenges in estimating uncertainty and determining when to trigger help, said Ren. More

Researchers report that they have developed a new composite material designed to change behaviors depending on temperature in order to perform specific tasks. These materials are poised to be part of the next generation of autonomous robotics that will interact with the environment.
The new study conducted by University of Illinois Urbana-Champaign civil and environmental engineering professor Shelly Zhang and graduate student Weichen Li, in collaboration with professor Tian Chen and graduate student Yue Wang from the University of Houston, uses computer algorithms, two distinct polymers and 3D printing to reverse engineer a material that expands and contracts in response to temperature change with or without human intervention.
The study findings are reported in the journal Science Advances.
“Creating a material or device that will respond in specific ways depending on its environment is very challenging to conceptualize using human intuition alone — there are just so many design possibilities out there,” Zhang said. “So, instead, we decided to work with a computer algorithm to help us determine the best combination of materials and geometry.”
The team first used computer modeling to conceptualize a two-polymer composite that can behave differently under various temperatures based on user input or autonomous sensing.
“For this study, we developed a material that can behave like soft rubber in low temperatures and as a stiff plastic in high temperatures,” Zhang said.
Once fabricated into a tangible device, the team tested the new composite material’s ability to respond to temperature changes to perform a simple task — switch on LED lights.

“Our study demonstrates that it is possible to engineer a material with intelligent temperature sensing capabilities, and we envision this being very useful in robotics,” Zhang said. “For example, if a robot’s carrying capacity needs to change when the temperature changes, the material will ‘know’ to adapt its physical behavior to stop or perform a different task.”
Zhang said that one of the hallmarks of the study is the optimization process that helps the researchers interpolate the distribution and geometries of the two different polymer materials needed.
“Our next goal is to use this technique to add another level of complexity to a material’s programmed or autonomous behavior, such as the ability to sense the velocity of some sort of impact from another object,” she said. “This will be critical for robotics materials to know how to respond to various hazards in the field.”
The National Science Foundation supported this research. More

A new kind of “wire” for moving excitons, developed at the University of Michigan, could help enable a new class of devices, perhaps including room temperature quantum computers.
What’s more, the team observed a dramatic violation of Einstein’s relation, used to describe how particles spread out in space, and leveraged it to move excitons in much smaller packages than previously possible.
“Nature uses excitons in photosynthesis. We use excitons in OLED displays and some LEDs and solar cells,” said Parag Deotare, co-corresponding author of the study in ACS Nano supervising the experimental work, and an associate professor of electrical and computer engineering. “The ability to move excitons where we want will help us improve the efficiency of devices that already use excitons and expand excitonics into computing.”
An exciton can be thought of as a particle (hence quasiparticle), but it’s really an electron linked with a positively-charged empty space in the lattice of the material (a “hole”). Because an exciton has no net electrical charge, moving excitons are not affected by parasitic capacitances, an electrical interaction between neighboring components in a device that causes energy losses. Excitons are also easy to convert to and from light, so they open the way for extremely fast and efficient computers that use a combination of optics and excitonics, rather than electronics.
This combination could help enable room temperature quantum computing, said Mackillo Kira, co-corresponding author of the study supervising the theory, and a professor of electrical and computer engineering. Excitons can encode quantum information, and they can hang onto it longer than electrons can inside a semiconductor. But that time is still measured in picoseconds (10-12 seconds) at best, so Kira and others are figuring out how to use femtosecond laser pulses (10-15 seconds) to process information.
“Full quantum-information applications remain challenging because degradation of quantum information is too fast for ordinary electronics,” he said. “We are currently exploring lightwave electronics as a means to supercharge excitonics with extremely fast processing capabilities.”
However, the lack of net charge also makes excitons very difficult to move. Previously, Deotare had led a study that pushed excitons through semiconductors with acoustic waves. Now, a pyramid structure enables more precise transport for smaller numbers of excitons, confined to one dimension like a wire.

It works like this:
The team used a laser to create a cloud of excitons at a corner of the pyramid’s base, bouncing electrons out of the valence band of a semiconductor into the conduction band — but the negatively charged electrons are still attracted to the positively charged holes left behind in the valence band. The semiconductor is a single layer of tungsten diselenide semiconductor, just three atoms thick, draped over the pyramid like a stretchy cloth. And the stretch in the semiconductor changes the energy landscape that the excitons experience.
It seems counterintuitive that the excitons should ride up the pyramid’s edge and settle at the peak when we imagine an energy landscape chiefly governed by gravity. But instead, the landscape is governed by how far apart the valence and conduction bands of the semiconductor are. The energy gap between the two, also known as the semiconductor’s band gap, shrinks where the semiconductor is stretched. The excitons migrate to the lowest energy state, funneled onto the pyramid’s edge where they then rise to its peak.
Usually, an equation penned by Einstein is good at describing how a bunch of particles diffuses outward and drifts. However, the semiconductor was imperfect, and those defects acted as traps that would nab some of the excitons as they tried to drift by. Because the defects at the trailing side of the exciton cloud were filled in, that side of the distribution diffused outward as predicted. The leading edge, however, did not extend so far. Einstein’s relation was off by more than a factor of 10.
“We’re not saying Einstein was wrong, but we have shown that in complicated cases like this, we shouldn’t be using his relation to predict the mobility of excitons from the diffusion,” said Matthias Florian, co-first-author of the study and a research investigator in electrical and computer engineering, working under Kira.
To directly measure both, the team needed to detect single photons, emitted when the bound electrons and holes spontaneously recombined. Using time-of-flight measurements, they also figured out where the photons came from precisely enough to measure the distribution of excitons within the cloud.
The study was supported by the Army Research Office (award no. W911NF2110207) and the Air Force Office of Scientific Research (award no. FA995-22-1-0530).
The pyramid structure was built in the Lurie Nanofabrication Facility.
The team has applied for patent protection with the assistance of U-M Innovation Partnerships and is seeking partners to bring the technology to market. More

Machine learning is now helping researchers analyze the makeup of unfamiliar cells, which could lead to more personalized medicine in the treatment of cancer and other serious diseases.
Researchers at the University of Waterloo developed GraphNovo, a new program that provides a more accurate understanding of the peptide sequences in cells. Peptides are chains of amino acids within cells and are building blocks as important and unique as DNA or RNA.
In a healthy person, the immune system can correctly identify the peptides of irregular or foreign cells, such as cancer cells or harmful bacteria, and then target those cells for destruction. For people whose immune system is struggling, the promising field of immunotherapy is working to retrain their immune systems to identify these dangerous invaders.
“What scientists want to do is sequence those peptides between the normal tissue and the cancerous tissue to recognize the differences,” said Zeping Mao, a PhD candidate in the Cheriton School of Computer Science who developed GraphNovo under the guidance of Dr. Ming Li.
This sequencing process is particularly difficult for novel illnesses or cancer cells, which may not have been analyzed before. While scientists can draw on an existing peptide database when analyzing diseases or organisms that have previously been studied, each person’s cancer and immune system are unique.
To quickly build a profile of the peptides in an unfamiliar cell, scientists have been using a method called de novo peptide sequencing, which uses mass spectrometry to rapidly analyze a new sample. This process may leave some peptides incomplete or entirely missing from the sequence.
Utilizing machine learning, GraphNovo significantly enhances the accuracy in identifying peptide sequences by filling these gaps with the precise mass of the peptide sequence. Such a leap in accuracy will likely be immensely beneficial in a variety of medical areas, especially in the treatment of cancer and the creation of vaccines for ailments such as Ebola and COVID-19. The researchers achieved this breakthrough due to Waterloo’s commitment to advances in the interface between technology and health.
“If we don’t have an algorithm that’s good enough, we cannot build the treatments,” Mao said. “Right now, this is all theoretical. But soon, we will be able to use it in the real world.” More

Particle accelerators hold great potential for semiconductor applications, medical imaging and therapy, and research in materials, energy and medicine. But conventional accelerators require plenty of elbow room — kilometers — making them expensive and limiting their presence to a handful of national labs and universities.
Researchers from The University of Texas at Austin, several national laboratories, European universities and the Texas-based company TAU Systems Inc. have demonstrated a compact particle accelerator less than 20 meters long that produces an electron beam with an energy of 10 billion electron volts (10 GeV). There are only two other accelerators currently operating in the U.S. that can reach such high electron energies, but both are approximately 3 kilometers long.
“We can now reach those energies in 10 centimeters,” said Bjorn “Manuel” Hegelich, associate professor of physics at UT and CEO of TAU Systems, referring to the size of the chamber where the beam was produced. He is the senior author on a recent paper describing their achievement in the journal Matter and Radiation at Extremes.
Hegelich and his team are currently exploring the use of their accelerator, called an advanced wakefield laser accelerator, for a variety of purposes. They hope to use it to test how well space-bound electronics can withstand radiation, to image the 3D internal structures of new semiconductor chip designs, and even to develop novel cancer therapies and advanced medical-imaging techniques.
This kind of accelerator could also be used to drive another device called an X-ray free electron laser, which could take slow-motion movies of processes on the atomic or molecular scale. Examples of such processes include drug interactions with cells, changes inside batteries that might cause them to catch fire, chemical reactions inside solar panels, and viral proteins changing shape when infecting cells.
The concept for wakefield laser accelerators was first described in 1979. An extremely powerful laser strikes helium gas, heats it into a plasma and creates waves that kick electrons from the gas out in a high-energy electron beam. During the past couple of decades, various research groups have developed more powerful versions. Hegelich and his team’s key advance relies on nanoparticles. An auxiliary laser strikes a metal plate inside the gas cell, which injects a stream of metal nanoparticles that boost the energy delivered to electrons from the waves.
The laser is like a boat skimming across a lake, leaving behind a wake, and electrons ride this plasma wave like surfers.

“It’s hard to get into a big wave without getting overpowered, so wake surfers get dragged in by Jet Skis,” Hegelich said. “In our accelerator, the equivalent of Jet Skis are nanoparticles that release electrons at just the right point and just the right time, so they are all sitting there in the wave. We get a lot more electrons into the wave when and where we want them to be, rather than statistically distributed over the whole interaction, and that’s our secret sauce.”
For this experiment, the researchers used one of the world’s most powerful pulsed lasers, the Texas Petawatt Laser, which is housed at UT and fires one ultra-intense pulse of light every hour. A single petawatt laser pulse contains about 1,000 times the installed electrical power in the U.S. but lasts only 150 femtoseconds, less than a billionth as long as a lightning discharge. The team’s long-term goal is to drive their system with a laser they’re currently developing that fits on a tabletop and can fire repeatedly at thousands of times per second, making the whole accelerator far more compact and usable in much wider settings than conventional accelerators.
The study’s co-first authors are Constantin Aniculaesei, corresponding author now at Heinrich Heine University Düsseldorf, Germany; and Thanh Ha, doctoral student at UT and researcher at TAU Systems. Other UT faculty members are professors Todd Ditmire and Michael Downer.
Hegelich and Aniculaesei have submitted a patent application describing the device and method to generate nanoparticles in a gas cell. TAU Systems, spun out of Hegelich’s lab, holds an exclusive license from the University for this foundational patent. As part of the agreement, UT has been issued shares in TAU Systems.
Support for this research was provided by the U.S. Air Force Office of Scientific Research, the U.S. Department of Energy, the U.K. Engineering and Physical Sciences Research Council and the European Union’s Horizon 2020 research and innovation program. More

Recent advances in generative artificial intelligence have spurred developments in realistic speech synthesis. While this technology has the potential to improve lives through personalized voice assistants and accessibility-enhancing communication tools, it also has led to the emergence of deepfakes, in which synthesized speech can be misused to deceive humans and machines for nefarious purposes.
In response to this evolving threat, Ning Zhang, an assistant professor of computer science and engineering at the McKelvey School of Engineering at Washington University in St. Louis, developed a tool called AntiFake, a novel defense mechanism designed to thwart unauthorized speech synthesis before it happens. Zhang presented AntiFake Nov. 27 at the Association for Computing Machinery’s Conference on Computer and Communications Security in Copenhagen, Denmark.
Unlike traditional deepfake detection methods, which are used to evaluate and uncover synthetic audio as a post-attack mitigation tool, AntiFake takes a proactive stance. It employs adversarial techniques to prevent the synthesis of deceptive speech by making it more difficult for AI tools to read necessary characteristics from voice recordings. The code is freely available to users.
“AntiFake makes sure that when we put voice data out there, it’s hard for criminals to use that information to synthesize our voices and impersonate us,” Zhang said. “The tool uses a technique of adversarial AI that was originally part of the cybercriminals’ toolbox, but now we’re using it to defend against them. We mess up the recorded audio signal just a little bit, distort or perturb it just enough that it still sounds right to human listeners, but it’s completely different to AI.”
To ensure AntiFake can stand up against an ever-changing landscape of potential attackers and unknown synthesis models, Zhang and first author Zhiyuan Yu, a graduate student in Zhang’s lab, built the tool to be generalizable and tested it against five state-of-the-art speech synthesizers. AntiFake achieved a protection rate of over 95%, even against unseen commercial synthesizers. They also tested AntiFake’s usability with 24 human participants to confirm the tool is accessible to diverse populations.
Currently, AntiFake can protect short clips of speech, taking aim at the most common type of voice impersonation. But, Zhang said, there’s nothing to stop this tool from being expanded to protect longer recordings, or even music, in the ongoing fight against disinformation.
“Eventually, we want to be able to fully protect voice recordings,” Zhang said. “While I don’t know what will be next in AI voice tech — new tools and features are being developed all the time — I do think our strategy of turning adversaries’ techniques against them will continue to be effective. AI remains vulnerable to adversarial perturbations, even if the engineering specifics may need to shift to maintain this as a winning strategy.” More

Health care organizations are looking to artificial intelligence (AI) tools to improve patient care, but their translation into clinical settings has been inconsistent, in part because evaluating AI in health care remains challenging. In a new article, researchers propose a framework for using AI that includes practical guidance for applying values and that incorporates not just the tool’s properties but the systems surrounding its use.
The article was written by researchers at Carnegie Mellon University, The Hospital for Sick Children, the Dalla Lana School of Public Health, Columbia University, and the University of Toronto. It is published in Patterns.
“Regulatory guidelines and institutional approaches have focused narrowly on the performance of AI tools, neglecting knowledge, practices, and procedures necessary to integrate the model within the larger social systems of medical practice,” explains Alex John London, K&L Gates Professor of Ethics and Computational Technologies at Carnegie Mellon, who coauthored the article. “Tools are not neutral — they reflect our values — so how they work reflects the people, processes, and environments in which they are put to work.”
London is also Director of Carnegie Mellon’s Center for Ethics and Policy and Chief Ethicist at Carnegie Mellon’s Block Center for Technology and Society as well as a faculty member in CMU’s Department of Philosophy.
London and his coauthors advocate for a conceptual shift in which AI tools are viewed as parts of a larger “intervention ensemble,” a set of knowledge, practices, and procedures that are necessary to deliver care to patients. In previous work with other colleagues, London has applied this concept to pharmaceuticals and to autonomous vehicles. The approach treats AI tools as “sociotechnical systems,” and the authors’ proposed framework seeks to advance the responsible integration of AI systems into health care.
Previous work in this area has been largely descriptive, explaining how AI systems interact with human systems. The framework proposed by London and his colleagues is proactive, providing guidance to designers, funders, and users about how to ensure that AI systems can be integrated into workflows with the greatest potential to help patients. Their approach can also be used for regulation and institutional insights, as well as for appraising, evaluating, and using AI tools responsibly and ethically. To illustrate their framework, the authors apply it to the development of AI systems developed for diagnosing more than mild diabetic retinopathy.
“Only a small majority of models evaluated through clinical trials have shown a net benefit,” says Melissa McCradden, a Bioethicist at the Hospital for Sick Children and Assistant Professor of Clinical and Public Health at the Dalla Lana School of Public Health, who coauthored the article. “We hope our proposed framework lends precision to evaluation and interests regulatory bodies exploring the kinds of evidence needed to support the oversight of AI systems.” More

Biomedical engineers at Duke University have developed a method using data from wearable devices such as smartwatches to digitally mimic an entire week’s worth of an individual’s heartbeats. The previous record covered only a few minutes.
Called the Longitudinal Hemodynamic Mapping Framework (LHMF), the approach creates “digital twins” of a specific patient’s blood flow to assess its 3D characteristics. The advance is an important step toward improving on the current gold standard in evaluating the risks of heart disease or heart attack, which uses snapshots of a single moment in time — a challenging approach for a disease that progresses over months to years.
The research was conducted in collaboration with computational scientists at Lawrence Livermore National Laboratory and was published on November 15, 2023, at the International Conference for High Performance Computing, Networking, Storage, and Analysis (SC23). The conference is the leading global conference in the field of high-performance computing.
“Modeling a patient’s 3D blood flow for even a single day would take a century’s worth of compute time on today’s best supercomputers,” said Cyrus Tanade, a PhD candidate in the laboratory of Amanda Randles, the Alfred Winborne and Victoria Stover Mordecai Associate Professor of Biomedical Sciences at Duke. “If we want to capture blood flow dynamics over long periods of time, we need a paradigm-shifting solution in how we approach 3D personalized simulations.”
Over the past decade, researchers have steadily made progress toward accurately modeling the pressures and forces created by blood flowing through an individual’s specific vascular geometry. Randles, one of the leaders in the field, has developed a software package called HARVEY to tackle this challenge using the world’s fastest supercomputers.
One of the most commonly accepted uses of such coronary digital twins is to determine whether or not a patient should receive a stent to treat a plaque or lesion. This computational method is much less invasive than the traditional approach of threading a probe on a guide wire into the artery itself.
While this application requires only a handful of heartbeat simulations and works for a single snapshot in time, the field’s goal is to track pressure dynamics over weeks or months after a patient leaves a hospital. To get even 10 minutes of simulated data on the Duke group’s computer cluster, however, they had to lock it down for four months.

“Obviously, that’s not a workable solution to help patients because of the computing costs and time requirements,” Randles said. “Think of it as taking three weeks to simulate what the weather will be like tomorrow. By the time you predict a rainstorm, the water would have already dried up.”
To ever apply this technology to real-world people over the long term, researchers must find a way to reduce the computational load. The new paper introduces the Longitudinal Hemodynamic Mapping Framework, which cuts what used to take nearly a century of simulation time down to just 24 hours.
“The solution is to simulate the heartbeats in parallel rather than sequentially by breaking the task up amongst many different nodes,” Tanade said. “Conventionally, the tasks are broken up spatially with parallel computing. But here, they’re broken up in time as well.”
For example, one could reasonably assume that the specifics of a coronary flow at 10:00 am on a Monday will likely have little impact on the flow at 2:00 pm on a Wednesday. This allowed the team to develop a method to accurately simulate different chunks of time simultaneously and piece them back together. This breakdown made the pieces small enough to be simulated using cloud computing systems like Amazon Web Services rather than requiring large-scale supercomputers.
To put the mapping framework to the test, researchers used tried and true methods to simulate 750 heartbeats — about 10 minutes of biological time — with the lab’s allotment of computing time on Duke’s computer cluster. Using continuous data on heart rate and electrocardiography from a smartwatch, it produced a complete set of 3D blood flow biomarkers that could correlate with disease progression and adverse events. It took four months to complete and exceeded the existing record by an order of magnitude.
They then compared these results to those produced by LHMF running on Amazon World Services and Summit, an Oak Ridge National Laboratory system, in just a few hours. The errors were negligible, proving that LHMF could work on a useful time scale.

The team then further refined LHMF by introducing a clustering method, further reducing the computational costs and allowing them to track the frictional force of blood on vessel walls — a well-known biomarker of cardiovascular disease — for over 700,000 heartbeats, or one week of continuous activity. These results allowed the group to create a personalized, longitudinal hemodynamic map, showing how the forces vary over time and the percentage of time spent in various vulnerable states.
“The results significantly differed from those obtained over a single heartbeat,” Tanade said. “This demonstrates that capturing longitudinal blood flow metrics provides nuances and information that is otherwise not perceptible with the previous gold standard approach.”
“If we can create a temporal map of wall stress in critical areas like the coronary artery, we could predict the risk of a patient developing atherosclerosis or the progression of such diseases,” Randles added. “This method could allow us to identify cases of heart disease much earlier than is currently possible.”
This work was supported by the National Science Foundation (ACI-1548562, DGE164486), the Department of Energy (DE-AC52-07NA27344, DE-AC05-00OR22725), Amazon Web Services, the National Insitutes of Health (DP1AG082343) and the Coulter Foundation. More