Aurelia Butler

As you scroll through any social media feed, you are likely to be prompted to follow or friend another person, expanding your personal network and contributing to the growth of the app itself. The person suggested to you is a result of link prediction: a widespread machine learning (ML) task that evaluates the links in a network — your friends and everyone else’s — and tries to predict what the next links will be.
Beyond being the engine that drives social media expansion, link prediction is also used in a wide range of scientific research, such as predicting the interaction between genes and proteins, and is used by researchers as a benchmark for testing the performance of new ML algorithms.
New research from UC Santa Cruz Professor of Computer Science and Engineering C. “Sesh” Seshadhri published in the journal Proceedings of the National Academy of Sciences establishes that the metric used to measure link prediction performance is missing crucial information, and link prediction tasks are performing significantly worse than popular literature indicates.
Seshadhri and his coauthor Nicolas Menand, who is a former UCSC undergraduate and masters student and a current Ph.D. candidate at the University of Pennsylvania, recommend that ML researchers stop using the standard practice metric for measuring link prediction, known as AUC, and introduce a new, more comprehensive metric for this problem. The research has implications for trustworthiness around decisionmaking in ML.
AUC’s ineffectiveness
Seshadhri, who works in the fields of theoretical computer science and data mining and is currently an Amazon scholar, has done previous research on ML algorithms for networks. In this previous work he found certain mathematical limitations that were negatively impacting algorithm performance, and in an effort to better understand the mathematical limitations in context, dove deeper into link prediction due to its importance as a testbed problem for ML algorithms.
‘”The reason why we got interested is because link prediction is one of these really important scientific tasks which is used to benchmark a lot of machine learning algorithms,” Seshadhri said. “What we were seeing was that the performance seemed to be really good… but we had an inkling that there seemed to be something off with this measurement. It feels like if you measured things in a different way, maybe you wouldn’t see such great results.”
Link prediction is based on the ML algorithm’s ability to carry out low dimensional vector embeddings, the process by which the algorithm represents the people within a network as a mathematical vector in space. All of the machine learning occurs as mathematical manipulations to those vectors.

AUC, which stands for “area under curve” and is the most common metric for measuring link prediction, gives ML algorithms a score from zero to one based on the algorithm’s performance.
In their research, the authors discovered that there are fundamental mathematical limitations to using low dimensional embeddings for link predictions, and that AUC can not measure these limitations. The inability to measure these limitations caused the authors to conclude that AUC does not accurately measure link prediction performance.
Seshadhri said these results call into question the widespread use of low dimensional vector embeddings in the ML field, considering the mathematical limitations that his research has surfaced on their performance.
Leading methods fall short
The discovery of AUC’s shortcomings led the researchers to create a new metric to better capture the limitations, which they call VCMPR. They used VCMPR to measure 12 ML algorithms chosen to be representative of the field, including algorithms such as DeepWalk, Node2vec, NetMF, GraphSage, and graph benchmark leader HOP-Rec, and found that the link prediction performance was worse using VCMPR as the metric rather than AUC.
“When we look at the VCMPR scores, we see that the scores of most of the leading methods out there are really poor,” Seshadhri said. “It looks like they’re actually not doing a good job when you measure things a different way.”
The results also showed that not only was performance lower across the board, some of the algorithms that performed worse than other algorithms when measured with AUC in turn performed better than the cohort with VCMPR, and vice versa.

Trustworthiness in machine learning
Seshadhri suggests that ML researchers use VCMPR to benchmark the link prediction performance of their algorithms, or at the very least stop using AUC as their measure. As metrics are so tightly connected to decision making in ML, using a flawed system to measure performance could lead to flawed decision making about which algorithms to employ in real world ML applications.
“Metrics are so closely tied to what we decide to deploy in the real world — people need to have some trust in that. If you have the wrong way of measuring, how can you trust the results?” Seshadri said. “This paper is in some sense cautionary: we have to be more careful about how we do our machine learning experiments, and we need to come up with a richer set of measures.”
In academia, using an accurate metric is crucial to creating progress in the ML field.
“This is in some sense a bit of a conundrum for scientific progress. A new result has to supposedly be better than everything previously, otherwise it’s not doing anything new — but that all depends on how you measure it.”
Beyond machine learning, there are researchers across a wide range of fields who use link prediction and ML to conduct their research, often with profound potential impact. For example, some biologists use link prediction to determine which proteins are likely to interact as a part of drug discovery. These biologists and other researchers outside of ML depend on the ML experts to create trustworthy tools, as they often cannot become ML experts themselves.
While he thinks these results may not be a huge surprise to those deeply involved in the field, he hopes that the larger community of ML researchers, and particularly graduate and Ph.D. students who use the current literature to learn best practices and common wisdom about the field, will take note of these results and take caution in their work. He sees this research that presents a skeptical view to be in somewhat contrast to a dominant philosophy in ML, which tends to accept a set of metrics and focuses on “pushing the bar” when it comes to progress in the field.
“It’s important that we have the skeptical view, are trying to understand deeper, and are constantly asking ourselves ‘Are we measuring things correctly?'”
This research was funded by the National Science Foundation and the Army Research Office. More

Soft robotics is the study of creating robots from soft materials, which has the advantage of flexibility and safety in human interactions. These robots are well-suited for applications ranging from medical devices to enhancing efficiency in various tasks. Additionally, using different forms of robotic movement may also serve us well in exploring the ocean or space, or doing certain jobs in those environments.
To broaden our understanding of locomotion, Richard Desatnik, who works in the labs of Philip LeDuc and Carmel Majidi at Carnegie Mellon University and collaborates with paleontologists from Europe, turns to the past. The team creates robots with the movement of ancient animals such as pleurocystitids, a sea creature that lived around 500 million years ago. Desatnik will present their findings from the process of building a soft robot based on pleurocystitids at the 68th Biophysical Society Annual Meeting, to be held February 10 — 14, 2024 in Philadelphia, Pennsylvania.
“We’ve learned a lot from modern creatures, but that’s only 1% of the animals that have existed during our planet’s history, and we want to see if there is something we can learn from the other 99% of creatures that once roamed the earth,” Desatnik said. He added, “there are animals that were very successful for millions of years and the reason they died out wasn’t from a lack of success from their biology — there may have been a massive environmental change or extinction event.”
Desatnik and colleagues started off with fossils of pleurocystitids, which are related to present-day sea stars and sea urchins but that had a muscular stem — a kind of tail — to move. They used CT scans to get a better idea of the 3D shape. Computer simulations suggested the ways it may have propelled itself through the water. Based on these data, they built a soft robot that mimics the prehistoric creature.
Their work suggests that a sweeping motion of the stem could have helped these animals glide along the ocean floor. They also found that a longer stem — which the fossil record suggests pleurocystitids developed over generations — could have made them faster without requiring much more energy.
These underwater soft robots may help in the future, “whether it’s geologic surveying, or fixing all the machinery that we have underwater,” Desatnik points out.
The researchers’ approach of using extinct animals to inform soft robotic design, which they call paleobionics, has the potential to further our understanding of evolution, biomechanics, and soft robot movements. More

With a brain the size of a pinhead, insects perform fantastic navigational feats. They avoid obstacles and move through small openings. How do they do this, with their limited brain power? Understanding the inner workings of an insect’s brain can help us in our search towards energy-efficient computing, physicist Elisabetta Chicca of the University of Groningen demonstrates with her most recent result: a robot that acts like an insect.
It’s not easy to make use of the images that come in through your eyes, when deciding what your feet or wings should do. A key aspect here is the apparent motion of things as you move. ‘Like when you’re on a train’, Chicca explains. ‘The trees nearby appear to move faster than the houses far away. Insects use this information to infer how far away things are. This works well when moving in a straight line, but reality is not that simple.
Moving in curves makes the problem too complex for insects. To keep things manageable for their limited brainpower, they adjust their behaviour: they fly in a straight line, make a turn, then make another straight line. Chicca explains: ‘What we learn from this is: if you don’t have enough resources, you can simplify the problem with your behaviour.’
Brains on wheels
In search of the neural mechanism that drives insect behaviour, PhD student Thorben Schoepe developed a model of its neuronal activity and a small robot that uses this model to navigate. All this was done under Chicca’s supervision, and in close collaboration with neurobiologist Martin Egelhaaf of Bielefeld University, who helped to identify the insects’ computational principles.
Schoepe’s model is based on one main principle: always steer towards the area with the least apparent motion. He had his robot drive through a long ‘corridor’ — consisting of two walls with a random print on it — and the robot centred in the middle of the corridor, as insects tend to do.
In other (virtual) environments, such as a space with obstacles or small openings, Schoepe’s model also showed similar behaviour to insects. ‘The model is so good’, Chicca concludes, ‘that once you set it up, it will perform in all kinds of environments. That’s the beauty of this result.’

Hardwired instead of learned
The fact that a robot can navigate in a realistic environment is not new. Rather, the model gives insight into how insects do the job, and how they manage to do things so efficiently. Chicca explains: ‘Much of Robotics is not concerned with efficiency. We humans tend to learn new tasks as we grow up and within Robotics, this is reflected in the current trend of machine learning. But insects are able to fly immediately from birth. An efficient way of doing that is hardwired in their brains.’
In a similar way, you could make computers more efficient. Chicca shows a chip that her research group has previously developed: a strip with a surface area that is smaller than a key on your keyboard. In the future, she hopes to incorporate this specific insect behaviour in a chip as well. She comments: ‘Instead of using a general-purpose computer with all its possibilities, you can build specific hardware; a tiny chip that does the job, keeping things much smaller and energy-efficient.’
Elisabetta Chicca is part of the Groningen Cognitive Systems and Materials Center (CogniGron). Its mission is to develop materials-centred systems paradigms for cognitive computing based on modelling and learning at all levels: from materials that can learn to devices, circuits, and algorithms. More

Children’s interest in, and competence perceptions of, mathematics are generally quite positive as they begin school, but turn less positive during the first three years. Changes in interest and self-concept are also associated with each other. In other words, if a child’s interest fades, so does their competence perception, and vice versa.
This is shown by a recent study from Finland exploring the development of children’s motivation for mathematics during the early school years, and how that development is associated with their mathematics competence. The researchers followed nearly three hundred children for three years.
“A significant observation was that both school beginners’ higher initial motivation, and less decline in motivation during the follow-up, predicted better competence in the third grade, after accounting for initial differences in competence,” says Professor Markku Niemivirta of the University of Eastern Finland.
There were no gender differences in school beginners’ motivation and competence, but at the end of the follow-up, girls’ motivation had, on average, declined more than that of boys.
Gendered development is starting to show
The study shows that children are able to assess their motivation for mathematics rather accurately already when beginning school. In addition, children’s assessments of their interest and competence are already differentiated, despite being closely related.
“It is only natural that children are more interested in things they feel good at. And vice versa, they may do better in something they’re interested in.”
On average however, school beginners’ positive motivation starts to decline during the early school years, and the scale of this decline is associated with later differences in competence. Although there are no gender differences in competence, girls’ more negative change in motivation on average reflects an unfortunate gendered development, the traces of which remain visible until much later.

Practices for maintaining interest and having experiences of success
Although the negative change observed in the study may partly reflect children’s more realistic self-assessment over time, the researchers suspect that a role is also played by mathematics gradually getting more difficult, and an emphasis being placed on performance.
“The observed association between a change in motivation and competence shows, however, the added value of positive interest and self-concept. It would be important to develop and apply teaching practices that support and maintain children’s interest in mathematics and strengthen their experiences of success,” Niemivirta points out.
In the three-year study conducted by the Motivation, Learning and Well-being research collective, MoLeWe, children assessed their interest in, and competence perceptions of, mathematics annually. Mathematics competence was assessed by tests and teacher evaluations. More

In several applications of computer vision, such as augmented reality and self-driving cars, estimating the distance between objects and the camera is an essential task. Depth from focus/defocus is one of the techniques that achieves such a process using the blur in the images as a clue. Depth from focus/defocus usually requires a stack of images of the same scene taken with different focus distances, a technique known as focal stack.
Over the past decade or so, scientists have proposed many different methods for depth from focus/defocus, most of which can be divided into two categories. The first category includes model-based methods, which use mathematical and optics models to estimate scene depth based on sharpness or blur. The main problem with such methods, however, is that they fail for texture-less surfaces which look virtually the same across the entire focal stack.
The second category includes learning-based methods, which can be trained to perform depth from focus/defocus efficiently, even for texture-less surfaces. However, these approaches fail if the camera settings used for an input focal stack are different from those used in the training dataset.
Overcoming these limitations now, a team of researchers from Japan has come up with an innovative method for depth from focus/defocus that simultaneously addresses the abovementioned issues. Their study, published in the International Journal of Computer Vision, was led by Yasuhiro Mukaigawa and Yuki Fujimura from Nara Institute of Science and Technology (NAIST), Japan.
The proposed technique, dubbed deep depth from focal stack (DDFS), combines model-based depth estimation with a learning framework to get the best of both the worlds. Inspired by a strategy used in stereo vision, DDFS involves establishing a ‘cost volume’ based on the input focal stack, the camera settings, and a lens defocus model. Simply put, the cost volume represents a set of depth hypotheses — potential depth values for each pixel — and an associated cost value calculated on the basis of consistency between images in the focal stack. “The cost volume imposes a constraint between the defocus images and scene depth, serving as an intermediate representation that enables depth estimation with different camera settings at training and test times,” explains Mukaigawa.
The DDFS method also employs an encoder-decoder network, a commonly used machine learning architecture. This network estimates the scene depth progressively in a coarse-to-fine fashion, using ‘cost aggregation’ at each stage for learning localized structures in the images adaptively.
The researchers compared the performance of DDFS with that of other state-of-the-art depth from focus/defocus methods. Notably, the proposed approach outperformed most methods in various metrics for several image datasets. Additional experiments on focal stacks captured with the research team’s camera further proved the potential of DDFS, making it useful even with only a few input images in the input stacks, unlike other techniques.
Overall, DDFS could serve as a promising approach for applications where depth estimation is required, including robotics, autonomous vehicles, 3D image reconstruction, virtual and augmented reality, and surveillance. “Our method with camera-setting invariance can help extend the applicability of learning-based depth estimation techniques,” concludes Mukaigawa.
Here’s hoping that this study paves the way to more capable computer vision systems. More

The properties of normal materials, such as stiffness and flexibility, are determined by the molecular composition of the material, but the properties of metamaterials are determined by the geometry of the structure from which they are built. Researchers design these structures digitally and then have it 3D-printed. The resulting metamaterials can exhibit unnatural and extreme properties. Researchers have, for instance, designed metamaterials that, despite being solid, behave like a fluid.
“Traditionally, designers use the materials available to them to design a new device or a machine. The problem with that is that the range of available material properties is limited. Some properties that we would like to have, just don’t exist in nature. Our approach is: tell us what you want to have as properties and we engineer an appropriate material with those properties. What you will then get, is not really a material but something in-between a structure and a material, a metamaterial,” says professor Amir Zadpoor of the Department of Biomechanical Engineering.
Inverse design
Such a material discovery process requires solving a so-called inverse problem: the problem of finding the geometry that gives rise to the properties you desire. Inverse problems are notoriously difficult to solve, which is where AI comes into the picture. TU Delft researchers have developed deep learning models that solve these inverse problems.
“Even when inverse problems were solved in the past, they have been limited by the simplifying assumption that the small-scale geometry can be made from an infinite number of building blocks. The problem with that assumption is that metamaterials are usually made by 3D-printing and real 3D-printers have a limited resolution, which limits the number of building blocks that fit within a given device,” says first author Dr. Helda Pahlavani.
The AI models developed by TU Delft researchers break new ground by bypassing any such simplifying assumptions. “So we can now simply ask: how many building blocks does your manufacturing technique allow you to accommodate in your device? The model then finds the geometry that gives you your desired properties for the number of building blocks that you can actually manufacture.”
Unlocking full potential
A major practical problem neglected in previous research, has been the durability of metamaterials. Most existing designs break once they are used a few times. That is because existing metamaterials design approaches do not take durability into account. “So far, it has been only about what properties can be achieved. Our study considers durability and selects the most durable designs from a large pool of design candidates. This makes our designs really practical and not just theoretical adventures,” says Zadpoor.
The possibilities of metamaterials seem endless, but the full potential is far from being realised, says assistant professor Mohammad J. Mirzaali, corresponding author of the publication. This is because finding the optimal design of a metamaterial is currently still largely based on intuition, involves trial and error and is therefore labour-intensive. Using an inverse design process, where the desired properties are the starting point of the design, is still very rare within the metamaterials field. “But we think the step we have taken, is revolutionary in the field of metamaterials. It could lead to all kinds of new applications.” There are possible applications in orthopaedic implants, surgical instruments, soft robots, adaptive mirrors, and exo-suits. More

Quantum computing has been hailed as a technology that can outperform classical computing in both speed and memory usage, potentially opening the way to making predictions of physical phenomena not previously possible.
Many see quantum computing’s advent as marking a paradigm shift from classical, or conventional, computing. Conventional computers process information in the form of digital bits (0s and 1s), while quantum computers deploy quantum bits (qubits) to store quantum information in values between 0 and 1. Under certain conditions this ability to process and store information in qubits can be used to design quantum algorithms that drastically outperform their classical counterparts. Notably, quantum’s ability to store information in values between 0 and 1 makes it difficult for classical computers to perfectly emulate quantum ones.
However, quantum computers are finicky and have a tendency to lose information. Moreover, even if information loss can be avoided, it is difficult to translate it into classical information — which is necessary to yield a useful computation.
Classical computers suffer from neither of those two problems. Moreover, cleverly devised classical algorithms can further exploit the twin challenges of information loss and translation to mimic a quantum computer with far fewer resources than previously thought — as recently reported in a research paper in the journal PRX Quantum.
The scientists’ results show that classical computing can be reconfigured to perform faster and more accurate calculations than state-of-the-art quantum computers.
This breakthrough was achieved with an algorithm that keeps only part of the information stored in the quantum state — and just enough to be able to accurately compute the final outcome.
“This work shows that there are many potential routes to improving computations, encompassing both classical and quantum approaches,” explains Dries Sels, an assistant professor in New York University’s Department of Physics and one of the paper’s authors. “Moreover, our work highlights how difficult it is to achieve quantum advantage with an error-prone quantum computer.”
In seeking ways to optimize classical computing, Sels and his colleagues at the Simons Foundation focused on a type of tensor network that faithfully represents the interactions between the qubits. Those types of networks have been notoriously hard to deal with, but recent advances in the field now allow these networks to be optimized with tools borrowed from statistical inference.

The authors compare the work of the algorithm to the compression of an image into a JPEG file, which allows large images to be stored using less space by eliminating information with barely perceivable loss in the quality of the image.
“Choosing different structures for the tensor network corresponds to choosing different forms of compression, like different formats for your image,” says the Flatiron Institute’s Joseph Tindall, who led the project. “We are successfully developing tools for working with a wide range of different tensor networks. This work reflects that, and we are confident that we will soon be raising the bar for quantum computing even further.”
The work was supported by the Flatiron Institute and a grant from the Air Force Office of Scientific Research (FA9550-21-1-0236). More

The past year has seen major advances in Large Language Models (LLMs) such as ChatGPT. The ability of these models to interpret and produce human text sources (and other sequence data) has implications for people in many areas of human activity. A new perspective paper in the journal Neuron argues that like many professionals, neuroscientists can either benefit from partnering with these powerful tools or risk being left behind.
In their previous studies, the authors showed that important preconditions are met to develop LLMs that can interpret and analyze neuroscientific data like ChatGPT interprets language. These AI models can be built for many different types of data, including neuroimaging, genetics, single-cell genomics, and even hand-written clinical reports.
In the traditional model of research, a scientist studies previous data on a topic, develops new hypotheses and tests them using experiments. Because of the massive amounts of data available, scientists often focus on a narrow field of research, such as neuroimaging or genetics. LLMs, however, can absorb more neuroscientific research than a single human ever could. In their Neuron paper, the authors argue that one day LLMs specialized in diverse areas of neuroscience could be used to communicate with one another to bridge siloed areas of neuroscience research, uncovering truths that would be impossible to find by humans alone. In the case of drug development, for example, an LLM specialized in genetics could be used along with a neuroimaging LLM to discover promising candidate molecules to stop neurodegeneration. The neuroscientist would direct these LLMs and verify their outputs.
Lead author Danilo Bzdok mentions the possibility that the scientist will, in certain cases, not always be able to fully understand the mechanism behind the biological processes discovered by these LLMs.
“We have to be open to the fact that certain things about the brain may be unknowable, or at least take a long time to understand,” he says. “Yet we might still generate insights from state-of-the-art LLMs and make clinical progress, even if we don’t fully grasp the way they reach conclusions.”
To realize the full potential of LLMs in neuroscience, Bzdok says scientists would need more infrastructure for data processing and storage than is available today at many research organizations. More importantly, it would take a cultural shift to a much more data-driven scientific approach, where studies that rely heavily on artificial intelligence and LLMs are published by leading journals and funded by public agencies. While the traditional model of strongly hypothesis-driven research remains key and is not going away, Bzdok says capitalizing on emerging LLM technologies might be important to spur the next generation of neurological treatments in cases where the old model has been less fruitful.
“To quote John Naisbitt, neuroscientists today are ‘drowning in information but starving for knowledge,'” he says. “Our ability to generate biomolecular data is eclipsing our ability to glean understanding from these systems. LLMs offer an answer to this problem. They may be able to extract, synergize and synthesize knowledge from and across neuroscience domains, a task that may or may not exceed human comprehension.” More