Computers Math

Chart captions that explain complex trends and patterns are important for improving a reader’s ability to comprehend and retain the data being presented. And for people with visual disabilities, the information in a caption often provides their only means of understanding the chart.
But writing effective, detailed captions is a labor-intensive process. While autocaptioning techniques can alleviate this burden, they often struggle to describe cognitive features that provide additional context.
To help people author high-quality chart captions, MIT researchers have developed a dataset to improve automatic captioning systems. Using this tool, researchers could teach a machine-learning model to vary the level of complexity and type of content included in a chart caption based on the needs of users.
The MIT researchers found that machine-learning models trained for autocaptioning with their dataset consistently generated captions that were precise, semantically rich, and described data trends and complex patterns. Quantitative and qualitative analyses revealed that their models captioned charts more effectively than other autocaptioning systems.
The team’s goal is to provide the dataset, called VisText, as a tool researchers can use as they work on the thorny problem of chart autocaptioning. These automatic systems could help provide captions for uncaptioned online charts and improve accessibility for people with visual disabilities, says co-lead author Angie Boggust, a graduate student in electrical engineering and computer science at MIT and member of the Visualization Group in the Computer Science and Artificial Intelligence Laboratory (CSAIL).
“We’ve tried to embed a lot of human values into our dataset so that when we and other researchers are building automatic chart-captioning systems, we don’t end up with models that aren’t what people want or need,” she says.

Boggust is joined on the paper by co-lead author and fellow graduate student Benny J. Tang and senior author Arvind Satyanarayan, associate professor of computer science at MIT who leads the Visualization Group in CSAIL. The research will be presented at the Annual Meeting of the Association for Computational Linguistics.
Human-centered analysis
The researchers were inspired to develop VisText from prior work in the Visualization Group that explored what makes a good chart caption. In that study, researchers found that sighted users and blind or low-vision users had different preferences for the complexity of semantic content in a caption.
The group wanted to bring that human-centered analysis into autocaptioning research. To do that, they developed VisText, a dataset of charts and associated captions that could be used to train machine-learning models to generate accurate, semantically rich, customizable captions.
Developing effective autocaptioning systems is no easy task. Existing machine-learning methods often try to caption charts the way they would an image, but people and models interpret natural images differently from how we read charts. Other techniques skip the visual content entirely and caption a chart using its underlying data table. However, such data tables are often not available after charts are published.

Given the shortfalls of using images and data tables, VisText also represents charts as scene graphs. Scene graphs, which can be extracted from a chart image, contain all the chart data but also include additional image context.
“A scene graph is like the best of both worlds — it contains almost all the information present in an image while being easier to extract from images than data tables. As it’s also text, we can leverage advances in modern large language models for captioning,” Tang explains.
They compiled a dataset that contains more than 12,000 charts — each represented as a data table, image, and scene graph — as well as associated captions. Each chart has two separate captions: a low-level caption that describes the chart’s construction (like its axis ranges) and a higher-level caption that describes statistics, relationships in the data, and complex trends.
The researchers generated low-level captions using an automated system and crowdsourced higher-level captions from human workers.
“Our captions were informed by two key pieces of prior research: existing guidelines on accessible descriptions of visual media and a conceptual model from our group for categorizing semantic content. This ensured that our captions featured important low-level chart elements like axes, scales, and units for readers with visual disabilities, while retaining human variability in how captions can be written,” says Tang.
Translating charts
Once they had gathered chart images and captions, the researchers used VisText to train five machine-learning models for autocaptioning. They wanted to see how each representation — image, data table, and scene graph — and combinations of the representations affected the quality of the caption.
“You can think about a chart captioning model like a model for language translation. But instead of saying, translate this German text to English, we are saying translate this ‘chart language’ to English,” Boggust says.
Their results showed that models trained with scene graphs performed as well or better than those trained using data tables. Since scene graphs are easier to extract from existing charts, the researchers argue that they might be a more useful representation.
They also trained models with low-level and high-level captions separately. This technique, known as semantic prefix tuning, enabled them to teach the model to vary the complexity of the caption’s content.
In addition, they conducted a qualitative examination of captions produced by their best-performing method and categorized six types of common errors. For instance, a directional error occurs if a model says a trend is decreasing when it is actually increasing.
This fine-grained, robust qualitative evaluation was important for understanding how the model was making its errors. For example, using quantitative methods, a directional error might incur the same penalty as a repetition error, where the model repeats the same word or phrase. But a directional error could be more misleading to a user than a repetition error. The qualitative analysis helped them understand these types of subtleties, Boggust says.
These sorts of errors also expose limitations of current models and raise ethical considerations that researchers must consider as they work to develop autocaptioning systems, she adds.
Generative machine-learning models, such as those that power ChatGPT, have been shown to hallucinate or give incorrect information that can be misleading. While there is a clear benefit to using these models for autocaptioning existing charts, it could lead to the spread of misinformation if charts are captioned incorrectly.
“Maybe this means that we don’t just caption everything in sight with AI. Instead, perhaps we provide these autocaptioning systems as authorship tools for people to edit. It is important to think about these ethical implications throughout the research process, not just at the end when we have a model to deploy,” she says.
Boggust, Tang, and their colleagues want to continue optimizing the models to reduce some common errors. They also want to expand the VisText dataset to include more charts, and more complex charts, such as those with stacked bars or multiple lines. And they would also like to gain insights into what these autocaptioning models are actually learning about chart data.
This research was supported, in part, by a Google Research Scholar Award, the National Science Foundation, the MLA@CSAIL Initiative, and the United States Air Force Research Laboratory. More

Children do better at maths when music is a key part of their lessons, an analysis of almost 50 years of research on the topic has revealed.
It is thought that music can make maths more enjoyable, keep students engaged and help any ease fear or anxiety they have about maths. Motivation may be increased and pupils may appreciate maths more, the peer-reviewed article in Educational Studies details.
Techniques for integrating music into maths lessons range from clapping to pieces with different rhythms when learning numbers and fractions, to using maths to design musical instruments.
Previous research has shown that children who are better at music also do better at maths. But whether teaching music to youngsters actually improves their maths has been less clear.
To find out more, Turkish researcher Dr. Ayça Akin, from the Department of Software Engineering, Antalya Belek University, searched academic databases for research on the topic published between 1975 and 2022.
She then combined the results of 55 studies from around the world, involving almost 78,000 young people from kindergarten pupils to university students, to come up with an answer.

Three types of musical intervention were included the meta-analysis: standardised music interventions (typical music lessons, in which children sing and listen to, and compose, music), instrumental musical interventions (lessons in which children learn how to play instruments, either individually or as part of a band) and music-maths integrated interventions, in which music is integrated into maths lessons.
Students took maths tests before and after taking part in the intervention and the change in their scores was compared with that of youngsters who didn’t take part in an intervention.
The use of music, whether in separate lessons or as part of maths classes, was associated with greater improvement in maths over time.
The integrated lessons had the biggest effect, with around 73% of students who had integrated lessons doing significantly better than youngsters who didn’t have any type of musical intervention.
Some 69% of students who learned how to play instruments and 58% of students who had normal music lessons improved more than pupils with no musical intervention.

The results also indicate that music helps more with learning arithmetic than other types of maths and has a bigger impact on younger pupils and those learning more basic mathematical concepts.
Dr Akin, who carried out the research while at Turkey’s National Ministry of Education and Antalya Belek University, points out that maths and music have much in common, such as the use of symbols symmetry. Both subjects also require abstract thought and quantitative reasoning.
Arithmetic may lend itself particularly well to being taught through music because core concepts, such as fractions and ratios, are also fundamental to music. For example, musical notes of different lengths can be represented as fractions and added together to create several bars of music.
Integrated lessons may be especially effective because they allow pupils to build connections between the maths and music and provide extra opportunities to explore, interpret and understand maths.
Plus, if they are more enjoyable than traditional maths lessons, any anxiety students feel about maths may be eased.
Limitations of the analysis include the relatively small number of studies available for inclusion. This meant it wasn’t possible to look at the effect of factors such as gender, socio-economic status and length of musical instruction on the results.
Dr Akin, who is now based at Antalya Belek University, concludes that while musical instruction overall has a small to moderate effect on achievement in maths, integrated lessons have a large impact.
She adds: “Encouraging mathematics and music teachers to plan lessons together could help ease students’ anxiety about mathematics, while also boosting achievement.” More

Even though our computers are now better than 15 years ago, they still malfunction between 11 and 20 per cent of the time, a new study from the University of Copenhagen and Roskilde University concludes. The researchers behind the study therefore find that there are major gains to be achieved for society by rethinking the systems and involving users more in their development.
An endlessly rotating beach ball, a program that crashes without saving data or systems that require illogical procedures or simply do not work. Unfortunately, struggling with computers is still a familiar situation for most of us. Tearing your hair out over computers that do not work remains very common among users, according to new Danish research.
In fact, so much that, on average, we waste between 11 and 20 per cent of our time in front of our computers on systems that do not work or that are so difficult to understand that we cannot perform the task we want to. And this is far from being good enough, says Professor Kasper Hornbæk, one of the researchers behind the study.
“It’s incredible that the figure is so high. However, most people experience frustration when using computers and can tell a horror story about an important PowerPoint presentation that was not saved or a system that crashed at a critical moment. Everyone knows that it is difficult to create IT systems that match people’s needs, but the figure should be much lower, and one thing that it shows is that ordinary people aren’t involved enough when the systems are developed,” he says.
Professor Morten Hertzum, the other researcher behind the study, emphasises that most frustrations are experienced in connection with the performance of completely ordinary tasks.
“The frustrations are not due to people using their computers for something highly advanced, but because they experience problems in their performance of everyday tasks. This makes it easier to involve users in identifying problems. But it also means that problems that are not identified and solved will probably frustrate a large number of users,” says Morten Hertzum.

The problems are only too recognisable
To examine this issue, the researchers have been assisted by 234 participants who spend between six and eight hours in front of a computer in their day-to-day work.
In one hour, the researchers told them to report the situations in which the computer would not work properly, or where the participants were frustrated about not being able to perform the task they wanted.
The problems most often experienced by the participants included that: “the system was slow,” “the system froze temporarily,” “the system crashed,” “it is difficult to find things.” The participants had backgrounds such as student, accountant, consultant, but several of them actually worked in the IT industry.
“A number of the participants in the survey were IT professionals, while most of the other participants were highly competent IT and computer users. Nevertheless, they encountered these problems, and it turns out that this involves some fundamental functions,” says Kasper Hornbæk.

The participants in the survey also responded that 84 per cent of the episodes had occurred before and that 87 per cent of the episodes could happen again. And, according to Kasper Hornbæk, we are having the same fundamental problems today that we had 15-20 years ago.
“The two biggest categories of problems are still about insufficient performance and lack of user-friendliness,” he says.
Morten Hertzum adds: “Our technology can do more today, and it has also become better, but, at the same time, we expect more from it. Even though downloads are faster now, they are often still experienced as frustratingly slow. ”
88 per cent use a computer at work
According to Statistics Denmark, 88 per cent of Danes used computers, laptops, smartphones, tablets or other mobile devices at work in 2018. In this context, the new study indicates that a half to a whole day of a normal working week may be wasted on computer problems.
“There is a lot of productivity lost in workplaces throughout Denmark because people are unable to perform their ordinary work because the computer is not running as it should. It also causes a lot of frustrations for the individual user,” says Kasper Hornbæk.
This means that there are major benefits to be gained for society if we experienced fewer problems in front of our computers. According to Kasper Hornbæk, the gains can, for example, be achieved if more resources are invested in rethinking how faults are presented to us on the computer.
“Part of the solution may be to shield us from knowing that the computer is working to solve a problem. In reality, there is no reason why we need to look at an incomprehensible box with commands or a frozen computer. The computer could easily solve the problems without displaying this, while it provided a back-up version of the system for us, so that we could continue to work with our tasks undisturbed,” says Kasper Hornbæk.
At the same time, IT developers should involve the users even more when designing the systems to make them as easy to use — and understand — as possible. For, according to the researcher, there are no poor IT users, only poor systems.
“When we’re all surrounded by IT systems that we’re cursing, it’s very healthy to ascertain that it’s probably not the users that are the problem, but those who make the systems. The study clearly shows that there is still much room for improvement, and we therefore hope that it can create more focus on making more user-friendly systems in the future,” concludes Kasper Hornbæk.
Facts: 234 participants, aged 10-69, participated in the survey. The majority of the participants spent between 6-8 hours a day in front of a computer. The participants reported an average of one computer problem or frustration per hour. The participants in the survey also responded that 84 per cent of the episodes had occurred before and that 87 per cent of the episodes could happen again. A large part of the problems concerned slow systems, systems that did not respond or crashed. The researchers have created a new version of a previous study conducted 15 years ago, which showed that the participants wasted as much as 40-50 per cent of their time on frustrations about the computer. The study has been conducted by Morten Hertzum from Roskilde University and Kasper Hornbæk from the University of Copenhagen. More

A new machine learning model uses electrocardiogram (ECG) readings to diagnose and classify heart attacks faster and more accurately than current approaches, according to a study led by University of Pittsburgh researchers that published today in Nature Medicine.
“When a patient comes into the hospital with chest pain, the first question we ask is whether the patient is having a heart attack or not. It seems like that should be straightforward, but when it’s not clear from the ECG, it can take up to 24 hours to complete additional tests,” said lead author Salah Al-Zaiti, Ph.D., R.N., associate professor in the Pitt School of Nursing and of emergency medicine and cardiology in the School of Medicine. “Our model helps address this major challenge by improving risk assessment so that patients can get appropriate care without delay.”
Among the peaks and valleys of an electrocardiogram, clinicians can easily recognize a distinct pattern that indicates the worst type of heart attack called STEMI. These severe episodes are caused by total blockage of a coronary artery and require immediate intervention to restore blood flow.
The problem is that almost two-thirds of heart attacks are caused by severe blockage, but do not have the telltale ECG pattern. The new tool helps detect subtle clues in the ECG that are difficult for clinicians to spot and improves classification of patients with chest pain.
The model was developed by co-author Ervin Sejdic, Ph.D., associate professor at The Edward S. Rogers Department of Electrical and Computer Engineering at the University of Toronto and the Research Chair in Artificial Intelligence for Health Outcomes at North York General Hospital in Toronto, with ECGs from 4,026 patients with chest pain at three hospitals in Pittsburgh. The model was then externally validated with 3,287 patients from a different hospital system.
The researchers compared their model to three gold standards for assessing cardiac events: experienced clinician interpretation of ECG, commercial ECG algorithms and the HEART score, which considers history at presentation — including pain and other symptoms — ECG interpretation, age, risk factors — such as smoking, diabetes, high cholesterol — and blood levels of a protein called troponin.

The model outperformed all three, accurately reclassifying 1 in 3 patients with chest pain as low, intermediate or high risk.
“In our wildest dreams, we hoped to match the accuracy of HEART, but we were surprised to find that our machine learning model based solely on ECG exceeded this score,” said Al-Zaiti.
According to co-author Christian Martin-Gill, M.D., M.P.H., chief of the Emergency Medical Services (EMS) division at UPMC, the algorithm will help EMS personnel and emergency department providers identify people having a heart attack and those with reduced blood flow to the heart in a much more robust way compared with traditional ECG analysis.
“This information can help guide EMS medical decisions such as initiating certain treatments in the field or alerting hospitals that a high-risk patient is incoming,” Martin-Gill added. “On the flip side, it’s also exciting that it can help identify low-risk patients who don’t need to go to a hospital with a specialized cardiac facility, which could improve prehospital triage.”
In the next phase of this research, the team is optimizing how the model will be deployed in partnership with City of Pittsburgh Bureau of Emergency Medical Services. Al-Zaiti said that they’re developing a cloud-based system that integrates with hospital command centers that receive ECG readings from EMS. The model will analyze the ECG and send back a risk assessment of the patient, guiding medical decisions in real-time.
Other authors who contributed to this research were Zeineb Bouzid, Stephanie Helman, M.S.N., R.N., Nathan Riek, Karina Kraevsky-Phillips, M.A., R.N., Gilles Clermont, M.D., Murat Akcakaya, Ph.D., Susan Sereika, Ph.D., Samir Saba, M.D., and Clifton Callaway, M.D., Ph.D., all of Pitt; Jessica Zègre-Hemsey, Ph.D., R.N., of the University of North Carolina; Ziad Faramand, M.D., of Northeast Georgia Health System; Mohammad Alrawashdeh, Ph.D., of Harvard Medical School; Richard Gregg, M.S., of Philips Healthcare; Peter Van Dam, of University Medical Center Utrecht; Stephen Smith, M.D., of Hennepin Healthcare and the University of Minnesota; and Yochai Birnbaum, M.D., of Baylor College of Medicine.
This research was supported by the National Heart, Lung, and Blood Institute, the National Center for Advancing Translational Sciences and the National Institute for Nursing Research through grants R01HL137761, UL1TR001857, K23NR017896 and KL2TR002490. More

Overheating of wearable skin-like electronic devices increases the risk of skin burning and results in performance degradation. A research team led by City University of Hong Kong (CityU) invented a photonic material-based “soft, ultrathin, radiative-cooling interface” that greatly enhances heat dissipation in devices, with temperature drops more than 56°C, offering an alternative for effective thermal management in advanced wearable electronics.
“Skin-like electronics are an emerging development in wearable devices,” said Dr Yu Xinge, Associate Professor in the Department of Biomedical Engineering (BME) at CityU, who co-led the research. “Effective thermal dissipation is crucial for maintaining sensing stability and a good user experience. Our ultrathin, soft, radiative-cooling interface, made of dedicatedly designed photonic material, provides a revolutionary solution to enable comfortable, long-term healthcare monitoring, and virtual and augmented reality (VR/AR) applications.”
In electronic devices, heat can be generated from both internal electronic components, when an electric current passes through a conductor, a process known as Joule heating, and external sources, such as sunlight and hot air. To cool down the devices, both radiative (i.e. thermal radiation — emitting heat energy from the device surface) and non-radiative (i.e. convection and conduction — losing heat to the layer of still air around the device and through direct contact with a cold object) heat-transfer processes can play a role.
However, the current technologies rely mostly on non-radiative means to dissipate the accumulated Joule heat. Moreover, the materials are usually bulky and rigid and offer limited portability, hindering the flexibility of wireless wearable devices.
To overcome these shortcomings, the research team developed a multifunctional composite polymer coating with both radiative and non-radiative cooling capacity without using electricity and with advances in wearability and stretchability.
The cooling interface coating is composed of hollow silicon dioxide (SiO2) microspheres, for improving infrared radiation, and titanium dioxide (TiO2) nanoparticles and fluorescent pigments, for enhancing solar reflection. It is less than a millimetre thick, lightweight (about 1.27g/cm2), and has robust mechanical flexibility.

When heat is generated in an electronic device, it flows to the cooling interface layer and dissipates to the ambient environment through both thermal radiation and air convection. The open space above the interface layer provides a cooler heat sink and an additional thermal exchange channel. The interface also exhibits excellent anti-ambient-interference capability due to its lower thermal conductivity, making it less susceptible to environmental heat sources that would affect the cooling effect and performance of the devices.
To examine its cooling capacity, the cooling interface layer was conformally coated onto a metallic resistance wire — a typical component causing a temperature rise in electronics. With a coating thickness of 75 μm, the temperature of the wire dropped from 140.5°C to 101.3°C, compared with uncoated wire at an input current of 0.5 A, and dropped to 84.2°C with 600 μm thickness, achieving a temperature drop of more than 56°C.
“It is necessary to keep the device temperature below 44°C to avoid skin burns,” said Dr Yu. “Our cooling interface can cool down the resistance wire from 64.1°C to 42.1°C with a 150 μm-thick coating.”
With the efficient passive radiative cooling capacity and the sophisticated nonradiative thermal design, the performance of several skin electronic devices developed by the team significantly improved, including the efficiency of wireless power transfer to light emitting diodes (LEDs) and the signal stability of a skin-interfaced wireless sensor under environmental obstructions (e.g. sunlight, hot wind and water).
“The intrinsically flexible nature of the cooling interface allows the electronic devices to undergo stable cooling even under extreme deformation, such as bending, twisting, folding and stretching many times,” said Dr Lei Dangyuan, Associate Professor in the Department of Materials Science and Engineering (MSE) at CityU, another co-leader of the study.
For example, their cooling-interface-integrated stretchable wireless-based epidermal lighting system showed higher illumination intensity and maintained stable performance even upon stretching from 5% to 50% 1,000 times.
The team submitted a US patent application for the invention. They won a Gold Medal, one of 36 awards won by CityU teams, the highest number of awards among local institutions at the 48th International Exhibition of Inventions Geneva, with their project named “Cooling Technology for Epidermal Electronics.”
Next, the research team will focus on practical applications of the cooling interfaces for advanced thermal management of wearable electronics in the healthcare monitoring, wireless communications and VR/AR fields. More

A savvy hacker can obtain secret information, such as a password, by observing a computer program’s behavior, like how much time that program spends accessing the computer’s memory.
Security approaches that completely block these “side-channel attacks” are so computationally expensive that they aren’t feasible for many real-world systems. Instead, engineers often apply what are known as obfuscation schemes that seek to limit, but not eliminate, an attacker’s ability to learn secret information.
To help engineers and scientists better understand the effectiveness of different obfuscation schemes, MIT researchers created a framework to quantitatively evaluate how much information an attacker could learn from a victim program with an obfuscation scheme in place.
Their framework, called Metior, allows the user to study how different victim programs, attacker strategies, and obfuscation scheme configurations affect the amount of sensitive information that is leaked. The framework could be used by engineers who develop microprocessors to evaluate the effectiveness of multiple security schemes and determine which architecture is most promising early in the chip design process.
“Metior helps us recognize that we shouldn’t look at these security schemes in isolation. It is very tempting to analyze the effectiveness of an obfuscation scheme for one particular victim, but this doesn’t help us understand why these attacks work. Looking at things from a higher level gives us a more holistic picture of what is actually going on,” says Peter Deutsch, a graduate student and lead author of an open-access paper on Metior.
Deutsch’s co-authors include Weon Taek Na, an MIT graduate student in electrical engineering and computer science; Thomas Bourgeat PhD ’23, an assistant professor at the Swiss Federal Institute of Technology (EPFL); Joel Emer, an MIT professor of the practice in computer science and electrical engineering; and senior author Mengjia Yan, the Homer A. Burnell Career Development Assistant Professor of Electrical Engineering and Computer Science (EECS) at MIT and a member of the Computer Science and Artificial Intelligence Laboratory (CSAIL). The research was presented last week at the International Symposium on Computer Architecture.

Illuminating obfuscation
While there are many obfuscation schemes, popular approaches typically work by adding some randomization to the victim’s behavior to make it harder for an attacker to learn secrets. For instance, perhaps an obfuscation scheme involves a program accessing additional areas of the computer memory, rather than only the area it needs to access, to confuse an attacker. Others adjust how often a victim accesses memory or another a shared resource so an attacker has trouble seeing clear patterns.
But while these approaches make it harder for an attacker to succeed, some amount of information from the victim still “leaks” out. Yan and her team want to know how much.
They had previously developed CaSA, a tool to quantify the amount of information leaked by one particular type of obfuscation scheme. But with Metior, they had more ambitious goals. The team wanted to derive a unified model that could be used to analyze any obfuscation scheme — even schemes that haven’t been developed yet.
To achieve that goal, they designed Metior to map the flow of information through an obfuscation scheme into random variables. For instance, the model maps the way a victim and an attacker access shared structures on a computer chip, like memory, into a mathematical formulation.

One Metior derives that mathematical representation, the framework uses techniques from information theory to understand how the attacker can learn information from the victim. With those pieces in place, Metior can quantify how likely it is for an attacker to successfully guess the victim’s secret information.
“We take all of the nitty-gritty elements of this microarchitectural side-channel and map it down to, essentially, a math problem. Once we do that, we can explore a lot of different strategies and better understand how making small tweaks can help you defend against information leaks,” Deutsch says.
Surprising insights
They applied Metior in three case studies to compare attack strategies and analyze the information leakage from state-of-the-art obfuscation schemes. Through their evaluations, they saw how Metior can identify interesting behaviors that weren’t fully understood before.
For instance, a prior analysis determined that a certain type of side-channel attack, called probabilistic prime and probe, was successful because this sophisticated attack includes a preliminary step where it profiles a victim system to understand its defenses.
Using Metior, they show that this advanced attack actually works no better than a simple, generic attack and that it exploits different victim behaviors than researchers previously thought.
Moving forward, the researchers want to continue enhancing Metior so the framework can analyze even very complicated obfuscation schemes in a more efficient manner. They also want to study additional obfuscation schemes and types of victim programs, as well as conduct more detailed analyses of the most popular defenses.
Ultimately, the researchers hope this work inspires others to study microarchitectural security evaluation methodologies that can be applied early in the chip design process.
“Any kind of microprocessor development is extraordinarily expensive and complicated, and design resources are extremely scarce. Having a way to evaluate the value of a security feature is extremely important before a company commits to microprocessor development. This is what Metior allows them to do in a very general way,” Emer says.
This research is funded, in part, by the National Science Foundation, the Air Force Office of Scientific Research, Intel, and the MIT RSC Research Fund. More

Imagine strapping on a virtual reality headset and “walking” through a long-gone neighborhood in your city — seeing the streets and buildings as they appeared decades ago.
That’s a very real possibility now that researchers have developed a method to create 3D digital models of historic neighborhoods using machine learning and historic Sanborn Fire Insurance maps.
But the digital models will be more than just a novelty — they will give researchers a resource to conduct studies that would have been nearly impossible before, such as estimating the economic loss caused by the demolition of historic neighborhoods.
“The story here is we now have the ability to unlock the wealth of data that is embedded in these Sanborn fire atlases,” said Harvey Miller, co-author of the study and professor of geography at The Ohio State University.
“It enables a whole new approach to urban historical research that we could never have imagined before machine learning. It is a game changer.”
The study was published today (June 28, 2023) in the journal PLOS ONE.

This research begins with the Sanborn maps, which were created to allow fire insurance companies to assess their liability in about 12,000 cities and towns in the United States during the 19th and 20th centuries. In larger cities, they were often updated regularly, said Miller, who is director of Ohio State’s Center for Urban and Regional Analysis (CURA).
The problem for researchers was that trying to manually collect usable data from these maps was tedious and time-consuming — at least until the maps were digitized. Digital versions are now available from the Library of Congress.
Study co-author Yue Lin, a doctoral student in geography at Ohio State, developed machine learning tools that can extract details about individual buildings from the maps, including their locations and footprints, the number of floors, their construction materials and their primary use, such as dwelling or business.
“We are able to get a very good idea of what the buildings look like from data we get from the Sanborn maps,” Lin said.
The researchers tested their machine learning technique on two adjacent neighborhoods on the near east side of Columbus, Ohio, that were largely destroyed in the 1960s to make way for the construction of I-70.

One of the neighborhoods, Hanford Village, was developed in 1946 to house returning Black veterans of World War II.
“The GI bill gave returning veterans funds to purchase homes, but they could only be used on new builds,” said study co-author Gerika Logan, outreach coordinator of CURA. “So most of the homes were lost to the highway not long after they were built.”
The other neighborhood in the study was Driving Park, which also housed a thriving Black community until I-70 split it in two.
The researchers used 13 Sanborn maps for the two neighborhoods produced in 1961, just before I-70 was built. Machine learning techniques were able to extract the data from the maps and create digital models.
Comparing data from the Sanford maps to today showed that a total of 380 buildings were demolished in the two neighborhoods for the highway, including 286 houses, 86 garages, five apartments and three stores.
Analysis of the results showed that the machine learning model was very accurate in recreating the information contained in the maps — about 90% accurate for building footprints and construction materials.
“The accuracy was impressive. We can actually get a visual sense of what these neighborhoods looked like that wouldn’t be possible in any other way,” Miller said.
“We want to get to the point in this project where we can give people virtual reality headsets and let them walk down the street as it was in 1960 or 1940 or perhaps even 1881.”
Using the machine learning techniques developed for this study, researchers could develop similar 3D models for nearly any of the 12,000 cities and towns that have Sanborn maps, Miller said.
This will allow researchers to re-create neighborhoods lost to natural disasters like floods, as well as urban renewal, depopulation and other types of change.
Because the Sanborn maps include information on businesses that occupied specific buildings, researchers could re-create digital neighborhoods to determine the economic impact of losing them to urban renewal or other factors. Another possibility would be to study how replacing homes with highways that absorbed the sun’s heat affected the urban heat island effect.
“There’s a lot of different types of research that can be done. This will be a tremendous resource for urban historians and a variety of other researchers,” Miller said.
“Making these 3D digital models and being able to reconstruct buildings adds so much more than what you could show in a chart, graph, table or traditional map. There’s just incredible potential here.” More

Engineers from Rice University and the University of Maryland have created full-motion video technology that could potentially be used to make cameras that peer through fog, smoke, driving rain, murky water, skin, bone and other media that reflect scattered light and obscure objects from view.
“Imaging through scattering media is the ‘holy grail problem’ in optical imaging at this point,” said Rice’s Ashok Veeraraghavan, co-corresponding author of an open-access study published today in Science Advances. “Scattering is what makes light — which has lower wavelength, and therefore gives much better spatial resolution — unusable in many, many scenarios. If you can undo the effects of scattering, then imaging just goes so much further.”
Veeraraghavan’s lab collaborated with the research group of Maryland co-corresponding author Christopher Metzler to create a technology they named NeuWS, which is an acronym for “neural wavefront shaping,” the technology’s core technique.
“If you ask people who are working on autonomous driving vehicles about the biggest challenges they face, they’ll say, ‘Bad weather. We can’t do good imaging in bad weather.'” Veeraraghavan said. “They are saying ‘bad weather,’ but what they mean, in technical terms, is light scattering. If you ask biologists about the biggest challenges in microscopy, they’ll say, ‘We can’t image deep tissue in vivo.’ They’re saying ‘deep tissue’ and ‘in vivo,’ but what they actually mean is that skin and other layers of tissue they want to see through, are scattering light. If you ask underwater photographers about their biggest challenge, they’ll say, ‘I can only image things that are close to me.’ What they actually mean is light scatters in water, and therefore doesn’t go deep enough for them to focus on things that are far away.
“In all of these circumstances, and others, the real technical problem is scattering,” Veeraraghavan said.
He said NeuWS could potentially be used to overcome scattering in those scenarios and others.

“This is a big step forward for us, in terms of solving this in a way that’s potentially practical,” he said. “There’s a lot of work to be done before we can actually build prototypes in each of those application domains, but the approach we have demonstrated could traverse them.”
Conceptually, NeuWS is based on the principle that light waves are complex mathematical quantities with two key properties that can be computed for any given location. The first, magnitude, is the amount of energy the wave carries at the location, and the second is phase, which is the wave’s state of oscillation at the location. Metzler and Veeraraghavan said measuring phase is critical for overcoming scattering, but it is impractical to measure directly because of the high-frequency of optical light.
So they instead measure incoming light as “wavefronts” — single measurements that contain both phase and intensity information — and use backend processing to rapidly decipher phase information from several hundred wavefront measurements per second.
“The technical challenge is finding a way to rapidly measure phase information,” said Metzler, an assistant professor of computer science at Maryland and “triple Owl” Rice alum who earned his Ph.D., masters and bachelors degrees in electrical and computer engineering from Rice in 2019, 2014 and 2013 respectively. Metzler was at Rice University during the development of an earlier iteration of wavefront-processing technology called WISH that Veeraraghavan and colleagues published in 2020.
“WISH tackled the same problem, but it worked under the assumption that everything was static and nice,” Veeraraghavan said. “In the real world, of course, things change all of the time.”
With NeuWS, he said, the idea is to not only undo the effects of scattering, but to undo them fast enough so the scattering media itself doesn’t change during the measurement.

“Instead of measuring the state of oscillation itself, you measure its correlation with known wavefronts,” Veeraraghavan said. “You take a known wavefront, you interfere that with the unknown wavefront and you measure the interference pattern produced by the two. That is the correlation between those two wavefronts.”
Metzler used the analogy of looking at the North Star at night through a haze of clouds. “If I know what the North Star is supposed to look like, and I can tell it is blurred in a particular way, then that tells me how everything else will be blurred.”
Veerarghavan said, “It’s not a comparison, it’s a correlation, and if you measure at least three such correlations, you can uniquely recover the unknown wavefront.”
State-of-the-art spatial light modulators can make several hundred such measurements per minute, and Veeraraghavan, Metzler and colleagues showed they could use a modulator and their computational method to capture video of moving objects that were obscured from view by intervening scattering media.
“This is the first step, the proof-of principle that this technology can correct for light scattering in real time,” said Rice’s Haiyun Guo, one of the study’s lead authors and a Ph.D. student in Veeraraghavan’s research group.
In one set of experiments, for example, a microscope slide containing a printed image of an owl or a turtle was spun on a spindle and filmed by an overhead camera. Light-scattering media were placed between the camera and target slide, and the researchers measured NeuWS ability to correct for light-scattering. Examples of scattering media included onion skin, slides coated with nail polish, slices of chicken breast tissue and light-diffusing films. For each of these, the experiments showed NeuWS could correct for light scattering and produce clear video of the spinning figures.
“We developed algorithms that allow us to continuously estimate both the scattering and the scene,” Metzler said. “That’s what allows us to do this, and we do it with mathematical machinery called neural representation that allows it to be both efficient and fast.”
NeuWS rapidly modulates light from incoming wavefronts to create several slightly altered phase measurements. The altered phases are then fed directly into a 16,000 parameter neural network that quickly computes the necessary correlations to recover the wavefront’s original phase information.
“The neural networks allow it to be faster by allowing us to design algorithms that require fewer measurements,” Veeraraghavan said.
Metzler said, “That’s actually the biggest selling point. Fewer measurements, basically, means we need much less capture time. It’s what allows us to capture video rather than still frames.”
The research was supported by the Air Force Office of Scientific Research (FA9550- 22-1-0208), the National Science Foundation (1652633, 1730574, 1648451) and the National Institutes of Health (DE032051), and partial funding for open access was provided by the University of Maryland Libraries’ Open Access Publishing Fund. More