Computers Math

Systems designed to detect deepfakes — videos that manipulate real-life footage via artificial intelligence — can be deceived, computer scientists showed for the first time at the WACV 2021 conference which took place online Jan. 5 to 9, 2021.
Researchers showed detectors can be defeated by inserting inputs called adversarial examples into every video frame. The adversarial examples are slightly manipulated inputs which cause artificial intelligence systems such as machine learning models to make a mistake. In addition, the team showed that the attack still works after videos are compressed.
“Our work shows that attacks on deepfake detectors could be a real-world threat,” said Shehzeen Hussain, a UC San Diego computer engineering Ph.D. student and first co-author on the WACV paper. “More alarmingly, we demonstrate that it’s possible to craft robust adversarial deepfakes in even when an adversary may not be aware of the inner workings of the machine learning model used by the detector.”
In deepfakes, a subject’s face is modified in order to create convincingly realistic footage of events that never actually happened. As a result, typical deepfake detectors focus on the face in videos: first tracking it and then passing on the cropped face data to a neural network that determines whether it is real or fake. For example, eye blinking is not reproduced well in deepfakes, so detectors focus on eye movements as one way to make that determination. State-of-the-art Deepfake detectors rely on machine learning models for identifying fake videos.
The extensive spread of fake videos through social media platforms has raised significant concerns worldwide, particularly hampering the credibility of digital media, the researchers point out. “”If the attackers have some knowledge of the detection system, they can design inputs to target the blind spots of the detector and bypass it,” ” said Paarth Neekhara, the paper’s other first coauthor and a UC San Diego computer science student.
Researchers created an adversarial example for every face in a video frame. But while standard operations such as compressing and resizing video usually remove adversarial examples from an image, these examples are built to withstand these processes. The attack algorithm does this by estimating over a set of input transformations how the model ranks images as real or fake. From there, it uses this estimation to transform images in such a way that the adversarial image remains effective even after compression and decompression.??

advertisement

The modified version of the face is then inserted in all the video frames. The process is then repeated for all frames in the video to create a deepfake video. The attack can also be applied on detectors that operate on entire video frames as opposed to just face crops.
The team declined to release their code so it wouldn’t be used by hostile parties.
High success rate
Researchers tested their attacks in two scenarios: one where the attackers have complete access to the detector model, including the face extraction pipeline and the architecture and parameters of the classification model; and one where attackers can only query the machine 
 learning model to figure out the probabilities of a frame being classified as real or fake. In the first scenario, the attack’s success rate is above 99 percent for uncompressed videos. For compressed videos, it was 84.96 percent. In the second scenario, the success rate was 86.43 percent for uncompressed and 78.33 percent for compressed videos. This is the first work which demonstrates successful attacks on state-of-the-art deepfake detectors.
“To use these deepfake detectors in practice, we argue that it is essential to evaluate them against an adaptive adversary who is aware of these defenses and is intentionally trying to foil these defenses,”? the researchers write. “We show that the current state of the art methods for deepfake detection can be easily bypassed if the adversary has complete or even partial knowledge of the detector.”
To improve detectors, researchers recommend an approach similar to what is known as adversarial training: during training, an adaptive adversary continues to generate new deepfakes that can bypass the current state of the art detector; and the detector continues improving in order to detect the new deepfakes.
Adversarial Deepfakes: Evaluating Vulnerability of Deepfake Detectors to Adversarial Examples
*Shehzeen Hussain, Malhar Jere, Farinaz Koushanfar, Department of Electrical and Computer Engineering, UC San Diego Paarth Neekhara, Julian McAuley, Department of Computer Science and Engineering, UC San Diego More

Floating sea macro-litter is a threat to the conservation of marine ecosystems worldwide. The largest density of floating litter is in the great ocean gyres — systems of circular currents that spin and catch litter — but the polluting waste is abundant in coastal waters and semi closed seas such as the Mediterranean.
MARLIT, an open access web app based on an algorithm designed with deep learning techniques, will enable the detection and quantification of floating plastics in the sea with a reliability over 80%, according to a study published in the journal Environmental Pollution and carried out by experts of the Faculty of Biology and the Biodiversity Research Institute of the University of Barcelona (IRBio).
This methodology results from the analysis through artificial intelligence techniques of more than 3,800 aerial images of the Mediterranean coast in Catalonia, and it will allow researchers to make progress in the assessment of the presence, density and distribution of the plastic pollutants in the seas and oceans worldwide. Among the participants in the study, published in the journal Environmental Pollution, are the experts of the Consolidated Research Group on Large Marine Vertebrates of the UB and IRBio, and the Research Group on Biostatistics and Bioinformatics (GRBIO) of the UB, integrated in the Bioinformatics Barcelona platform (BIB).
Litter that floats and pollutes the ocean
Historically, direct observations (boats, planes, etc.) are the base for the common methodology to assess the impact of floating marine macro-litter (FMML). However, the great ocean area and the volume of data make it hard for the researchers to advance with the monitoring studies.
“Automatic aerial photography techniques combined with analytical algorithms are more efficient protocols for the control and study of this kind of pollutants,” notes Odei Garcia-Garin, first author of the article and member of the CRG on Large Marine Mammals, led by Professor Àlex Aguilar.

advertisement

“However,” he continues, “automated remote sensing of these materials is at an early stage. There are several factors in the ocean (waves, wind, clouds, etc.) that harden the detection of floating litter automatically with the aerial images of the marine surface. This is why there are only a few studies that made the effort to work on algorithms to apply to this new research context.”
The experts designed a new algorithm to automate the quantification of floating plastics in the sea through aerial photographs by applying the deep learning techniques, automatic learning methodology with artificial neuronal networks able to learn and take the learning to higher levels.
“The great amount of images of the marine surface obtained by drones and planes in monitoring campaigns on marine litter -also in experimental studies with known floating objects- enabled us to develop and test a new algorithm that reaches a 80% of precision in the remote sensing of floating marine macro-litter,” notes Garcia-Garin, member of the Department of Evolutionary Biology, Ecology and Environmental Sciences of the UB and IRBio.
Preservation of the oceans with deep learning techniques
The new algorithm has been implemented to MARLIT, an open access web app described in the article and which is available to all managers and professionals in the study of the detection and quantification of floating marine macro-litter with aerial images. In particular, this is a proof of concept based on an R Shiny package, a methodological innovation with great interest to speed up the monitoring procedures of floating marine macro-litter.
MARLIT enables the analysis of images individually, as well as to divide them into several segments, according to the user’s guidelines, identify the presence of floating litter in each certain area and estimate their density with the image metadata (height, resolution). In the future, it is expected to adapt the app to a remote sensor (for instance, a drone) to automate the remote sensing process.
At a European level, the EU Marine Strategy Framework Directive indicates the application of FMML monitoring techniques to fulfill the continuous assessment of the environmental state of the marine environment. “Therefore, the automatization of monitoring processes and the use of apps such as MARLIT would ease the member states’ fulfilment of the directive,” conclude the authors of the study. More

A new machine-learning framework uses reported test results and death rates to calculate estimates of the actual number of current COVID-19 infections within all 50 U.S. states and 50 countries. Jungsik Noh and Gaudenz Danuser of the University of Texas Southwestern Medical Center present these findings in the open-access journal PLOS ONE on February 8, 2021.
During the ongoing pandemic, U.S. states and many countries have reported daily counts of COVID-19 infections and deaths confirmed by testing. However, many infections have gone undetected, resulting in under-counting of the total number of people currently infected at any given point in time — an important metric to guide public health efforts.
Now, Noh and Danuser have developed a computational model that uses machine-learning strategies to estimate the actual daily number of current infections for all 50 U.S. states and the 50 most-infected countries. To make the calculations, the model draws on previously published pandemic parameters and publicly available daily data on confirmed cases and deaths. Visualizations of these daily estimates are freely available online.
The model’s estimates indicate severe undercounting of cases across the U.S. and worldwide. The cumulative number of actual cases in 9 out of 50 countries is estimated to be at least five times higher than confirmed cases. Within the U.S., estimates of the cumulative number of actual cases within states were in line with the results of an antibody testing study conducted in 46 states.
For some countries, such as the U.S., Belgium, and the U.K., estimates indicate that more than 20 percent of the total population has experienced infection. As of January 31, 2021, some U.S. states — including Pennsylvania, Arizona, and Florida — have currently active cases totaling more than 5 percent of the state’s entire population. In Washington, the active cases were estimated to be one percent of the population that day.
Looking ahead, the model has been estimating current COVID-19 case counts within communities, which could help inform contact-tracing and other public health efforts.
The authors add: “Given that the confirmed cases only capture the tip of the iceberg in the middle of the pandemic, the estimated sizes of current infections in this study provide crucial information to determine the regional severity of COVID-19 that can be misguided by the confirmed cases.”

Story Source:
Materials provided by PLOS. Note: Content may be edited for style and length. More

Cold Spring Harbor Laboratory (CSHL) Assistant Professor Peter Koo and collaborator Matt Ploenzke reported a way to train machines to predict the function of DNA sequences. They used “neural nets,” a type of artificial intelligence (AI) typically used to classify images. Teaching the neural net to predict the function of short stretches of DNA allowed it to work up to deciphering larger patterns. The researchers hope to analyze more complex DNA sequences that regulate gene activity critical to development and disease.
Machine-learning researchers can train a brain-like “neural net” computer to recognize objects, such as cats or airplanes, by showing it many images of each. Testing the success of training requires showing the machine a new picture of a cat or an airplane and seeing if it classifies it correctly. But, when researchers apply this technology to analyzing DNA patterns, they have a problem. Humans can’t recognize the patterns, so they may not be able to tell if the computer identifies the right thing. Neural nets learn and make decisions independently of their human programmers. Researchers refer to this hidden process as a “black box.” It is hard to trust the machine’s outputs if we don’t know what is happening in the box.
Koo and his team fed DNA (genomic) sequences into a specific kind of neural network called a convolutional neural network (CNN), which resembles how animal brains process images. Koo says:
“It can be quite easy to interpret these neural networks because they’ll just point to, let’s say, whiskers of a cat. And so that’s why it’s a cat versus an airplane. In genomics, it’s not so straightforward because genomic sequences aren’t in a form where humans really understand any of the patterns that these neural networks point to.”
Koo’s research, reported in the journal Nature Machine Intelligence, introduced a new method to teach important DNA patterns to one layer of his CNN. This allowed his neural network to build on the data to identify more complex patterns. Koo’s discovery makes it possible to peek inside the black box and identify some key features that lead to the computer’s decision-making process.
But Koo has a larger purpose in mind for the field of artificial intelligence. There are two ways to improve a neural net: interpretability and robustness. Interpretability refers to the ability of humans to decipher why machines give a certain prediction. The ability to produce an answer even with mistakes in the data is called robustness. Usually, researchers focus on one or the other. Koo says:
“What my research is trying to do is bridge these two together because I don’t think they’re separate entities. I think that we get better interpretability if our models are more robust.”
Koo hopes that if a machine can find robust and interpretable DNA patterns related to gene regulation, it will help geneticists understand how mutations affect cancer and other diseases.

Story Source:
Materials provided by Cold Spring Harbor Laboratory. Original written by Jasmine Lee. Note: Content may be edited for style and length. More

Researchers have identified a new form of magnetism in so-called magnetic graphene, which could point the way toward understanding superconductivity in this unusual type of material.
The researchers, led by the University of Cambridge, were able to control the conductivity and magnetism of iron thiophosphate (FePS3), a two-dimensional material which undergoes a transition from an insulator to a metal when compressed. This class of magnetic materials offers new routes to understanding the physics of new magnetic states and superconductivity.
Using new high-pressure techniques, the researchers have shown what happens to magnetic graphene during the transition from insulator to conductor and into its unconventional metallic state, realised only under ultra-high pressure conditions. When the material becomes metallic, it remains magnetic, which is contrary to previous results and provides clues as to how the electrical conduction in the metallic phase works. The newly discovered high-pressure magnetic phase likely forms a precursor to superconductivity so understanding its mechanisms is vital.
Their results, published in the journal Physical Review X, also suggest a way that new materials could be engineered to have combined conduction and magnetic properties, which could be useful in the development of new technologies such as spintronics, which could transform the way in which computers process information.
Properties of matter can alter dramatically with changing dimensionality. For example, graphene, carbon nanotubes, graphite and diamond are all made of carbon atoms, but have very different properties due to their different structure and dimensionality.
“But imagine if you were also able to change all of these properties by adding magnetism,” said first author Dr Matthew Coak, who is jointly based at Cambridge’s Cavendish Laboratory and the University of Warwick. “A material which could be mechanically flexible and form a new kind of circuit to store information and perform computation. This is why these materials are so interesting, and because they drastically change their properties when put under pressure so we can control their behaviour.”
In a previous study by Sebastian Haines of Cambridge’s Cavendish Laboratory and the Department of Earth Sciences, researchers established that the material becomes a metal at high pressure, and outlined how the crystal structure and arrangement of atoms in the layers of this 2D material change through the transition.

advertisement

“The missing piece has remained however, the magnetism,” said Coak. “With no experimental techniques able to probe the signatures of magnetism in this material at pressures this high, our international team had to develop and test our own new techniques to make it possible.”
The researchers used new techniques to measure the magnetic structure up to record-breaking high pressures, using specially designed diamond anvils and neutrons to act as the probe of magnetism. They were then able to follow the evolution of the magnetism into the metallic state.
“To our surprise, we found that the magnetism survives and is in some ways strengthened,” co-author Dr Siddharth Saxena, group leader at the Cavendish Laboratory. “This is unexpected, as the newly-freely-roaming electrons in a newly conducting material can no longer be locked to their parent iron atoms, generating magnetic moments there — unless the conduction is coming from an unexpected source.”
In their previous paper, the researchers showed these electrons were ‘frozen’ in a sense. But when they made them flow or move, they started interacting more and more. The magnetism survives, but gets modified into new forms, giving rise to new quantum properties in a new type of magnetic metal.
How a material behaves, whether conductor or insulator, is mostly based on how the electrons, or charge, move around. However, the ‘spin’ of the electrons has been shown to be the source of magnetism. Spin makes electrons behave a bit like tiny bar magnets and point a certain way. Magnetism from the arrangement of electron spins is used in most memory devices: harnessing and controlling it is important for developing new technologies such as spintronics, which could transform the way in which computers process information.

advertisement

“The combination of the two, the charge and the spin, is key to how this material behaves,” said co-author Dr David Jarvis from the Institut Laue-Langevin, France, who carried out this work as the basis of his PhD studies at the Cavendish Laboratory. “Finding this sort of quantum multi-functionality is another leap forward in the study of these materials.”
“We don’t know exactly what’s happening at the quantum level, but at the same time, we can manipulate it,” said Saxena. “It’s like those famous ‘unknown unknowns’: we’ve opened up a new door to properties of quantum information, but we don’t yet know what those properties might be.”
There are more potential chemical compounds to synthesise than could ever be fully explored and characterised. But by carefully selecting and tuning materials with special properties, it is possible to show the way towards the creation of compounds and systems, but without having to apply huge amounts of pressure.
Additionally, gaining fundamental understanding of phenomena such as low-dimensional magnetism and superconductivity allows researchers to make the next leaps in materials science and engineering, with particular potential in energy efficiency, generation and storage.
As for the case of magnetic graphene, the researchers next plan to continue the search for superconductivity within this unique material. “Now that we have some idea what happens to this material at high pressure, we can make some predictions about what might happen if we try to tune its properties through adding free electrons by compressing it further,” said Coak.
“The thing we’re chasing is superconductivity,” said Saxena. “If we can find a type of superconductivity that’s related to magnetism in a two-dimensional material, it could give us a shot at solving a problem that’s gone back decades.” More

An important class of challenging computational problems, with applications in graph theory, neural networks, artificial intelligence and error-correcting codes can be solved by multiplying light signals, according to researchers from the University of Cambridge and Skolkovo Institute of Science and Technology in Russia.
In a paper published in the journal Physical Review Letters, they propose a new type of computation that could revolutionise analogue computing by dramatically reducing the number of light signals needed while simplifying the search for the best mathematical solutions, allowing for ultra-fast optical computers.
Optical or photonic computing uses photons produced by lasers or diodes for computation, as opposed to classical computers which use electrons. Since photons are essentially without mass and can travel faster than electrons, an optical computer would be superfast, energy-efficient and able to process information simultaneously through multiple temporal or spatial optical channels.
The computing element in an optical computer — an alternative to the ones and zeroes of a digital computer — is represented by the continuous phase of the light signal, and the computation is normally achieved by adding two light waves coming from two different sources and then projecting the result onto ‘0’ or ‘1’ states.
However, real life presents highly nonlinear problems, where multiple unknowns simultaneously change the values of other unknowns while interacting multiplicatively. In this case, the traditional approach to optical computing that combines light waves in a linear manner fails.
Now, Professor Natalia Berloff from Cambridge’s Department of Applied Mathematics and Theoretical Physics and PhD student Nikita Stroev from Skolkovo Institute of Science and Technology have found that optical systems can combine light by multiplying the wave functions describing the light waves instead of adding them and may represent a different type of connections between the light waves.

advertisement

They illustrated this phenomenon with quasi-particles called polaritons — which are half-light and half-matter — while extending the idea to a larger class of optical systems such as light pulses in a fibre. Tiny pulses or blobs of coherent, superfast-moving polaritons can be created in space and overlap with one another in a nonlinear way, due to the matter component of polaritons.
“We found the key ingredient is how you couple the pulses with each other,” said Stroev. “If you get the coupling and light intensity right, the light multiplies, affecting the phases of the individual pulses, giving away the answer to the problem. This makes it possible to use light to solve nonlinear problems.”
The multiplication of the wave functions to determine the phase of the light signal in each element of these optical systems comes from the nonlinearity that occurs naturally or is externally introduced into the system.
“What came as a surprise is that there is no need to project the continuous light phases onto ‘0’ and ‘1’ states necessary for solving problems in binary variables,” said Stroev. “Instead, the system tends to bring about these states at the end of its search for the minimum energy configuration. This is the property that comes from multiplying the light signals. On the contrary, previous optical machines require resonant excitation that fixes the phases to binary values externally.”
The authors have also suggested and implemented a way to guide the system trajectories towards the solution by temporarily changing the coupling strengths of the signals.
“We should start identifying different classes of problems that can be solved directly by a dedicated physical processor,” said Berloff. “Higher-order binary optimisation problems are one such class, and optical systems can be made very efficient in solving them.”
There are still many challenges to be met before optical computing can demonstrate its superiority in solving hard problems in comparison with modern electronic computers: noise reduction, error correction, improved scalability, guiding the system to the true best solution are among them.
“Changing our framework to directly address different types of problems may bring optical computing machines closer to solving real-world problems that cannot be solved by classical computers,” said Berloff. More

The lung is a complex organ whose main function is to exchange gases. It is the largest organ in the human body and plays a key role in the oxygenation of all the organs. Due to its structure, cellular composition and dynamic microenvironment, is difficult to mimic in vitro.
A specialized laboratory of the ARTORG Center for Biomedical Engineering Research, University of Bern, headed by Olivier Guenat has developed a new generation of in-vitro models called organs-on-chip for over 10 years, focusing on modeling the lung and its diseases. After a first successful lung-on-chip system exhibiting essential features of the lung, the Organs-on-Chip (OOC) Technologies laboratory has now developed a purely biological next-generation lung-on-chip in collaboration with the Helmholtz Centre for Infection Research in Germany and the Thoracic Surgery and Pneumology Departments at Inselspital.
A fully biodegradable life-sized air-blood-barrier
Pauline Zamprogno, who developed the new model for her PhD thesis at the OOC, summarizes its characteristics: “The new lung-on-chip reproduces an array of alveoli with in vivo like dimensions. It is based on a thin, stretchable membrane, made with molecules naturally found in the lung: collagen and elastin. The membrane is stable, can be cultured on both sides for weeks, is biodegradable and its elastic properties allow mimicking respiratory motions by mechanically stretching the cells.”
By contrast to the first generation, which was also built by the team around Olivier Guenat, the developed system reproduces key aspects of the lung extracellular matrix (ECM): Its composition (cells support made of ECM proteins), its structure (array of alveoli with dimension similar to those found in vivo + fiber structure) and its properties (biodegradability, a key aspect to investigating barrier remodeling during lung diseases such as IPF or COPD). Additionally, the fabrication process is simple and less cumbersome than that of a polydimethylsiloxane stretchable porous membrane from the first-generation lung-on-chip.
Broad potential clinical applications
Cells to be cultured on the new chip for research are currently obtained from cancer patients undergoing lung resections at the Inselspital Department of Thoracic Surgery. Department Head Ralph Schmid sees a double advantage in the system: “The second generation lung-on-chip can be seeded with either healthy or diseased lung alveolar cells. This provides clinicians with both a better understanding of the lung’s physiology and a predictive tool for drug screening and potentially also for precision medicine, identifying the specific therapy with the best potential of helping a particular patient.”

advertisement

“The applications for such membranes are broad, from basic science investigations into lung functionalities and pathologies, to identifying new pathways, and to a more efficient discovery of potential new therapies,” says Thomas Geiser, Head of the Department of Pneumology at the Inselspital and Director of Teaching and Research of the Insel Gruppe.
Powerful alternative to animal models in research
As an additional plus, the new lung-on-chip can reduce the need for pneumological research based on animal models. “Many promising drug candidates successfully tested in preclinical models on rodents have failed when tested in humans due to differences between the species and in the expression of a lung disease,” explains Olivier Guenat. “This is why, in the long term, we aim to reduce animal testing and provide more patient-relevant systems for drug screening with the possibility of tailoring models to specific patients (by seeding organs-on-chip with their own cells).”
The new biological lung-on-chip will be further developed by Pauline Zamprogno and her colleagues from the OOC Technologies group to mimic a lung with idiopathic pulmonary fibrosis (IPF), a chronic disease of the lung leading to progressive scarring of the lung tissue within the framework of a research project funded by the Swiss 3R Competence Center (3RCC). “My new project consists in the development of an IPF-on- chip model based on the biological membrane. So far, we have develop a healthy air-blood barrier. Now it’s time to use it to investigate a real biological question,” says Zamprogno.
Research group Organs-On-Chip Technologies of the ARTORG Center
This specialized group of the ARTORG Center for Biomedical Engineering Research develops organs-on-chip, focusing on the lung and its diseases, in collaboration with the Departments of Pulmonary Medicine and Thoracic Surgery of the Inselspital. The group combines engineering, in particular microfluidics and microfabrication, cell biology and tissue engineering methods, material sciences and medicine.
Their first development of a breathing lung-on-chip is further developed in collaboration with the start-up AlevoliX, with the aim to revolutionize preclinical research. Recently the group has developed an entirely biological second-generation lung-on-chip focusing on recreating the air-blood barrier of the lung. A second research direction aims at developing a functional lung microvasculature. Here, lung endothelial cells are seeded in a micro-engineered environment, where they self-assemble to build a network of perfusable and contractile microvessels of only a few tens of micrometers in diameter.
Next to pharmaceutical applications, organs-on-chip are seen as having the potential to be used in precision medicine to test the patient’s own cells in order to tailor the best therapy. Furthermore, such systems have the significant potential to reduce animal testing in medical and life-science research. The OOC group operates the Organs-on-Chip Facility, providing scientists from the University of Bern, the University Hospital of Bern and beyond an infrastructure and equipment to produce microfluidic devices and test organs-on-chips. More

Online searches for mobile and isolated activities can help to predict later surges and declines in COVID-19 cases, a team of researchers has found. Its findings, based on a four-month analysis of online searches, offer a potential means to anticipate the pathways of the pandemic — before new infections are reported.
“This is a first step towards building a tool that can help predict COVID case surges by capturing higher-risk activities and intended mobility, which searches for gyms and in-person dining can illuminate,” says Anasse Bari, a clinical assistant professor in computer science at New York University’s Courant Institute of Mathematical Sciences and one of the authors of the paper, which appears in the journal Social Network Analysis and Mining. “Using such ‘alternative data’ is nothing new and has been applied for other purposes — for instance, alternative data has been used in finance to generate data-driven investments, such as studying satellite images of cars in parking lots to predict businesses earnings.”
“Our research shows the same techniques could be applied to combatting a pandemic by spotting, ahead of time, where outbreaks are likely to occur,” adds Megan Coffee, a clinical assistant professor in the Division of Infectious Disease & Immunology at NYU Grossman School of Medicine. “Developing a barometer of behavior would, with further work and validation, allow policymakers and epidemiologists to track the impact of social interventions and brace for rising surges.”
The research also showed an association between intended activities outside the home after lockdown restrictions were lifted, pointing to how the effects of policy decisions can be measured using alternative data.
Since the onset of the pandemic, governments have restricted activities, often based on surges of COVID-19 cases, then loosened these restrictions after declines. However, these actions are in response to infection rates and are designed to limit the spread of future cases.
In the Social Network Analysis and Mining study, the researchers sought to determine if there were ways to spot behaviors known to be risky during the pandemic (e.g., visits to barbershops and nail salons) ahead of local and regional outbreaks — and conversely, identify behaviors known to be less risky (e.g., exercising at home) prior to declines in coronavirus cases.

advertisement

“Our goal was to capture the underlying social dynamics of an unprecedented pandemic using alternative data sources that are new to infectious disease epidemiology,” explains Bari. “When someone searches the closing time of a local bar or looks up directions to a local gym, they give some insight into what future risks they may have.”
To examine this, they studied online searches from March through June in 2020 in all 50 states. Here, they divided searches into two categories — or “tracks”: a mobility index track, which categorized searches linked to interactions with others outside the home (e.g., “theaters near me,” “flight tickets”), and an isolation index track, which categorized searches linked to activities done at home (“food delivery,” “at-home yoga”).
The team’s choice of search keywords was informed by a recent Democracy Fund + UCLA Nationscape survey that tracked activities individuals reported they would prioritize attending if “restrictions were lifted on the advice of public health officials regarding activities.” The most popular results included “going to a stadium/concert,” “going to the movies,” and “attending a sports event.”
Using Google Trends data, the researchers tracked search trends related to mobility and isolation to develop mobility and isolation indexes. They complemented these with a “net movement index,” which was the difference between the mobility index and the isolation index.
The researchers then looked at COVID-19 case growth 10 to 14 days later — the expected lag between exposure and symptoms — at the state level by examining data from state and local health agencies.

advertisement

Overall, they found that the net movement index correlated with new COVID-19 cases — reported weekly — in 42 of 50 states over the studied period (March-June 2020).
The researchers also looked more closely at five states (Arizona, California, Florida, New York, and Texas) to determine the impact of the ending of stay-at-home orders on searches. In all of these states, the mobility index, which decreased during the initial lockdown phase, increased as re-openings began. Subsequently, COVID-19 cases rose again nationwide in June 2020 and surged in Arizona, California, Florida, and Texas.
By contrast, an earlier sharp decline in mobility indices was followed by a sharp decline in the case growth data in these same five states.
“From this work, we hope to build a knowledge base on human behavior change from alternative data during the life cycle of the pandemic in order to allow machine learning to predict behavior in future epidemics,” says Aashish Khubchandani, an NYU undergraduate and one of the paper’s authors.
The researchers recognize that search-based methods to predict infection outbreaks raise privacy concerns. However, they emphasize that their tool uses large volumes of search queries, not individual ones, and relies on anonymized data in order to offer health-related projections.
The paper’s other authors were Courant researchers Matthias Heymann and Junzhang Wang, who are part of the Predictive Analytics and AI research lab at the Courant Institute.
The study was supported, in part, by an Amazon AI research grant and an NYU COVID-19 Catalyst Research grant. More