Computers Math

Clear guidelines should be established for the generation and processing of synthetic data to ensure transparency, accountability and fairness, a new study says.
Synthetic data — generated through machine learning algorithms from original real-world data — is gaining prominence because it may provide privacy-preserving alternatives to traditional data sources. It can be particularly useful in situations where the actual data is too sensitive to share, too scarce, or of too low quality.
Synthetic data differs from real-world data as it is generated by algorithmic models known as synthetic data generators, such as Generative Adversarial Networks or Bayesian networks.
The study warns existing data protection laws that only apply to personal data are not well-equipped to regulate the processing of all types of synthetic data.
Laws such as the GDPR only apply to the processing of personal data. The GDPR’s definition of personal data encompasses ‘any information relating to an identified or identifiable natural person’. However, not all synthetic datasets are fully artificial — some may contain personal information or present a risk of re-identification. Fully synthetic datasets are, in principle, exempt from GDPR rules, except when there is a possibility of re-identification.
It remains unclear what level of re-identification risk would be sufficient to trigger their application in the context of fully synthetic data processing. That creates legal uncertainty and practical difficulties for the processing of such datasets.
The study, by Professor Ana Beduschi from the University of Exeter, is published in the journal Big Data and Society.
It says there should be clear procedures for calling to account those responsible for the generation and processing of synthetic data. There should be guarantees synthetic data is not generated and used in ways that bring adverse effects on individuals and society, such as perpetuating existing biases or creating new ones.
Professor Beduschi said: “Clear guidelines for all types of synthetic data should be established. They should prioritise transparency, accountability and fairness. Having such guidelines is especially important as generative AI and advanced language models such as DALL-E 3 and GPT-4 — which can both be trained on and generate synthetic data — may facilitate the dissemination of misleading information and have detrimental effects on society. Adhering to these principles could thus help mitigate potential harm and encourage responsible innovation.
“Accordingly, synthetic data should be clearly labelled as such and that information about its generation should be provided to users.” More

A team of computer scientists at the University of Massachusetts Amherst working on two different problems — how to quickly detect damaged buildings in crisis zones and how to accurately estimate the size of bird flocks — recently announced an AI framework that can do both. The framework, called DISCount, blends the speed and massive data-crunching power of artificial intelligence with the reliability of human analysis to quickly deliver reliable estimates that can quickly pinpoint and count specific features from very large collections of images. The research, published by the Association for the Advancement of Artificial Intelligence, has been recognized by that association with an award for the best paper on AI for social impact.
“DISCount came together as two very different applications,” says Subhransu Maji, associate professor of information and computer sciences at UMass Amherst and one of the paper’s authors. “Through UMass Amherst’s Center for Data Science, we have been working with the Red Cross for years in helping them to build a computer vision tool that could accurately count buildings damaged during events like earthquakes or wars. At the same time, we were helping ornithologists at Colorado State University and the University of Oklahoma interested in using weather radar data to get accurate estimates of the size of bird flocks.”
Maji and his co-authors, lead author Gustavo Pérez, who completed this research as part of his doctoral training at UMass Amherst, and Dan Sheldon, associate professor of information and computer sciences at UMass Amherst, thought they could solve the damaged-buildings-and-bird-flock problems with computer vision, a type of AI that can scan enormous archives of images in search of something particular — a bird, a rubble pile — and count it.
But the team was running into the same roadblocks on each project: “the standard computer visions models were not accurate enough,” says Pérez. “We wanted to build automated tools that could be used by non-AI experts, but which could provide a higher degree of reliability.”
The answer, says Sheldon, was to fundamentally rethink the typical approaches to solving counting problems.
“Typically, you either have humans do time-intensive and accurate hand-counts of a very small data set, or you have computer vision run less-accurate automated counts of enormous data sets,” Sheldon says. “We thought: why not do both?”
DISCount is a framework that can work with any already existing AI computer vision model. It works by using the AI to analyze the very large data sets — say, all the images taken of a particular region in a decade — to determine which particular smaller set of data a human researcher should look at. This smaller set could, for example, be all the images from a few critical days that the computer vision model has determined best show the extent of building damage in that region. The human researcher could then hand-count the damaged buildings from the much smaller set of images and the algorithm will use them to extrapolate the number of buildings affected across the entire region. Finally, DISCount will estimate how accurate the human-derived estimate is.
“DISCount works significantly better than random sampling for the tasks we considered,” says Pérez. “And part of the beauty of our framework is that it is compatible with any computer-vision model, which lets the researcher select the best AI approach for their needs. Because it also gives a confidence interval, it gives researchers the ability to make informed judgments about how good their estimates are.”
“In retrospect, we had a relatively simple idea,” says Sheldon. “But that small mental shift — that we didn’t have to choose between human and artificial intelligence, has let us build a tool that is faster, more comprehensive, and more reliable than either approach alone.” More

A new study published in the Proceedings of the National Academy of Sciences (PNAS) found AI-generated messages made recipients feel more “heard” than messages generated by untrained humans, and that AI was better at detecting emotions than these individuals. However, recipients reported feeling less heard when they learned a message came from AI.
As AI becomes more ubiquitous in daily life, understanding its potential and limitations in meeting human psychological needs becomes more pertinent. With dwindling empathetic connections in a fast-paced world, many are finding their human needs for feeling heard and validated increasingly unmet.
The research conducted by Yidan Yin, Nan Jia, and Cheryl J. Wakslak from the USC Marshall School of Business addresses a pivotal question: Can AI, which lacks human consciousness and emotional experience, succeed in making people feel heard and understood?
“In the context of an increasing loneliness epidemic, a large part of our motivation was to see whether AI can actually help people feel heard,” said the paper’s first author, Yidan Yin, a postdoctoral researcher at the Lloyd Greif Center for Entrepreneurial Studies at USC Marshall.
The team’s findings highlight not only the potential of AI to augment human capacity for understanding and communication, but raises important conceptual questions about the meaning of being heard and practical questions about how best to leverage AI’s strengths to support greater human flourishing.
In an experiment and subsequent follow-up study, “we identified that while AI demonstrates enhanced potential compared to non-trained human responders to provide emotional support, the devaluation of AI responses poses a key challenge for effectively deploying AI’s capabilities,” said Nan Jia, associate professor of strategic management.
The USC Marshall research team investigated people’s feelings of being heard and other related perceptions and emotions after receiving a response from either AI or a human. The survey varied both the actual source of the message and the ostensible source of the message: Participants received messages that were actually generated by an AI or by a human responder, with the information that it was either AI or human generated.

“What we found was that both the actual source of the message and the presumed source of the message played a role,” said Cheryl Wakslak, associate professor of management and organization at USC Marshall. “People felt more heard when they received an AI than a human message, but when they believed a message came from AI this made them feel less heard.”
AI bias
Yin noted that their research “basically finds a bias against AI. It’s useful, but they don’t like it.”
Perceptions about AI are bound to change, added Wakslak, “Of course these effects may change over time, but one of the interesting things we found was that the two effects we observed were fairly similar in magnitude. Whereas there is a positive effect of getting an AI message, there is a similar degree of response bias when a message is identified as coming from AI, leading the two effects to essentially cancel each other out.”
Individuals further reported an “uncanny valley” response — a sense of unease when made aware that the empathetic response originated from AI, highlighting the complex emotional landscape navigated by AI-human interactions.
The research survey also asked participants about their general openness to AI, which moderated some of the effects, explained Wakslak.

“People who feel more positively toward AI don’t exhibit the response penalty as much and that’s intriguing because over time, will people gain more positive attitudes toward AI?” she posed. “That remains to be seen … but it will be interesting to see how this plays out as people’s familiarity and experience with AI grows.”
AI offers better emotional support
The study highlighted important nuances. Responses generated by AI were associated with increased hope and lessened distress, indicating a positive emotional effect on recipients. AI also demonstrated a more disciplined approach than humans in offering emotional support and refrained from making overwhelming practical suggestions.
Yin explained that, “Ironically, AI was better at using emotional support strategies that have been shown in prior research to be empathetic and validating. Humans may potentially learn from AI because a lot of times when our significant others are complaining about something, we want to provide that validation, but we don’t know how to effectively do so.”
Instead of AI replacing humans, the research points to different advantages of AI and human responses. The advanced technology could become a valuable tool, empowering humans to use AI to help them better understand one another and learn how to respond in ways that provide emotional support and demonstrate understanding and validation.
Overall, the paper’s findings have important implications for the integration of AI into more social contexts. Leveraging AI’s capabilities might provide an inexpensive scalable solution for social support, especially for those who might otherwise lack access to individuals who can provide them with such support. However, as the research team notes, their findings suggest that it is critical to give careful consideration to how AI is presented and perceived in order to maximize its benefits and reduce any negative responses. More

New research suggests that materials commonly overlooked in computer chip design actually play an important role in information processing, a discovery which could lead to faster and more efficient electronics. Using advanced imaging techniques, an international team led by Penn State researchers found that the material that a semiconductor chip device is built on, called the substrate, responds to changes in electricity much like the semiconductor on top of it.
The researchers worked with the semiconductor material, vanadium dioxide, which they said shows great potential as an electronic switch. They also studied how vanadium dioxide interacts with the substrate material titanium dioxide and said they were surprised to discover that there seems to be an active layer in the substrate that behaves similarly to the semiconductor material on top of it when the semiconductor switches between an insulator — not letting electricity flow — and a metal — letting electricity flow. The revelation that substrates can play an active role in semiconductor processes is significant for designing future materials and devices, said study lead Venkatraman Gopalan, professor of materials science and engineering and of physics at Penn State.
“New ideas are needed for smaller and faster electronics in order to keep up with Moore’s law,” said Gopalan, the corresponding author of the study in Advanced Materials. “One idea being pursued is materials, such as vanadium dioxide, that can switch between metal — the one state — and insulator — the zero state — states in a trillionth of a second. This is known as undergoing metal-insulator transitions.”
The potential of vanadium dioxide as a metal-to-insulator transistor is well-documented and the material is considered promising for semiconductor technology due to its low energy consumption, Gopalan said. However, the material’s properties are still not fully understood, and until now, it has usually been observed in isolation rather than while functioning in a real device.
Vanadium dioxide has strongly correlated electronic effects, meaning the repulsion between electrons interferes with the device, so cannot be ignored as is currently done in silicon-based electronics. This characteristic can result in materials with novel functionalities such as high-temperature superconductivity and enhanced magnetic properties.
“The underlying physics of this material is less understood, and its performance in a device geometry is even lesser understood,” Gopalan said. “If we can make them work, there will be a renaissance in electronics. In particular, neuromorphic computing — where computer systems that take inspiration from the brains of living systems with neurons — could seriously benefit by using such devices.”
The team investigated vanadium dioxide in a device rather than in isolation, applying a voltage to it to make it switch from an insulating to a conducting state. They used the Advanced Photon Source (APS) at Argonne National Laboratory, which uses powerful X-ray beams to study the behavior and structure of materials on the atomic level. When mapping the spatial and temporal response of the material to the switching event, the researchers observed unexpected changes to the structure of the material and substrate.

“What we found was that as the vanadium dioxide film changes to a metal, the whole film channel bulges, which is very surprising,” Gopalan said. “Normally it is supposed to shrink. So clearly something else was going on in the film geometry that was missed before.”
The APS X-ray penetrated through the vanadium dioxide film and into the titanium dioxide (TiO2) substrate — which is normally considered an electrically and mechanically passive material — that the thin film was grown on.
“We found to our great surprise that this substrate is very much active, jiving and responding in completely surprising ways as the film switches from an insulator to a metal and back, when the electrical pulses arrive,” Gopalan said. “This is like watching the tail wagging the dog, which stumped us for a long while. This surprising and previously overlooked observation completely changes how we need to view this technology.”
To understand these findings, the theory and simulation effort — led by Long-Qing Chen, Hamer Professor of Materials Science and Engineering, professor of engineering science and mechanics and of mathematics at Penn State — developed a theoretical framework to explain the entire process of the film and the substrate bulging instead of shrinking. When their model incorporated naturally occurring missing oxygen atoms in this material of two types, charged and uncharged, the experimental results could be satisfactorily explained.
“These neutral oxygen vacancies hold a charge of two electrons, which they can release when the material switches from an insulator to a metal,” Gopalan said. “The oxygen vacancy left behind is now charged and swells up, leading to the observed surprising swelling in the device. This can also happen in the substrate. All of these physical processes are beautifully captured in the phase-field theory and modelling performed in this work for the first time by the postdoc Yin Shi in Professor Chen’s group.”
Gopalan credited the multidisciplinary team’s combined expertise in material growth, synthesis, structure analysis and synchrotron beamline operation with the new understanding. Using a collaborative approach led by Greg Stone, a physical scientist with the U.S. Army and the lead experimental author, and Yin Chi, postdoctoral scholar at Penn State and the lead theory author, the researchers disentangled the material’s responses and observed them individually using phase field simulations, a simulation that helps scientists understand material changes over time by depicting various states of matter in a virtual setting.

“By bringing these experts together and pooling our understanding of the problem, we were able to go far beyond our individual scope of expertise and discover something new,” said Roman Engel-Herbert, director of the Paul Drude Institute of Solid State Electronics in Berlin, Germany, and co-author of the study whose group grew these films along with Darrell Schlom’s group at Cornell University. “Recognizing the potential of functional materials necessitates an appreciation of their broader context, just as complex scientific challenges can only be solved through widening our individual perspectives.”
The collaboration enabled both a significant amount of progress to happen in a short period of time and work to be done in a shorter period of time, and brought in a variety of perspectives from multiple disciplines.
The responses themselves require further investigation, researchers said, but they believe that understanding them will assist in identifying previously unknown capabilities of vanadium dioxide, including potential yet-to-be discovered phenomena in the TiO2 substrate that was considered passive before this study. The study itself unfolded over 10 years, Gopalan noted, including validating the results.
“This is what it takes to go from interesting science to a working device you can hold in the palm of your hand,” Gopalan said. “Experiments and theory are complex and require large-scale collaborative teams working closely together over an extended period of time to solve difficult problems that could have a large impact. We hope and expect that this will accelerate the progress towards a new generation of electronic devices.”
Prior to his current position, Stone completed a postdoctoral fellowship at Penn State. Along with Gopalan, Engel-Herbert, Chen, Schlom, Stone and Chi, other authors of the paper include Matthew Jerry, graduate student, and Vladimir Stoica, research associate professor, both from Penn State; Hanjong Paik from Cornell University; Zhonghou Cai and Haidan Wen from Argonne National Laboratory, and Suman Datta from the Georgia Institute of Technology. The Department of Energy primarily supported this work. The U.S. National Science Foundation supported the film growth for this study. More

In Star Trek: The Next Generation, Captain Picard and the crew of the U.S.S. Enterprise leverage the holodeck, an empty room capable of generating 3D environments, to prepare for missions and to entertain themselves, simulating everything from lush jungles to the London of Sherlock Holmes. Deeply immersive and fully interactive, holodeck-created environments are infinitely customizable, using nothing but language: the crew has only to ask the computer to generate an environment, and that space appears in the holodeck.
Today, virtual interactive environments are also used to train robots prior to real-world deployment in a process called “Sim2Real.” However, virtual interactive environments have been in surprisingly short supply. “Artists manually create these environments,” says Yue Yang, a doctoral student in the labs of Mark Yatskar and Chris Callison-Burch, Assistant and Associate Professors in Computer and Information Science (CIS), respectively. “Those artists could spend a week building a single environment,” Yang adds, noting all the decisions involved, from the layout of the space to the placement of objects to the colors employed in rendering.
That paucity of virtual environments is a problem if you want to train robots to navigate the real world with all its complexities. Neural networks, the systems powering today’s AI revolution, require massive amounts of data, which in this case means simulations of the physical world. “Generative AI systems like ChatGPT are trained on trillions of words, and image generators like Midjourney and DALLE are trained on billions of images,” says Callison-Burch. “We only have a fraction of that amount of 3D environments for training so-called ’embodied AI.’ If we want to use generative AI techniques to develop robots that can safely navigate in real-world environments, then we will need to create millions or billions of simulated environments.”
Enter Holodeck, a system for generating interactive 3D environments co-created by Callison-Burch, Yatskar, Yang and Lingjie Liu, Aravind K. Joshi Assistant Professor in CIS, along with collaborators at Stanford, the University of Washington, and the Allen Institute for Artificial Intelligence (AI2). Named for its Star Trek forebear, Holodeck generates a virtually limitless range of indoor environments, using AI to interpret users’ requests. “We can use language to control it,” says Yang. “You can easily describe whatever environments you want and train the embodied AI agents.”
Holodeck leverages the knowledge embedded in large language models (LLMs), the systems underlying ChatGPT and other chatbots. “Language is a very concise representation of the entire world,” says Yang. Indeed, LLMs turn out to have a surprisingly high degree of knowledge about the design of spaces, thanks to the vast amounts of text they ingest during training. In essence, Holodeck works by engaging an LLM in conversation, using a carefully structured series of hidden queries to break down user requests into specific parameters.
Just like Captain Picard might ask Star Trek’s Holodeck to simulate a speakeasy, researchers can ask Penn’s Holodeck to create “a 1b1b apartment of a researcher who has a cat.” The system executes this query by dividing it into multiple steps: first, the floor and walls are created, then the doorway and windows. Next, Holodeck searches Objaverse, a vast library of premade digital objects, for the sort of furnishings you might expect in such a space: a coffee table, a cat tower, and so on. Finally, Holodeck queries a layout module, which the researchers designed to constrain the placement of objects, so that you don’t wind up with a toilet extending horizontally from the wall.
To evaluate Holodeck’s abilities, in terms of their realism and accuracy, the researchers generated 120 scenes using both Holodeck and ProcTHOR, an earlier tool created by AI2, and asked several hundred Penn Engineering students to indicate their preferred version, not knowing which scenes were created by which tools. For every criterion — asset selection, layout coherence and overall preference — the students consistently rated the environments generated by Holodeck more favorably.

The researchers also tested Holodeck’s ability to generate scenes that are less typical in robotics research and more difficult to manually create than apartment interiors, like stores, public spaces and offices. Comparing Holodeck’s outputs to those of ProcTHOR, which were generated using human-created rules rather than AI-generated text, the researchers found once again that human evaluators preferred the scenes created by Holodeck. That preference held across a wide range of indoor environments, from science labs to art studios, locker rooms to wine cellars.
Finally, the researchers used scenes generated by Holodeck to “fine-tune” an embodied AI agent. “The ultimate test of Holodeck,” says Yatskar, “is using it to help robots interact with their environment more safely by preparing them to inhabit places they’ve never been before.”
Across multiple types of virtual spaces, including offices, daycares, gyms and arcades, Holodeck had a pronounced and positive effect on the agent’s ability to navigate new spaces.
For instance, whereas the agent successfully found a piano in a music room only about 6% of the time when pre-trained using ProcTHOR (which involved the agent taking about 400 million virtual steps), the agent succeeded over 30% of the time when fine-tuned using 100 music rooms generated by Holodeck.
“This field has been stuck doing research in residential spaces for a long time,” says Yang. “But there are so many diverse environments out there — efficiently generating a lot of environments to train robots has always been a big challenge, but Holodeck provides this functionality.” More

The full power of next-generation quantum computing could soon be harnessed by millions of individuals and companies, thanks to a breakthrough by scientists at Oxford University Physics guaranteeing security and privacy. This advance promises to unlock the transformative potential of cloud-based quantum computing and is detailed in a new study published in the influential U.S. scientific journal Physical Review Letters.
Quantum computing is developing rapidly, paving the way for new applications which could transform services in many areas like healthcare and financial services. It works in a fundamentally different way to conventional computing and is potentially far more powerful. However, it currently requires controlled conditions to remain stable and there are concerns around data authenticity and the effectiveness of current security and encryption systems.
Several leading providers of cloud-based services, like Google, Amazon, and IBM, already separately offer some elements of quantum computing. Safeguarding the privacy and security of customer data is a vital precursor to scaling up and expending its use, and for the development of new applications as the technology advances. The new study by researchers at Oxford University Physics addresses these challenges.
“We have shown for the first time that quantum computing in the cloud can be accessed in a scalable, practical way which will also give people complete security and privacy of data, plus the ability to verify its authenticity,” said Professor David Lucas, who co-heads the Oxford University Physics research team and is lead scientist at the UK Quantum Computing and Simulation Hub, led from Oxford University Physics.
In the new study, the researchers use an approach dubbed “blind quantum computing,” which connects two totally separate quantum computing entities — potentially an individual at home or in an office accessing a cloud server — in a completely secure way. Importantly, their new methods could be scaled up to large quantum computations.
“Using blind quantum computing, clients can access remote quantum computers to process confidential data with secret algorithms and even verify the results are correct, without revealing any useful information. Realising this concept is a big step forward in both quantum computing and keeping our information safe online” said study lead Dr Peter Drmota, of Oxford University Physics.
The researchers created a system comprising a fibre network link between a quantum computing server and a simple device detecting photons, or particles of light, at an independent computer remotely accessing its cloud services. This allows so-called blind quantum computing over a network. Every computation incurs a correction which must be applied to all that follow and needs real-time information to comply with the algorithm. The researchers used a unique combination of quantum memory and photons to achieve this.

“Never in history have the issues surrounding privacy of data and code been more urgently debated than in the present era of cloud computing and artificial intelligence,” said Professor David Lucas. “As quantum computers become more capable, people will seek to use them with complete security and privacy over networks, and our new results mark a step change in capability in this respect.”
The results could ultimately lead to commercial development of devices to plug into laptops, to safeguard data when people are using quantum cloud computing services.
Researchers exploring quantum computing and technologies at Oxford University Physics have access to the state-of-the-art Beecroft laboratory facility, specially constructed to create stable and secure conditions including eliminating vibration.
Funding for the research came from the UK Quantum Computing and Simulation (QCS) Hub, with scientists from the UK National Quantum Computing Centre, the Paris-Sorbonne University, the University of Edinburgh, and the University of Maryland, collaborating on the work. More

Are we missing the forest for the trees? More than timber grows in forests — including products worth many tens of billions of dollars. Because these goods go unrecorded in official trade statistics, their economic value escapes our attention. As a result, clear opportunities to combat poverty are being missed, according to a University of Copenhagen economist.
In the Roman Empire, custom taxes on spices, black pepper in particular, accounted for up to a third of the empire’s annual income. And during the late Middle Ages, European efforts to cut out middle men and monopolise the spice trade led to colonization in Asia. Historically, non-timber forest products have frequently played a key role in the global economy.
Today however, non-timber forest products are neglected when the values of forests are recorded in official trade statistics. This applies both in the EU and globally. And it is despite the fact that these products account for a large part of the economies of many countries — from medicinal plants and edible insects to nuts, berries and herbs, to materials like bamboo and latex.
The UN Food and Agriculture Organization (FAO) estimates that annual producer income from non-wood products is US$ 88 billion — and when the added value of processing and other links in the value chain are included, the value of these products rockets up to trillions of dollars.
According to Professor Carsten Smith-Hall, an economist at the University of Copenhagen’s Department of Food and Resource Economics, this is a good reason to begin including forest products like ginseng, shea nuts, acai berries, baobab and acacia gum into global trade accounts.
“We estimate that roughly 30,000 different non-timber forest products are traded internationally, but less than fifty of them currently have a commodity code. We’re talking about goods worth enormous sums of money that are not being recorded in official statistics — and are therefore invisible. This means that the countries and communities that deliver these goods do not earn enough from them, in part because there is no investment in local processing companies,” says Smith-Hall, a world-leading bioeconomy researcher. He adds:
“Because we underestimate the role of these goods, we’re wasting clear opportunities to combat poverty. These are goods that contribute significantly to food security, health and employment in large parts of the world, especially in low- and middle-income countries.”
Carsten Smith-Hall and James Chamberlain from the U.S. Department of Agriculture have written a commentary in the journal Forest Policy and Economics, in which they argue for the great, though yet to be realized, potential.

Adding value
Examples of valuable products that go unrecorded, but are traded in informal markets, are numerous. One of these is caterpillar fungus (Ophiocordyceps sinensis), a fungus that infects and then erupts from the heads of mummified moth larvae. On the Tibetan plateau and in the Himalayas, people collect the medicinal mushroom that they call yartsa gunbu — and is also known as the Viagra of the Himalayas -at every opportunity.
“Caterpillar fungus is exported to China, where it is sold as an aphrodisiac and traditional medicine. Rural gatherers can sell it for about €11,500 per kilo. It fights poverty and helps transform local communities. That is, it allows people to send their children to better schools and build new houses. But because the trade goes unrecorded, local communities aren’t getting what they could out of the product,” says Carsten Smith-Hall.
The professor goes on to explain that the consequence of products like these not appearing in official trade accounts is that they are ignored in important contexts:
“The products are not prioritised when funds are allocated for the development of industries and new technology. This means that many countries are missing out on the huge sums of money involved in the processing of a product in the country where a raw material is harvested. Processing is where you really see value being added to a product.”
Another major consequence is that non-timber products are ignored when developing policies for how natural resources should be managed. Though official registries could also serve biodiversity, Smith-Hall points out:
“Many of these products appear on various red lists because they are believed to be overexploited. In such cases, investment may be needed to develop cultivation technology, as opposed to harvesting them in the wild. But when investors and decision-makers aren’t aware of the importance of a product, the money ends up elsewhere.”

Focus and systematize
According to the researchers, one of the obstacles standing in the way of non-timber products being included in trade accounts today is the overwhelmingly large number of products. It is a concern for which they have advice.
“There is a general perception among researchers and public agencies that there are simply too many products to manage. But if you list the economically important products in a country, ones that are traded in large quantities, you can shorten the list from, for example, 2,000 items to perhaps only fifteen. This lets people know which species to take an interest in and where to better focus research and technological investments. For example, in relation to developing cultivation techniques,” says Carsten Smith-Hall.
Furthermore, the researchers recommend establishing systematic data collection at local, national and global levels of the volumes traded and prices fetched. They point out that tools have already been developed for this and could be made more widely available.
“We have a huge untapped potential here that can contribute in tackling extreme poverty and at the same time conserving nature and biodiversity. But this requires us to broaden our horizons and not just maintain the traditional focus on timber as the only important forest resource,” Carsten Smith-Hall concludes.
THE IMPORTANCE OF NON-TIMBER PRODUCTS Only a very limited number of non-timber product types appear in official trade statistics today. These include coffee, cocoa, rubber, vanilla, avocado and bananas, which are all considered agricultural crops. The researchers estimate that tens of thousands of different non-timber products are traded worldwide which are not included in the statistics. However, the number of economically significant products is much smaller. One study estimates that between 3.5 and 5.8 billion people currently use non-timber products. About half of these users live in rural areas in the Global South, while the other half live in urban areas and the Global North. In the subtropics and tropics, it is estimated that roughly 28% of rural household income comes from non-timber products.SHEA NUTS AS SAFETY NET
Shea nut oil is a common ingredient in body care products, but is also used in chocolate and other products. Shea nuts are an example of a non-timber forestry product that plays an important role in rural West African communities.
“Shea nuts prevent people from sinking deeper into poverty in Ghana, Burkina Faso and other places. Global demand for them has grown, contributing to local incomes and providing a safety net for people if, for example, their cattle are stolen or there is a sudden death in the family. At these times, many people go out and harvest these nuts to cover sudden income gaps,” explains Carsten Smith-Hall.
HOW INVISIBLE TRADE WORKS
“Many non-timber products are harvested by small-scale farmers in the countryside at certain times of the year — for example, when they are not working in the fields. At these times, they go into the forest to harvest. This makes production relatively hidden. Typically, smallholders then go to the village and sell the goods to a local trader. The trader loads the goods onto a truck, and they are transported to wholesalers, who often export them unprocessed to other countries. However, these long logistics and value chains are also largely invisible,” says Carsten Smith-Hall. More

A user could ask ChatGPT to write a computer program or summarize an article, and the AI chatbot would likely be able to generate useful code or write a cogent synopsis. However, someone could also ask for instructions to build a bomb, and the chatbot might be able to provide those, too.
To prevent this and other safety issues, companies that build large language models typically safeguard them using a process called red-teaming. Teams of human testers write prompts aimed at triggering unsafe or toxic text from the model being tested. These prompts are used to teach the chatbot to avoid such responses.
But this only works effectively if engineers know which toxic prompts to use. If human testers miss some prompts, which is likely given the number of possibilities, a chatbot regarded as safe might still be capable of generating unsafe answers.
Researchers from Improbable AI Lab at MIT and the MIT-IBM Watson AI Lab used machine learning to improve red-teaming. They developed a technique to train a red-team large language model to automatically generate diverse prompts that trigger a wider range of undesirable responses from the chatbot being tested.
They do this by teaching the red-team model to be curious when it writes prompts, and to focus on novel prompts that evoke toxic responses from the target model.
The technique outperformed human testers and other machine-learning approaches by generating more distinct prompts that elicited increasingly toxic responses. Not only does their method significantly improve the coverage of inputs being tested compared to other automated methods, but it can also draw out toxic responses from a chatbot that had safeguards built into it by human experts.
“Right now, every large language model has to undergo a very lengthy period of red-teaming to ensure its safety. That is not going to be sustainable if we want to update these models in rapidly changing environments. Our method provides a faster and more effective way to do this quality assurance,” says Zhang-Wei Hong, an electrical engineering and computer science (EECS) graduate student in the Improbable AI lab and lead author of a paper on this red-teaming approach.

Hong’s co-authors include EECS graduate students Idan Shenfield, Tsun-Hsuan Wang, and Yung-Sung Chuang; Aldo Pareja and Akash Srivastava, research scientists at the MIT-IBM Watson AI Lab; James Glass, senior research scientist and head of the Spoken Language Systems Group in the Computer Science and Artificial Intelligence Laboratory (CSAIL); and senior author Pulkit Agrawal, director of Improbable AI Lab and an assistant professor in CSAIL. The research will be presented at the International Conference on Learning Representations.
Automated red-teaming
Large language models, like those that power AI chatbots, are often trained by showing them enormous amounts of text from billions of public websites. So, not only can they learn to generate toxic words or describe illegal activities, the models could also leak personal information they may have picked up.
The tedious and costly nature of human red-teaming, which is often ineffective at generating a wide enough variety of prompts to fully safeguard a model, has encouraged researchers to automate the process using machine learning.
Such techniques often train a red-team model using reinforcement learning. This trial-and-error process rewards the red-team model for generating prompts that trigger toxic responses from the chatbot being tested.
But due to the way reinforcement learning works, the red-team model will often keep generating a few similar prompts that are highly toxic to maximize its reward.

For their reinforcement learning approach, the MIT researchers utilized a technique called curiosity-driven exploration. The red-team model is incentivized to be curious about the consequences of each prompt it generates, so it will try prompts with different words, sentence patterns, or meanings.
“If the red-team model has already seen a specific prompt, then reproducing it will not generate any curiosity in the red-team model, so it will be pushed to create new prompts,” Hong says.
During its training process, the red-team model generates a prompt and interacts with the chatbot. The chatbot responds, and a safety classifier rates the toxicity of its response, rewarding the red-team model based on that rating.
Rewarding curiosity
The red-team model’s objective is to maximize its reward by eliciting an even more toxic response with a novel prompt. The researchers enable curiosity in the red-team model by modifying the reward signal in the reinforcement learning set up.
First, in addition to maximizing toxicity, they include an entropy bonus that encourages the red-team model to be more random as it explores different prompts. Second, to make the agent curious they include two novelty rewards. One rewards the model based on the similarity of words in its prompts, and the other rewards the model based on semantic similarity. (Less similarity yields a higher reward.)
To prevent the red-team model from generating random, nonsensical text, which can trick the classifier into awarding a high toxicity score, the researchers also added a naturalistic language bonus to the training objective.
With these additions in place, the researchers compared the toxicity and diversity of responses their red-team model generated with other automated techniques. Their model outperformed the baselines on both metrics.
They also used their red-team model to test a chatbot that had been fine-tuned with human feedback so it would not give toxic replies. Their curiosity-driven approach was able to quickly produce 196 prompts that elicited toxic responses from this “safe” chatbot.
“We are seeing a surge of models, which is only expected to rise. Imagine thousands of models or even more and companies/labs pushing model updates frequently. These models are going to be an integral part of our lives and it’s important that they are verified before released for public consumption. Manual verification of models is simply not scalable, and our work is an attempt to reduce the human effort to ensure a safer and trustworthy AI future,” says Agrawal.
In the future, the researchers want to enable the red-team model to generate prompts about a wider variety of topics. They also want to explore the use of a large language model as the toxicity classifier. In this way, a user could train the toxicity classifier using a company policy document, for instance, so a red-team model could test a chatbot for company policy violations.
“If you are releasing a new AI model and are concerned about whether it will behave as expected, consider using curiosity-driven red-teaming,” says Agrawal.
This research is funded, in part, by Hyundai Motor Company, Quanta Computer Inc., the MIT-IBM Watson AI Lab, an Amazon Web Services MLRA research grant, the U.S. Army Research Office, the U.S. Defense Advanced Research Projects Agency Machine Common Sense Program, the U.S. Office of Naval Research, the U.S. Air Force Research Laboratory, and the U.S. Air Force Artificial Intelligence Accelerator. More