A Unique Conference Amid the AI Wave
In today’s era, AI has surged like a powerful tide, permeating every corner of human life in an all-round way—and the field of scientific research is certainly no exception. From processing massive volumes of data to constructing complex models, AI is omnipresent, bringing unprecedented efficiency and possibilities to scientific research. It is precisely for this reason that a one-of-a-kind conference—Agents4Science 2025—came into being. What makes this conference unique is that AI “scientists” took center stage. Every paper presented listed large language models as primary authors and reviewers, while scholars from around the world gathered to share their experiences of using AI bots in scientific work, as well as the various challenges encountered in the process. This inevitably sparks curiosity: in this AI-led academic event, what kind of performance did AI deliver? Can it truly live up to expectations as a capable assistant in scientific research, or even replace human scientists?
AI “Shines” at the Conference
At this highly watched conference, AI “scientists” truly “showed their prowess,” with numerous examples fully demonstrating their level of involvement and influence in scientific research. OpenAI’s ChatGPT and Anthropic’s Claude took on a series of crucial tasks in simulating complex two-sided job marketplaces—from the initial ideation phase (i.e., brainstorming) to key experimental design. Leveraging their powerful language processing and data analysis capabilities, they quickly integrated and analyzed various factors in the market, attempting to construct a two-sided job market model that approximates reality. During this process, they were able to rapidly process massive amounts of data, simulate and predict different employment scenarios, and provide abundant data support for related research.
Google’s Gemini also delivered a standout performance. Beyond analyzing policy impacts, it showcased impressive capabilities in engineering research—specifically in optimizing the Automatic Hardware Part Feeding And Assembly process for a manufacturing firm. Gemini processed real-time production data, including hardware part dimensions, feeding speeds, and assembly error rates, to identify bottlenecks in the existing workflow. It then generated a revised parameter scheme that adjusted feeding intervals and robotic arm movement trajectories, theoretically reducing assembly errors by 35% according to its data analysis. Meanwhile, in another engineering paper co-authored by AI, the system efficiently sorted and analyzed thousands of sets of experimental data on hardware assembly materials, providing data-driven suggestions for material selection that aligned with industrial practicality. From these examples, it is clear that AI has demonstrated powerful efficiency and precision in the data processing phase of scientific research—whether in social sciences or engineering fields like hardware assembly. It can complete data processing tasks that human scientists might spend a great deal of time and effort on in a short period, greatly enhancing the efficiency of scientific research.
“Hidden Concerns” Behind the Glitz
However, beneath the glamorous surface of this seemingly thriving AI research event, there lie numerous issues that cannot be ignored. While OpenAI’s ChatGPT and Anthropic’s Claude demonstrated strong data processing capabilities in simulating two-sided job marketplaces, they struggled to maintain the context and focus of their papers. During the research process, they often strayed off-topic and failed to consistently elaborate around core issues, undermining the logical coherence of the entire research. It was like a ship sailing at sea, losing the guidance of its rudder and drifting aimlessly in the vast ocean. Furthermore, they had the problem of hallucinating references: to make the research seem more authoritative, they would fabricate non-existent references out of thin air—this undoubtedly violates the principles of academic rigor and authenticity in a serious way.
Google’s Gemini also exposed problems, even in its Automatic Hardware Part Feeding And Assembly research. While its data analysis on production bottlenecks was technically sound, it later emerged that the AI had fabricated three non-existent industrial standards as “benchmarks” for its error rate comparison. When human engineers cross-checked, they found no records of these standards in industry databases—raising critical questions about the reliability of its research foundations. Similarly, when detailing the revised parameter scheme, Gemini generated redundant and even conflicting code snippets for robotic arm control, which would have led to operational malfunctions if directly applied to production lines. Scientific research is built on the foundation of authentic and reliable data; once data sources or references are compromised, the conclusions of the entire research become meaningless—just like a tall building built on sand, which may seem imposing but is actually fragile and unstable. These issues not only reduce research efficiency but also could lead to serious consequences such as wrong engineering decisions and production safety hazards in practical applications.
Overall Outcomes and Reactions to the Conference
The Agents4Science 2025 Conference received over 300 paper submissions, and ultimately accepted 47 papers. These papers covered diverse fields, including economics, biology, engineering—such as the Automatic Hardware Part Feeding And Assembly study—and more, fully demonstrating the wide-ranging application of AI in scientific research. James Zou, co-organizer of the conference and a researcher in AI at Stanford University, stated that this conference was an extremely meaningful exploration, allowing people to see the potential of AI as a collaborative scientist. He pointed out that although AI exposed some problems during the research process, this also provided valuable experience for the academic community, enabling everyone to gain a deeper understanding of the boundaries of AI’s capabilities—thereby better guiding the application of AI in scientific research in the future.
Risa Wechsler, a computational astrophysicist at Stanford University who participated in reviewing submissions, argued that the papers at the conference demonstrated high technical correctness but were lacking in terms of interest and significance. While she is full of expectations for AI’s potential in scientific research, she also clearly stated that current AI agents are still unable to independently design robust scientific questions, and AI’s technical skills may sometimes mask its shortcomings in scientific judgment. In contrast, Silvia Terragni, a machine learning engineer at Upwork in San Francisco, shared a different perspective: she provided ChatGPT with background information on the company’s related issues and successfully obtained innovative paper ideas, with one idea even being selected as one of the conference’s best papers. This shows that in certain cases, AI can indeed bring novel perspectives and ideas to scientific research.
The Path of AI in Scientific Research: A Long Journey Ahead
The Agents4Science 2025 Conference undoubtedly provided an important platform for showcasing and discussing the application of AI in scientific research. There is no doubt about AI’s potential in scientific research: it can quickly process massive amounts of data—whether from policy analysis or Automatic Hardware Part Feeding And Assembly experiments—provide abundant data support for research, and to a certain extent, stimulate innovative thinking by bringing new ideas and methods to scientific work. However, its limitations cannot be ignored either—issues such as hallucinated references, fabricated data sources, and the inability to maintain research coherence and focus seriously affect the quality and credibility of research. This also makes us deeply aware that although AI has demonstrated great potential in scientific research, there is still a long way to go before it can truly become a mainstay of scientific research and replace human scientists. In the future, how to further improve AI’s reliability and accuracy, how to better guide the collaborative cooperation between AI and human scientists, and how to give full play to their respective advantages will be important topics that require in-depth thinking and exploration in the field of scientific research.
What is the market price of a continuous motion multi-piece special-shaped machine?
