With the rapid iteration of artificial intelligence (AI), relying solely on computing power stacking can no longer achieve a synchronous linear improvement in model performance during the large model era. The collaborative effect between model algorithms and underlying hardware, software stacks, and development frameworks ultimately affects training and inference efficiency and performance. This highlights the importance of AI hardware and software ecosystems. The “Expert Talk” column of the China Academy of Information and Communications Technology (CAICT) has launched a special series on “Artificial Intelligence,” inviting senior industry experts to discuss the development status of China’s AI industry from multiple perspectives.
In this issue, Li Lun, Director of the Hardware, Software and Innovation Ecosystem Department at the AI Research Institute of CAICT, shares insights on AI hardware and software product innovation and ecosystem construction, aiming to help China build a technologically leading and soundly developing hardware and software ecosystem as soon as possible.
I. Innovation Iteration of Large Models and Deep Coupling with Hardware and Software Systems
The upgrading and iteration of large models require experiments on massive hardware and software systems. Original innovation of models and the implementation of application iterations now heavily rely on an advanced technology ecosystem of hardware-software collaboration. The collaboration between frameworks, chips, clusters, networks, and algorithms is becoming increasingly close. Currently, models represented by DeepSeek have achieved significant performance improvements through engineering innovation—especially by optimizing and coordinating model architectures, hardware computing power, and interconnection communication to the extreme, thereby enhancing training and inference efficiency. Globally, leading companies such as Meta and ByteDance are accelerating hardware-software collaborative optimization for larger-scale clusters, such as innovating cluster network architectures, building advanced operation and maintenance tools, and optimizing distributed parallel strategies. We predict that in the next stage, hardware-software collaboration and ecosystem construction will be the focus of global competition in large model innovation and computing power infrastructure development.
II. Current Development Status of China in Hardware-Software Collaboration and Adaptation
The adaptation between models and underlying hardware is key to fully releasing hardware capabilities. Taking DeepSeek, a typical open-source large model in the industry, as an example, running the DeepSeek model on hardware and software products of major domestic manufacturers still requires in-depth optimization based on actual application scenarios to meet extended needs such as accuracy, performance, and product services.
We divide hardware-software adaptation into three stages:
- The first stage is to complete basic adaptation of hardware and software products, ensuring that models run on hardware-software systems without errors, reaching a “basically usable” state.
- The second stage is to optimize system performance, such as aligning accuracy and improving indicators like throughput and latency, initially achieving a “user-friendly” state.
- The third stage is to integrate with actual scenarios, further improving application effectiveness through collaboration among users, service providers, and suppliers.
Against this backdrop, CAICT has promoted localized adaptation testing by building the AI hardware and software benchmark system AISHPerf. We found that a large number of domestic hardware and software systems can already support DeepSeek. Through a series of optimizations, they can achieve “usability” within days. Some manufacturers have even achieved accuracy alignment with official technical reports in model adaptation. Meanwhile, various toolkits, all-in-one machines, and public cloud services have emerged in the short term, continuously improving model usability.
III. Development Path of AI Computing Power Infrastructure and Hardware-Software Products
From an overall development perspective, we believe that the scaling law will continue in the next few years. The industry mainly follows two development approaches:
- Expanding the scale of training computing power, increasing model size, and improving model accuracy.
- Focusing on model transformation and efficient training within a certain range of parameters—especially under computing constraints, improving training efficiency through effective matching of model parameters with training data and algorithm architecture transformation, which is a key area of technical competition among manufacturers.
Along this development path, there are three key innovation focuses:
- First, focusing on system performance and improving computing power utilization. Although computing power demand is still growing significantly, the computing power utilization rate of many current clusters is still less than 50%. Deep hardware-software collaboration in computing, storage, and communication is needed to improve system performance.
- Second, focusing on innovation in inference optimization. Large model inference has become a key link in application implementation. Integrating hardware and software, and adopting technologies such as distributed parallel inference can effectively improve inference efficiency, leading to new product and service forms such as all-in-one machines and inference clusters.
- Third, adapting to diverse scenarios is critical for implementation. Open-source model ecosystems like DeepSeek are expected to further drive growth in industry application demand. However, performance indicators vary across scenarios, and there are numerous point-to-point adaptation issues that require in-depth optimization.
IV. China Needs to Continuously Promote Hardware-Software Product Innovation and Ecosystem Construction Based on Application Needs
The efficient deployment of application models and the establishment of ecosystem collaboration paradigms are key to AI product innovation in the next stage. To further consolidate industry consensus, reduce technical difficulties in hardware-software selection and development thresholds, CAICT, relying on the AI Standardization Technical Committee of the Ministry of Industry and Information Technology, has promoted the formulation of more than ten standards based on the AISHPerf Working Group for AI hardware and software benchmarks, covering operators, computing devices, training and inference clusters, framework tools, edge systems, intelligent computing interconnection, and networks.
In addition, CAICT has built the first AI Hardware-Software Collaborative Innovation and Adaptation Verification Center in Yizhuang, Beijing. It provides experimental verification, testing, and industrial cultivation services for AI hardware and software products, including chips, servers, various computing devices, clusters, development frameworks and platforms, and all-in-one machines.
In the next step, we will continue to carry out a series of testing and verification work on large-model hardware-software integrated devices, framework software and tools, and large-scale intelligent computing clusters. We will also host large-scale events such as the “Xingzhi Cup” National AI Innovation Application Competition and forums on large-model innovation and hardware-software collaborative ecosystems. We hope to guide collaborative innovation between AI model providers and hardware-software manufacturers in an orderly manner, gather efforts from all parties, and build a technologically leading and soundly developing industrial ecosystem as soon as possible.
