标题： SortedNet：一种可扩展且通用的模块化深度神经网络训练框架 显示英文标题

标题： SortedNet: A Scalable and Generalized Framework for Training Modular Deep Neural Networks

摘要： 深度神经网络（DNNs）必须满足具有不同性能需求和预算的各种用户，导致训练、存储和维护大量用户/任务特定模型的昂贵实践。 文献中有一些解决方案用于处理单个动态模型或一个模型应对多种情况，而不是多个独立网络；然而，它们存在性能显著下降、在不同模型架构或不同维度（例如深度、宽度、注意力块）上泛化能力不足、训练期间需要大量的模型搜索以及只能训练有限数量的子模型等问题。 为了解决这些限制，我们提出了SortedNet，这是一种通用且可扩展的训练解决方案，以利用DNN的固有模块化特性。 得益于一种通用的嵌套架构（我们在本文中将其称为\textit{已排序}架构）以及结合随机子模型采样和新的梯度累积机制的新颖更新方案，SortedNet能够在训练主模型的同时同时训练子模型（没有任何显著的额外训练或推理开销），简化动态模型选择，在推理期间实现定制化部署，并显著减少模型存储需求。 通过各种架构和任务（包括LLaMA、BERT、RoBERTa（自然语言处理任务）、ResNet和MobileNet（图像分类））验证了SortedNet的多功能性和可扩展性，证明了其优于现有的动态训练方法。 例如，我们引入了一种基于排序训练的新型自适应推测方法，以加速大型语言模型的解码。 此外，SortedNet能够同时训练160个子模型，达到原始模型至少96%的性能。 显示英文摘要

摘要： Deep neural networks (DNNs) must cater to a variety of users with different performance needs and budgets, leading to the costly practice of training, storing, and maintaining numerous user/task-specific models. There are solutions in the literature to deal with single dynamic or many-in-one models instead of many individual networks; however, they suffer from significant drops in performance, lack of generalization across different model architectures or different dimensions (e.g. depth, width, attention blocks), heavy model search requirements during training, and training a limited number of sub-models. To address these limitations, we propose SortedNet, a generalized and scalable training solution to harness the inherent modularity of DNNs. Thanks to a generalized nested architecture (which we refer as \textit{sorted} architecture in this paper) with shared parameters and its novel update scheme combining random sub-model sampling and a new gradient accumulation mechanism, SortedNet enables the training of sub-models simultaneously along with the training of the main model (without any significant extra training or inference overhead), simplifies dynamic model selection, customizes deployment during inference, and reduces the model storage requirement significantly. The versatility and scalability of SortedNet are validated through various architectures and tasks, including LLaMA, BERT, RoBERTa (NLP tasks), ResNet and MobileNet (image classification) demonstrating its superiority over existing dynamic training methods. For example, we introduce a novel adaptive self-speculative approach based on sorted-training to accelerate large language models decoding. Moreover, SortedNet is able to train 160 sub-models at once, achieving at least 96\% of the original model's performance.