标题： 用于分子构象生成的标准变压器和带有线性偏差的注意力 显示英文标题

标题： A standard transformer and attention with linear biases for molecular conformer generation

摘要： 从二维分子图生成低能分子构象，即分子中原子的空间排列，是药物发现和优化过程中许多不同计算的关键任务。 已经设计了许多专门的等变网络，以从二维分子图生成分子构象。 最近，由于非等变变换器模型能够扩展以提高泛化能力，因此作为可行的替代方案出现。 然而，人们担心非等变模型需要较大的模型规模来弥补缺乏等变偏差的问题。 在本文中，我们证明了选择合适的定位编码可以有效解决这些规模限制。 一个标准的变换器模型，当其参数规模扩展到2500万时，结合分子图的相对定位编码，在GEOM-DRUGS基准测试中超越了当前最先进的非等变基础模型（参数规模为6400万）。 我们将相对定位编码实现为一种负注意力偏差，该偏差根据图节点之间最短路径距离的不同斜率线性增加，类似于ALiBi，这是一种在自然语言处理领域广泛采用的相对定位编码技术。 这种架构有可能成为一类新型分子构象生成模型的基础。 显示英文摘要

摘要： Sampling low-energy molecular conformations, spatial arrangements of atoms in a molecule, is a critical task for many different calculations performed in the drug discovery and optimization process. Numerous specialized equivariant networks have been designed to generate molecular conformations from 2D molecular graphs. Recently, non-equivariant transformer models have emerged as a viable alternative due to their capability to scale to improve generalization. However, the concern has been that non-equivariant models require a large model size to compensate the lack of equivariant bias. In this paper, we demonstrate that a well-chosen positional encoding effectively addresses these size limitations. A standard transformer model incorporating relative positional encoding for molecular graphs when scaled to 25 million parameters surpasses the current state-of-the-art non-equivariant base model with 64 million parameters on the GEOM-DRUGS benchmark. We implemented relative positional encoding as a negative attention bias that linearly increases with the shortest path distances between graph nodes at varying slopes for different attention heads, similar to ALiBi, a widely adopted relative positional encoding technique in the NLP domain. This architecture has the potential to serve as a foundation for a novel class of generative models for molecular conformations.