Трансформеры в условиях ограниченных ресурсов: масштабируемость и эффективность обучения и инференса тема диссертации и автореферата по ВАК РФ 00.00.00, кандидат наук Мезенцев Глеб Владимирович

• Мезенцев Глеб Владимирович
• кандидат наук
• 2026

Оглавление диссертации кандидат наук Мезенцев Глеб Владимирович

Contents

Page

Introduction

Chapter 1. Background

1.1 Transformer model

1.1.1 Sequence modeling

1.1.2 Computational resources: memory and time

1.1.3 Model training

1.1.4 Model inference

1.1.5 Transformer Blocks Architecture

1.2 Related work

1.2.1 Embeddings

1.2.2 Attention

1.2.3 Feed-forward

1.2.4 Output Head

1.2.5 Cross-cutting

Chapter 2. Parameter-efficient fine-tuning via sparsified gradients

2.1 Related Work

2.2 Method

2.2.1 Finding Transition Matrices

2.2.2 Signal Propagation in SparseGradLinear Layer

2.2.3 Sparse-by-Dense Matrix Multiplication

2.2.4 Time and Memory Consumption per Training Iteration

2.3 Experiments

2.3.1 Natural Language Understanding with BERT and RoBERTa

2.3.2 Conversations with LLaMa-2

2.4 Conclusions to chapter

Chapter 3. Reduced Cross-Entropy calculation for Large Catalog

Sequential Transformer-based Recommender Systems

3.1 Related work

Page

3.2 Method

3.2.1 Optimal bucket size

3.3 Experiments and Results

3.3.1 Datasets

3.3.2 Evaluation

3.3.3 Model and Baselines

3.3.4 Results

3.4 Conclusions to chapter

Chapter 4. Scalable Cross-Entropy calculation for Large Catalog

Sequential Transformer-based Recommender Systems

4.1 Method

4.1.1 Scalable Cross-Entropy

4.1.2 Bucket Collapse Mitigation

4.1.3 Method Applicability

4.2 Experiments and Results

4.2.1 Datasets

4.2.2 Evaluation

4.2.3 Model and Baselines

4.2.4 Dependence on SCE Hyperparameters

4.2.5 Influence of Mix Operation

4.2.6 Small Catalog Effects

4.2.7 Model Performance Under Memory Constraints

4.2.8 Large catalog evaluation

4.2.9 Evaluating Against Contemporary Models

4.2.10 Beyond Cross-Entropy approximation

4.3 Conclusions to chapter

Chapter 5. Non-autoregressive Text Generation with Frozen

Large Language Models

5.1 Related work

5.2 Method

5.2.1 Exact scheme

5.2.2 Metrics

Page

5.2.3 Solution space connectivity

5.2.4 Token sequences similarity

5.3 Experiments and results

5.3.1 Models

5.3.2 Data

5.3.3 Training details

5.3.4 Proto-token arrangement

5.3.5 Token sharing

5.3.6 Generation capacity

5.3.7 Proto-tokens interpretation

5.3.8 Proto-tokens embedding space structure

5.4 Conclusions to chapter

Conclusions

List of Figures

List of Tables

Appendix A. Russian translation of the dissertation

Введение диссертации (часть автореферата) на тему «Трансформеры в условиях ограниченных ресурсов: масштабируемость и эффективность обучения и инференса»

Introduction

The relevance of the research area. The degree of its development.

Transformer architecture is the backbone of most of the modern discrete sequence processing models, including Large Language Models (LLMs). The particular example of LLMs is vital due to both their high success in various tasks [1—8] and the rapid growth of model sizes [9], hardware requirements [10], and overall usage [11]. This makes improving the efficiency of LLM-transformer training and inference very important. However, even in domains with smaller model sizes, the resource requirements of the model - training and inference time and memory - are still critically important, as reductions in any of those directly translate to lower overall costs and the ability to achieve better results with the same resources. This makes the problems of constrained transformer training and inference universal. Essentially, any transformer training/inference is resource-constrained.

The problem of memory and time reduction is widely addressed in the literature, with some methods being almost universally adopted in modern models. The optimizations are usually made on several levels of abstraction.

On the lowest level, some hardware-aware optimization could be made to either ensure a reduced number of data transfers between different parts of the memory hierarchy [12; 13], or to utilize more hardware-efficient data types [14—20], or to enable low-level efficient code [21; 22]. These methods usually do not fundamentally alter the model's architecture (what is computed) and instead focus on how to compute it more efficiently.

On the intermediate level, methods are focused on optimizing specific blocks of the transformer architecture, often by approximating the existing component in a more efficient way [23—33]. Such methods use low-parameter or low-compute approximations that offer a functional trade-off between reduced quality and saved resources, and sometimes even improve model quality.

On the highest level of abstraction, the fundamental underlying mathematical model behind sequence modeling could be altered, [34—42], targeting the autoregressive nature of transformer models and replacing it with parallel alternatives.

Typically, a combination of optimizations at all levels is utilized in modern models [43; 44].

While advances at the system, architectural, and algorithmic levels have remarkably improved efficiency, important spaces remain under-explored. Without implying exclusivity or primacy, we highlight three representative fronts where further progress can yield significant practical gains. The first two fronts are intermediate-level abstract in our classification, while the final third is the high-level.

Most of the modern transformer parameters are introduced by Multilayer per-ceptron (MLP) blocks [43]; however, during fine-tuning (a post-training on a small portion of data tailored for a specific end-task), they often remain frozen or trained in a simple representation with a limited expressiveness [31; 45; 46]. This makes developing an efficient and expressive MLP fine-tuning method an open problem.

When applying the transformer to a sequential recommendation task, the primary factor limiting direct transition is the size of the vocabulary (catalog). It makes Cross-Entropy loss computation, the main bottleneck of the training procedure, and, in practice, forces the replacement of Cross-Entropy loss with alternatives that harm performance [32; 47]. More powerful, quality-preserving variants exist [48—56], but they suffer from limited GPU compatibility and are thus not practical. To this end, developing a method that is both hardware-efficient and achieves quality similar to full Cross-Entropy loss remains an important and unsolved problem. The problem is relevant beyond the Recommender Systems (RecSys) domain and is essential in any domain with an extensive vocabulary.

The autoregressive nature of transformers is a fundamental limiting factor on the inference speed of transformer-based models. This factor is widely acknowledged and addressed both in research and production-level models. However, all existing methods either require additional auxiliary models [34—37] or extensive retraining [38—42] and may degrade quality. So, it remains an open question whether a pre-trained autoregressive transformer could be upgraded for multi-token generation with little to no additional training.

Goals and problems addressed. The goal of this work is to advance the efficiency of resource-constrained Transformers by designing and evaluating new resource-aware training approaches and characterizing under-explored efficiency-relevant mechanisms.

To achieve this goal, the following problems are addressed in the dissertation:

1. Development of a method for transformer fine-tuning targeting the updates of MLP parameters, performing these updates in an efficient, expressive

manner, and outperforming existing MLP-targeted parameter-efficient fine-tuning (PEFT) methods on a wide range of benchmarks.

2. Development of methods for training large-catalog transformer-based sequential recommender systems that are hardware-efficient (outperform existing methods in training time), memory-efficient (outperform existing methods in memory requirements), and at the same time achieve State-Of-The-Art (SOTA) performance in recommendation metrics.

3. Characterization of non-autoregressive one-step decoding capabilities of frozen autoregressively pretrained LLMs and the description of the relation between those capabilities and autoregressive capabilities.

Scientific novelty.

1. The work introduces a novel PEFT method, that is, to the best of our knowledge, the first to demonstrate that the parameter updates of MLP layers during fine-tuning are sparse in some basis that is universal to all layers across the model. By leveraging this observation, the method achieves performance comparable to or better than SOTA PEFT methods, updating only around 1% of the parameters of the MLP layers.

2. The thesis also presents a novel adaptation of a locality-sensitive hashing (LSH)-based attention approximation method to the task of Cross-Entropy loss approximation in the context of an extensive catalog transformer-based sequential recommender system. To the best of our knowledge, the proposed method is the first LSH-based CE loss approximation method that enables efficient GPU computation via batching. The developed method outperforms the full CE loss and several popular approximation alternatives in a quality-memory trade-off paradigm across a wide range of memory-constrained regimes on a variety of datasets. Moreover, a different algorithm that improves upon this one has been developed. It uses a novel bucketing approach that yields empirically better approximation quality and overall performance. This algorithm demonstrates even better quality, outperforming alternatives in a quality-memory trade-off paradigm by a large margin.

3. This thesis also demonstrates a previously unknown phenomenon: non-au-toregressive decoding abilities in pretrained autoregressive LLMs. It specifies the requirements for such decoding and describes the structure of the underlying embedding space that enables it.

Theoretical and practical significance. Both the method developed for parameter-efficient fine-tuning and the methods for extensive catalogue Cross-Entropy loss approximation achieve State-Of-The-Art performance and thus have high practical significance. Besides that, the latter method demonstrates performance superior to the unapproximated Cross-Entropy loss, which raises questions about its optimally and the desired qualities of the suggested approximation. For example, this could shift the balance in the large-catalog vs. long sequence trade-off in LLM input tokenization. Lastly, the unexpected property of non-autoregressive decoding of autoregressively trained LLMs raises both practically and theoretically important questions of underlying non-autoregressive capabilities or, at the very least, information storage and decoding capabilities of the modern LLMs.

Methodology and research methods. The research mainly uses methods from Machine Learning and Deep Learning in particular, with substantial usage of methods and techniques from Numerical Linear Algebra. The methods and the code for the experiments are implemented in Python and the corresponding Deep Learning frameworks.

Main results submitted for the defense.

1. A novel selective PEFT method, called SparseGrad, has been developed. The technique is demonstrated to be on par, or, on most models/tasks tested, superior to other PEFT methods in terms of fine-tuning metrics.

2. A novel Cross-Entropy loss approximation method, called Reduced Cross-Entropy (RECE), is developed. The proposed method is evaluated on datasets from various domains. It outperforms existing approximating methods and the unapproximated Cross-Entropy loss in terms of the memory and training time required to achieve a given validation metric value.

3. A novel Cross-Entropy loss approximation method, called Scalable Cross-Entropy (SCE), that improves upon RECE, is developed. The method is based on a novel bucketing approach. It is demonstrated to be superior to other Cross-Entropy loss approximation methods, including RECE and the unapproximated Cross-Entropy loss, in both training memory vs. quality and training time vs. quality paradigms across a variety of datasets from various domains.

4. A novel phenomenon of non-autoregressive decoding of frozen autoregres-sively trained LLMs is demonstrated. Some properties of the underlying text embeddings are analyzed and described.

Validity of the obtained results. All the suggested methods and described phenomena were tested across multiple datasets and base models. Moreover, the experiments were conducted using multiple random initializations, reporting average results and their standard deviation, which is a standard practice for validating results. The source code for all the presented methods is published in open repositories. The main results of the dissertation were presented at the core A and A* conferences and published in four publications (the peer-reviewed conference proceedings). The detailed list of publications is presented below. Each of the chapters of the dissertation corresponds to one of these publications.

1. Chekalina V. A., Rudenko A., Mezentsev G., Mikhalev A., Panchenko A., SparseGrad: A Selective Method for Efficient Fine-tuning of MLP Layers. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. - 2024. - C. 14929-14939. [CORE A*]

2. Gusak D.*, Mezentsev G.*, Oseledets I., Frolov E., RECE: Reduced Cross-Entropy Loss for Large-Catalogue Sequential Recommenders. Proceedings of the 33rd ACM International Conference on Information and Knowledge Management. - 2024. - C. 3772-3776. [CORE A]

3. Mezentsev G.*, Gusak D.*, Oseledets I., Frolov E., Scalable cross-entropy loss for sequential recommendations with large item catalogs. Proceedings of the 18th ACM Conference on Recommender Systems. - 2024. - C. 475-485. [CORE A]

4. Mezentsev G., Oseledets I. Exploring the Hidden Capacity of LLMs for One-Step Text Generation. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. - 2025. - C. 22891-22900. [CORE A*]

Personal contribution of the author. The personal contribution of the author in the published works with the co-authors is as follows:

1. Chapter 2: Design and implementation of the main experiments (comparing with alternative methods (sections 2.3.1, & 2.3.2); implementation of the alternative methods.

2. Chapter 3: Authorship of the idea; method implementation; hyperparameter optimality section (section 3.2.1); joint participation (with second main author) in the development and implementation of performance experiments (section 3.3.4).

3. Chapter 4: Authorship of the main idea; implementation of the basic method variation; development, and implementation of experiments on hyperparameter selection (section 4.2.4); joint participation (with second main author) in development and implementation of performance experiments (sections 4.2.7, & 4.2.8, & 4.2.9); design and implementation of experiments concerning method performance explanation (section 4.2.10).

4. Chapter 5: All the results were obtained by the author personally.

Approbation. The main results of the dissertation were reported at the following scientific conferences:

1. The 2024 Conference on Empirical Methods in Natural Language Processing (Core A* rating), Miami, Florida, USA, November 2024;

2. 33rd ACM International Conference on Information and Knowledge Management (Core A rating), Boise, Idaho, USA, October 2024;

3. 18th ACM Conference on Recommender Systems (Core A rating), Bari, Italy, October 2024;

4. The 2025 Conference on Empirical Methods in Natural Language Processing (Core A* rating), Suzhou, China, November 2025;

5. 3rd Fall into ML conference on machine learning, Moscow, Russia, October 2024;

6. Recommender systems: new algorithms and modern practice, Moscow, Russia, June 2025;

7. 4th Fall into ML conference on machine learning, Moscow, Russia, October 2025;

Dissertation structure. The dissertation consists of an introduction, 5 chapters, and a conclusion. The dissertation is 140 pages long, including 25 figures, 5 algorithms, and 28 tables. The list of references contains 215 titles, including 4 publications by the author.

Organization of the Dissertation. Chapter 1 of the dissertation introduces the Transformer architecture, tasks that it used for, the primary building blocks on different levels of granularity, and the description of qualities of interest: time and memory, in the context of transformer training and inference. It also presents a review of the literature relevant to the dissertation topic. Chapter 2 introduces SparseGrad, a selective method of parameter-efficient fine-tuning targeting the parameters of MLP blocks. Chapters 3 and 4 present RECE and SCE - methods of Cross-Entropy loss approximation in the context of large-catalog transformer-based recommender systems. Finally, Chapter 5 introduces the phenomena of non-autore-gressive decoding of autoregressively trained LLMs and discusses the underlying text embeddings.

Заключение диссертации по теме «Другие cпециальности», Мезенцев Глеб Владимирович

Заключение

В данной диссертации рассматривается проблема эффективности моделей на основе трансформеров. Более конкретно, работа фокусируется на следующих трёх аспектах: эффективное селективное дообучение параметров MLP-блоков, эффективное обучение трансформерных рекомендательных систем для больших каталогов и неавторегрессионная генерация для авторегрессионно обученных LLM. Основные результаты, соответствующие этим направлениям, заключаются в следующем:

1. Разработан новый селективный PEFT-метод, названный SparseGrad. Метод использует тот факт, что обновления параметров всех MLP-сло-ёв становятся разреженными при переходе к специально подобранному базису. Метод обновляет лишь около 1% параметров, демонстрируя передовое (SOTA) качество на ряде задач при использовании BERT, RoBERTa и LLaMa-2 в качестве базовых моделей и превосходя аддитивный метод LoRA и селективный метод MeProp.

2. Предложен новый аппаратно-эффективный метод аппроксимации кросс-энтропийной функции потерь в контексте рекомендательных систем с большим каталогом, названный RECE. Этот метод сохраняет только наиболее информативные части вычислений с точки зрения будущего шага градиентного спуска. Это позволяет использовать преимущества функции потерь CE, известной своим передовым качеством, для больших каталогов, обучение на которых в противном случае было бы практически невозможно из-за высоких требований к памяти. RECE почти совпадает по качеству и требованиям к памяти с современными методами negative sampling на датасетах с небольшими каталогами и потребляет до 12 раз меньше памяти на датасетах с большими каталогами. В качестве альтернативы, при наличии расширенного бюджета по памяти он может улучшать качество (NDCG@10) до 8.19% по сравнению с другими подходами.

3. Предложен метод SCE, развивающий подход RECE. Улучшение основано на новом подходе к разбиению на «бакеты» (bucketing), обеспечивающем более эффективную аппроксимацию функции потерь. Благодаря этому SCE почти повсеместно превосходит альтернативы как по соот-

ношению «память при обучении / качество», так и по соотношению «время обучения / качество». В частности, мы продемонстрировали, что SCE достигает того же качества, что и другие недавно предложенные методы negative sampling, при этом используя до 100 раз меньше памяти и обучаясь до 6.7 раз быстрее на нескольких популярных датасе-тах. Либо, при одинаковом бюджете по памяти, SCE может обеспечить до 17.6% прироста качества (NDCG@10). На практике это означает, что SCE открывает широкий спектр возможностей для выбора конфигурации, соответствующей требованиям пользователя. Базовые принципы SCE могут быть распространены за пределы моделей рекомендательных систем и соответствующих функций потерь, потенциально принося пользу и другим областям, где распространены большие словари. Тот факт, что SCE в ряде случаев даже превосходит полную функцию потерь на основе кросс-энтропии, ставит под вопрос оптимальность CE и задаёт направление для будущих исследований: изучение скрытых преимуществ подобных рандомизированных методов аппроксимации, к которым относится SCE.

4. Наконец, продемонстрирован новый феномен неавторегрессионных возможностей декодирования у авторегрессионно обученных LLM. Предложена специальная схема предсказания с двумя вводимыми «прототокенами», делающая возможным такое поведение. Показано, что эта конкретная схема принципиально важна для декодирования почти 1000 точных токенов вне зависимости от источника текста. Высказывается гипотеза о различии функциональных ролей этих двух прототокенов. Показано, что прототокены содержат некоторую информацию, выходящую за пределы самого декодированного текста. Наконец, мы обнаруживаем, что пространство встраиваний, в котором существуют прототокены, обладает важными структурными свойствами: прототокены, соответствующие одному и тому же тексту, образуют локализованные, связные области, допускающие плавные переходы с помощью квадратичной интерполяции. Эти результаты позволяют предположить, что возможно построить энкодер, способный отображать в это пространство, что открывает путь к дальнейшим исследованиям в области неавторегрессионного инференса и обучения представлений.