Разработка алгоритмов построения пайплайнов машинного обучения методами обработки естественного языка, основанными на данных тема диссертации и автореферата по ВАК РФ 00.00.00, кандидат наук Трофимова Екатерина Алексеевна

• Трофимова Екатерина Алексеевна
• кандидат наук
• 2025

Оглавление диссертации кандидат наук Трофимова Екатерина Алексеевна

Contents

Chapter 1. Introduction

1.1 Topic of the thesis

1.2 Relevance

1.3 Contributions of the thesis

1.4 Structure of the thesis

Chapter 2. Research landscape in ML automation and code generation

2.1 Automated machine learning (AutoML)

2.1.1 Foundations of AutoML

2.1.2 Techniques in AutoML

2.1.3 AutoML systems

2.2 Code Generation from natural language descriptions

2.2.1 Early methods for code generation

2.2.2 Large language models for code generation

2.2.3 Scoring metrics and benchmarks for code generation frameworks

2.3 Large language models planning in ML task execution

2.4 Datasets and taxonomies for ML automation

2.4.1 ML-related datasets

2.4.2 ML-related taxonomies

2.4.3 ML code annotation models

Chapter 3. Method for automated machine learning code generation

3.1 Dataset creation: Code4ML

3.1.1 Methodology for dataset construction

3.1.2 Metadata enrichment and contextualization

3.1.3 Dataset usage and limitations

3.2 Taxonomy design and classification

3.2.1 Development of the taxonomy for ML task classification

3.2.2 Automatic classification of code using taxonomy

3.2.3 Augmentation algorithm

3.3 Instruction Generation Inspired by Taxonomy

3.3.1 Motivation for an Intermediate Instruction Step

3.3.2 Structured instruction generation from task descriptions

3.4 Code generation framework: Linguacodus

3.4.1 Overview and key components of Linguacodus

3.4.2 Integration of instruction generation into Linguacodus

Chapter 4. Method validation

4.1 Evaluation criteria

4.1.1 Taxonomy-based classification metrics

4.1.2 Code generation evaluation methods

4.2 Performance of taxonomy-based classification

4.2.1 Experimental setup

4.2.2 Classification methods

4.2.3 Augmentation Algorithm

4.2.4 Results and analysis

4.3 Experiments in Instruction Generation for ML Pipelines

4.3.1 Experimental Setup

4.3.2 Results and analysis

4.4 Code generation comparative results

4.4.1 Experimental Setup

4.4.2 Results and analysis

4.5 Discussion

4.6 Framework limitations

Conclusion

Bibliography

List of Figures

List of Tables

Appendix A. Russian translation of the dissertation / Перевод диссертации на русский язык

Введение диссертации (часть автореферата) на тему «Разработка алгоритмов построения пайплайнов машинного обучения методами обработки естественного языка, основанными на данных»

Chapter 1. Introduction

1.1 Topic of the thesis

Automated code generation from natural language, commonly known as natural language programming (NLP), has the potential to simplify programming tasks and enhance the software development process [1], [2], [3], particularly in the domain of machine learning (ML) [4], emerging as a key tool across various domains and fundamentally changing how tasks are executed and problems are solved [5]. In this context, the ability to rapidly and accurately translate ambiguous task descriptions into functional code is becoming more essential. This dissertation advances the field of data-driven NLP by addressing the challenge of autonomously synthesizing ML pipelines from plain English task descriptions.

The problem of description-based ML code synthesis arises from the variability and abstraction inherent in human language. General task descriptions, such as "It is your job to predict the sales price for each house. For each ID in the test set, you must predict the value of the SalePrice variable," from competitions like Kaggle's "House Prices - Advanced Regression Techniques", are often provided without explicit instructions on data preprocessing, feature engineering, or specific model selection. Kaggle is an online platform widely used by data scientists and machine learning practitioners to access datasets, share solutions, and participate in competitions that tackle real-world ML problems.

The desired system takes a natural language description of a problem relying on machine learning as input and outputs Python code that implements a solution to address it. This functionality necessitates overcoming challenges posed by incomplete and ambiguous task descriptions, which often omit details regarding data preprocessing, model architecture, or evaluation criteria. To address these challenges, this thesis proposes the creation of a system capable of translating machine learning tasks described in natural

language into Python programming code. The following objectives are identified:

• Investigate existing ML workflows to identify recurring patterns and common practices that can be formalized into generative models capable of reproducing key patterns based on task descriptions in natural language.

• Develop methodologies for transforming identified patterns into executable Python code.

• Construct a comprehensive dataset encompassing ML task descriptions, corresponding pipelines, and code snippets to support system development and evaluation.

ML code generation by natural task description is crucial for democratizing access to machine learning by enabling both non-experts and experienced practitioners to efficiently develop ML applications. A system capable of interpreting natural language descriptions and generating accurate, modular, and adaptable ML pipelines lowers the barrier to entry for leveraging machine learning, fostering innovation across diverse domains. Addressing these challenges requires innovations in NLP modeling, task-specific contextual understanding, and the design of adaptable, error-resilient systems capable of generating high-quality ML pipelines from natural language inputs.

Existing approaches lack the flexibility to generalize across diverse datasets and tasks and often fail to adapt to variations in task descriptions. They also provide limited capabilities for assessing the quality and efficiency of the generated code. This dissertation proposes a novel framework that enhances the generation of ML pipelines by leveraging task-specific instructions reflecting the main stages of ML pipelines. These instructions, derived from natural language inputs, guide the transformative nature of the framework.

Figure 1 presents the systematic approach to generating ML pipelines from natural language descriptions. The process begins with constructing a comprehensive dataset of ML tasks and corresponding code snippets. Dimensionality reduction techniques are employed to organize the learning space, while a taxonomy tree enriches the dataset through classification-based annotations. Although the classification algorithm is primarily used for corpus enrichment and to inspire the transformative principles of the framework, the final system depends on large language models (LLMs) to infer and execute task-specific instructions. These LLM-inferred instructions drive the automatic generation of modular

Goal

How?

Subtasks

Solution artifacts/ frameworks

Data collection Construct a corpus of ML tasks & code snippets Code4ML: large corpus of ML code

ML code generation from task description Learning space dimensionality reduction

Use data and dimensionality - reduction -technique to generate code

Propose a taxonomy tree

Create a framework for ML

code generation by task description

ML code taxonomy tree

Automative annotation of ML code

Classify code into

taxonomy types

Linguacodus framework

Figure 1: Schematic of ML pipeline synthesis from plain natural language task descriptions.

and adaptable code tailored to diverse ML tasks. Future integration of classification-based techniques could further enhance the framework by ensuring the consistency and correctness of inferred instructions, thereby increasing its robustness. This potential extension highlights the adaptability of the proposed system and its ability to incorporate advancements in classification and validation methodologies.

1.2 Relevance

The automation of machine learning (ML) pipeline synthesis from natural language is a pivotal step in democratizing access to machine learning tools. As ML models are increasingly integrated into various industries, there is a growing need for systems that simplify the model development process for non-experts while retaining the flexibility and precision required for more complex tasks. This dissertation addresses these challenges by focusing on two critical aspects: the creation of a large-scale, annotated corpus of ML code and the development of a robust framework capable of transforming task descriptions into executable code. While current AutoML systems streamline ML workflows, they still lack the ability to generate code directly from natural language inputs, underscoring the necessity for novel solutions that can handle this transition effectively.

One of the primary challenges in ML code generation is the lack of comprehensive, high-quality datasets that accurately capture the relationship between natural language descriptions and their corresponding domain-specific code implementations. Existing datasets often exhibit limited domain generalization, making them insufficient for the diverse and nuanced requirements of ML pipeline synthesis. This domain specificity exacerbates domain shift, where models trained on one dataset struggle to generalize effectively to new domains, particularly in ML workflows. Generic datasets, such as CoNaLa [6] or BigCode [7], while broad in scope, typically lack the depth and variety needed for learning the complex patterns inherent to ML tasks. These limitations hinder the development of

robust models capable of generating accurate and context-aware ML pipelines from natural language inputs.

The lack of such tailored datasets further complicates the ability of current AutoML systems to generate robust and reliable ML code directly from high-level descriptions. Without datasets that link natural language descriptions to specific code solutions in the ML context, existing models may fail to produce code that aligns with the user's intended task or the underlying complexities of ML workflows. This gap in training data is one of the primary factors limiting the current ability to automate ML code generation at scale.

To address these gaps, this dissertation introduces the Code4ML corpus, which directly tackles the challenge by curating a dataset that combines a wide range of ML tasks with annotated code snippets and metadata. By designing this corpus to include various ML tasks, this work aims to mitigate the problem of domain shift and enhance the model's ability to accurately map task descriptions to ML code. This annotated corpus is key to improving the generalization of models trained on it, allowing them to handle a broader range of ML tasks and provide accurate, context-aware outputs for ML pipeline generation.

This research builds upon prior approaches and introduces solutions informed by modern transformer-based architectures. By integrating the latest advancements in large language models (LLMs) with a carefully curated ML code corpus, the work advances the state-of-the-art in automating ML tasks from natural language descriptions. Recent developments, particularly in LLMs such as CodeBERT [8] and CoditT5 [9], have shown impressive capabilities in code generation and recommendation, laying the foundation for more sophisticated ML applications. These advancements, exemplified by models like GPT-3 [10] and Codex [11], have proven effective in generating code from natural language prompts. However, challenges remain in managing the inherent ambiguity of natural language and the specificity needed for accurate code generation. This research takes these limitations into account and seeks to advance the state-of-the-art in ML pipeline generation.

This research's relevance is further heightened within the context of automated machine learning (AutoML), where existing solutions such as AutoGluonTabular [12], H2O

AutoML [13], and LightAutoML [14] are designed to streamline structured workflows and offer tailored solutions for specific domains. Additionally, frameworks like Hugging-GPT [15] demonstrate the increasing versatility of LLMs in tasks such as task planning, model selection, subtask execution, and result summarization across a wide array of AI challenges.

While AutoML focuses on optimizing structured workflows, the challenge of generating ML code directly from natural language descriptions represents a crucial frontier for seamless system integration and domain-specific customization. The proposed dissertation addresses this evolving landscape of ML code generation, acknowledging both the strengths and limitations of current LLMs and aims to advance the state-of-the-art in ML pipeline synthesis from natural language descriptions.

Although other frameworks, such as Data Interpreter [16] and MapCoder [17], have showcased their strengths in competitive problem-solving through retrieval-based techniques [18] and iterative refinement during code generation, they fall short in addressing key aspects. The Data Interpreter, for instance, excels in automating end-to-end workflows with its hierarchical graph modeling, dynamic task adaptation, and programmable node generation capabilities. It has demonstrated significant performance improvements in various benchmarks, including a 25% accuracy boost on InfiAgent-DABench and a notable 26% improvement on the MATH dataset. However, its focus on small-scale datasets and limited application to ML pipelines underscores the need for broader evaluation benchmarks and more extensive scalability tests. Similarly, MapCoder's iterative refinement is effective but lacks a specialized focus on data science-specific challenges. This highlights the necessity for robust, versatile frameworks tailored to address the demands of ML task automation while ensuring scalability, dynamic adaptability, and comprehensive evaluation criteria.

1.3 Contributions of the thesis

The goal of this research is to develop an efficient method for converting plain language descriptions of machine learning tasks into executable Python code.

Key results and conclusions

The dissertation presents a series of scientific contributions that advance the automation of machine learning pipeline synthesis by leveraging novel algorithms and methodologies. These contributions emphasize the integration of natural language processing techniques with ML code generation and task execution. The key contributions are as follows:

• The dissertation introduces Linguacodus, a framework for generating executable ML code directly from natural language descriptions of ML tasks. The framework utilizes a two-step process: transforming unstructured task descriptions into structured instructions and synthesizing these into Python code. Linguacodus demonstrates competitive performance, successfully generating structured, compilable code in 10 out of 10 sampled Kaggle competitions, outperforming GPT-3.5, which compiled in 7 out of 10 tasks, and Data Interpreter, which compiled in only 4. Moreover, Linguacodus consistently achieved top Kaggle leaderboard percentiles, demonstrating its feasibility in automating ML pipeline creation with superior metrics.

• A novel approach is proposed for leveraging LLMs to plan ML task execution. This method converts unstructured natural language inputs into actionable instructions, generating problem statements and execution plans tailored to specific ML tasks. This contribution highlights the role of LLMs as intelligent agents in ML automation.

• A taxonomy-based classification system for ML tasks is introduced, enabling robust generalization across diverse domains. This taxonomy achieves anF1 score of 0.871 on test data and underpins the task interpretation and instruction generation phases of the Linguacodus framework. It ensures accurate alignment with intended task objectives.

• The dissertation provides Code4ML, a comprehensive collection comprising approxi-

mately 2.5 million Python code snippets derived from public Kaggle notebooks. These snippets are systematically enriched with metadata that contextualizes the corresponding machine learning tasks, providing key insights into the nature of each task and its execution. Each notebook is also accompanied by detailed natural language descriptions of the associated competition.

• The proposed algorithms and frameworks are designed to accommodate a wide range of ML tasks, from standard regression and classification to complex, domain-specific challenges. This scalability ensures the practical applicability of the methods in both academic research and industrial settings.

Theoretical and practical significance. The dissertation presents the first framework demonstrating the capability to generate executable code for machine learning tasks using deep learning methods, achieving competitive quality metrics. A method for generating textual descriptions of machine learning tasks based on their source code was developed, enabling the augmentation and expansion of the Code4ML corpus. The proposed approach to automatic task category annotation, utilizing a custom taxonomy and state-of-the-art language models, demonstrates a high level of accuracy and opens new possibilities for classifying and systematizing machine learning tasks. The use of a combination of pre-trained and fine-tuned language models for code generation, evaluation, and ranking is unique and achieves significant improvements in key quality metrics compared to existing AutoML systems.

The practical significance of the work is substantial. The results obtained during the research open new opportunities for automating the development of ML pipelines. The developed Code4ML corpus and the proposed taxonomy of machine learning tasks provide a foundation for advancing methods of task classification and code synthesis. The Lingua-codus framework, which for the first time demonstrates the ability to generate executable code with competitive quality metrics, offers users a tool to transform textual descriptions into ready-to-use ML solutions, lowering barriers to adopting machine learning in practice. This is particularly relevant for professionals without deep programming expertise.

The results can be applied in various areas, including the development of educa-

tional platforms, automation of scientific research, data engineering, and the creation of applications utilizing AutoML. The universality and reproducibility of the proposed solutions are ensured by open access to the code and detailed instructions for reproducing the experiments. This makes the results accessible to a broad audience and facilitates their integration into practical projects.

Publications and probation of the work

The thesis author is the corresponding author for the initial two articles on the thesis topic and holds the primary authorship position for the major article central to the thesis subject.

• Anastasia Drozdova, Ekaterina Trofimova, Polina Guseva, Anna Scherbakova, An-drey Ustyuzhanin. Code4ML: a large-scale dataset of annotated Machine Learning code. //PeerJ Computer Science 9:e1230, 2023. DOI: 10.7717/peerj-cs.1230 (Q1, Impact Factor 2024 3.8) [Scopus]

• Valeriy Berezovskiy, Anastasia Gorodilova, Ekaterina Trofimova, Andrey Ustyuzhanin. Machine learning code snippets semantic classification. //PeerJ Computer Science 9:e1654, 2023. DOI: 10.7717/peerj-cs.1654 (Q1, Impact Factor 2024 3.8) [Scopus]

• Ekaterina Trofimova, Emil Sataev and Andrey Ustyuzhanin. Linguacodus: a syn-ergistic framework for transformative code generation in machine learning pipelines. //PeerJ Computer Science 10:e2328, 2024. DOI: 10.7717/peerj-cs.2328 (Q1, Impact Factor 2024 3.8) [Scopus]

Personal contribution In the first two papers, the candidate served as the lead author and took full responsibility for the entire research process. She developed the concept and design of the experiments, conducted the experiments, performed data analysis, carried out computational work, and wrote the manuscript. In the third paper, the candidate also designed and planned the experiments, conducted them, analyzed the data, performed computational work, prepared figures and tables, drafted the manuscript, and approved the final version. Thus, the candidate played a key role in the methodological development and execution of the experiments, as well as made a significant contribution to the preparation and publication of the results.

Reports at conferences

The 11th International Conference on Analysis of Images, Social Networks and Texts, September 29, 2023, poster. "Machine learning pipelines synthesis with LLMs".

Reports at seminars

• Seminar of the Laboratory of methods for big data analysis research group, HSE University, Moscow, 06.03.2023. Topic: "Natural language to machine learning (NL2ML): automatic generation of machine learning pipelines".

• Seminar ofthe Institute for Functional Intelligent Materials, NUS, Singapore, 12.04.2024. Topic: "Automatic code generation: material properties prediction."

1.4 Structure of the thesis

The chapters of this dissertation are structured to provide a logical progression from the foundational theory to the practical implementation of the proposed system. Chapter 1, Introduction, introduces the research problem, outlines the objectives of the thesis, and highlights the significance of automating ML pipeline generation from natural language descriptions. It presents the key research questions, the methodology adopted, and the contributions of the thesis. The chapter sets the stage by discussing the relevance of this research to the field of ML automation and its potential to democratize access to machine learning tools.

Chapter 2 provides a comprehensive overview of state-of-the-art ML automation and code generation from natural language descriptions. Section 2.1 explores the foundations and techniques of Automated Machine Learning (AutoML), examining current AutoML systems and their limitations, particularly their inability to directly generate executable code from natural language descriptions. Section 2.2 reviews early methods of code generation, including latent predictor networks, transformer architectures, abstract syntax tree-based models, and long-context history-conditioned models. The chapter further discusses the role of pre-trained large language models in advancing code generation

capabilities. Section 2.3 introduces the concept of using large language models for planning in ML task execution, emphasizing their potential for improving automation in ML workflows. Finally, Section 2.4 addresses the importance of datasets and taxonomies for enriching ML automation systems.

Chapter 3 delves into the core methodology employed in this dissertation. Section 3.1 discusses the creation of the Code4ML dataset, which forms the backbone of the code generation system. It details the methodology for dataset construction, metadata enrichment, and contextualization. Section 3.2 describes the development of a taxonomy for ML task classification, exploring the automatic classification of ML code and the augmentation algorithm used to enhance the dataset. Section 3.3 introduces the generation of intermediate instructions based on the taxonomy, discussing the motivations for this step and the process of generating structured task instructions that guide code generation. Section 3.4 presents Linguacodus, the integrated framework that leverages the Code4ML corpus, taxonomy, and instruction generation to transform natural language task descriptions into executable ML code.

Chapter 4 presents the validation and evaluation of the proposed methodology. Section 4.1 defines the evaluation criteria for taxonomy-based classification, instruction generation, and code generation. Instruction evaluation is based on criteria such as relevance, clarity, completeness, and actionability. Section 4.2 evaluates the performance of taxonomy-based classification, providing details of experimental setups, classification methods, and results from classical machine learning models and models like BERT and CodeBERT. Section 4.3 focuses on instruction generation, evaluating the quality of instructions inferred by models such as Code Llama and Fine-Tuned Llama 2. Section 4.4 compares the performance of the code generation component with existing systems, discussing the experimental setup, results, and their implications. Section 4.5 provides a discussion of the results, analyzing the strengths and limitations of the framework, while Section 4.6 outlines the framework's limitations and directions for future research.

Finally, Conclusion summarizes the key findings of the dissertation. The chapter reflects on the significance of the proposed system in advancing the automation of ML

pipeline generation, highlighting its potential to democratize access to machine learning by enabling non-experts to effectively use ML tools.

Заключение диссертации по теме «Другие cпециальности», Трофимова Екатерина Алексеевна

Заключение

В этой диссертации представлены несколько ключевых вкладов в область автоматизации машинного обучения, особенно в области генерации исполняемого кода машинного обучения из описаний на естественном языке. Основные вклады обобщены следующим образом:

• Разработан и опубликован в журнале Q1 (автор диссертации является основным соавтором) новый фреймворк, который генерирует исполняемый код для задач машинного обучения (ML) непосредственно из их описаний на естественном языке. Это первая система такого рода, достигающая конкурентоспособных результатов. Фреймворк основан на сочетании предварительно обученных и дообученных больших языковых моделей (LLM).

В выборке из 10 соревнований Kaggle предложенная система успешно скомпилировала код для всех 10 задач, демонстрируя надежную архитектуру и адаптивность. Для сравнения, GPT-3.5 справился с компиляцией кода только в 7 из 10 случаев, а автономный графово-иерархический фреймворк Data Interpreter — лишь в 4. Помимо более высокой частоты успешной компиляции, Linguacodus стабильно превосходил GPT-3.5 и Data Interpreter по ключевым метрикам машинного обучения.

• Code4ML — масштабный корпус, включающий около 2,5 миллиона фрагментов Python-кода, извлеченных из публичных ноутбуков Kaggle, был предложен и опубликован в журнале Q1 (автор диссертации является вторым соавтором). Эти фрагменты обогащены метаданными, а каждый ноутбук сопровождается текстовыми описаниями соответствующих соревнований.

• Предложен новый метод аннотирования кода ML, основанный на таксономическом дереве ML, которое отражает основные этапы ML-конвейера. Кроме того, предоставлен инструмент для аннотирования, облегчающий непрерывное обогащение и расширение набора данных, поддерживая его развитие для исследований

и практического применения в автоматизации ML (журнал Q1, автор диссертации является вторым соавтором).

• Введена система классификации ML-задач на основе таксономии, опубликованная в журнале Q1, которая обеспечивает надежную генерализацию в различных областях. Эта система достигает значения Fl-меры 0,871 на тестовых данных, демонстрируя высокую точность в классификации ML-кода. Она поддерживает фреймворк Linguacodus, обеспечивая точное соответствие между интерпретацией задач и генерацией инструкций. Благодаря улучшенной классификации задач этот вклад повышает надежность и масштабируемость автоматического синтеза кода.