Generative AI Advance Fine-Tuning for LLMs

IBM via Coursera

Coursera based on 71 ratings

Likes

Overview

Fine-tuning a large language model (LLM) is crucial for aligning it with specific business needs, enhancing accuracy, and optimizing its performance. In turn, this gives businesses precise, actionable insights that drive efficiency and innovation. This course gives aspiring gen AI engineers valuable fine-tuning skills employers are actively seeking. During this course, youll explore different approaches to fine-tuning and causal LLMs with human feedback and direct preference. Youll look at LLMs as policies for probability distributions for generating responses and the...

Syllabus

Different Approaches to Fine-Tuning

In this module, youll begin by defining instruction-tuning and its process. Youll also gain insights into loading a dataset, generating text pipelines, and training arguments. Further, youll delve into reward modeling, where youll preprocess the dataset and apply low-rank adaptation (LoRA) configuration. Youll also learn to quantify quality responses, guide model optimization, and incorporate reward preferences. Youll also describe reward trainer, an advanced training technique to train a model, and reward model loss using Hugging Face. The labs, in this module will allow practice on instruction-tuning and reward models.

Fine-Tuning Causal LLMs with Human Feedback and Direct Preference

In this module, youll describe the applications of large language models (LLMs) to generate policies and probabilities for generating responses based on the input text. Youll also gain insights into the relationship between the policy and the language model as a function of omega to generate possible responses. Further, this module will demonstrate how to calculate rewards using human feedback incorporating reward function, train response samples, and evaluate agents performance. Youll also define the scoring function for sentiment analysis using PPO with Hugging Face. Youll also explain the PPO configuration class for specific models and learning rate for PPO training and how the PPO trainer processes the query samples to optimize the chatbots policies to get high-quality responses. This module delves into direct preference optimization (DPO) concepts to provide optimal solutions for the generated queries based on human preferences more directly and efficiently using Hugging Face. The labs in this module provide hands-on practice on human feedback and DPO. Methods like PPO and reinforcement learning are quite involved and could be considered subjects of study on their own. While we have provided some references for those interested, you are not expected to understand them in depth for this course