Supervised Fine-Tuning (SFT)

Supervised Fine-Tuning (SFT) is the process of taking a general-purpose, pre-trained language model and further training it on a smaller, labeled dataset to specialize it for a specific downstream task. The pre-trained model has already learned a vast amount of information about language, grammar, and world knowledge from its initial training on a massive corpus. SFT leverages this existing knowledge as a starting point, which is far more efficient than training a model from scratch for every new task. The fine-tuning dataset consists of input-output pairs that are examples of the desired behavior. For a sentiment analysis task, the dataset would contain sentences (input) paired with their sentiment labels like 'positive', 'negative', or 'neutral' (output). For a text summarization task, it would contain articles (input) paired with human-written summaries (output). The fine-tuning process is similar to the pre-training phase but operates on a much smaller scale. The model processes the new data, makes predictions, compares them to the true labels using a loss function, and uses backpropagation to update its weights. A key decision in SFT is how much of the model to update. In 'full fine-tuning,' all the billions of parameters in the model are updated. While this can lead to the best performance, it is computationally expensive and requires a lot of memory. Alternatively, one might choose to 'freeze' the earlier layers of the model and only update the final few layers. The assumption is that the initial layers have learned general linguistic features that are broadly useful, while the later layers learn more task-specific representations.