Large Language Models - an Overview

The presentation traces the evolution of language modeling from early neural-network concepts and rule-based systems through n-gram models, recurrent neural networks, LSTMs, GRUs, word embeddings, transformers, and pretrained models such as BERT, GPT, and LLaMA.</p><p dir="ltr">The presentation explains how transformer architectures and self-attention overcome limitations of sequential models by allowing language models to capture relationships across longer passages of text.

It also compares encoder-based and decoder-based architectures, with particular emphasis on BERT and its bidirectional representation of language, masked-language-model training, and ability to capture context in complex biomedical text.</p><p dir="ltr">A major focus is the application of language models to clinical information extraction. Using pathology reports as an example, the presentation demonstrates how Clinical BERT can represent medical concepts such as p16-positive and p16-negative findings as embeddings and use semantic similarity to classify information from new clinical text.