Foundation models for natural language processing : pre-trained language models integrating media

Covers basic natural language processing models, pre-trained language models BERT, GPT, and sequence-to-sequence converters, as well as the concepts of self-attention and context-sensitive embedding. Various approaches to improving these models are then discussed, such as expanding the pre-training parameters, increasing the length of input texts, or incorporating additional knowledge. An overview of the best performing models is then provided for about twenty application areas, e.g., question answering, translation, story generation, dialogue systems, image generation from text, etc. For each application area, the strengths and weaknesses of existing models are discussed, and an overview of further developments is provided. In addition, links to freely available code are provided. The concluding chapter summarizes the economic opportunities, risk mitigation, and potential developments of AI.