Original Paper: https://arxiv.org/abs/2305.09137
By: Yuxian Gu, Li Dong, Furu Wei, Minlie Huang
Abstract:
In-context learning, where pre-trained language models learn to perform tasks from task examples and instructions in their contexts, has attracted much attention in the NLP community. However, the ability of in-context learning is not fully exploited because language models are not explicitly trained to learn in context. To this end, we propose PICL (Pre-training for In-Context Learning), a framework to enhance the language models' in-context learning ability by pre-training the model on a large collection of "intrinsic tasks" in the general plain-text corpus using the simple language modeling objective. PICL encourages the model to infer and perform tasks by conditioning on the contexts while maintaining task generalization of pre-trained models. We evaluate the in-context learning performance of the model trained with PICL on seven widely-used text classification datasets and the Super-NaturalInstrctions benchmark, which contains 100+ NLP tasks formulated to text generation. Our experiments show that PICL is more effective and task-generalizable than a range of baselines, outperforming larger language models with nearly 4x parameters. The code is publicly available at
As artificial intelligence (AI) continues to advance, the importance of pre-trained language models (PLMs) like GPT and BERT has become increasingly evident.
These models are excellent at mimicking human text, but their ability to learn new tasks from context, known as in-context learning (ICL), has been a challenge.
This has led to the creation of the Pre-training for In-Context Learning (PICL) framework, aimed at improving PLMs' adaptability.
This blog post breaks down the PICL framework, its methodology, benefits, and its implications for AI in the enterprise.
While PLMs have been a breakthrough for natural language processing (NLP), their performance in adapting to new tasks through context alone has been less than ideal.
This limitation affects their ability to adjust to specific, real-world tasks without extensive additional training.