سيدمحسن طبسي

Language models (LMs) are one of the core components in natural language processing (NLP). Nowadays, the development of language models based on deep neural networks and the emergence of pre-trained LM have led to dramatic advances in solving various tasks in this field. This improvement can be seen in a variety of issues related to language understanding, including question answering and various types of text classification. In this direction, the most important step has been taken with the introduction of transformer architecture and then the development of language models based on it, including BERT and GPT. One of the key features of transformer architecture is the of training on a large volume of textual data, scalability in terms of number of parameters and ease of transferring linguistic knowledge to various tasks. After observing the performance improvement in various NLP tasks, a new approach was developed in the use of transformer-based LMs. This approach tries to make the most of the knowledge embedded in the model by pro‎mp‎ting it using natural language. For example, the task of emotion recognition, the model is asked in natural language: "_Input text_ I feel very __" and the language model makes remarks about the emotion of the input text by predicting the next word (for example "happy"). The important point about this approach (which is called pro‎mp‎ting) is its remarkable performance in few-shot setting, suggesting that the language model has the knowledge needed to solve these tasks -to some extent- and it performs very good if the task description is well understood. To date, the most successful classification methods based on pro‎mp‎ting have used masked language models (MLMs). The main limitation of these methods is the constant number of mask tokens (usually one) and thus the limitation of the LM in response. In this study, we propose a new few-shot text classification method, which uses a causal language model (CLM) such as GPT2 to provide more flexibility for language model pro‎mp‎ting. In the proposed method, several descriptions for each class are first generated automatically or manually. Then, by asking the LM how well each description relates to the input text, a decision is made about the output. The key features of this method include explainability, flexibility for language model response, possibility of description engineering by expert and the use of standard language models without any assumption. Quantitative and qualitative analyzes show the acceptable performance of the proposed method in four classification tasks including news topic classification, sentiment analysis, emotion recognition, and intention detection.

پردازش زبان طبيعي , مدل زباني از پيش آموزش ديده , يادگيري با نمونه هاي معدود , طبقه بندي متن

Natural Language Processing , Pre-trained Language Model , few-shot learning , Text classificatioد

Seyed Mohsen Tabasi

Mohammad Taher Pilehvar