pro‎mp‎t engineering for avatar generation in the metaverse using language models

1/1/1900 12:00:00 AM

With the increasing use of language models, the need to develop models aligned with recent ad vancements has emerged. Due to its complex morphological–syntactic structure, variability in writ ing conventions, an‎d the lack of large-scale clean datasets, the Persian language still faces challenges in natural language processing. This study aims to mitigate these limitations by developing an‎d pre training an encoder model with a modern architecture, based on the ModernBERT framework, on a large corpus of Persian textual data, an‎d introducing it as an infrastructure for designing textual inputs in metaverse avatar generation systems. A large dataset containing more than 100 billion tokens from Persian texts across news, books, articles, blogs, an‎d social media was first collected. Thedatathenunderwentamulti-stagecleaningprocess—includingnormalization, removal of struc tural noise such as symbols an‎d emojis, an‎d elimination of duplicate sentences—for language model training. A new tokenizer based on the Byte Pair Encoding algorithm with a 50k vocabulary size was developed an‎d trained to optimally segment words. The encoder model was trained in three separate phases with progressively increasing input sequence lengths from 512 to 1024 an‎d even tually to 8192 tokens, enabling the processing of long documents. eva‎luation results on natural language processing tasks—including sentiment analysis, text classification, named entity recogni tion, natural language inference, part-of-speech tagging, an‎d question answering—showed that the developed model achieves competitive performance in most tasks an‎d provides up to a 6% improve ment in tasks such as text classification an‎d sentiment analysis. Furthermore, the increased effective context length significantly improved the model’s ability to process long documents. This study rep resents the first attempt to train a Persian model with a state-of-the-art architecture an‎d can serve as a foundational platform for developing encoder-based applications such as semantic search engines, text classification an‎d analysis models, avatar generation systems, an‎d information retrieva‎l tools. Finally, for future research directions, the development of mixture-of-experts architectural variants, expansion of domain-specific datasets, an‎d enhancement of eva‎luation using stan‎dardized Persian benchmarks are proposed.

مدل‌هاي رمزگذار , پيكره متني فارسي , پردازش زبان طبيعي

Encoder-only , Persian Corpus , Natural Language Processing

Mahmood Kalantari khalil abad

Nasser Mozayani