MLM

A pre-training objective that trains a model to predict randomly masked tokens in a sentence using bidirectional context. A Masked Language Model (MLM) is trained by randomly masking (hiding) a percentage of tokens in input text and requiring the model to predict the original tokens using the surrounding context in both directions.

The MLM objective, used in BERT and its variants, forces the model to develop rich bidirectional representations because it must integrate context from both the left and right of each masked position.

Masked Language Model is a foundational concept in Training Technique. A pre-training objective that trains a model to predict randomly masked tokens in a sentence using bidirectional context.