llm.trainers.bert.convert

llm/trainers/bert/convert.py

Convert pretraining checkpoints to HuggingFace pretrained models.

Example

After training, use the CLI to convert the latest checkpoint. Note that this uses the same configuration file used for training.

python -m llm.trainers.bert.convert --config /path/to/config.py --model-path /path/to/output/dir

Then, use AutoModel.from_pretrained to load the model back. Learn more about AutoModels here.

from transformers import AutoModelForMaskedLM

model = AutoModelForMaskedLM.from_pretrained('/path/to/output/dir')