Skip to content

LLM-PyTorch

llm.preprocess

Initializing search

gpauloski/llm-pytorch

Home
Installation
Guides
API Reference
Contributing

LLM-PyTorch

gpauloski/llm-pytorch

Home
Home
Installation
Installation
- Polaris
Guides
Guides
API Reference
API Reference
- LLM PyTorch
  LLM PyTorch
  - llm
    
    llm
    
    llm.checkpoint
    
    llm.config
    
    llm.datasets
    
    llm.datasets
    
    llm.datasets.bert
    
    llm.datasets.roberta
    
    llm.datasets.sharded
    
    llm.engine
    
    llm.engine
    
    llm.engine.accumulation
    
    llm.engine.amp
    
    llm.engine.base
    
    llm.engine.initialize
    
    llm.environment
    
    llm.initialize
    
    llm.kfac
    
    llm.loss
    
    llm.models
    
    llm.models
    
    llm.models.bert
    
    llm.optimizers
    
    llm.preprocess
    
    llm.preprocess
    
    llm.preprocess.download
    
    llm.preprocess.format
    
    llm.preprocess.roberta
    
    llm.preprocess.shard
    
    llm.preprocess.tokenizer
    
    llm.preprocess.utils
    
    llm.schedulers
    
    llm.timer
    
    llm.trainers
    
    llm.trainers
    
    llm.trainers.bert
    
    llm.trainers.bert
    
    llm.trainers.bert.convert
    
    llm.trainers.bert.data
    
    llm.trainers.bert.main
    
    llm.trainers.bert.utils
    
    llm.trainers.gpt
    
    llm.trainers.gpt
    
    llm.trainers.gpt.arguments
    
    llm.trainers.gpt.data
    
    llm.trainers.gpt.main
    
    llm.trainers.gpt.model
    
    llm.trainers.gpt.optimizer
    
    llm.utils
- CLI Reference
Contributing
Contributing
- Style Guide
- Issues and Pull Requests

Table of contents

preprocess

llm.preprocess

llm/preprocess/__init__.py

Pretraining preprocessing CLIs.

See the CLI Reference for usage instructions.

Copyright © 2023-Present Greg Pauloski

Made with Material for MkDocs