Contributing
Getting Started for Local Development¶
We recommend using Tox
to setup the development environment. This will create a new virtual
environment with all of the required packages installed
and llm installed in editable mode with the necessary extras options.
$ git clone https://github.com/gpauloski/llm-pytorch
$ cd llm-pytorch
$ tox --devenv venv -e py310
$ . venv/bin/activate
Warning
Running Tox in a Conda environment is possible but it may conflict with
Tox's ability to find the correct Python versions. E.g., if your
Conda environment is Python 3.9, running $ tox -e p38 may still use
Python 3.9.
To install manually:
$ git clone https://github.com/gpauloski/llm-pytorch
$ cd llm-pytorch
$ python -m venv venv
$ . venv/bin/activate
$ pip install -e .[dev,docs,colossalai]
Continuous Integration¶
This package uses pre-commit and Tox for continuous integration (test, linting, etc.).
Linting and Type Checking (pre-commit)¶
To use pre-commit, install the hook and then run against files.
Tests (tox)¶
The entire CI workflow can be run with $ tox.
This will test against multiple versions of Python and can be slow.
Module-level unit-test are located in the tests/ directory and its
structure is intended to match that of llm/.
E.g. the tests for llm/preprocess/shard.py are located in
tests/preprocess/shard.py; however, additional test files can be added
as needed. Tests should be narrowly focused and target a single aspect of the
code's functionality, tests should not test internal implementation details of
the code, and tests should not be dependent on the order in which they are run.
Code that is useful for building tests but is not a test itself belongs in the
testing/ directory.
# Run all tests in tests/
$ tox -e py39
# Run a specific test
$ tox -e py39 -- tests/preprocess/shard.py::test_cli
Docs¶
If code changes require an update to the documentation (e.g., for function signature changes, new modules, etc.), the documentation can be built using MKDocs.
# With tox (will only build, does not serve)
$ tox -e docs
# Or manually
$ pip install -e .[docs]
$ mkdocs build --strict # Build only to site/index.html
$ mkdocs serve # Serve locally
Docstrings are automatically generated, but it is recommended to check the generated docstrings to make sure details/links/etc. are correct.