Composable PyTorch transformers for research.
Build vision, spatial, and sequence models with interchangeable attention, positional encodings, normalization, and feedforward components. Start with a preset, then replace individual components as your experiments evolve.
- Experiment with architecture: explicit, serializable configurations and structural protocols let you supply your own components.
- Work with variable-length data: run the same encoder weights on padded or packed token batches, with explicit token coordinates.
- Connect research and deployment: examples cover image patches, point clouds, causal modeling, custom components, and cached ONNX Runtime decoding.
Install from PyPI with pip:
pip install --upgrade stackformersOr add it to your uv project:
uv add stackformersUse --pre to include newer prereleases when upgrading:
pip install --upgrade --pre stackformersWith uv:
uv add --upgrade --prerelease allow stackformersFor an existing virtual environment, uv pip install --upgrade --pre stackformers
is also supported. These commands select the newest compatible published version,
including prereleases. Pin the version you validate for an experiment; see the
research guide.
import torch
import stackformers as sf
model = sf.TransformerEncoder(
sf.plain_encoder_config(dim=64, heads=1, num_layers=2, ff_mult=3.0)
).eval()
sequences = [torch.randn(8, 64), torch.randn(5, 64)]
padded = sf.PaddedTensor.from_list(sequences)
with torch.no_grad():
padded_output = model(padded) # (2, 8, 64)
packed_output = model(padded.to_packed()) # (13, 64)The two calls share model weights. Packed output contains valid tokens only; padded output retains batch and sequence axes. See the layout parity example for a numerical comparison. Packed input does not guarantee faster execution: backend, hardware, dtype, and length distribution matter.
Browse complete examples for common tasks:
| Your task | Example |
|---|---|
| Image patches and 2-D positions | Image encoder |
| Continuous spatial coordinates | Point-cloud encoder |
| Unequal sequence lengths | Padded and packed |
| Replace a model component | Custom encoders |
| Autoregressive token modeling | Causal language model |
| Export an encoder–decoder | ONNX Runtime guide |
The examples guide includes commands for running the repository examples.
These are integration examples with synthetic inputs, not trained models or task-accuracy benchmarks. Stackformers provides transformer components; your application owns tokenization, data processing, training, and task heads.
| Component | Included choices |
|---|---|
| Attention | Global self/cross attention, local self-attention, GQA/MQA |
| Token layout | Padded or packed, with explicit positions |
| Positional encoding | RoPE-1D, RoPE-2D, continuous-coordinate RoPE-ND, or none |
| Feedforward | SwiGLU, HardSwishGLU, GEGLU, GELU, ReLU² |
| Normalization | RMSNorm, LayerNorm; pre, post, sandwich, or reordered placement |
| Model assembly | Encoders, decoders, cross-attenders, variable-width encoders |
| Deployment | Projection packing, compile/export paths, cached decoder adapters |
Presets build components from discriminated Pydantic configurations. For deeper changes, wire modules directly or implement an owning module's protocol. The configuration and API guide covers both approaches.
- Global attention uses PyTorch SDPA. Eligible CUDA fp16/bf16 packed workloads use PyTorch's variable-length attention; other paths can fall back to padded execution.
- Compile and export support depend on the component and input path. Eager list/nested adapters and deployment cache strategies have separate contracts.
- Cached deployment supports specified built-in classes; custom forward overrides need their own deployment integration.
- No speedup or downstream accuracy improvement is claimed without a matching benchmark.
See compatibility evidence and inference contracts before selecting a deployment path.
- Examples and expected behavior
- Research reproducibility and citation
- API, layouts, configuration, and migration
- Deployment and inference
- Release history
- Contributing
- Adoption and evidence roadmap
git clone https://github.com/Red-Eyed/stackformers.git
cd stackformers
just sync
just checkThe Justfile owns formatting, linting, type-checking, tests, and builds.
If Stackformers contributes to your research implementation, cite the software using CITATION.cff and report the exact version or commit used. No associated research paper or conference endorsement is implied.
Released under the MIT license.