Skip to content

About

Composable PyTorch transformers for vision, spatial, and sequence research.

Topics

Resources

Contributing

Stars

3 stars

Watchers

0 watching

Forks

Repository files navigation

Stackformers

CI PyPI License: MIT

Composable PyTorch transformers for research.

Build vision, spatial, and sequence models with interchangeable attention, positional encodings, normalization, and feedforward components. Start with a preset, then replace individual components as your experiments evolve.

  • Experiment with architecture: explicit, serializable configurations and structural protocols let you supply your own components.
  • Work with variable-length data: run the same encoder weights on padded or packed token batches, with explicit token coordinates.
  • Connect research and deployment: examples cover image patches, point clouds, causal modeling, custom components, and cached ONNX Runtime decoding.

Install

Install from PyPI with pip:

pip install --upgrade stackformers

Or add it to your uv project:

uv add stackformers

Try the latest prerelease

Use --pre to include newer prereleases when upgrading:

pip install --upgrade --pre stackformers

With uv:

uv add --upgrade --prerelease allow stackformers

For an existing virtual environment, uv pip install --upgrade --pre stackformers is also supported. These commands select the newest compatible published version, including prereleases. Pin the version you validate for an experiment; see the research guide.

Run your first encoder

import torch
import stackformers as sf

model = sf.TransformerEncoder(
    sf.plain_encoder_config(dim=64, heads=1, num_layers=2, ff_mult=3.0)
).eval()

sequences = [torch.randn(8, 64), torch.randn(5, 64)]
padded = sf.PaddedTensor.from_list(sequences)
with torch.no_grad():
    padded_output = model(padded)              # (2, 8, 64)
    packed_output = model(padded.to_packed())  # (13, 64)

The two calls share model weights. Packed output contains valid tokens only; padded output retains batch and sequence axes. See the layout parity example for a numerical comparison. Packed input does not guarantee faster execution: backend, hardware, dtype, and length distribution matter.

Explore examples

Browse complete examples for common tasks:

Your task Example
Image patches and 2-D positions Image encoder
Continuous spatial coordinates Point-cloud encoder
Unequal sequence lengths Padded and packed
Replace a model component Custom encoders
Autoregressive token modeling Causal language model
Export an encoder–decoder ONNX Runtime guide

The examples guide includes commands for running the repository examples.

These are integration examples with synthetic inputs, not trained models or task-accuracy benchmarks. Stackformers provides transformer components; your application owns tokenization, data processing, training, and task heads.

What can you change?

Component Included choices
Attention Global self/cross attention, local self-attention, GQA/MQA
Token layout Padded or packed, with explicit positions
Positional encoding RoPE-1D, RoPE-2D, continuous-coordinate RoPE-ND, or none
Feedforward SwiGLU, HardSwishGLU, GEGLU, GELU, ReLU²
Normalization RMSNorm, LayerNorm; pre, post, sandwich, or reordered placement
Model assembly Encoders, decoders, cross-attenders, variable-width encoders
Deployment Projection packing, compile/export paths, cached decoder adapters

Presets build components from discriminated Pydantic configurations. For deeper changes, wire modules directly or implement an owning module's protocol. The configuration and API guide covers both approaches.

Compatibility and limits

  • Global attention uses PyTorch SDPA. Eligible CUDA fp16/bf16 packed workloads use PyTorch's variable-length attention; other paths can fall back to padded execution.
  • Compile and export support depend on the component and input path. Eager list/nested adapters and deployment cache strategies have separate contracts.
  • Cached deployment supports specified built-in classes; custom forward overrides need their own deployment integration.
  • No speedup or downstream accuracy improvement is claimed without a matching benchmark.

See compatibility evidence and inference contracts before selecting a deployment path.

Documentation

Development

git clone https://github.com/Red-Eyed/stackformers.git
cd stackformers
just sync
just check

The Justfile owns formatting, linting, type-checking, tests, and builds.

Citation and license

If Stackformers contributes to your research implementation, cite the software using CITATION.cff and report the exact version or commit used. No associated research paper or conference endorsement is implied.

Released under the MIT license.

About

Composable PyTorch transformers for vision, spatial, and sequence research.

Topics

Resources

Contributing

Stars

3 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages