retnet

Phi-3, -3.5, and -4 for Mac: Locally-run Vision and Language Models for Apple Silicon

an autotuner for the flash version of attention and retnet

This is the repo for the MixKABRN Neural Network (Mixture of Kolmogorov-Arnold Bit Retentive Networks), and an attempt at first adapting it for training on text, and later adjust it for other modalities.

PyTorch research scaffold for RetNet, Block Attention Residuals, and hashed Engram memory

Byte-level RetNet experiments: a from-scratch, softmax-free retention layer and a 28-notebook research archive.

Torch and TF implementation of Retention Network modules.

Point-free layer composition for PyTorch research — build neural networks as expressions, not Module subclasses.

Train-your-own 250M local LM kit — Gated Retention (RetNet family), pure PyTorch, own BPE