Every cell in your body reads from the same genome, but each cell type switches on a different set of genes. netformer learns that pattern in four plain steps.
1
Turn a cell into a sentence
Sort the genes from most to least active. The order is the sentence.
→
2
Learn by fill-in-the-blank
Hide some genes and ask the model to guess them from the rest.
→
3
Name the cell type
Add a small head to the learned model and fine-tune it to label cells.
→
4
Ask “what if?”
Delete a gene from the sentence and see how far the cell moves toward another type.
Why rank, not raw counts? Sequencing machines read some cells more deeply than others. Ranking removes that effect exactly — doubling a cell's counts gives an identical sentence (this is checked by a test).
Try it
Two small interactive demos. The first is a toy; the second uses the project's real results.
① Depth doesn't change the sentence
Raw counts (scaled by depth)
Resulting sentence (genes ranked by activity vs. their typical level)
② Which gene matters most for being an alpha cell?
Bar = how far the alpha cell moves toward a beta cell. Real output of results/perturbation_*.csv.
Results
Every number comes from make all — nothing is hand-copied.
Starting point
Accuracy
Macro-F1 (14 types)
Macro-F1 (9 common types)
Pretrained
0.971
0.713
0.931
From scratch
0.971
0.653
0.933
Honest reading: on common cell types, pretraining makes no measurable difference — 8.5k cells of one tissue is too little to show a benefit. The small gap on rare types rests on 1–6 test cells and is noise. The scratch control is in the repo so this can be checked, not assumed.
Pretraining: held-out masked-gene accuracy 9.9% top-1, 13.3% top-10.Fine-tuning vs. from-scratch control.Deleting GCG (glucagon) moves alpha cells most — found without being told it's a marker.Attention network in beta cells: INS is the hub, linked to IAPP, SLC30A8.Cell embeddings cluster by type, with donors mixed.Tokenisation quality checks.
Limits: this is an engineering demo, not a biological result. Attention isn't causation, and the perturbation ranking is a hypothesis generator. See the model card.
Quickstart
Python 3.10+. The full pipeline takes about 90 minutes on a laptop CPU.
git clone https://github.com/ashokdwivedi/netformer && cd netformer
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
make data # download GSE84133 + tokenise (~3 min)
make pretrain # masked-gene pretraining (~60 min)
make finetune # cell-type head + control (~25 min)
make eval # metrics, perturbation, attention network
make figures # rebuild every figure
make smoke # 1-minute synthetic run to check your setup
Tokenizer
Rank-value encoding, fit on the training split only so nothing leaks.
Transformer
Pre-LN encoder with GEGLU, written from scratch; attention maps recoverable.
Trainer
DDP, mixed precision, resumable checkpoints.
HPC
Slurm scripts for 4 nodes × 8 GPUs with auto-resume.
Interpretability
In-silico knock-out / activation and attention networks.
Tests
27 tests, including depth invariance and padding isolation.
