ART: a modular transformer that abstains instead of guessing

I started github.com/swadhinbiswas/Aurora on 14 January 2026 with Tuhin Sheikh and Md Shahriar Shakil at Daffodil International University in Dhaka. I had spent enough time watching language models answer in the same calm tone when right and when made up. For casual chat that habit is harmless. For medical, legal, or financial questions a confident wrong answer does more damage than no answer. I wanted a model that could not reach the user without passing a quality gate first.

The repo holds art-transformer version 0.1.0, MIT licensed, Python package name art-transformer. It is a PyTorch library for language models that carry uncertainty as an explicit output. The paper draft lives in paper/paper.md and paper/full_paper.tex, full result tables live in result/table1.json through result/table15.json, and figures regenerate from python -m art.benchmark.analysis --input result. Everything below traces to those files.

What we built

ART splits a reasoning pass into six stages:

Perceive to Reason to Plan to Execute to Verify to Answer
                                              or abstain

The reasoning core reads the question and produces three separate signals: what it understands (R), what it wants to do next (C), and how unsure it is (U). A deterministic router turns those signals into an acyclic task graph, drops any sub-task whose uncertainty crosses a threshold, and dispatches the rest to stateless experts. Every expert result passes through a verification anchor before it counts. When too few results survive, the system says so explicitly.

I keep coming back to a hospital analogy. I would not want my surgeon, pharmacist, and radiologist to share one brain. I want specialists, with someone checking the work before anything reaches the patient.

flowchart TD
  BPL[Binary perception layer] --> CTC[Cognitive core R C U]
  CTC --> D3R[Dynamic DAG router]
  D3R --> EX[Stateless experts]
  EX --> CRA[Reality anchor accept reject abstain]
  CRA --> SYN[Synthesis head]
  CTC -. U channel .-> D3R
  CTC -. U channel .-> CRA
  SYN --> OUT[Answer]
  SYN --> ABS[Abstain]

Story of one forward pass

A dense model reads a question and starts writing at once. Understanding, planning, computing, and checking tangle into one stream of hidden states. ART walks the question through stations, and nothing reaches the user until the last station signs off.

The journey starts at the Binary Perception Layer in art/modules/bpl.py. Tokenisation happens offline. Text pre-compiles into fixed width 32-bit records that memory map at training time, so the runtime loop reads contiguous integer arrays instead of parsing strings. This removes the CPU bound data bottleneck that throttles accelerators. The trade is less vocabulary flexibility than byte level or unigram schemes, and the repo docs state that trade plainly.

Next comes the Cognitive Transformer Core in art/modules/ctc.py. It has no decoder and no vocabulary softmax. Its final states project into three disjoint channels: reasoning representations R, control signals C, and a sigmoid bounded uncertainty channel U, trained jointly with a lightweight auxiliary correctness objective. Confidence is learned as a dedicated feature here. I did this because generation softmaxes miscalibrate under shift and quantization, and I did not trust them to say when to stop.

Control passes to the Dynamic DAG Router in art/modules/d3r.py. Five control primitives select by deterministic argmax against projection vectors frozen after training: EXECUTE, COMPOSE, BRANCH, AGGREGATE, VERIFY. Routing carries no sampling variance. The router assembles acyclic graphs with hard caps on depth and width, and it suppresses any sub-task whose uncertainty exceeds threshold. Acyclicity is enforced on insertion, so the recursive runaway loops I have seen in agent frameworks cannot start here.

Each surviving node dispatches through an ExpertRegistry in art/modules/experts/ to stateless domain experts for math, code, retrieval, and passthrough. Experts share no memory and cannot call one another. They return structured results only, so cross talk between sub-tasks is impossible by construction. The math expert routes parseable expressions through SymPy for exact symbolic evaluation and falls back to a neural stand in otherwise.

Before any result counts, the Context and Reality Anchor in art/modules/cra.py checks it against R and U and returns ACCEPT, REJECT, or ABSTAIN. Finally the Synthesis Head in art/modules/syn.py aggregates only accepted results through confidence weighted attention over verified candidates. It produces the final answer or an explicit abstention when too little survives.

For higher stakes I run Multi-ART. It runs M independent instances over the same input and merges verified outputs through uncertainty weighted voting, without shared weights or joint training. Disagreement triggers abstention rather than forced consensus. Our configs use M equal to 3.

Every call returns an ARTOutput dataclass with answer, abstained flag, calibrated confidence, pooled uncertainty, and graph diagnostics:

import torch
from art import ART

model = ART(vocab_size=50257, hidden_dim=768, num_layers=12, num_heads=12)
out = model(input_ids=torch.randint(0, 50257, (2, 128)))

print(out.abstained)    # True when evidence was insufficient
print(out.confidence)   # calibrated confidence estimate
print(out.num_tasks)    # sub-tasks executed after uncertainty gating

Downstream code can act on refusals. It can escalate to a reviewer, retry with retrieval, or log for audit, instead of parsing prose for hesitation that was never there.

What ships in the box

Beyond the six modules, the repo has the full loop to train, evaluate, and serve:

• Training pipeline in art/training/, driven by configs/art_base.yaml, configs/art_full.yaml, and configs/multi_art.yaml, with staged schedules that pre-train the core before freezing routing projections.
• Benchmark harness in art/benchmark/ covering GSM8K, MATH, MMLU, HumanEval, MedQA, and LegalBench, plus a failure injection framework and latency measurement. Results write as structured JSON so figures regenerate exactly.
• NVFP4 utilities in art/utils/quantization.py with block scaled 4-bit simulation and bounded initialisation safeguards, alongside optional hardware paths for Blackwell GPUs.
• Serving in art/api/serve.py as a FastAPI app with /v1/infer, /v1/batch, and health endpoints.
• Checkpoint export that integrates with Hugging Face tooling.
• Unit tests in tests/test_modules.py covering shape contracts, cycle rejection, depth and width bounds, uncertainty gating, verification outcomes, and abstention logic. The suite runs on CPU in seconds.

The package metadata states Python 3.9 or newer in pyproject.toml, while the README asks for Python 3.10 or newer. I develop on 3.10 and above. Dependencies include torch>=2.0.0, transformers>=4.30.0, datasets>=2.14.0, numpy>=1.24.0, pyyaml>=6.0, tqdm>=4.65.0, fastapi>=0.100.0, uvicorn>=0.23.0, and pydantic>=2.0.0.

Evaluate a trained model under simulated NVFP4 with ensemble consensus with one command:

uv run python -m art.benchmark.run \
  --checkpoint checkpoints/best.pt \
  --nvfp4 --multi --limit 500 --output-dir result

Main results

I report the NVFP4 numbers first because that is the point of the design. ART was built from scratch for aggressive 4-bit math, with bounded activations and no recursion. Quantization noise shows up as detectable uncertainty instead of silent hallucination. Dense models degrade sharply at the same precision. ART barely moves.

System	GSM8K	MATH	MMLU	HumanEval
Dense FP16	47.2	12.1	45.2	21.3
Dense NVFP4	40.5	9.8	38.6	17.4
ART FP16	56.4	18.2	49.3	25.4
ART NVFP4	55.6	17.5	48.6	24.8
Multi-ART NVFP4	57.9	19.1	50.4	26.5

The table above is result/table2.json. GSM8K climbs from 47.2 percent dense FP16 to 55.6 percent ART NVFP4, and to 57.9 percent with three way consensus. MATH moves from 12.1 to 17.5, and to 19.1 with consensus. The same pattern holds on MMLU and HumanEval.

{
  "type": "bar",
  "data": {
    "labels": ["Dense FP16", "Dense NVFP4", "ART FP16", "ART NVFP4", "Multi-ART NVFP4"],
    "datasets": [{
      "label": "GSM8K accuracy percent",
      "data": [47.2, 40.5, 56.4, 55.6, 57.9]
    }]
  },
  "options": {
    "responsive": true,
    "scales": { "y": { "beginAtZero": true, "title": { "display": true, "text": "percent" } } }
  }
}

Confident errors and leakage

I care more about confident error rate than raw accuracy. A wrong answer with high confidence is the failure that hurts users.

System	FP16 confident error percent	NVFP4 confident error percent
Dense text	12.4	19.8
Dense binary	10.1	14.5
ART binary	2.1	2.9
Multi-ART M equal 3	1.4	1.8

The table above is result/table4.json. Dense text at NVFP4 answers confidently and wrong 19.8 percent of the time. ART at NVFP4 does that 2.9 percent of the time. Three way consensus brings it to 1.8 percent.

Error leakage tells a similar story. This is the share of errors that slip through verification, from result/table5.json: Dense text NVFP4 82 percent, Dense binary 67 percent, ART single 14 percent, Multi-ART 5 percent. I read that as the verification anchor doing its job. Most bad expert outputs never reach synthesis.

Repeated runs stay tight. From result/table3.json, GSM8K with standard deviation: Dense text 48.5 plus minus 0.8 FP16 and 42.1 plus minus 1.4 NVFP4, Dense binary 49.7 plus minus 0.6 and 45.3 plus minus 1.1, ART binary 56.4 plus minus 0.4 and 55.6 plus minus 0.5, Multi-ART 58.7 plus minus 0.3 and 57.9 plus minus 0.3.

Cost and memory

System	Relative cost
Dense FP16 text	1.0
Dense NVFP4 text	0.72
ART NVFP4	0.29
Multi-ART NVFP4	0.43

The table above is result/table6.json. ART at NVFP4 runs at 0.29x relative inference cost. Even three parallel instances at 0.43x stay below half the dense FP16 baseline.

Memory and latency come from result/table13.json:

Format	Accuracy percent	Confident error percent	Memory GB	Latency ms per query
FP16 baseline	56.4	2.1	5.4	98
FP8	56.1	2.4	2.8	61
INT4 GPTQ	52.1	4.8	1.8	47
NVFP4 ART	55.6	2.9	1.6	38

ART at NVFP4 needs 1.6 GB against 5.4 GB at FP16, and answers in 38 ms per query against 98 ms in our harness. INT4 GPTQ uses 1.8 GB and 47 ms but drops to 52.1 percent accuracy with 4.8 percent confident errors. I chose NVFP4 because it kept accuracy while cutting memory and latency together.

Errors do not snowball

In dense models small numerical errors compound layer after layer. In ART they stay flat. That property makes everything else possible.

From result/table14.json, error magnitude by depth at NVFP4: Dense goes 0.18 at depth 4, 0.54 at depth 8, 1.41 at depth 12, 3.87 at depth 16. ART goes 0.12, 0.19, 0.23, 0.28 across the same depths. I plotted this in result/figures/table14_chart.png. The dense curve climbs. Ours stays level.

Calibration

When ART says 80 percent sure, it is right about 80 percent of the time. The reliability diagram in result/figures/reliability_diagram.png shows that alignment. The abstention curve in result/figures/abstention_accuracy.png climbs steeply as coverage shrinks, which is what I want from a system allowed to pass on hard questions.

Numbers from result/table11.json:

System	ECE	Brier	AUROC
Dense FP16 text	0.168	0.341	0.72
Dense NVFP4 text	0.243	0.428	0.64
MoE FP16 calibrated	0.112	0.295	0.78
ART NVFP4 binary	0.024	0.142	0.94
Multi-ART NVFP4 M equal 3	0.018	0.118	0.97

Lower ECE and Brier are better. Higher AUROC is better. Dense at NVFP4 degrades to 0.243 ECE. ART at NVFP4 sits at 0.024, and consensus at 0.018.

Ablations

I removed each piece in turn to see what breaks. From result/table7.json, confident error rate at ART binary plus NVFP4 starts at 2.9 percent with high stability. Without binary tokenization it rises to 6.9 percent with medium stability. Without the uncertainty channel it rises to 8.1 percent with low stability. Without D3R routing it rises to 10.4 percent with low stability. Without CRA verification it rises to 12.8 percent with poor stability.

No single stage carries the result alone. Verification matters most in that table, which matches my experience debugging it. When I loosened the anchor, bad expert outputs reached synthesis and confidence numbers stopped meaning much.

Router bounds show a clear sweet spot in result/table8.json. K max 4 and D max 2 gives 52.6 percent at 0.21 relative cost. K max 8 and D max 4 gives 55.6 percent at 0.29. K max 16 and D max 4 gives 55.8 percent at 0.38. K max 8 and D max 6 gives 55.5 percent at 0.34. K max 16 and D max 8 gives 55.3 percent at 0.52. I ship 8 and 4. Wider and deeper graphs cost more without helping accuracy.

Tokenization and routing

Offline binary records change the input path. From result/table1.json: text pipeline is CPU limited, high memory amplification, linear latency growth in batch size, high latency variance, poor accelerator utilization. BPL is bandwidth limited, minimal amplification near 1, sublinear latency growth, low variance, near saturation utilization.

Accuracy follows the same split in result/table9.json:

Tokenization	Accuracy	Confident error	Prep cost
BPE text baseline	47.9	12.4	Runtime
Byte level	46.1	11.8	Runtime
Unigram SentencePiece	48.3	11.2	Runtime
BPL binary offline	55.6	2.9	Offline

Routing comparison from result/table10.json:

Routing	Accuracy	Confident error	Failure pass	Deterministic
MoE learned gating	46.5	10.2	58	No
Tool augmented	48.2	9.4	45	No
D3R deterministic	55.6	2.9	14	Yes

Deterministic argmax against frozen projections removes sampling variance from routing. That choice cost some flexibility during training, and it repaid that cost in reproducibility. I can rerun a graph and get the same graph.

Failure injection

I injected retrieval noise, tool misuse, and multi step cascading failures to see which architecture holds up. From result/table12.json, accuracy under each stress:

Dense text holds 41.3, 38.7, 34.2. Tool augmented holds 52.6, 43.1, 39.8. ART single holds 64.8, 66.2, 61.5. Multi-ART holds 69.1, 70.4, 67.3.

The gap widens under stress. Stateless experts cannot call one another, so a poisoned retrieval result stays contained in its own node. The anchor rejects it, synthesis never sees it, and the answer either uses surviving evidence or abstains.

High stakes suites

MedQA and LegalBench matter to me because they proxy the domains where a wrong answer costs real money or health. From result/table15.json:

Dense FP16 scores MedQA 41.2 accuracy with 18.4 confident error, LegalBench 44.5 with 16.1. Dense NVFP4 falls to 33.8 with 27.6, and 36.3 with 24.9. ART NVFP4 scores 44.8 with 3.6, and 47.1 with 3.3. Multi-ART reaches 46.2 with 2.4, and 48.6 with 2.1.

Quantization hurts dense models most where I can afford it least. On MedQA dense confident errors climb from 18.4 to 27.6 percent when moving to NVFP4. ART stays at 3.6 percent at the same precision.

Getting started

You need Python 3.10 or newer per README, with pyproject.toml allowing 3.9 and above:

git clone https://github.com/swadhinbiswas/Aurora
cd Aurora
pip install .

Or with uv:

uv sync

Minimal forward pass:

import torch
from art import ART

model = ART(vocab_size=50257, hidden_dim=768, num_layers=12, num_heads=12)
out = model(input_ids=torch.randint(0, 50257, (2, 128)))

print(out.abstained)
print(out.confidence)
print(out.num_tasks)

Run the test suite, CPU only, finishes in seconds:

pytest tests/ -v

Train from YAML:

python scripts/train.py --config configs/art_full.yaml

Evaluate standard suites:

python -m art.benchmark.run --checkpoint checkpoints/best.pt --nvfp4 --limit 500

Regenerate figures from structured results:

python -m art.benchmark.analysis --input result

Hardware note and reproduction

Native NVFP4 acceleration needs NVIDIA Blackwell class GPUs, compute capability 10.x. On older GPUs and CPUs the library runs in faithful simulation mode with the same numerics and behavior, without the speed and memory wins. I did most early debugging in simulation and moved to Blackwell for timed runs.

The full 8B configuration in reproduce.sh asks for an 8x Blackwell B200 cluster for native NVFP4 tensor cores and BPL DMA bypass. The training corpus is 198 GB of proprietary expert reasoning and coding text that I cannot redistribute, so reproduce.sh skips raw dataset preparation by design. Tracked checkpoints/export/ entries are placeholders to replace with a locally trained checkpoint from python scripts/train.py --config configs/art_base.yaml.

reproduce.sh runs four steps: prepare dataset and offline tokenization, distributed training from configs/art_base.yaml, full evaluation across GSM8K, MATH, MMLU, HumanEval, MedQA, and LegalBench with --limit 200 to checkpoints/art-8b-nvfp4-instruct.pt, and figure generation to result/. When training exits non zero on non Blackwell hardware, the script falls back to the pre trained 8B checkpoint path. The README warns that paper --checkpoint commands assume a checkpoint exists at checkpoints/, otherwise the harness re-plots committed results without evaluating a model. Point it at a real checkpoint before claiming a fresh number.

About pretrained weights: an 8B instruction tuned checkpoint with GGUF, MLX, and PyTorch exports was trained on that proprietary expert corpus. Those weight files are not in the repo. Architecture, training pipeline, tokenizers, benchmark harness, and evaluation logs are all there. Train your own model with the provided configs, or wire heavier expert backends through the registry interface.

Honest limitations

ART is a research testbed. The bundled experts are lightweight neural stand ins behind a stable interface. Only the math expert dispatches to a real symbolic backend today. Users bring heavier tools for code, retrieval, and domain work.

Because the architecture refuses to guess, it abstains more often than dense baselines. That behavior helps in a clinic or a law office and annoys in a chatbot. I consider that trade the point of the project, but I track abstention rate alongside accuracy so the cost stays visible.

The repo history showed 10 commits on main and 3 stars when I pulled these numbers, language Python, license MIT. Citation metadata is version 0.1.0, released 23 August 2026, DOI 10.5281/zenodo.22067754:

@software{biswas2026art,
  title = {ART: AURORA-Transformer, a modular transformer library for uncertainty-aware language reasoning},
  author = {Biswas, Swadhin and Sheikh, Tuhin and Shakil, Md Shahriar},
  year = {2026},
  doi = {10.5281/zenodo.22067754},
  url = {https://github.com/swadhinbiswas/Aurora}
}

Authors are Swadhin Biswas with ORCID 0009-0005-2980-6651, Tuhin Sheikh, and Md Shahriar Shakil, Department of Computer Science and Engineering, Daffodil International University, Dhaka, Bangladesh. Contact is swadhinbiswas.cse@gmail.com.

If you work on calibration, routing, or low precision training, the repo gives you a place to swap one stage and measure the effect in isolation. Start with configs/art_base.yaml, run the CPU tests, then try a single ablation from the table above and see whether your change moves confident error rate or just accuracy.