Shared Reference Transformer Weights and Cross-layer KV-Cache
A Conceptual Architecture with Preliminary Experimental Evidence on Grouped Shared-Reference Transformer Weights and Cross-Layer KV-Cache Reuse
Research Working Paper
Author: Christopher Soans
Date: October 2026
Download PDF
Download Zip Archive
Status: Exploratory, proof-of-concept research. Not a claim of production readiness, invention of cross-layer sharing, or parity with large pretrained language models.
|
| Figure 1. Shared Reference Transformer Weights and Cross-layer KV-Cache |
Abstract
This paper investigates reducing two separate sources of AI inference memory: static transformer model weights and dynamically growing key–value (KV) attention caches. The proposed design shares corresponding attention and feed-forward reference tensors among short groups of transformer layers and derives each layer’s effective weights using fixed, normalized positional multipliers. A complementary cross-layer attention mechanism reuses a single KV representation across each group, optionally together with grouped-query attention (GQA). In local, CPU-based proof-of-concept studies, an eight-reference, 32-layer language model used 69.7% fewer trainable parameters than an equal-width, equal-depth conventional model (435,328 versus 1,436,800); its held-out perplexity was 348.51 versus 330.55 after 110 training steps. Separate KV studies established a structural 75% reduction in raw cached KV elements with four-layer reuse, or 93.75% relative to eight-head multi-head attention when also using two-head GQA. A short real-text next-token pilot did not show an obvious quality penalty for the combined KV arrangement, but it was limited to 30 training steps and one complete matched seed. These results support further study, not claims of equal general-purpose capability, faster GPU inference, or lower energy consumption.
1. Motivation and research questions
Large transformer deployments are constrained by model weights, optimizer state during training, attention KV caches during autoregressive serving, and bandwidth required to move these tensors. Reducing static parameters alone cannot guarantee lower inference FLOPs; likewise a smaller KV cache does not automatically produce faster attention kernels. The engineering objective is therefore a measured quality–memory–compute trade-off, with minimal new mechanisms.
- RQ1 — Can grouped reuse of trained weight tensors substantially lower parameter memory while preserving competitive language-model loss at fixed depth and width?
- RQ2 — Can one KV representation per small group of layers lower autoregressive cache storage without unacceptable language-model degradation?
- RQ3 — Does combining cross-layer KV reuse with established GQA remain operationally useful after accounting for latency, bandwidth, and implementation overhead?
2. Architecture and notation
2.1 Baseline transformer
A conventional transformer layer l uses independent attention projections WQ,l, WK,l, WV,l and WO,l, plus independently trained feed-forward matrices. In decoder-only inference, each layer typically retains its own sequence of K and V states (unless the architecture already uses head-sharing or cross-layer techniques).
2.2 Grouped shared-reference weights
Partition L layers into G consecutive groups, each containing m = L/G layers. For every corresponding matrix type T (Q, K, V, O and feed-forward projections), group g learns a distinct reference tensor RT,g. The effective tensor in within-group position j is:
WT,g,j = αj RT,g where αj = j / ((m + 1) / 2), j = 1,…,m.
For the primary L = 32, G = 8 configuration, each group contains four layers with multipliers 0.4, 0.8, 1.2, and 1.6. Attention and feed-forward matrix TYPES remain separate: Q, K, V, and O are not constrained to be scalar multiples of one another. Normalization parameters are shared within each group in the tested implementation without positional multiplication. Embedding and output-head parameters remain independent.
The effective linear operation can be evaluated as αj(RT,g x), without materializing and permanently storing the full per-layer matrix. Identical references at different layers do not imply identical hidden states or identical computations. The approach is learned from random initialization; it is not a lossless transformation of a conventionally trained model.
2.3 Optional limited two-hop connection
An optional variant adds a learned scalar skip from the hidden representation two layers earlier: hl = Fl(hl−1 + gl hl−2). Two-hop connectivity is orthogonal to weight-reference grouping and is not required for the main memory-saving proposal. A 32-layer proof-of-concept required 30 additional scalar parameters in the tested implementation.
2.4 Cross-layer KV-cache reuse
The first layer in each four-layer group generates a reference key and value representation from its input. Subsequent layers reuse those references, optionally applying the corresponding positional scale factor while retaining their own queries and attention outputs. The group holds one actual cached K/V tensor pair per token instead of four. This requires a model trained with the cross-layer constraint; naive replacement of arbitrary pretrained KV states is not information-preserving.
With L = 32, m = 4, Hq query heads, Hkv KV heads, head width dh, sequence length S, batch B and b bytes per cached scalar, the raw cache footprint is:
M_KV = 2 × B × S × (L / m) × Hkv × dh × b.
When head width is held fixed, four-layer sharing reduces cache elements by 75% relative to the same number of KV heads per layer. Reducing KV heads from 8 to 2 further lowers raw elements by a factor of four: the combined cache is 6.25% of the 32-layer eight-KV-head MHA baseline. This is an architectural accounting identity, not a measured end-to-end serving-memory reduction.
3. Scope and experimental method
The recorded experiments were local, exploratory PyTorch CPU tests. Models were trained from scratch using a common small decoder-only codebase rather than modifying a pretrained foundation model. Real-text trials used locally available Python technical documentation. One language-model study used SentencePiece BPE with a 768-token vocabulary; a separate KV study used byte tokens. Both used held-out portions of real documentation, but limited context windows and short training schedules.
|
Study |
Common configuration |
Primary comparison |
|
Equal-width language-model pilot |
32 layers; width 64; 4 heads; BPE vocab 768; context 48; 110 steps; seed 11 |
Independent weights vs eight groups of four |
|
Optional two-hop variant |
Same LM setup; 110 steps; seed 11 |
Conventional vs eight-group weights plus two-hop |
|
KV quality pilot |
32 layers; width 64; 8 query heads; byte tokens; context 64; 30 steps; seed 11 |
MHA, GQA, cross-layer KV, combined |
|
Long-context KV microbenchmark |
32 layers; widths 64/256/512; contexts up to 2,048; CPU; random tensors |
Raw KV bytes and token decode timing |
The studies do not constitute a large-model scaling law. Their benchmarks differ and therefore their perplexities must not be compared across rows as though they were the same dataset, vocabulary, or training run.
4. Findings: model-weight reference sharing
|
Architecture (32 × 64) |
Parameters |
FP32 weights |
Validation perplexity ↓ |
|
Conventional / independent |
1,436,800 |
5.75 MB |
330.55 |
|
Eight reference groups |
435,328 |
1.74 MB |
348.51 |
|
Eight groups + two-hop |
435,358 |
1.74 MB |
347.13 |
All three figures are from the same 110-step, seed-11, BPE-based proof-of-concept setting. The eight-group architecture stored 69.7% fewer trainable weights than the conventional baseline. Its perplexity was approximately 5.4% higher; two-hop reduced this to approximately 5.0% higher. These perplexities are high in absolute terms, reflecting short training, tiny models and the task/tokenizer. The marginal two-hop difference is not statistically established.
Separate 32-layer width-48 synthetic-association tests (160 steps; two seeds) produced 93.45% mean task accuracy for independent weights and 91.49% for eight reference groups at 69.9% fewer parameters. Those synthetic results are background evidence only; real-text perplexity is the more relevant quality metric.
5. Findings: KV-cache sharing and GQA
|
KV configuration |
KV heads |
Independent cache sets |
Relative raw KV storage |
|
Independent MHA |
8 |
32 |
100.00% |
|
GQA only |
2 |
32 |
25.00% |
|
Cross-layer sharing only |
8 |
8 |
25.00% |
|
Cross-layer sharing + GQA |
2 |
8 |
6.25% |
For an illustrative width-256, FP32, batch-one, 1,024-token cache, the recorded values were 64 MiB (MHA), 16 MiB (GQA), 16 MiB (cross-layer), and 4 MiB (combined). These counts scale linearly with sequence length, batch size and numeric bytes per element. They exclude allocator, metadata, workspace, fragmentation and other serving allocations.
|
KV configuration (real-text pilot) |
Validation perplexity ↓ |
Training steps |
|
MHA, per-layer KV |
32.80 |
30 |
|
GQA, per-layer KV |
30.49 |
30 |
|
MHA, four-layer KV reuse |
32.28 |
30 |
|
GQA + four-layer KV reuse |
30.15 |
30 |
The complete four-way comparison shown above is seed 11 on the byte-token real-document next-token pilot. GQA configurations also change the projection parameter count; models did not have identical initial perplexity. The combined model’s lower observed perplexity is therefore not evidence that sharing improves language modeling. It shows that this constrained architecture can learn under the short tested regimen without an obvious collapse in validation quality.
A separate 65-step training and incremental-decoding test of shared versus nonshared KV within the eight-reference weight model reported perplexity 22.56 versus 22.72 and 96 KiB versus 384 KiB of 24-token FP32 raw KV tensors. Incremental and full-sequence logits matched within approximately 1.7 × 10−6 for each respective architecture. This verifies cache implementation consistency for the test inputs, not equivalence of the two models.
5.1 Integrated-system comparison: independent weights and KV versus grouped weights and shared KV
To test whether both proposed optimizations operate together, a matched 32-layer, width-64 decoder-only real-text pilot compared (A) a conventional transformer with independently learned weight matrices and per-layer KV caches against (B) eight groups of normalized shared-reference weights, each spanning four layers, with one KV reference reused within each group. Both used four attention heads, a byte-level vocabulary of 256, context length 32, batch size 8, 65 training steps, and random seed 17. GQA was excluded to isolate the proposed combined mechanisms.
Table 5. Integrated comparison, one seed (17), after 65 training steps.
|
Metric |
Conventional independent |
Combined grouped weights + KV |
|
Independent trainable parameters |
1,376,384 |
381,056 |
|
FP32 parameter storage |
5.51 MB |
1.52 MB (72.3% lower) |
|
Raw FP32 KV cache, 24 tokens, batch 1 |
384 KiB |
96 KiB (75% lower) |
|
Final validation negative log-likelihood |
3.16528 |
3.11633 |
|
Final validation perplexity (lower is better) |
23.695 |
22.563 |
|
Incremental CPU decode, 24 tokens |
205.82 ms |
158.75 ms (22.9% lower) |
|
Train + evaluation CPU wall time |
10.59 s |
9.34 s |
|
Max |full-sequence − incremental logits| |
1.67 × 10⁻⁶ |
1.43 × 10⁻⁶ |
The combined architecture exhibited a 4.8% lower measured perplexity than the conventional baseline and lower CPU decoding time in this exploratory run. The storage reductions are structural properties of the parameterization and cache layout; the observed perplexity and timing advantages are not established as repeatable improvements. Initial perplexities differed (297.05 conventional versus 353.82 combined), and only one random seed and a short training schedule were completed. The evaluation used a small technical-documentation corpus, short contexts and custom PyTorch CPU execution; it does not demonstrate GPU throughput, power savings, or general-purpose model parity.
The logit-consistency checks compare full-sequence execution with incremental decoding separately for each trained model; they do not imply that the two architectures yield identical logits. Weight storage and cache storage represent distinct memory pools and their reductions should not be summed. This paired experiment provides preliminary integrated-system evidence, strengthening—but not replacing—the need for longer, multi-seed and quality-matched evaluation.
5.2 Independent laptop CUDA replication: three random seeds
A Python 3.10.11 CUDA-enabled laptop benchmark with less capable hardware independently repeated the integrated comparison at 32 layers, width 64, four attention heads, eight shared-weight groups, and four layers per shared-KV group. Conventional and combined models were trained separately for 65 steps under paired seeds 17, 23 and 41. Three timed decoding repetitions were used for 24-, 64-, and 128-token sequences. The comparison isolates a fixed architecture and training-step budget; it does not establish matched total compute or broadly equivalent language-model quality.
Table 6. Three-seed laptop replication: aggregate measurements. Inference figures are mean within-seed reductions in elapsed time; validation perplexity is lower-is-better.
|
Measurement |
Conventional |
Combined / observed effect |
|
Stored model weights |
1,376,384 parameters |
381,056 parameters (72.31% fewer) |
|
Raw KV cache |
32 independent layer caches |
8 shared groups (75.00% fewer KV bytes) |
|
Validation perplexity, mean of 3 seeds |
26.45 |
27.82 (mean paired gap +5.30%) |
|
Training-time reduction, 3 seeds |
Reference |
22.93% mean (SD 4.91 pp) |
|
Decoding elapsed-time reduction, 24 tokens |
Reference |
22.35% mean (SD 8.46 pp) |
|
Decoding elapsed-time reduction, 64 tokens |
Reference |
22.28% mean (SD 10.31 pp) |
|
Decoding elapsed-time reduction, 128 tokens |
Reference |
17.14% mean (SD 13.05 pp) |
The combined model was faster in all nine seed/context decoding comparisons and in all three training-time comparisons. Its validation perplexity was worse in every seed (paired gaps +6.46%, +1.12%, +8.32%). Structural memory reductions were invariant to the random seed. These are descriptive observations from three seeds, not confidence intervals or production throughput/energy estimates. Timing variability is material, especially at 128 tokens.
5.3 Extended laptop run: counted FLOPs and attempted GPU energy measurement
A further paired CUDA run used an older laptop NVIDIA GPU, Python 3.10.11 and PyTorch 2.14.1+cu126 to check for efficiencies on less capable hardware. Both models were trained for 250 steps using seed 17, sequence context 32, batch size 8, and an equal layer/width configuration. PyTorch profiler FLOP counts were collected separately from timing and should not be described as complete training FLOP totals.
Table 7. Extended 250-step CUDA benchmark and instrumentation outcome.
|
Measurement |
Conventional |
Combined architecture |
|
Training time (250 steps) |
34.14 s |
36.13 s (+5.85%; slower) |
|
Counted profiler FLOPs / step |
2.049 billion |
1.747 billion (14.72% fewer) |
|
Extrapolated counted FLOPs / 250 steps |
512.23 billion |
436.83 billion |
|
Validation perplexity |
21.19 |
21.78 (+2.80%) |
|
FP32 model-weight memory |
5.51 MB |
1.52 MB (72.31% reduction) |
|
KV cache at 128 tokens |
2.00 MiB |
0.50 MiB (75.00% reduction) |
|
GPU energy, joules |
Not measured |
Not measured |
|
NVML power samples |
0 |
0 |
The profiler counted approximately 14.72% fewer supported operations for the combined architecture, yet the observed training elapsed time was 5.85% longer. This illustrates that reduced profiler-counted FLOPs do not guarantee lower wall time. Kernel launches, scaling operations, scheduling, implementation details and measurement variability may contribute, but their individual effects were not isolated. The previous 65-step, three-seed benchmark reported faster combined training; accordingly, speed outcomes are mixed across training duration and test conditions.
The energy measurement attempt did not produce valid GPU power samples for either model (power_samples = 0; energy_j = null). The hardware/driver telemetry cause was not established. No claim of GPU energy reduction, GPU power efficiency, or whole-laptop electricity reduction follows from this run. The FLOP profiler documents supported operations and can omit fused attention, portions of backward propagation and other kernels; its counts are a comparative proxy, not complete measured training FLOPs.
Interpretation. The multi-seed results add preliminary evidence of repeatable memory savings and decoding-time improvements on one laptop implementation; the longer follow-up identifies a meaningful computational-count reduction but no corresponding training-speed advantage. Both runs retain a validation-perplexity disadvantage for the combined model. Further independent GPU and dataset replication remains necessary before making generalized efficiency or energy claims.
6. Efficiency interpretation
6.1 What is structurally reduced
- Static parameter and corresponding optimizer-state tensor counts can fall when reference matrices replace multiple independently trained sets. Actual complete training memory falls by less because per-layer activations and other buffers remain.
- Raw autoregressive KV memory can fall in proportion to the number of reused layer states and the number of KV heads; for four-layer sharing and eight-to-two GQA, the idealized reduction is 93.75% against the specified MHA baseline.
6.2 What is not automatically reduced
- Transformer depth and width are unchanged; each layer still performs its own query/attention/output and feed-forward work. Sharing trained weights is therefore not a proportional reduction in forward-pass matrix-multiplication FLOPs.
- Attention kernels may incur additional gather, reshape, scaling or synchronization overhead. CPU microbenchmarks showed configurations with smaller caches were not always fastest; these results cannot be used as estimates of GPU serving performance.
- GPU energy measurement was attempted in a later laptop run but yielded no valid power samples; aggregate throughput, data-center energy, and whole-system energy remain unmeasured. Cross-layer KV sharing is an architectural constraint rather than lossless compression of otherwise independent KV values.
7. Existing methods, differentiation and novelty boundary
Cross-layer parameter sharing has precedents, notably ALBERT and Universal Transformers. MQA and GQA reduce KV redundancy among query heads; cross-layer attention and shared-KV designs, including approaches discussed in the YOCO research line, explore reuse across depth. Parameter tying also appears in recurrent and iterative models. Accordingly, neither cross-layer weight sharing nor cross-layer KV reuse nor GQA should be presented as new discoveries.
The proposed research contribution is the particular grouped construction: per-matrix-type, normalized positional scaling across short blocks of layers; optional limited two-hop connectivity; a compatible KV-reference reuse scheme; and a controlled empirical evaluation of the combined memory–quality–compute trade-offs. Novelty relative to recent published cross-layer KV schemes and production systems remains to be established with an up-to-date literature review.
8. Risks, limitations and adversarial checks
- Capacity and training divergence: tying reference matrices reduces independent degrees of freedom; larger domains and long-range reasoning may exhibit greater perplexity or quality gaps than short local-document tests.
- KV incompatibility across depth: later layers receive different contextual inputs, so reusing early-layer K/V may suppress useful transformations; long-range retrieval, tool use and factual accuracy require independent evaluation.
- Baseline fairness: fixed layer count and width isolate weight sharing, but parameter budgets differ; equal-compute and quality-matched alternatives remain useful secondary comparisons.
- Weak experimental power: early real-text comparisons were limited to one complete seed, short training, small corpora and validation-token samples. A later laptop replication added three paired CUDA seeds and a 250-step single-seed follow-up, but neither establishes statistical certainty, billion-parameter scaling, nor realistic long-context quality.
- Training/inference separation: training backpropagation storage is not identical to inference KV cache. Accuracy under teacher-forced evaluation does not establish generation quality, throughput or cache-eviction robustness.
- Implementation assumptions: any shared-group KV design must be explicitly compatible with causal masking, positional encoding, decoding order, group boundaries, continuous batching and cache paging.
9. Proposed validation program
Follow-up work should use an established decoder-only framework and documented dataset splits, retaining a conventional equal-width/depth baseline and the grouped-weights-plus-shared-KV candidate. At minimum, assess independent and combined effects of (a) weight sharing, (b) GQA and (c) cross-layer KV reuse. Preserve the simplest configuration that demonstrates a measurable quality-adjusted efficiency gain.
|
Priority |
Study |
Success criterion |
|
1 |
Longer real-text LM training; multiple seeds; held-out corpora |
Stable validation-loss trade-off; no widening unexplained gap |
|
2 |
Long-context generation, retrieval and perplexity |
Quality retention at realistic sequence lengths |
|
3 |
Optimized GPU decode/prefill with GQA and paging |
Measured cache resident bytes, tokens/s, p50/p95 latency |
|
4 |
Full training-memory and compute accounting |
Peak VRAM, optimizer/activation splits, time-to-quality |
|
5 |
Larger established pretrained-model comparisons |
Reproducible scaling evidence; clear retraining needs |
Where possible, measure total lifecycle resources to reach a target quality rather than comparing parameter counts alone. A shared-reference model trained from scratch must not be evaluated as though it were a lossless post-training codec.
10. Conclusion
The preliminary evidence supports the feasibility of a grouped shared-reference transformer architecture that reduces static weight memory and can be combined with cross-layer KV reuse to lower raw autoregressive cache requirements. In a tiny real-text model, grouped weight sharing reduced stored parameters by approximately 69.7%, with a modest but real validation-perplexity penalty. A separate short real-text KV pilot demonstrated learnability of GQA plus cross-layer KV sharing while the cache design provided a structural 93.75% reduction in raw KV elements versus the specified MHA reference. These results are promising proof-of-concept observations, not proof of comparable capabilities in billion-parameter models, faster GPU inference, or lower energy consumption. The next decision should follow rigorous quality-matched, large-scale system measurements and comparison against established methods. A subsequent direct conventional-versus-combined comparison (Section 5.1) measured 72.3% fewer stored FP32 parameters and 75% less raw KV memory with competitive perplexity in a single small pilot, but does not establish reproducibility or scaled-model quality. Follow-up laptop CUDA measurements additionally showed 72.31% less parameter storage and 75% less raw KV-cache storage; the three-seed replication showed faster measured decoding but consistently higher validation perplexity. A longer 250-step run showed 14.72% fewer profiler-counted FLOPs while training 5.85% more slowly; GPU energy was unavailable because no power samples were collected. Consequently, the proposed architecture remains preliminary and hardware efficiency must not be inferred from memory reductions alone.
Appendix A. Experimental provenance
This paper synthesizes recorded locally generated outputs of the following reproducibility files associated with the research conversation: real_language_shared_reference_scaleup_results.json; conventional_vs_twohop_eight_reference_lm_results.json; matched_width_reference_followup_results.json; cross_layer_kv_shared_reference_pilot_results.json; kv_sharing_gqa_long_context_results.json; real_text_gqa_crosslayer_quality_results.json. Matching Python scripts were generated for each study. Reported outcomes have been transcribed from those records; they have not been externally peer-reviewed or independently reproduced. Additional integrated-system comparison: full_conventional_vs_combined_weight_kv.py and full_conventional_vs_combined_weight_kv_results.json; execution depends on cross_layer_kv_shared_reference_pilot.py. The recorded integrated comparison is a one-seed, 65-step CPU proof of concept. Additional laptop benchmark provenance: comparison.json and comparison.csv (65-step seed-17 CUDA benchmark); three_seed_summary.json and paired_comparisons.csv (paired seeds 17/23/41); energy_flops_comparison.json and energy_flops_comparison.csv (250-step CUDA run on an older laptop Nvidia GPU; zero power telemetry samples). These laptop datasets measurements are represented as recorded, with hardware and profiling limitations explicitly acknowledged.
Appendix B. Selected related work for literature follow-up
Lan et al., “ALBERT: A Lite BERT for Self-supervised Learning of Language Representations” (2019/2020). Dehghani et al., “Universal Transformers” (2018/2019). Shazeer, “Fast Transformer Decoding: One Write-Head is All You Need” (2019). Ainslie et al., “GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints” (2023). Sun et al., “You Only Cache Once: Decoder-Decoder Architectures for Language Models” (2024). These are contextual references, not claims that the exact proposed combination is absent from the literature.
© 2026 Christopher Soans. All rights reserved.
This work is licensed under a Creative Commons Attribution 4.0 International License (CC BY 4.0).
