Two Hop Connectivity Transformer Architecture

A Conceptual Alternative To Conventional Sequential Transformer Connectivity

Conceptual Research Paper

Author: Christopher Soans
Date: October 2026

Download PDF
Download Zip Archive

Status: Preliminary experimental findings; not a production-validated architecture.

Two-Hop Connectivity Transformer Architecture
Figure 1. Two-Hop Connectivity Transformer Architecture

Abstract

This paper proposes a limited two-hop connectivity pattern for transformer networks. In addition to the conventional input from the immediately preceding layer, each applicable block receives a learnable contribution from the representation two layers earlier. The design aims to make earlier intermediate representations more directly accessible while avoiding densely connected architectures and preserving standard attention, feed-forward, normalization and residual mechanisms. Small-scale comparisons of conventional and two-hop 32-layer transformers, jointly trained on synthetic general, technical and relational-reasoning tasks, found that the two-hop architecture achieved higher accuracy on the deepest reasoning task in two completed matched-seed runs. A post-training connection ablation also reduced reasoning accuracy in the two-hop model, indicating that the learned paths affected predictions. Benefits were not universal: technical-task accuracy was mixed, catastrophic-forgetting protection was not demonstrated, and total training-compute savings are unproven. The proposal is therefore presented as a promising, low-parameter-overhead architectural alternative requiring further independent validation.

1. Motivation and problem statement

Transformer blocks normally process hidden states sequentially. Although residual connections already transport information through depth, a subsequent block ordinarily receives a transformed aggregate from its immediate predecessor, not an independently accessible copy of a selected earlier layer output. The motivating hypothesis is that limited access to an earlier intermediate representation may help some deeper computations preserve useful relationships and improve next-token prediction. A second, separate hypothesis is that shorter gradient paths may improve optimization and reduce steps to a target accuracy.

The proposal does not assume that each subject or fact is stored in a unique layer. Transformer knowledge is distributed, and the benefit of an extra path must be demonstrated empirically. The goal is a restrained modification that can be compared directly against the conventional architecture.

2. Architecture

2.1 Connectivity rule

Let Hᵢ be the output of transformer block i and Fᵢ a standard transformer block, including its internal residual paths. The conventional form is Hᵢ = Fᵢ(Hᵢ₋₁). The tested two-hop form is:

Hᵢ = Fᵢ(Hᵢ₋₁ + gᵢ · Hᵢ₋₂)

Here gᵢ is a learned scalar initialized to 0.1 in the experiments. The first eligible block with two preceding block outputs is block 3. The 32-layer implementation therefore adds 30 learned scalar coefficients. An additional path from the output of block i to the input of block i+2 gives the network the requested Layer 1 → Layer 3, Layer 2 → Layer 4, etc., connectivity.

2.2 Logical dataflow

Standard path: H₁ → H₂ → H₃ → H₄ → … → H₃₂. Additional two-hop paths: H₁ ↗ H₃; H₂ ↗ H₄; H₃ ↗ H₅; …; H₃₀ ↗ H₃₂. Every block still executes in order to compute its state; two-hop routing is not a mechanism for skipping transformer blocks.

2.3 Design boundaries

  • Retain the same standard transformer block structure and internal residual connections as the baseline.
  • Add exactly one nonadjacent source per eligible block (two layers earlier); do not connect to every preceding layer.
  • Use a scalar learned gate rather than adding attention modules or a separate routing network.
  • Keep ordinary optimizers, automatic differentiation and joint multi-objective supervision.
  • Evaluate actual compute, activation-memory and numerical effects before making efficiency claims.

3. Hypothesized mechanisms and counterarguments

3.1 Representation accessibility

A direct contribution from Hᵢ₋₂ may expose features that are less directly available in Hᵢ₋₁. Because the representations are added before Fᵢ in the tested implementation, the receiving block does not separately interrogate two independent memories. Conventional residual connections may already provide much of the desired access, making incremental benefit task-dependent.

3.2 Optimization

The two-hop path changes forward dynamics and provides additional gradient pathways. Faster convergence might therefore arise from gradient-flow changes rather than an intrinsic improvement in representational capacity. The experiments do not separate these mechanisms causally.

3.3 Stability and limits

Repeated addition of earlier hidden states can change activation magnitudes and normalization behavior. Scalar gates are low-cost but do not guarantee numerical stability or improved predictions. Extra paths can also carry distracting information. Neither reduced catastrophic forgetting nor broad language generalization follows automatically from the connectivity rule.

4. Experimental method

The core comparisons used 32-layer, 64-dimensional causal transformer models with three supervised intermediate output heads: General at layer 16, Advanced Technical at layer 24, and Maximum Reasoning at layer 32. The tasks were small, template-generated synthetic domains: ownership extraction; technical protocol/port relationships presented in text; and two-link relationships between nodes. Training used AdamW (learning rate 0.002, no weight decay), FP32, equally weighted task losses and simultaneous examples from all three domains. For the joint cross-domain experiments, each step included 12 examples per domain (36 total), with 1,200 synthetic training and 96 validation examples per domain. The two architectures used matched seeds, common initialization for shared parameters and matched minibatches.

This setup probes relative learning under tightly constrained conditions; the subject labels do not imply general-purpose language understanding, professional technical expertise or unconstrained multi-step reasoning.

5. Experimental findings

5.1 Matched-seed joint training

Validation after 120 steps

Conventional

Two-hop

General — seed 11

100.00%

100.00%

General — seed 19

100.00%

100.00%

Technical — seed 11

58.33%

53.12%

Technical — seed 19

60.42%

64.58%

Reasoning — seed 11

63.54%

86.46%

Reasoning — seed 19

39.58%

87.50%

Reasoning — two-run mean

51.56%

86.98%

In the two completed replications, the two-hop model outperformed the conventional baseline on the deepest reasoning task by 22.92 and 47.92 percentage points, respectively (35.42 points on the two-run mean). General accuracy was equal at 100%, and the technical results alternated in which architecture performed better. Two completed seeds are insufficient for a stable performance estimate; the third planned replication was not completed and is not included.

5.2 Earlier convergence in a relational-language test

In a separate 32-layer synthetic language-like relational task, the two-hop model achieved 100% Maximum-exit validation accuracy at the recorded 40-step checkpoint, whereas the conventional model recorded 71.88% at that checkpoint and reached 100% by 80 steps. This supports a task-specific earlier-convergence observation, not a verified reduction in training FLOPs. The task used constrained phrase templates and distinct heads trained to predict different target relations from the same prompt.

5.3 Inference-time ablation

Seed 11 trained two-hop model

Connections on

Connections off

Restored

General

100.00%

100.00%

100.00%

Technical

53.12%

54.17%

53.12%

Reasoning

86.46%

75.00%

86.46%

Disabling the additional paths without changing other learned weights reduced reasoning accuracy by 11.46 percentage points, and restoring the paths recovered the prior accuracy. The learned scalar gates averaged approximately 0.148 in that run. This intervention demonstrates that the trained model relied on the paths for some predictions; it cannot by itself prove superiority to a baseline separately trained without those paths.

5.4 Forgetting and joint training

Sequential-learning experiments with different sentence orders and with distinct synthetic domains yielded mixed retention results. Two-hop connectivity did not consistently protect previously learned performance. By contrast, simultaneously supervising all three exits maintained General-task accuracy in both architectures and produced the reasoning differences shown above. Joint supervision tests multitask learning and interference, not classical catastrophic forgetting from sequential exposure.

6. Efficiency assessment

The connectivity rule adds 30 scalar parameters to the tested 32-layer variant—about 0.003% relative to an approximately 1.09-million-parameter baseline. This small parameter increase is attractive but does not make the added operations free. Each applicable block accesses a second hidden-state tensor, combines it with the primary input, and retains sufficient activations for backward differentiation. GPU memory traffic, kernel launch behavior and optimizer-step efficiency therefore require separate measurement. The CPU timings from small experiments were similar and varied by run; no reliable speedup was demonstrated.

The appropriate total-cost measure is compute, energy and elapsed time required to reach a specified quality target, rather than number of optimizer steps alone. Comparison should control architecture size, training data, seeds, learning-rate tuning and hardware.

7. Relationship to early-exit inference

The two-hop modification is architecturally independent of early exits. It can coexist with an adaptive-depth system having output heads at selected layer depths, provided that each executed block has its required predecessor states and that KV-cache consistency is maintained for autoregressive generation. Early exits were useful experimental supervision points, but do not establish that two-hop connectivity independently lowers inference compute.

8. Positioning relative to existing ideas

The proposal belongs to the broader family of skip-connected and cross-layer information-flow designs. Residual networks already provide bypass paths; DenseNet-style architectures use extensive cross-layer connectivity; other approaches reuse or aggregate states across depth. The differentiating design choice here is a deliberately minimal, single two-layer-span connection, added to otherwise conventional transformer blocks and controlled by a learnable scalar. This is a specific configuration and preliminary evaluation, not a claim that skip connections or cross-layer access are fundamentally new.

9. Validation limitations

  • Synthetic, templated datasets do not establish real-world language modeling or transferable subject expertise.
  • The strongest comparison has only two completed matched-seed trials, and statistical uncertainty remains large.
  • Different tasks showed different effects; Advanced Technical improvement was not reliable.
  • Mechanism is unresolved: improved information availability and changed gradient flow are both plausible.
  • No conclusive evidence of reduced catastrophic forgetting or total compute/energy savings.
  • No validated production throughput, accelerator-memory, distributed-training or robustness figures.

10. Conclusions

A limited two-hop connection is a modest, implementable variation on standard transformer architecture. Across selected small synthetic experiments it produced encouraging gains in early learning and deepest-exit relational reasoning, while adding very few trainable parameters. A reversible inference ablation found that the trained network made use of those paths. Results were not universal across tasks, and no claim is made that the design replaces conventional transformers, prevents forgetting, or reduces overall training cost. The present evidence supports positioning the design as a conceptual alternative with promising task-specific preliminary findings, suitable for future independent comparison when broader validation becomes worthwhile.

Appendix A. Experimental artifacts and reproducibility

Supporting executable records generated during this exploration:

Limited_Two_Hop_Transformer_Python_Experiments-v1.2.zip

  • 32_layer_two_hop_joint_test.py
  • 32_layer_two_hop_language_three_level.py
  • 32_layer_cross_domain_joint_training_test.py
  • 32_layer_two_hop_multiseed_joint_validation.py
  • 32_layer_two_hop_inference_ablation.py
  • 32_layer_two_hop_forgetting_test.py
  • 32_layer_cross_domain_continual_test.py

Companion JSON files containing measurements are stored under matching *_results.json filenames. Note that the baseline comparisons use controlled, small-scale synthetic data and should not be treated as a public benchmark.

Related research context (background, not independently verified here): He et al., “Deep Residual Learning for Image Recognition” (2016); Huang et al., “Densely Connected Convolutional Networks” (2017); Vaswani et al., “Attention Is All You Need” (2017).

© 2026 Christopher Soans. All rights reserved.

This work is licensed under a Creative Commons Attribution 4.0 International License (CC BY 4.0).