Graph Structure As Transformer Context

Summary

This page is the synthesis handle for a narrow design question: how should a mostly standard Transformer receive graph structure as context, especially for Kubernetes OTEL Control Gym?

The local answer is not “use a graph Transformer” as a single category. The useful design space is a set of graph-context interfaces:

• pairwise structural attention bias;
• explicit node and edge tokens;
• reversible graph serialization or graph-to-sequence traversal;
• learned graph-token vocabularies;
• implicit graph inference from transition sequences supplied in context;
• visibility masks or sparse relation masks when a sourced baseline is added.

These interfaces can make graph structure visible to a Transformer, but they are not by themselves evidence for an action-conditioned world model. The OTEL target remains:

observation + graph context + action/control input
  -> next observation + reward/label

Mechanism Map

Pattern	Main source	How graph structure enters the Transformer	OTEL adaptation	Main risk
Pairwise attention bias	Graphormer	Shortest-path distance, degree/centrality, and edge-path features become attention-bias terms.	Bias attention by service distance, dependency direction, edge type, and call-path structure.	Strong graph prior, but modifies attention and has quadratic scaling pressure.
Node and edge tokens	TokenGT	Nodes and edges become continuous Transformer input tokens; endpoint and type identifiers expose incidence without changing attention.	Services/resources become node tokens; directed dependencies become edge tokens; node- or edge-targeted actions can reuse the same identifier pairs.	Full attention scales with $n+m$ tokens, and snapshot-local ORF/Laplacian identifiers are not a temporal identity contract.
Reversible graph serialization	GraphGPT	Eulerian or semi-Eulerian paths serialize nodes, edges, and attributes into a reversible sequence.	Serialize whole service graphs, ego-subgraphs around affected services, trace-induced subgraphs, or action-targeted subgraphs.	Path stochasticity and subgraph sampling can disrupt temporal alignment and incident localization.
Reversible serialization plus BPE	Graph Tokenization	Frequency-guided graph serialization plus BPE turns recurring substructures into discrete graph tokens.	Learn reusable service-topology motifs such as fan-out, queue, cache, database dependency, or rollout target neighborhoods.	Continuous telemetry and timestamps still need typed numeric/time-series channels.
Learned graph-token vocabulary	GQT	A graph-specialized tokenizer learns quantized hierarchical tokens before Transformer processing.	Use as an optional graph-front-end for service topology and local neighborhoods.	Not the purest no-GNN path: the tokenizer can use graph-specialized machinery and may hide rare operational edges.
Visibility or adjacency mask	Winning the L2RPN Challenge	Grid adjacency constrains which power-grid elements attend to each other inside a graph-structured policy/value model.	Use visibility masks for physically or operationally reachable service/resource neighborhoods before action-effect prediction.	Domain-specific policy evidence; not a general graph tokenizer or proof of graph-context transfer.
Intervention-derived influence graph	Guided Machine Learning for power grid segmentation	Simulator interventions define a directed weighted graph over line-flow variables before clustering.	Use controlled rollouts or incident/action logs to derive effect graphs, then compare them with service topology.	Representation evidence; not a Transformer architecture or policy benchmark.
Heterogeneous domain graph	Graph Neural Networks for Transmission Grid Topology Control	Separates current and potential busbar connectivity to avoid hiding action-relevant topology information.	Distinguish current service dependencies from possible routing/deployment/control targets.	Power-grid GNN evidence; the Transformer interface still needs a tested token/bias design.
Line-graph local observation encoder	Graph-based distributed RL	Converts grid observations into a homogeneous line graph so a shared GNN enriches each line agent’s local view.	Build action-targeted local graph views for subcontrollers.	GNN policy evidence; no Transformer interface or learned transition rollout.
Graph action/risk surrogate	Soft-label topology actions, Gibbs priors, and Varying Grid Topology	Graph encoders help rank candidate topology actions or predict post-action overload risk.	Use graph context to score candidate remediations before expensive simulation or live execution.	Mostly GNN/surrogate evidence; multi-step action-conditioned Transformer world models remain untested.
In-context graph inference	ICLR: In-Context Learning of Representations	No explicit graph object is provided; a random-walk event stream makes transition adjacency inferable, and late-layer token activations develop graph-like geometry as context grows.	Test whether trace/event streams can induce a latent service-influence graph when the declared topology is incomplete or stale.	Induction-style retrieval and previous-token mixing can explain local prediction and visible geometry; degree-dependent visitation frequency is a confounder.

Cell-Complex Boundary

Topological Neural Operators is not a Transformer graph-context method, but it sharpens the boundary for graph time-series interfaces. Some physical quantities naturally live on vertices, edges, faces, or volumes, and their interactions are governed by incidence, codifferential, Hodge, and harmonic operators rather than arbitrary node-feature mixing. For power-grid, robotics, or operational telemetry experiments, this is a warning against flattening every structured quantity into anonymous channels: the model interface should preserve typed support, directionality, and action affordances when those structures determine future-state dynamics.

Explicit Graph Context Versus Inferred Transition Structure

ICLR: In-Context Learning of Representations adds an interface where graph structure is not serialized or supplied as pairwise bias. The context is only a sequence of graph-valid transitions. With enough context, a frozen LLM’s mean token activations become locally smooth over the hidden graph, while output probability shifts toward valid neighbors.

This is useful for event streams because operational influence graphs are often only partially declared. It is not evidence that explicit topology can be discarded. The benchmark can be solved through induction-style successor retrieval, and previous-token mixing can create graph-like class means without a reusable planner. A stronger OTEL experiment SHOULD therefore compare declared topology, transition-inferred topology, and intervention-derived influence graphs, then require multi-step and counterfactual queries that cannot be answered by copied bigrams.

Building-Telemetry Substrate: BTS

BTS supplies a concrete non-OTEL graph/time-series substrate. Its three buildings expose irregular numeric streams linked by StreamID to Brick 1.2.1 RDF graphs over points, equipment, and locations. The channel inventory differs across buildings, so the graph can act as a system descriptor rather than relying on a fixed channel position.

The released paper benchmarks do not yet use that graph as a model interface. Ontology classification predicts Brick labels from resampled stream chunks, and the cross-building forecasting appendix falls back to channel-wise Transformer processing when the input inventory changes. A stronger experiment would compare numeric-only, class-text-conditioned, and graph-conditioned models under the same cross-building split, then test missing channels, unseen equipment types, rare classes, and topology perturbations.

BTS also sharpens the action-conditioned boundary. Command, Setpoint, and Status are useful semantic roles, but the release does not validate command issuance, target, execution receipt, NOOP, or downstream outcome. The Brick graph is therefore context for a passive dynamics model until those streams are aligned into a typed control-input contract.

OTEL Graph Time-Series Contract

For k8s-otel-control-gym, graph structure SHOULD be treated as context, not as an anonymous channel order. The minimum model input contract is:

graph segment:
  graph.json or topology snapshot
  service/resource metadata
  edge type, direction, protocol, endpoint, and ownership metadata
 
observation segment:
  node_features.parquet time patches
  edge_features.parquet time patches
  selected event/log/trace streams
 
action segment:
  action_type
  target_service or target_edge
  parameters / control input
  status, precheck, postcheck
 
target:
  next node/edge observations
  reward
  labels or diagnosis fields

The model can then be tested on whether graph context improves action-effect prediction, not merely passive forecasting.

Recommended Baselines

The first k8s-otel-control-gym model suite SHOULD compare graph-context interfaces under matched data, context length, and compute:

1. No-graph / ID-only baseline. Flatten node and edge observations with service IDs only. This proves whether graph structure is actually helping.
2. TokenGT-style node/edge tokens. Make services/resources node tokens and directed dependencies edge tokens. Add time patches and action/control-input tokens explicitly.
3. Graphormer-style attention bias. Keep the token layout simple but add directed shortest-path distance, dependency direction, edge type, and action-target relation as pairwise biases.
4. Graph serialization baseline. Serialize graph snapshots or subgraphs with GraphGPT/Graph Tokenization-style traversal, then interleave telemetry and action segments.
5. Learned graph-tokenizer front-end. Try a GQT-like vocabulary only after the simpler baselines establish what graph details must be preserved.

The pragmatic first implementation should be a hybrid of explicit node/edge/ action tokens plus a small relation bias. It keeps the Transformer ordinary enough for standard sequence tooling, preserves entity-aligned output heads, and makes service topology visible before investing in learned graph vocabularies. The bias SHOULD cover only sourced or schema-defined relations such as node—edge incidence, directed one/two-hop distance, action target, topology version, and failure domain. Pure TokenGT MUST remain an ablation: otherwise the experiment cannot tell whether finite OTEL data actually needs the stronger inductive bias.

For time, the first implementation SHOULD encode each persistent node or edge’s recent numeric history into one temporal patch/state token before graph mixing. Repeating every graph token at every timestep creates $H(n+m)$ tokens and makes full attention quadratic in both history length and graph size. A later factorized model can alternate per-entity temporal attention with cross-entity graph attention if one summary token erases action transients.

TokenGT identifiers also need to be split into distinct contracts. Episode-local entity keys track the same logical service, workload, or dependency through a trajectory; semantic metadata supports cross-system transfer; node/edge/action type identifiers expose role; and optional Laplacian features describe the current topology version. Pod UIDs, array positions, independently resampled ORF vectors, and Laplacian coordinates MUST NOT be treated as durable service identity across topology snapshots.

For power-grid or OTEL graph experiments, include a direct message-passing baseline such as Graph Neural Solver or LEAP-style models when applicable. This tests whether Transformer graph-context interfaces beat a domain-specific graph neural surrogate, but it should not be counted as Transformer evidence.

IO-Aware GNN Layers makes that direct-GNN baseline a performance-hygiene requirement rather than a throwaway comparison: if the baseline uses slow DGL/PyG composition that materializes edge-wise tensors, it may be measuring framework overhead more than the value of message passing. For OTEL experiments, report wall-clock latency, peak memory, preprocessing/reordering cost, graph degree distribution, and whether the GNN layer used fused attention, degree-aware reductions, cached cuSPARSE, or default framework kernels.

Action-Conditioned Boundary

This source cluster is graph-learning or graph-context evidence. It does not by itself test controlled DevOps trajectories, interventions, rewards, or counterfactual rollouts.

That boundary matters. A model that encodes the service graph well is still only a graph-aware passive dynamics model until it receives logged actions and is evaluated on:

• next node/edge feature prediction after an action;
• action delta versus NOOP;
• candidate-action ranking by reward;
• counterfactual prediction under alternative actions;
• closed-loop regret or recovery quality on the live stand.

For a non-OTEL graph-control precedent, see Grid2Op: graph-structured numeric observations, topology/control inputs, simulator-backed next states, and safety/cost outcomes. The transferable point is the action-conditioned graph contract, not the power-grid domain itself.

Graph context should distinguish physical topology from functional influence graphs. The latter may group non-adjacent components and can be more relevant for action-effect prediction, but it depends on the simulator, intervention set, or incident/action logs used to derive the graph.

The fresh Grid2Op papers add a second warning: graph context must preserve action affordances, not only current adjacency. A representation that only says which objects are connected now can omit which objects could become connected after a busbar/topology action. For OTEL, the analogous distinction is current dependency topology versus possible deployment, routing, scaling, rollback, feature-flag, or traffic-shaping targets.

Recent Grid2Op graph papers also disagree on the best representation route for busbar/action-affordance information: heterogeneous current/potential connectivity versus homogeneous line-graph preprocessing. Treat this as a benchmarked design choice, not a settled rule.

Relation To Foundation TSFM Agenda

Agenda slot	Verdict	Evidence	Missing pieces
Context interface	partially closes	The source cluster gives concrete ways to encode graph structure as Transformer context.	Needs an OTEL schema that joins graph context with observation windows, events, and action history.
Native multivariate encoding and high-channel scaling	adjacent	Node/edge tokens, graph serialization, and graph-token vocabularies avoid flattening service telemetry into anonymous channels.	Needs graph time-series experiments with topology drift, missing streams, and high-cardinality telemetry.
Control and counterfactuals	insufficient evidence	Graph context is necessary for action-conditioned observability world models.	Needs observation + graph + action/control input -> next observation/reward experiments.
Online structure inference	adjacent	In-context graph tracing shows that a transition event stream can reorganize internal geometry without weight updates.	Must rule out local induction/retrieval shortcuts and test unseen multi-step or counterfactual transitions.

Open Questions

• Which interface wins under matched compute for OTEL episodes: node/edge tokens, pairwise attention bias, graph serialization, learned graph tokens, or a hybrid?
• Which IO-aware direct-GNN baseline is the right matched-compute comparator for each graph-context interface: cached SpMM, fused CSR graph attention, degree-aware reductions, or block-sparse Tensor Core attention?
• How should time be represented: one graph-token set per timestep, temporal patches per node/edge token, or a flattened sequence over (time, graph element) pairs?
• What identifier scheme survives service renames, topology changes, autoscaling, and ephemeral Kubernetes objects?
• Does a target-swap probe change the predicted action effect while preserving the same observation and action type, or is the model ignoring TokenGT-style action-to-entity incidence?
• How much action-transient information is lost when each entity’s time window is compressed before graph mixing, and when is factorized space-time attention necessary?
• Should the graph tokenizer learn motifs from one OpenTelemetry Demo graph, from many generated graph variants, or from external production topologies?
• How should graph time-series benchmarks distinguish physical topology from intervention-derived influence or effect graphs when they disagree?
• How do we test that graph compression does not erase rare but intervention-relevant edges?
• Which graph-context benchmark forces inferred transition structure rather than local successor retrieval from repeated event-stream bigrams?

Related Pages

• Kubernetes OTEL Control Gym
• Observability Time Series
• High-Dimensional Time Series Forecasting
• Latent Tokenization
• Tokenizer Transfer
• World Models
• Foundation Time-Series Model Research Agenda
• Graph Observability Benchmarks
• Grid2Op