Spenso parsing flow

Lifecycle

current

Owner

Ownership decision required

Review

2026-10-01 · jj:xuoqsyop

Freshness

reviewed

Evidence

Not an evidence record

Verified scopes

• spenso::network::parsing::ShorthandParsing
  crates/spenso/src/network/parsing/mod.rs · digest bd148e50a697
  entire file

This document describes how symbolic atoms are parsed into tensor networks in crates/spenso/src/network/parsing. It focuses on the control flow, shorthand expansion, opaque leaves, structure inference, and edge cases that affect Schoonschip-style notation.

Audit status: reviewed 2026-10-01 against jj change xuoqsyop. Lifecycle: current implementation architecture.

Entry Points

Parsing starts from NetworkParse methods such as parse_to_atom_net. The parser first rejects chains or traces nested inside another chain or trace, because they share one global in/out placeholder scope. It then creates a fresh ParseState, reserves the serialized names of input slot indices, and calls try_from_view_impl on the input AtomView. Parser clones share both the allocator and reservation set, so newly generated dummies cannot collide with written indices or each other.

Settings

When scalar precontraction is enabled, sum parsing collects borrowed pure-scalar summands and constructs their sum once with Atom::add_many only if tensor summands remain. An entirely scalar sum reuses the already normalized input atom. This avoids repeatedly copying a growing scalar sum; compatibility checks and tensor summands retain their existing meaning.

Option Examples

Main Dispatch

try_from_view_impl dispatches by atom kind:

Mul  -> try_from_mul
Fun  -> try_from_fun
Add  -> try_from_add
Pow  -> try_from_pow
else -> scalar fallback

Products, sums, and powers may hit the depth leaf boundary before they recurse. Function parsing validates dot arity, handles broadcast wrappers, applies is_tensorial(strict_tensor_filter), and uses the opaque tensor-expression boundary only for recognized shorthand roots.

crates/spenso/src/network/parsing/mod.rs:362

match value {
    AtomView::Mul(m) => Self::try_from_mul(...),
    AtomView::Fun(f) => Self::try_from_fun(...),
    AtomView::Add(a) => Self::try_from_add(...),
    AtomView::Pow(p) => Self::try_from_pow(...),
    a => Ok(Network::from_scalar(a.try_into()?)),
}

Function Flow

Function parsing has the most important ordering rules.

1. Reject malformed dot syntax first. Dot has exactly two arguments in parser syntax.
2. If the head has the broadcast tag, require one argument, parse that argument recursively, and apply the head to the result.
3. Apply is_tensorial(settings.strict_tensor_filter). If it rejects the root, parse it as a scalar; pure_scalar(x) is unwrapped in that scalar path.
4. If shorthand_parsing is Opaque and the root is recognized shorthand (chain, trace, dot, or compact Schoonschip syntax), parse the whole function as one inferred tensor leaf. Ordinary tensor heads do not take this branch.
5. Otherwise enter expanded-function parsing. bracket parses its arguments and multiplies them; other roots enter the shorthand materializer.
6. The Expand fields independently decide whether chain, trace, and compact Schoonschip syntax materialize. A disabled shorthand root is retained as a fast-inferred leaf. An unchanged ordinary head uses regular leaf parsing.
7. Try S::parse. If it succeeds, use a library tensor when possible, otherwise parse a rank-zero result as a scalar or concretize a tensor shell.
8. If structure parsing reports StructureError::EmptyStructure, parse the function as a scalar expression. Propagate other structure errors.

crates/spenso/src/network/parsing/mod.rs:606

if symbol == SPENSO_TAG.dot && value.get_nargs() != 2 {
    return Err(TensorNetworkError::InvalidDotFunction(...));
}

if symbol.has_tag(&SPENSO_TAG.broadcast) {
    return Self::parse_broadcast_function(...);
}

if !value.as_view().is_tensorial(settings.strict_tensor_filter) {
    return Self::parse_scalar_function(value);
}

crates/spenso/src/network/parsing/structure_inference.rs:50

pub trait AtomStructureExt {
    fn infer_structure<S: StructureFromAtom>(...);
    fn is_tensorial(&self, filter: StrictTensorFilter) -> bool;
}

crates/spenso/src/network/parsing/mod.rs:626

if let Some(inference) = settings.shorthand_parsing.opaque_inference()
    && Self::is_shorthand_function(value)
{
    return Self::as_inferred_leaf(...);
}

crates/spenso/src/network/parsing/mod.rs:729

Self::materialize_shorthand(
    value,
    state,
    library,
    function_library,
    settings,
)

Important: materialization must run before the generic leaf parser. Otherwise F(p(rep), slot) can be accepted as a lower-rank tensor with p(rep) stored as metadata, losing the required p(dummy) factor.

Expand vs Opaque

Expand

Selected shorthand families are rewritten into explicit parser input. The schoonschip, trace, and chain fields can be configured independently. Expanded chain and trace roots allocate links; compact vector arguments allocate fresh slots.

Opaque

A recognized shorthand root that survives strict tensor filtering and broadcast handling becomes one leaf. Structure is inferred first, then TensorFromExpression constructs the target tensor from the original expression and inferred structure. Ordinary tensor heads still use regular leaf parsing.

Opaque mode is checked before chain or trace expansion, but after strict tensor classification and broadcast handling. Non-tensor functions stay scalar in opaque mode; depth-limited composite expressions use the separate leaf boundary described below.

Opaque Leaf Boundary

Opaque shorthand parsing is split into two steps.

1. StructureFromAtom::structure_from_atom determines the exposed slots. Fast uses a syntactic walk. Expanded builds an expanded shorthand network and reads dangling slots.
2. TensorFromExpression::tensor_from_expression turns the original atom plus inferred structure into a tensor. Symbolic tensors keep the original expression on a tensor with the inferred structure. Concrete and parametric tensor classes parse the same atom again with ShorthandParsing::expand_all(), execute that nested network, and return the resulting tensor or tensor scalar.

crates/spenso/src/network/parsing/structure_inference.rs:162

match mode {
    StructureInferenceMode::Fast => Self::leaf_structure_from_atom(value),
    StructureInferenceMode::Expanded =>
        Self::expanded_shorthand_structure_from_atom(value),
}

crates/spenso/src/network/parsing/tensor_from_expression.rs:25

/// Builds a tensor leaf from an opaque expression once its structure is known.
pub trait TensorFromExpression<S, Sc, K, FK, Aind, Lib, FunLib> {
    fn tensor_from_expression(...);
}

crates/spenso/src/network/parsing/tensor_from_expression.rs:120

let mut expanded_settings = settings.clone();
expanded_settings.shorthand_parsing = ShorthandParsing::expand_all();

let mut network = Network::try_from_view_with_function_library(
    expression,
    tensor_library,
    function_library,
    &expanded_settings,
)?;
network.execute::<Sequential, SmallestDegree, ...>(...)?;

crates/idenso/src/tensor/mod.rs:118

let mut tensor = structure.into_canonical();
...
tensor.expression = expression.to_owned();
Ok(tensor)

Structure Inference

Fast structure inference is syntactic and intentionally non-semantic.

Materialization

The parser's materialize_shorthand method is the configurable shorthand boundary. It always returns a network. Enabled chain and trace roots materialize topology directly. Enabled compact dot/Schoonschip syntax rewrites to ordinary parser input and recurses. A disabled shorthand root becomes a fast-inferred leaf; an unchanged root containing disabled compact shorthand also remains a leaf. An unchanged ordinary function uses regular leaf parsing.

SchoonschipMaterializer is the narrower atom rewriter used by that network boundary. It stores a current atom plus additional_factors. Additional factors are multiplied beside the current atom. Compact scalar factors inside a compact-vector product wrapper are recursively materialized before the complete expression re-enters ordinary parsing.

crates/spenso/src/network/parsing/materialization.rs:457

fn materialize_shorthand(...)
    -> Result<Self, TensorNetworkError<K, Symbol>>
{
    if symbol == SPENSO_TAG.chain && !root_chain_disabled {
        return Self::materialize_chain_shorthand(...);
    }
    if symbol == SPENSO_TAG.trace && !root_trace_disabled {
        return Self::materialize_trace_shorthand(...);
    }

    let materialized = SchoonschipMaterializer::with_mode(&state, mode)
        .materialize_shorthand(value.as_view());
    if materialized == value.as_view().to_owned() {
        if root_chain_disabled || root_trace_disabled || has_schoonschip_shorthand {
            return Self::as_inferred_leaf(..., StructureInferenceMode::Fast, ...);
        }
        return Self::parse_regular_function_leaf(...);
    }
    Self::try_from_view_impl(materialized.as_view(), ...)
}

crates/spenso/src/network/parsing/materialization.rs:114

pub(super) fn materialize_shorthand(&self, value: AtomView<'_>) -> Atom {
    self.materialize_shorthand_root(value)
        .map(SchoonschipMaterialization::into_expression)
        .unwrap_or_else(|| value.to_owned())
}

crates/spenso/src/network/parsing/materialization.rs:171

let lhs_rep = Self::compact_vector_rep(*lhs)?;
let rhs_rep = Self::compact_vector_rep(*rhs)?;
if !lhs_rep.matches(&rhs_rep) {
    return None;
}

let lhs_slot = self.state.slot(&lhs_rep);
let rhs_slot = rhs_rep.slot::<Aind, _>(lhs_slot.aind());

crates/spenso/src/network/parsing/materialization.rs:20

// Chain and trace materialization chooses the symbolic in/out
// replacements, then lets this Schoonschip helper expand compact
// arguments inside each factor.

Compact Vector Argument

F(..., p(rep), ...)
-> F(..., rep(dummy), ...) * p(rep(dummy))

The dummy slot is allocated from the active ParseState, so it shares the namespace of the surrounding network parse.

Compact Scalar Product

g(p(rep), q(rep.dual()))
-> p(rep(dummy)) * q(rep.dual()(dummy))

The compact representations must match under duality, including dimension. For self-dual representations, rep.dual() is rep. dot(p(rep), q(rep.dual())) follows the same rule.

The stripped representation in p(rep) means that the vector replaces an omitted tensor slot. It is valid only where such a slot exists: inside another tensor argument, or inside a recognized compact scalar product head such as g or dot. For example, p!(q!(mink!(4))) is valid shorthand and materializes like p(mink(4, d)) * q(mink(4, d)). A bare product such as p!(mink!(4)) * q!(mink!(4)) is malformed input, because the stripped vectors are not replacing slots.

Compact Vector Detection

A compact vector is a function that:

• has a tensor-tagged head;
• is not metric or dot;
• is not itself a representation;
• has no explicit slot argument;
• has exactly one direct argument matching the representation wildcard convention.

Sums are accepted only when every summand is a compact vector with the same representation. Unary broadcasts, projectors, and products preserve compact-vector syntax when they contain exactly one compatible vector and otherwise scalar factors. Powers are not compact vector syntax.

Chain Expansion

When chain expansion is enabled, chain(start, end, factors...) first tries to parse each endpoint as a concrete slot. If Schoonschip expansion is enabled, a compact vector endpoint can instead materialize into a fresh endpoint slot plus an additional rank-one factor. The network-level materializer then chooses internal links, rewrites each factor's in/out placeholders, and optionally expands compact arguments inside the factor.

1. Materialize compact endpoints when configured and retain their additional factors.
2. With no chain factors, add g(start, end).
3. Each factor gets a left and right link slot.
4. in and out placeholders are replaced syntactically.
5. The factor is passed through shorthand materialization.
6. Each materialized factor is parsed into one or more factor networks.
7. The factor networks are multiplied in chain order.

If a materialized chain factor is a product with scalar factors and exactly one tensor factor, scalar factors are split out so they can be handled separately. Otherwise the whole factor product is parsed as one expression.

chain(s, e, F(in, out, p(rep)))
-> replace placeholders: F(s, e, p(rep))
-> materialize factor:   F(s, e, d) * p(d)
-> parse product into the chain network

crates/spenso/src/network/parsing/materialization.rs:540

let start = Self::materialize_chain_endpoint(args[0], "start", ...)?;
let end = Self::materialize_chain_endpoint(args[1], "end", ...)?;
...
let factor = ChainExpansion::replace_placeholders(...);
let factor = SchoonschipMaterializer::with_mode(&state, factor_mode)
    .materialize_shorthand(factor.as_view());

Trace Expansion

When trace expansion is enabled, trace(rep, factors...) starts from a representation, not endpoint slots. The network-level shorthand materializer returns the representation dimension for an empty trace and otherwise assigns cyclic links uniformly across all factors. Depending on the Schoonschip expansion settings, it can materialize compact arguments inside each factor after placeholder replacement and before parsing.

1. No factors: parse the representation dimension.
2. For a non-empty trace, allocate one cyclic link per factor.
3. Replace each factor's placeholders with its link and the dual of the next cyclic link.
4. Materialize shorthand inside each factor when configured.
5. Multiply the factor networks.

trace(rep, F(in, out, p(rep)))
-> allocate one cyclic link
-> replace placeholders: F(link, link*, p(rep))
-> materialize factor:   F(link, link*, d) * p(d)

crates/spenso/src/network/parsing/materialization.rs:701

let rep = Representation::try_from(*rep_view)?;
...
let links = (0..factors.len())
    .map(|_| state.slot(&rep))
    .collect::<Vec<_>>();
let left = links[position].to_atom();
let right = links[(position + 1) % factors.len()].dual().to_atom();
let factor = ChainExpansion::replace_placeholders(factor, &left, &right);

Products, Sums, Powers

When a positive integer power greater than one lowers shorthand that allocates internal dummy indices, each remaining copy is parsed from the original base with the shared allocator. For a self-dual representation, dot(p(rep), q(rep))^2 therefore has two contracted pairs, p(d1) * q(d1) and p(d2) * q(d2), with distinct internal indices. Explicit boundary slots stay unchanged. Bases that allocate no new indices use the ordinary network power path; scalar precontraction and negative-power restrictions still apply.

Edge Cases

Short Algorithm

parse(atom):
  reject nested chain/trace placeholder consumers
  dispatch by atom kind

parse_fun(fun):
  if fun is dot with arg count != 2:
    return InvalidDotFunction

  if fun is a broadcast wrapper:
    return its parsed single argument with broadcast handling

  if !fun.is_tensorial(strict_tensor_filter):
    return scalar(fun)

  if opaque mode and fun is recognized shorthand:
    infer structure
    return TensorFromExpression(original expression, inferred structure)

  return materialize_shorthand(fun)

materialize_shorthand(fun):
  if chain expansion is enabled and fun is chain:
    materialize compact endpoints when configured
    allocate links
    replace in/out placeholders
    materialize configured shorthand inside each factor
    parse factor expressions
    return chain network

  if trace expansion is enabled and fun is trace:
    return the dimension for zero factors
    allocate one cyclic link per factor
    replace in/out placeholders
    materialize configured shorthand inside each factor
    parse factor expressions
    return trace network

  select the effective Schoonschip expansion mode
  rewrite the function if that mode is enabled
  if an enabled shorthand remains unchanged:
    return an inferred tensor leaf
  if an ordinary function remains unchanged:
    return a regular tensor leaf
  parse the rebuilt expression recursively

Factorized scalar and symbolic-dimension boundaries

Scalar factors and compatible sums remain grouped while tensor slots are inferred. A scalar-weighted compact vector is accepted only when exactly one factor supplies its compact axis and every other factor has scalar structure; unsupported products remain opaque. Library lookup retains the canonical layout. Missing tensor leaves use the target’s fallible Concretize::concretize_logical boundary: symbolic targets may keep symbolic dimensions, while component targets report materialization errors.