αDeveloper notes
Research docs
On this page

For developers

CI test failure summary

Archived investigationCI investigation · 2026-03-25archived

A 2026-03-25 workspace failure inventory retained as historical triage evidence rather than a current CI status page.

View the source note

Lifecycle
archived
Owner
Ownership decision required
Review
No review record
Freshness
frozen-evidence
Evidence
captured 2026-03-25
Verified scopes
No verified code scopes recorded

Lifecycle: Archived investigation captured 2026-03-25

This record predates later fixes and does not identify one immutable source revision for every run. Test names, snapshots, and environment behavior may have changed. Preserve it as triage evidence; use the current CI workflow and test results for the present contract.

Date: 2026-03-25

Scope

  • CI workflow inspected: .github/workflows/continuous-integration.yml
  • Test command used by CI on Ubuntu and macOS: just test-ci
  • Local Nix equivalent exposed by the repo: just test-nix

Commands Run

  1. SYMBOLICA_LICENSE=${SYMBOLICA_LICENSE:-GAMMALOOP_USER} just test-ci
  2. SYMBOLICA_LICENSE=${SYMBOLICA_LICENSE:-GAMMALOOP_USER} just test-nix
  3. Follow-up isolation run: SYMBOLICA_LICENSE=${SYMBOLICA_LICENSE:-GAMMALOOP_USER} cargo nextest run --workspace --exclude linnet-py --profile ci --locked --no-fail-fast
  4. Fix validation run: cargo test -p linnet-py --no-default-features --no-run --locked

Top-Level Result

just test-ci

  • Failed before the test run could start.
  • Immediate blocker: linnet-py fails to link on macOS/arm64 with unresolved Python symbols such as _Py_InitializeEx, _Py_IsInitialized, __Py_DecRef, __Py_IncRef.

  • Final error:

    • error: could not compile 'linnet-py' (lib) due to 1 previous error
    • ld: symbol(s) not found for architecture arm64

just test-nix

  • Failed after the Nix build reached the gammaloop-nextest check derivation.

  • Final failure:

    • Missing SYMBOLICA_LICENSE environment variable
    • emitted from the Nix checkPhase of gammaloop-workspace-nextest-0.1.0.drv

  • Reading:

    • the flake uses builtins.getEnv "SYMBOLICA_LICENSE" for the nextest check input.
    • the local just test-nix command does not pass --impure to nix build, so the environment variable is not visible at Nix evaluation/build time.

Follow-up run excluding linnet-py

  • Purpose: determine whether the workspace has additional failures behind the linnet-py linker blocker.
  • Result: yes, multiple additional failures are exposed.
  • Final result: completed with exit code 100 and a broad failure surface across integration tests and crate unit tests.

Confirmed Failures And Likely Fixes

  • Failure source:

    • crates/linnet-py/Cargo.toml

  • Relevant manifest detail:

    • default = ["extension-module"]

  • Why this is likely the cause:

    • PyO3 extension-module suppresses normal libpython linking and is correct for packaging a Python extension.
    • cargo test --workspace / cargo nextest run --workspace still builds the crate in a Rust test context, where those Python symbols must resolve.
    • Validation: cargo test -p linnet-py --no-default-features --no-run --locked succeeded locally.

  • Possible fixes:

    • Remove extension-module from linnet-py default features and enable it only in packaging commands.
    • Keep a dedicated packaging path such as maturin develop --features extension-module or equivalent.
    • If linnet-py is not meant to participate in the Rust workspace test matrix, exclude it from the CI nextest workspace invocation.
    • Secondary cleanup: align pyo3-build-config in crates/linnet-py/Cargo.toml with the workspace version (0.28) instead of the current standalone 0.25.

  • Status on this branch:

    • Fixed.
    • linnet-py no longer enables extension-module by default, pyo3-build-config now uses the workspace version, and crates/linnet-py/pyproject.toml enables extension-module explicitly for maturin packaging.

    • Validation:

      • cargo test -p linnet-py --no-run --locked
      • SYMBOLICA_LICENSE=${SYMBOLICA_LICENSE:-GAMMALOOP_USER} cargo nextest list --workspace --profile ci --locked

2. gammaloop-api REPL completion regression

  • Test:

    • gammaloop-api repl::tests::completion_does_not_repeat_used_options_by_short_or_long_form

  • Failure:

    • assertion failed at crates/gammaloop-api/src/repl.rs:4364
    • !values.contains(&"-p".to_string())

  • Reading:

    • completion candidates still include a short option that should have been filtered out after the option was already used.

  • Possible fixes:

    • Inspect option de-duplication in crates/gammaloop-api/src/repl.rs around the completion filtering logic near line 4364.
    • Check whether recent clap metadata changes now expose both short and long forms separately and the filter only removes one representation.
    • Add normalization so -p and --process share the same “already used” key.

3. State roundtrip regression: required file missing

  • Test:

    • gammaloop-api state::tests::state_manifest_roundtrip_current_version

  • Failure:

    • panic at crates/gammaloop-api/src/state.rs:2112
    • underlying error at crates/gammaloop-api/src/state.rs:833
    • missing file: symbolica_state.bin

  • Reading:

    • the roundtrip test expects a saved state layout that includes symbolica_state.bin, but the saved state in the temp directory no longer has it.

  • Possible fixes:

    • Reconcile the test with the current saved-state contract if symbolica_state.bin is no longer supposed to exist.
    • If the file is still required by the manifest/current version, inspect the save path in State::save and any recent changes that stopped writing it.
    • Avoid unwrap() in the test until the contract is settled; assert on the expected error or the expected saved files explicitly.

4. Histogram/event-processing regression

  • Test:

    • gammaloop-integration-tests::test_differential event_processing_runtime_merges_worker_results_and_batch_histograms

  • Failure:

    • Error: Rebinning factor 2 does not divide the 2 histogram bins exactly.
    • location: crates/gammalooprs/src/observables/mod.rs:1063

  • Reading:

    • either the rebin divisibility check is wrong, or the test is now feeding a histogram shape that no longer matches the implementation assumptions.

  • Possible fixes:

    • Inspect the divisibility logic around crates/gammalooprs/src/observables/mod.rs:1063.
    • Confirm whether the check is using the total bin count, visible bins, or excluding underflow/overflow incorrectly.
    • Compare the test fixture’s histogram configuration to the new sparse histogram semantics documented in the repo instructions.

5. Vakint / analytical evaluation regression

  • Test:

    • gammaloop-integration-tests::test_evaluation_command evaluate_1l_scalar_vacuum

  • Failure:

    • Could not find muV in graph
    • location: crates/gammalooprs/src/processes/amplitude.rs:744

  • Reading:

    • the vakint analytical path appears to assume a graph component or variable (muV) that is not present in the generated scalar vacuum topology anymore.

  • Possible fixes:

    • Inspect the graph fixture used by tests/tests/test_evaluation_command.rs.
    • Check recent changes in graph import or vakint topology generation that renamed or dropped muV.
    • If this command is expected to work numerically without a vakint analytical topology assumption, separate the analytical and numerical preconditions more clearly.

6. Feyngen snapshot regressions

  • Confirmed failing snapshot-based tests:

    • gammaloop-integration-tests::test_feyngen from_symbolica
    • gammaloop-integration-tests::test_feyngen example_graph_count
    • gammaloop-integration-tests::test_feyngen test_generate_sm_a_ddx
    • gammaloop-integration-tests::test_feyngen cp_fix_from_symbolica

  • Observed changes:

    • renamed/sign-expanded factors:

      • ExtFerm -> ExternalFermionOrderingSign
      • IntFerm -> InternalFermionLoopSign

    • graph serialization differences:

      • quoted attribute values
      • different hedge/edge numbering and projector ordering

    • color-factor / grouping changes:

      • several snapshots flip Nc vs Nc^(-1) placement/sign ordering
      • one case flips the sign of the 12*G^2*ee^(-2) contribution

  • Possible fixes:

    • First decide whether the new outputs are semantically equivalent or physically changed.
    • If only naming/serialization changed, update the insta snapshots.
    • If the Nc / Nc^(-1) flips represent a real color-factor regression, inspect the numerator grouping / canonized numerator comparison path before updating snapshots.
    • The strongest suspects are the fermion-ordering sign and graph canonization / serialization code used by feyngen snapshots.

7. Feyngen workspace/path regression

  • Test:

    • gammaloop-integration-tests::test_feyngen test_generate_aa_ttx_cross_section_slow_filter

  • Failure:

    • No such file or directory (os error 2)
    • location: crates/gammalooprs/src/utils/serde_utils.rs:41
    • call path shows State::save trying to create model_parameters.json

  • Reading:

    • the test’s selected workspace root is not created before save, or save no longer ensures parent directories exist.

  • Possible fixes:

    • Ensure State::save creates parent directories before File::create(...).
    • If directory creation is intentionally the caller’s job, update the test helper in tests/src/lib.rs / tests/tests/test_feyngen.rs to create the workspace first.

8. Python API tests are not prepared in the Rust CI path

  • Confirmed failing tests:

    • python_evaluate_sample_honors_generate_events_and_observable_snapshots
    • python_evaluate_sample_preserves_graph_grouping_and_incoming_pdgs
    • python_evaluate_samples_batch_and_momentum_space_have_expected_shape
    • more may follow in the still-running partial pass

  • Failure:

    • No module named 'gammaloop'

    • test message explicitly says:

      • run maturin develop or just build-api first

  • Reading:

    • this matches the repo’s own API interop note: these subprocess tests assume the extension was already built and installed into the active Python environment.
    • the current Rust just test-ci path does not appear to satisfy that prerequisite.

  • Possible fixes:

    • Build/install the Python extension before running these tests in CI.
    • Or gate/skip these tests unless the extension is present.
    • Or split them into a separate CI job that explicitly runs just build-api first.

  • Status on this branch:

    • Fixed by moving the subprocess-based Python API tests behind an explicit python-api-tests feature on the integration-test crate.
    • Default just test-ci no longer registers those tests at all, so a green CI run does not silently pass them without execution.
    • They can be run explicitly after preparing the Python environment with just test-python-api.

    • Validation:

      • cargo nextest list --workspace --profile ci --locked
      • cargo nextest list -p gammaloop-integration-tests --features python-api-tests --test test_python_api --profile ci --locked

9. Additional test_runs stack overflows

  • Confirmed failing tests:

    • gammaloop-integration-tests::test_runs addbar
    • gammaloop-integration-tests::test_feyngen simple_epem_ddx_generation

  • Failure:

    • thread '<unknown>' has overflowed its stack
    • process aborts with SIGABRT

  • Reading:

    • these are not ordinary assertion failures; something in graph generation / rendering / recursion is blowing the thread stack.
    • simple_epem_ddx_generation reaches a large feyngen grouping run before aborting.

  • Possible fixes:

    • inspect recent recursion-depth growth in feyngen grouping / canonization paths.
    • consider replacing deep recursion with an explicit work stack in the offending traversal.
    • if these tests run on a deliberately small stack in nextest, confirm whether the code now depends on unusually deep nesting.

10. More feyngen snapshot regressions appeared

  • Additional confirmed failing snapshot tests:

    • gammaloop-integration-tests::test_feyngen test_generate_aa_ttx_amplitude
    • gammaloop-integration-tests::test_runs photons_1l_inspect
    • gammaloop-integration-tests::test_runs trees
    • gammaloop-integration-tests::test_feyngen test_vaccuum_amplitude_generation_full_sm

  • Observed concrete diff:

    • generate_aa_ttx_amplitude-3 changed from 152 | 28 = 28 to 144 | 36 = 36

  • Reading:

    • at least some of these are not mere serialization/naming changes; they change graph counts or grouped totals.

  • Possible fixes:

    • inspect whether the new numbers are expected due to a deliberate grouping/filtering change.
    • if expected, update snapshots only after confirming the new counting logic is correct.
    • if not expected, focus on recent changes in feyngen grouping, zero-detection, and symmetry-factor accumulation.

11. Run-card schema incompatibility

  • Confirmed failing test:

    • gammaloop-integration-tests::test_runs oak

  • Failure:

    • Error parsing run history toml
    • unknown field enable_thresholds
    • location: crates/gammaloop-api/src/state.rs:744

  • Reading:

    • older run cards under tests/resources/run_cards still contain enable_thresholds, but the current deserializer no longer accepts it.
    • this is consistent with the repo policy of allowing backward-incompatible state/config changes, but then tests and example cards must be updated in lockstep.

  • Possible fixes:

    • update the affected run cards to the current schema.
    • or add a migration/alias only if preserving old run-card compatibility is now desired.

12. Missing example command-card files

  • Confirmed failing tests:

    • gammaloop-integration-tests::test_runs test_advanced_integration_example_card
    • gammaloop-integration-tests::test_runs test_simple_workflow_example_card

  • Failure:

    • could not open:

      • examples/command_cards/advanced_integration.toml
      • examples/command_cards/simple_workflow.toml

  • Reading:

    • either the tests still reference files that were renamed/removed, or the example-card locations changed without updating test expectations.

  • Possible fixes:

    • restore the missing files.
    • or update the tests to the new example-card names/paths.

13. More run/integration failures are present but not fully triaged yet

  • Additional confirmed failing test names seen in the background follow-up run:

    • photonic_amplitudes
    • photons_2l_inspect
    • scalar_box
    • test_epem_tth_inspect_nlo_gl18
    • multiple gammalooprs unit tests in cff, settings, uv, utils::serde_utils
    • multiple idenso, spenso, and spenso-hep-lib tests

  • Reading:

    • the failure surface is substantially broader than the initial workspace blocker.
    • many of the later failures are SIGABRT after panic reporting, so the current log is enough to identify names but not yet enough to classify every root cause.

  • Suggested next step:

    • rerun failing subsets individually once the first-layer blockers are addressed, so the aborting panic hook does not hide later diagnostics.

14. Completed follow-up run surfaced more concrete regressions

  • Additional confirmed failures from the finished --exclude linnet-py run:

    • gammaloop-integration-tests::test_feyngen simple_epem_ddx_generation

      • stack overflow after generating 86 amplitudes

    • gammaloop-integration-tests::test_feyngen test_generate_aa_ttx_amplitude

      • snapshot generate_aa_ttx_amplitude-3 changed from 152 | 28 = 28 to 144 | 36 = 36

    • gammaloop-integration-tests::test_runs oak

      • run-card parse failure on unknown field enable_thresholds

    • gammaloop-integration-tests::test_runs test_advanced_integration_example_card

      • missing examples/command_cards/advanced_integration.toml

    • gammaloop-integration-tests::test_runs test_simple_workflow_example_card

      • missing examples/command_cards/simple_workflow.toml

    • gammaloop-integration-tests::test_runs addbar

      • stack overflow

    • gammalooprs::test_renormalization finite_part_ghost_3loop

      • panic at crates/gammalooprs/src/uv/forest.rs:135
      • message: not implemented: Union not implemented

  • Additional confirmed snapshot failures from the finished run:

    • photons_1l_inspect
    • trees
    • vaccuum_amplitude_generation_full_sm-2

    • several spenso / spenso-hep-lib snapshot tests:

      • infinite_execution
      • parse_ratio
      • parse_scalar_tensors_step_by
      • parse_val
      • to_symbolic
      • exterior_prod_simple

  • Additional confirmed aborting unit/integration tests from the finished run:

    • multiple gammalooprs::cff::* tests
    • multiple gammalooprs::settings::* tests
    • multiple gammalooprs::uv::* tests
    • multiple idenso::* tests
    • linnet permutation::tests::test_iter_slice_panic_too_short
    • linnet permutation::tests::test_iter_slice_panic_map_idx_out_of_bounds

15. Nix-specific likely fix

  • Local just test-nix likely needs one of:

    • nix build --impure ...
    • or a different mechanism to inject SYMBOLICA_LICENSE into the flake check
  • Without that, the local Nix path fails before it can validate the actual nextest workload.

Prioritized Fix Order

  1. Done on this branch: fix linnet-py default-feature behavior so the workspace test build can start.
  2. Done on this branch: move Python API subprocess tests behind an explicit feature so just test-ci does not report unexecuted tests as passing.

  3. Fix the obvious non-snapshot runtime regressions:

    • REPL completion de-duplication
    • missing symbolica_state.bin contract mismatch
    • histogram rebin failure
    • muV missing in scalar vacuum evaluation
    • missing directory creation in save path

  4. Fix CI/test-environment issues that are not product regressions but currently break the matrix:

    • Python API tests without a built extension
    • missing example command-card files
    • stale run-card schema (enable_thresholds)
  5. Review the feyngen snapshot diffs and determine which are intended output updates versus real sign/canonization regressions.
  6. Fix the Nix-local invocation so SYMBOLICA_LICENSE is visible to the flake checks.
  7. Triage the remaining broad SIGABRT set in gammalooprs, idenso, spenso, and related crates after the first blockers are reduced.

Notes

  • The exclusion-based nextest run was intentionally non-CI and only used to see what is behind the first workspace blocker.
  • The initial version of this file was written before the long-running background jobs completed; the later sections record the final outcomes of those runs.

Local Validation Update

  • Additional local validation on 2026-03-25 used:

    • direnv exec . cargo nextest run --workspace --profile ci --locked --no-fail-fast --status-level all

  • Confirmed non-gammalooprs failures in that run:

    • gammaloop-api repl::tests::completion_does_not_repeat_used_options_by_short_or_long_form

      • panic at crates/gammaloop-api/src/repl.rs:4463
      • -p is still offered after integrate --process triangle

    • gammaloop-api state::tests::state_manifest_roundtrip_current_version

      • panic due to missing symbolica_state.bin
      • call path reaches crates/gammaloop-api/src/state.rs:833

    • gammaloop-integration-tests::test_evaluation_command evaluate_1l_scalar_vacuum

      • Could not find muV in graph
      • location: crates/gammalooprs/src/processes/amplitude.rs:744

    • gammaloop-integration-tests::test_feyngen example_graph_count

      • snapshot drift in color-factor ordering/sign placement

    • gammaloop-integration-tests::test_feyngen graph_count_from_amplitude_load

      • No such file or directory (os error 2)
      • location: crates/gammalooprs/src/utils/serde_utils.rs:41

    • gammaloop-integration-tests::test_feyngen from_symbolica

      • snapshot drift including renamed sign factors (ExtFerm/IntFerm -> expanded names)

    • gammaloop-integration-tests::test_feyngen test_generate_sm_a_ddx

      • snapshot drift in Nc vs Nc^(-1) ordering/sign placement

    • gammaloop-integration-tests::test_feyngen test_generate_aa_ttx_amplitude

      • snapshot generate_aa_ttx_amplitude-3 changed from 152 | 28 = 28 to 144 | 36 = 36

    • gammaloop-integration-tests::test_feyngen simple_epem_ddx_generation

      • long-running test aborts with stack overflow
      • same run also emitted not yet implemented at crates/gammalooprs/src/uv/approx/local_3d.rs:151
      • and InvalidIntegralFormat(...) at crates/gammalooprs/src/uv/approx/integrated.rs:581

  • Confirmed improvements relative to the earlier summary:

    • linnet-py compiles in the workspace run and no longer blocks the test matrix.
    • gammaloop-integration-tests::test_differential event_processing_runtime_merges_worker_results_and_batch_histograms passed.
    • gammaloop-integration-tests::test_clustering passed after the local C++ link fix.
Developer architecture · documented revision e51747446aa7