On this page

For developers

Three-Loop aa to aa Performance Smoke

Archived · partially reproduciblePerformance investigation · 2026-06-15archived

A 2026-06-15 generation and integration smoke record with retained cards, DOTs, scripts, and CSV summaries, explicit missing-state boundaries, and a warning for 400--570 GiB attempts.

View the source note

Lifecycle
archived
Owner
Ownership decision required
Review
No review record
Freshness
frozen-evidence
Evidence
captured 2026-06-15 · 4fdbf430b29edd24b9e1292c07ab24a151427dc7
Verified scopes
No verified code scopes recorded

Status: Archived, partially reproducible performance evidence

This record was captured on 2026-06-15 at evidence revision 4fdbf430b29edd24b9e1292c07ab24a151427dc7. The cards, runner, 339 ungrouped and 155 grouped DOT files, and compact CSV summaries remain. The generated saved states, workspaces, and /tmp logs named below do not. These are historical measurements, not current performance guarantees. The retained cards and runner were later schema-migrated in revision 21cefaeebe9de3f15be8efcf4f871aa6a1275954; use the evidence revision above for the exact capture inputs.

Resource warning: The recorded whole-amplitude attempts reached about 400–570 GiB RSS. Do not repeat those commands on a shared host without an explicit memory guard and resource authorization.

This file records the first whole-amplitude three-loop aa -> aa smoke test using the lightweight SingleParametric runtime-evaluation path. Compilation, summed evaluators, and summed function maps are disabled in the card.

Card and Artifacts

  • Card: examples/cli/aa_aa/3L/aa_aa.toml
  • DOT export: examples/cli/aa_aa/3L/graphs/processes/amplitudes/aa_aa/3L/GL000.dot through GL338.dot
  • DOT count: 339 graph files
  • Runtime evaluator: SingleParametric
  • Function maps/compilation: disabled (compile=false, summed=false, summed_function_map=false)
  • UV/threshold/tropical generation: disabled for this raw runtime smoke test
  • Kinematics/helicity: kinematics A, helicity +-+-
  • Final card integration smoke settings after this pass: n_start=20, n_max=20, --batch-size 1

Diagram Generation

The card uses feyngen with

generate amp a a > a a | a t t~ g ghG ghG~ QCD==4 QED==4 [{3}]
  --numerator-grouping no_grouping
  --symmetrize-initial-states=true --symmetrize-final-states=true
  -p aa_aa -i 3L --only-diagrams

Observed counts from /tmp/aa_aa_3l_full_sequence_nolimit.log:

StageGraphs
Symbolica generation6192
After vetoed topologies4284
After complete-graph filters1296
After closed-fermion-chain analysis1296
After external-state symmetrization339
After canonization339
After numerator-aware grouping (no_grouping)339

The 339 graphs contain 189 isomorphically unique graph shapes. The sum of generated graph symmetry factors reported by feyngen is -444.

A previous attempt with numerator-aware scalar-rescaling grouping reached the same canonical 339-graph stage but was not kept as the default because it was much heavier. The retained 3L card uses no_grouping to keep this first all-graph generation pass tractable.

Release-Build Scalar-Rescaling Grouping Attempt

A follow-up card was added at examples/cli/aa_aa/3L/aa_aa_grouped.toml to isolate the one-time feyngen pass with

--numerator-grouping group_identical_graphs_up_to_scalar_rescaling

The card saves DOTs to examples/cli/aa_aa/3L/graphs_grouped and would save the grouped state to examples/cli/aa_aa/3L/gammaloop_state_grouped if the feyngen grouping pass completed.

Command used with a freshly built release binary:

target/release/gammaloop --clean-state examples/cli/aa_aa/3L/aa_aa_grouped.toml run generate_diagrams

The release binary was built with:

nix develop --no-write-lock-file -c cargo build -p gammaloop-api --bin gammaloop --features ufo_support --release

Observed behavior from /tmp/aa_aa_3l_grouped_release.log and process monitoring:

QuantityValue
Symbolica generation6192 graphs
After vetoed topologies4284 graphs
After complete-graph filters1296 graphs
After closed-fermion-chain analysis1296 graphs
After external-state symmetrization339 graphs
After canonization339 graphs
Time spent after canonization before manual stopabout 80 min
Peak observed RSS during groupingabout 166 GiB
CPU usage during groupingabout 9-10 cores
Grouped DOT files produced0
Usable grouped saved state producedno

The run was interrupted after about 80 minutes because it remained inside the scalar-rescaling numerator-aware grouping stage with no progress log after the 339 canonized graph line. A tiny startup state scaffold was removed afterward to avoid confusing it with a usable grouped saved state. This confirms that using the release binary does not make the scalar-rescaling grouping pass tractable for the full 3L aa -> aa graph set as currently configured.

Integrand Generation

Command used for the no-cap run:

./gammaloop --clean-state -s /tmp/gammaloop_aa_aa_3l_perf_nolimit /tmp/aa_aa_3l_noauto.toml run generate_diagrams generate_integrands integrate_diagrams

Generation summary:

QuantityValue
Wall time, generate existingabout 29m 58s
Peak RAM reported by GammaLoop9.59 GiB
Generation cores10
Per-graph timing sum291.11 min
Mean graph generation time51.52 s
Median graph generation time43.30 s
Fastest graph119 ms
Slowest graph3.15 min
Graphs above 60 s112
Graphs at least 120 s25
Graphs below 1 s38

Timing share over the 339 per-graph timings:

PhaseSumShare
Expression build277.33 min95.27%
Spenso6.32 min2.17%
Symbolica eval7.46 min2.56%
Compile0 ms0.00%

Expression construction is the dominant generation cost. Spenso and Symbolica runtime evaluation are subdominant in this light SingleParametric setup.

Whole-Amplitude Integration Smoke

Two all-graph integration attempts were made.

  1. With a 100 GiB virtual-memory cap, integrand generation completed, but integration aborted before the first iteration with memory allocation of 16896 bytes failed. The process had reached the virtual-memory cap during integration setup and still held about 77 GiB RSS when stopped.
  2. Without the cap, the same setup reached actual integration over 3 nested discrete grids, 339 graphs, and 9 continuous dimensions using Monte Carlo over graphs, orientations, and LMBs. Integration-time memory rose well beyond the capped limit: about 51 GiB RSS at integration setup, then above 570 GiB RSS before any 1000-sample iteration completed. The run was stopped manually after roughly 7 minutes of sampling to avoid unnecessary pressure on the shared machine.

No completed integration iteration was obtained from the 1000-sample, --batch-size 100 smoke run, so this record contains no reliable MC uncertainty or ms/sample/core number for the full 339-graph amplitude. The capture-time performance conclusion was that all-graph raw 3L SingleParametric integration was memory dominated before it was statistics dominated. The retained card uses n_start=20, n_max=20, and --batch-size 1 for a smaller whole-amplitude smoke attempt.

Slowest Generation Graphs

RankGraphExpression buildSpensoSymbolica evalTotal
1GL2963m 0.9s3.95s4.08s3.15 min
2GL2993m 2.0s2.86s2.54s3.12 min
3GL1912m 30.2s2.63s2.88s2.60 min
4GL1692m 28.5s3.19s3.17s2.58 min
5GL1792m 30.1s2.18s2.49s2.58 min
6GL2132m 24.6s2.25s2.29s2.49 min
7GL1922m 21.0s2.40s2.49s2.43 min
8GL2162m 17.9s3.15s2.44s2.39 min
9GL2192m 18.6s2.30s2.24s2.39 min
10GL2262m 17.3s2.56s2.63s2.37 min
11GL2152m 16.7s2.49s2.46s2.36 min
12GL1782m 16.4s2.41s2.72s2.36 min
13GL1702m 15.1s2.27s2.78s2.34 min
14GL1862m 14.8s2.69s2.59s2.33 min
15GL2202m 15.2s2.46s2.33s2.33 min
16GL1872m 14.0s2.75s2.59s2.32 min
17GL2142m 12.3s2.75s2.31s2.29 min
18GL1982m 10.6s2.63s2.34s2.26 min
19GL1972m 8.2s2.84s2.26s2.22 min
20GL3092m 6.0s2.78s2.09s2.18 min

The complete 339-row timing matrix is preserved as aa_aa_generation_timings.csv rather than embedded in this page.

Follow-Up

  • Re-run the checked-in card with n_start=20 and --batch-size 1 to see whether the integration memory blow-up was primarily batch-size driven.
  • Use the saved DOT files to build one graph at a time and establish per-graph runtime/memory behavior before returning to the full all-graph Monte Carlo setup.
  • If the batch-size-one all-graph run still allocates hundreds of GiB before the first iteration, inspect the integration-state layout for graph/orientation/LMB Monte Carlo to identify what scales with all 339 graph evaluators.

Grouped Rescaling Follow-Up

This follow-up used the release binary target/release/gammaloop built with

nix develop --no-write-lock-file -c cargo build -p gammaloop-api --bin gammaloop --features ufo_support --release

The grouped card is examples/cli/aa_aa/3L/aa_aa_grouped.toml. The default symbolic scalar-rescaling comparison remained impractical, so the successful one-time feyngen pass used the same scalar-rescaling grouping mode but with a single fully numerical numerator-comparison sample:

--numerator-grouping group_identical_graphs_up_to_scalar_rescaling
--number-of-samples-for-numerator-comparisons 1
--fully-numerical-substitution-when-comparing-numerators=true
--compare-canonized-numerator=false

Grouped Diagram Generation

At capture time, the grouped feyngen pass completed and saved reusable DOT/state artifacts. Only the DOT export is retained in this checkout:

QuantityValue
Symbolica generation6192 graphs
After vetoed topologies4284 graphs
After complete-graph filters1296 graphs
After external-state symmetrization339 graphs
After canonization339 graphs
After numerator-aware grouping155 graphs
Isomorphically unique graphs163
Color zeros58
Grouped by scalar rescaling118
Cancellations8
Wall time for grouped feyngen stageabout 7 min
Peak observed RSS during numeric groupingabout 37 GiB
DOT exportexamples/cli/aa_aa/3L/graphs_grouped
Saved state at capture timeexamples/cli/aa_aa/3L/gammaloop_state_grouped (not retained)

The earlier release-build attempt with the default symbolic rescaling comparison remained useful as a negative control: it was stopped after about 80 min and about 166 GiB peak RSS without producing grouped DOTs/state.

Grouped Integrand Generation

Command:

target/release/gammaloop --override-state \
  -s examples/cli/aa_aa/3L/gammaloop_state_grouped \
  generate existing -p aa_aa -i 3L

Observed grouped integrand generation:

QuantityValue
Graph groups155
Wall time11m36s
Peak RAM reported by GammaLoop9.15 GiB
Generation cores10
Saved integrand payload153.15 MiB
Slowest generated graphsGL299, GL296, GL191, GL226, GL215

All-Grouped-Amplitude Smoke

The grouped state was then integrated with kinematics A, helicity +-+-, SingleParametric, Monte Carlo over graphs/orientations/LMBs, and inverse-Jacobian LMB channel weights. The 20-sample all-graph smoke reached the integration loop over 155 graphs and 9 continuous dimensions, but it was stopped before the first iteration completed because RSS kept increasing:

QuantityValue
Workspace/tmp/aa_aa_3l_grouped_overall_smoke_mc_workspace
Requested samples20
Batch size1
RSS after setup/sampling startabout 52 GiB
RSS when stoppedabout 400 GiB
Completed iterations0

At capture time, grouping reduced the previous 339-graph all-amplitude setup, but the combined grouped evaluator still aggregated too much runtime state to be useful in the tested all-at-once SingleParametric workflow.

Per-Diagram Grouped Smoke Scan

A separate single-DOT runner was added at examples/cli/aa_aa/3L/run_grouped_diagram_smokes.py. It imports one grouped DOT at a time, generates the SingleParametric integrand, and runs a 20-sample integration with kinematics A and helicity +-+-. The compact per-graph table is stored in examples/cli/aa_aa/3L/grouped_diagram_smokes_summary.csv; phase-probe runs are stored in examples/cli/aa_aa/3L/grouped_phase_probe_summary.csv.

All 155 grouped diagrams completed and emitted integration results.

QuantityMinMedianMeanMax
End-to-end wall time per graph15.79 s61.26 s66.58 s169.54 s
Peak RSS per graph2.88 GiB17.79 GiB18.11 GiB46.55 GiB
Avg total runtime per sample0.176 ms0.402 ms0.447 ms1.627 ms
Avg integrand runtime per sample0.102 ms0.267 ms0.314 ms1.417 ms
NAN or unstable samples0.0%0.0%0.065%5.0%

Top peak-RSS graphs:

GraphWall sRSS GiBAvg ms/sampleUnstable %
GL309149.646.60.6380.0
GL301163.746.51.6270.0
GL303140.046.50.5210.0
GL307135.346.50.4900.0
GL244126.735.91.6050.0

Top runtime-per-sample graphs:

GraphWall sRSS GiBAvg ms/sampleUnstable %
GL301163.746.51.6270.0
GL244126.735.91.6050.0
GL296169.525.31.4065.0
GL191137.324.51.2780.0
GL00032.99.51.2040.0

Only GL296 and GL299 reported nonzero instability in this 20-sample smoke scan, each with one unstable/NAN-class evaluation out of 20 samples. Both still emitted integration results.

Real vs Imaginary Training Probe

The user pointed out that the three-loop setup may need real-part rather than imaginary-part training. I ran separate high-stat single-graph probes using the same runner with explicit integrator.integrated_phase choices.

The first probes (GL026, GL001, GL047) were not reliable phase discriminators: at higher statistics, rare-weight variance dominated and neither real nor imaginary converged cleanly. A lower-max-weight graph, GL105, was pushed to 10M samples with real-part training:

ProbeGraphSamplesReRe errImIm err
real-trainedGL10510,000,000−5.31e-71.19e-66.84e-71.37e-6

This probe is compatible with zero in both components. I therefore did not restart the full per-diagram sweep with real-part training. The completed smoke scan should be read primarily as a runtime/memory/stability scan; its 20-sample central values are too noisy for a phase conclusion. Both real and imaginary columns are retained in the CSV summaries for follow-up.

Developer architecture · documented revision e51747446aa7