Skip to content

Correctness release 4.0.0: deterministic core, weights, frozen inductive updates, real-data benchmarks - #81

Open
jaroslawkrolewski wants to merge 37 commits into
masterfrom
fix/correctness-determinism-real-benchmarks
Open

jaroslawkrolewski wants to merge 37 commits into
masterfrom
fix/correctness-determinism-real-benchmarks

Conversation

@jaroslawkrolewski

@jaroslawkrolewski jaroslawkrolewski commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

A correctness pass over the Rust core and the Python layer, plus the tests and benchmarks that were missing. The public API keeps its names and signatures; behaviour changes are listed below. Proposed as 4.0.0 because embeddings differ from 3.x (the initial vectors changed) and Python 3.7–3.9 are dropped.

Everything claimed here is covered by a test or by a script in the repository.

What was wrong, and what this changes

Problem in 3.2.1 Evidence Now
Initial vectors nearly collinear: one multiplicative hash step per value pairwise cosine std 0.45 at 256 dims (independent vectors: 0.06), effective rank ≈ 5 for any dimension independent uniform values from SplitMix64(entity hash, column, seed); statistics tested for 64/256/1024 dims
Graph construction depended on thread scheduling (float summation order; which entities survive hyperedge trimming) different matrices from the same input with num_workers > 1 exact fixed-point accumulation, trimming by global occurrence with a fixed tie-break, entities numbered by first appearance: bit-identical for any num_workers
embed_weighted ignored its weights weighted vs unweighted result differ by 8e-6 (float noise) weights are part of the matrix (weights= / weight:: column); integer weight k ≡ repeating the line k times
embed_inductive / embed_streaming moved all existing vectors median cosine old-vs-new vector 0.04; classifier trained before the update: 0.82 → 0.27 default mode="frozen": old vectors untouched, new entities at the weighted average of their neighbours; refresh_every for periodic aligned re-embedding
Bad input accepted silently missing file → empty graph; malformed lines dropped; wrong separator → empty graph FileNotFoundError, RuntimeWarning naming the first bad line, ValueError when nothing is usable
Fortran-ordered / strided arrays crash left_markov_propagate Rust panic (PanicException) any layout accepted, readable errors for wrong shape/dtype
pycleora.datasets served synthetic graphs under real names (cora, citeseer, pubmed, ppi, …) the README benchmark table was computed on them real graphs from their original sources, verified against published sizes; synthetic ones only via load_synthetic()
ProNE and RandNE at chance level Cora, same protocol: 0.30 and 0.32 (largest class: 0.30) baselines rewritten after the papers, on the adjacency matrix instead of Cleora's normalised matrix: 0.79 and 0.71; a test checks every baseline recovers planted communities
Whitening with no more entities than dimensions all vectors equidistant (cosine −1/(N−1)): the README quick start returned the same similarity for every entity RuntimeWarning; README examples fixed
Wheels declared no dependencies import pycleora needs NumPy (and needed SciPy) NumPy declared; SciPy optional and imported lazily (a test runs the core API with SciPy blocked)
get_entity_column_mask all False for a reflexive column — fixed
benchmark memory via tracemalloc does not see Rust allocations peak RSS on Linux

Performance (2-core VM, 100k nodes / 500k edges, 256 dims; 3.2.1 wheel from PyPI vs this branch)

3.2.1 this branch
embed() default, 40 iterations 32.0 s, 6 matrices of working memory 16.5 s, 2 matrices
classic (whiten=False), 40 iterations 6.1 s 3.7 s
from_iterator 0.66 s 0.34 s (from_edges on int arrays: 0.25 s)
one propagation step 148 ms 107 ms
find_most_similar 129 ms 34 ms
1000 × graph.entity_ids[i] 7.5 s (list rebuilt per access) 6 ms (cached tuple)

Benchmarks

benchmarks/run_benchmarks.py reproduces the tables now in the README: 8 public graphs, 9 methods, logistic regression on the embeddings, several random splits, time and memory measured in fresh processes.

Accuracy / micro-F1 at 256 dimensions (full tables with time and memory are in the README):

Dataset Cleora default Cleora classic (8 it.) best other method
Cora 0.800 0.812 0.813 (DeepWalk)
CiteSeer 0.648 0.617 0.625 (NetMF)
PubMed 0.818 0.730 0.827 (DeepWalk)
email-Eu-core 0.630 0.753 0.759 (RandNE)
Amazon Photo 0.933 0.932 0.929 (DeepWalk)
Amazon Computers 0.908 0.899 0.905 (DeepWalk)
PPI (micro-F1) 0.216 0.234 0.240 (GraRep)
Wikipedia POS (micro-F1) 0.472 0.477 0.538 (HOPE)
ogbn-arxiv (1 split) 0.691 0.548 0.646 (ProNE)
  • One of the two Cleora modes is best, or within one standard deviation of the best, on 5 of the 8 graphs; it is clearly behind on Wikipedia POS.
  • Classic Cleora is the fastest and smallest method everywhere (0.01–0.13 s and 2–38 MB on these graphs; 45–130× faster than ProNE). roadNet-CA (1.97M nodes) takes 8 s with classic mode and 276 s with the default, on 2 cores, in 3.8 GB.
  • Neither Cleora mode dominates the other, which is why the default was left alone.

The previous README table ("#1 on every dataset", perfect accuracy on PPI) could not be reproduced on the real datasets and has been replaced by measured numbers. The same claims on cleora.ai/benchmarks are outside this repository and still need updating.

Behaviour changes to be aware of

  • Embeddings differ from 3.x (new initial vectors). Recompute stored embeddings; do not mix.
  • SparseMatrix.entity_ids is a tuple (was a new list on every access); entities are ordered by first appearance.
  • embed_inductive / embed_streaming default to frozen updates (mode="retrain" restores the old behaviour).
  • Unusable input raises or warns instead of producing an empty graph.
  • Datasets: real data under the real names; flickr, yelp, dblp, reddit raise until someone adds a loader.
  • Python ≥ 3.10, abi3 wheels (PyO3 0.30), maturin ≥ 1.9.4. Pickles written by 3.2 still load (fixture in the tests).
  • whiten=True stays the default. On the benchmark graphs neither mode dominates the other; the README now shows both.

Tests

  • cargo test: 23 unit tests + the 4 snapshot tests (snapshots regenerated: 7 of 12,800 values moved by one unit in the last printed digit, because weights are now accumulated exactly).
  • pytest: 164 tests (6 more download datasets and are opt-in via PYCLEORA_TEST_DOWNLOADS=1). They compare graph construction with a brute-force reference (trimming, weights, multiple relations), check bit-identical results across worker counts, the init formula, propagation against dense algebra, whitening against a NumPy reference, frozen updates, link prediction and similarity search against brute force, legacy pickles, and that the core works without SciPy.
  • CI on this PR: the new test job passes on Python 3.10 and 3.13. The wheels build and pass the smoke test on Linux x86_64, Windows x64, macOS x86_64 and macOS arm64; they also build for Linux aarch64 and Windows x86, where they are not executed.

Not verified — please look at these

  • CI: the workflow gains a test job, smoke tests of the built wheels and a macOS x86_64 runner, and drops --find-interpreter (abi3). Two things had to follow what GitHub offers today, independently of this change: the macos-14 runner was retired in November 2026 (now macos-latest), and the 32-bit Linux wheel can no longer be built, because the manylinux2014_i686 image (CentOS 7) cannot reach its package mirror. That job is non-blocking now; whether to keep an i686 Linux wheel at all is your call.
  • scripts/post-merge.sh (Replit): adapted to the abi3 file name (pycleora.abi3.so) and to the maturin version PyO3 0.30 needs; untested.
  • Version: bumped to 4.0.0 in Cargo.toml (now the single source; pyproject.toml reads it). No tag was created — tags trigger the PyPI upload.
  • SciPy is an optional extra here. If you prefer it mandatory (as the old setup.py had it), it is a one-line change in pyproject.toml.
  • COMPARISON.md was only updated where this change affects it (datasets, dependencies, install size); the rest still describes the state of pycleora 2.3.

Not done

  • Out-of-core / memory-mapped embedding matrices and half-precision or int8 output.
  • A time-decay helper for edge weights (weights themselves are in).
  • Using the symmetry of the covariance matrix in the whitening step (worth roughly another 1.5× on the default path).

Generated by Claude Code

jaroslawkrolewski and others added 30 commits November 18, 2026 17:46
…e updates

Rust core
- Initial vectors are independent uniform values (SplitMix64 of the entity
  hash, the column and the seed). The previous single multiplicative hash
  step produced nearly collinear vectors: effective rank ~5 for any
  dimension, pairwise cosine std 0.45 at 256 dimensions instead of 0.06.
- Graph construction is exact and independent of the number of threads:
  fixed-point accumulation, hyperedge trimming by global occurrence with a
  fixed tie-break, entities numbered by first appearance in the input.
- Edge / hyperedge weights (weights= argument or a weight:: column).
- from_edges(): build a graph from arrays of integer ids or names.
- CSR storage with one weight per entry, normalised on the fly.
- Missing files raise, malformed lines warn (naming the first one), input
  without a usable line is an error. Any file extension is accepted.
- The propagation methods accept any array layout (Fortran-ordered and
  strided input used to panic) and take out= and rows=.
- Covariance and projection kernels for whitening, deterministic and in
  place; embed_fast() runs classic Cleora without leaving Rust.
- PyO3 0.30, abi3 wheels for Python >= 3.10. The version is defined in
  Cargo.toml only and exposed as pycleora.__version__.
- get_entity_column_mask() works for reflexive columns.
- Pickles written by 3.2 still load.

Python layer
- embed(): in-place loop over two buffers; whiten_embeddings() runs on the
  Rust kernels with a fixed sign convention. A RuntimeWarning is raised
  when whitening cannot work (no more entities than dimensions + 1: all
  vectors end up equidistant).
- embed_weighted() uses its weights; they cancelled in the normalisation.
- embed_inductive() / embed_streaming() keep existing vectors where they
  are and place new entities next to their neighbours (mode="retrain"
  gives the previous behaviour).
- io_utils: from_arrays(); weights in from_pandas, from_edge_list,
  from_networkx, from_scipy_sparse and from_numpy.
- entity_ids is a cached tuple instead of a list rebuilt on every access.
- NumPy is declared as a dependency; SciPy is optional and imported only
  by the functions that need it.

Embeddings differ from 3.x because the initial vectors changed.
The snapshot tests were regenerated: 7 of 12,800 values moved by one unit
in the last printed digit, since weights are now accumulated exactly.
- pycleora.datasets downloads the real graphs (Planetoid, SNAP, OGB,
  node2vec, GraphSAGE, Newman) instead of generating synthetic graphs
  under their names. Every dataset reports "synthetic" and "source".
  Names without a loader yet (flickr, yelp, dblp, reddit) raise an
  explanatory error; the old generators are available, labelled as
  synthetic, through load_synthetic().
- pycleora.algorithms is rewritten after the papers and works on the
  adjacency matrix instead of Cleora's row-normalised matrix. ProNE and
  RandNE were at chance level on real graphs (Cora: 0.30 and 0.32
  accuracy, now 0.79 and 0.71). DeepWalk and node2vec are documented as
  the PPMI + SVD variants they are.
- pycleora.benchmark reports peak resident memory on Linux; tracemalloc
  does not see the allocations of the Rust core.
- CLI: `embed --no-whiten`; `benchmark` adds classic Cleora, adapts the
  dimension to small graphs and explains why multi-label datasets are
  not supported there.
tests/python covers what used to be untested:
- graph construction against a brute-force reference (hyperedge
  trimming, weights, several relations, all column layouts),
- bit-identical matrices and embeddings for any number of workers,
- the formula and the statistics of the initial vectors,
- propagation against dense algebra, for every array layout,
- whitening and embed() against NumPy reference implementations,
- frozen inductive updates, streaming, alignment,
- similarity search and link prediction against brute force,
- input errors and warnings, pickles (including one written by 3.2.1),
- datasets, baseline algorithms, packaging metadata, and that the core
  API works without SciPy.

Tests that download datasets are opt-in (PYCLEORA_TEST_DOWNLOADS=1).
tests/wheel_smoke.py is a minimal check for a freshly built wheel.
…ARISON

benchmarks/run_benchmarks.py embeds eight public graphs with nine
methods and evaluates node classification with logistic regression over
several random splits. Time and memory are measured in fresh processes.

The README benchmark section now shows the numbers this script produces.
The previous table was computed on synthetic graphs that carried the
names of real datasets and could not be reproduced on the real ones.

README: installation and dependencies, classic mode, weights, inductive
updates, determinism, algorithm descriptions and FAQ follow what the
code does; the examples no longer whiten graphs that are too small.
- New job: cargo test, build, pytest on Python 3.10 and 3.13.
- Wheels are abi3 now (one per platform), so --find-interpreter goes.
- The wheels that can run on their build machine are smoke-tested.
- macOS x86_64 wheels on the macos-15-intel runner.
- A release needs the test job to pass.
The macos-14 runner image was retired in November 2026, so the arm64
macOS job was cancelled before it started. Wheel builds of one platform
no longer cancel the others when one of them fails.
manylinux2014_i686 is based on CentOS 7, whose package mirror no longer
resolves, so the libatomic install that the i686 Rust toolchain needs
fails before the build starts. The job keeps running but its failure no
longer fails the workflow or blocks a release.
…rmalisation

The propagation kernel handed rayon equal *row* ranges. Entities are numbered by
first appearance, so the busiest ones sit together at the start of the matrix, and
rayon's adaptive splitter left that first range to a single thread: on a power-law
graph the whole product waited for one core.

* Tasks now hold about the same number of matrix entries (TASK_NNZ) and each can
  be stolen on its own.
* A row with more than HUB_NNZ entries is summed in parallel in fixed-size
  segments that are combined in order, so the result is still independent of the
  number of threads. Rows below that threshold are bit-identical to before (the
  stored snapshots are unchanged).
* Four source rows are added per pass over the destination row, with a prefetch of
  the rows needed next; every element still receives its terms in entry order.
* The embedding loop normalises (and mixes the residual into) every row while it
  is still in the cache instead of in separate passes over the matrix.
* propagate() and propagate_selected() share one scheduler.

R-MAT graph, 573k entities, 20.6M entries, 1024 dimensions, 24 threads (quiet
machine, best of 3): one propagation 3.39 s -> 1.22 s; four rounds of
propagation + normalisation 14.0 s -> 5.0 s.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…earn

pycleora.metrics had no tests and several of its numbers were not what their
names say:

* average_precision was the share of positives above the median negative;
* clustering_scores assigned every vector to the nearest centroid of the *true*
  classes and compared that with the true classes, so vectors without any
  structure scored well; its 'purity' was a best-permutation accuracy that
  enumerated k! permutations;
* map_at_k / ndcg_at_k looped over all entities in Python for every query;
* silhouette_score built an n x n distance matrix;
* AUC was a trapezoid over individually sorted scores, wrong with ties.

The module now has one definition of every metric, on arrays (labels and scores
in, numbers out), used by the convenience functions and by the benchmark:
classification_metrics (accuracy, balanced accuracy, macro / weighted F1, MCC,
Cohen's kappa, top-k accuracy, one-vs-rest ROC-AUC and average precision),
multilabel_metrics (micro / macro / samples F1 at the model's threshold and with
the known label count of the DeepWalk protocol, subset accuracy, Hamming loss,
ROC-AUC, average precision, label ranking average precision, precision@1),
ranking_metrics (MRR, mean / median rank, Hits@k), retrieval_metrics (Recall,
Precision, hit rate, NDCG, MAP at k), roc_auc, average_precision,
normalized_mutual_info, adjusted_rand_index, purity.

The convenience functions keep their names, arguments and result keys.
clustering_scores now runs spherical k-means and additionally reports 'ari';
link_prediction_scores no longer samples graph edges or self-pairs as negatives.

Every metric is checked against scikit-learn (or brute force where scikit-learn
has no equivalent), including ties and degenerate inputs. scikit-learn joins
the 'test' extra.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Three pieces of the internal library's interface that the public one lacked:

* SparseMatrix.from_cliques(cliques, hyperedge_trim_n, num_workers, weights):
  hyperedges given as sequences of entity names instead of text lines, so names
  may contain any character. Same matrix as the equivalent lines of a
  'complex::reflexive' column.
* SparseMatrix.isolated_entities: indices of the entities without a neighbour
  other than themselves.
* SparseMatrix.left_markov_propagate_normalized_inplace(x, num_iterations) and
  its symmetric twin: repeated propagation + row normalisation of a caller's
  array in place, with one matrix of working memory for any number of rounds.
  Unlike the internal version, a row of zeros stays zero instead of becoming NaN.

embedding::embed() and the in-place methods share one loop (embedding::iterate),
which leaves its result in the caller's buffer for odd and even round counts.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…dist

* The 'linux (x86)' job could not pass any more: maturin-action installs
  libatomic with yum in the manylinux2014_i686 image, whose package mirror no
  longer resolves. It had been marked continue-on-error, so a release would
  silently have shipped without that wheel. The target is removed instead:
  NumPy has not published a 32-bit manylinux wheel since 1.22, so a pycleora
  wheel for it could not be installed anyway.
* New lint job: cargo fmt --check and cargo clippy --all-targets -D warnings
  (the existing warnings are fixed).
* The test suite also runs on Linux aarch64 and on macOS arm64; the aarch64
  wheel is built natively and smoke-tested like the others.
* The sdist job installs the sdist and runs the smoke test, so a source
  distribution that cannot be built is caught.
* Cargo.toml excludes repository-only files (images, benchmarks, the legacy
  CLI, the Replit scripts, the workflow) from the sdist: 1.6 MB -> 0.2 MB.
* Runs of a pull request that are superseded by a new push are cancelled.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The wheels are built against the stable ABI, so one wheel per platform is meant
to load on every CPython from 3.10 on. CI only ran the tests on 3.10 and 3.13.
The matrix now covers 3.10, 3.11, 3.12, 3.13, 3.14 and 3.15 on Linux x86_64 and
the oldest and newest of them on Linux aarch64 and on Apple silicon, and the
package metadata lists the versions.

Checked beforehand with one locally built abi3 wheel installed into a fresh
environment of each version: 187 passed, 6 skipped on all six.

The PyPy classifier is dropped: PyPy cannot load abi3 wheels.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Twelve signatures spelled the default of `whiten` as a literal and the
hetero module repeated 256 and 40. They now refer to DEFAULT_WHITEN,
DEFAULT_FEATURE_DIM and DEFAULT_NUM_ITERATIONS. No change in behaviour.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
mixing = steps(purpose) / ((1 - s) * sqrt(mean neighbours))   (x 1.5 symmetric)
forget = ln(1/256) / ln(s)                                     (0 when s = 0)
rounds = max(mixing, forget, 2), at most 64

s is the share of the weight on the diagonal; steps is 12 for similarity,
16 in general and 24 for classification.

Every round is one step of a random walk. Denser graphs mix faster (the
square root). A matrix with weight on its diagonal - a reflexive column -
is a lazy walk: it moves with probability 1 - s, and keeps a share s of the
entity's own random initial vector in every round, so it needs at least
"forget" rounds however dense the graph (8 for two-entity hyperedges).

Form and constants were fitted leave-one-graph-out on iteration sweeps over
graphs with 3 to 500 neighbours per node, both ways of feeding a homogeneous
graph and both normalisations; the benchmark documentation has the analysis.

SparseMatrix gains mean_neighbours and self_weight_share (Rust) and
suggest_iterations(propagation, purpose) (Python).

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
num_iterations="auto" asks SparseMatrix.suggest_iterations; whiten=False.

Why: the 3.2 default (40 iterations, PCA whitening after each) needs many
more entities than dimensions and loses to plain Cleora on small graphs at
any dimension, e.g. node classification accuracy at 256 / 1024 dimensions
with 10% of the labels: CiteSeer 0.48 / 0.37 against 0.57, Cora 0.45 at 1024
against 0.79 - while costing 10 to 100 times the time (ogbn-arxiv, 1024
dimensions: 104 s against 1-4 s). embed(graph, num_iterations=40,
whiten=True) still gives the old behaviour, and with whitening "auto" keeps
meaning 40.

- _resolve_iterations is the one place that turns "auto" into a number;
  every function with its own loop calls it.
- With initial vectors and nothing to observe between iterations, embed()
  runs all of them in the core (normalisation fused into the propagation).
- CLI: -n accepts auto (the default); --whiten replaces --no-whiten, which
  had not been released.
- Tests for the defaults, the in-core path with initial vectors and the
  embed command.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The table of rayon pools is inherited by a forked child, the worker threads
behind it are not. Every call in the child (building a graph, propagating,
embedding) then waited forever for the parent's threads. The table now
records the process that filled it and starts afresh in another one; the
parent's pools are leaked in the child rather than dropped, since their
threads are not there to be stopped.

Test: a forked child builds a graph, embeds and propagates after the parent
did; it failed (no answer) before the change.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…rguments

- find_most_similar(exclude_self=True) returned the query itself, with
  similarity -1, when top_k was at least the number of entities. The query
  is now excluded from the candidates, not just given a low score.
- predict_links returned every pair in both directions: top_k=10 gave five
  distinct pairs. A pair comes back once.
- embed_directed gave every entity without outgoing edges a zero vector, and
  the zeros spread upstream by one step per iteration (a chain of five was
  all zero after five iterations). A sink now keeps its own vector.
- embed(residual_weight=...) silently ignored values outside [0, 1); they
  are an error now.
- embed_with_node_features takes seed and whiten like its siblings.

Tests that pinned the old behaviour (the query among its own neighbours,
mirrored pairs, the zero row of a sink) assert the corrected one.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
mlp_classify and gcn_classify scored the test entities every tenth epoch,
kept the weights with the best test accuracy and reported that maximum.
A fifth of the training entities is now held out to choose the checkpoint,
and the test entities are scored once. Flipping the test labels turns an
accuracy a into 1 - a, which the new test checks; before, the two summed to
1.03-1.07 on the test graph.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The published POS.mat is a weighted co-occurrence matrix (values from 0.5 to
1941.5, two thirds of them not 1); the loader returned the edges only.
load_wikipedia_pos now returns "weights" aligned with "edges".

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…figures

- pycleora embed --algorithm node2vec ran with p = q = 1, which is DeepWalk
  bit for bit. --p and --q (defaults 1 and 0.5) are passed on, and the
  benchmark command's node2vec row uses the same values.
- pycleora.benchmark reported in-process "memory" figures that were
  meaningless (0.00 MB for an embedding that allocates megabytes). The
  measurement and its column are removed; a failing algorithm is reported
  in its row with the type and message of the error.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The script it replaces ran pycleora's own re-implementations of the
baselines at settings their authors do not use, evaluated one probe with one
metric on node classification only, picked classic Cleora's 8 iterations
without validation, and timed Cleora as the best of three runs against one
run of everything else.

benchmarks/cleora_bench:
  datasets        26 public graphs from their original sources, one canonical
                  form on disk, sizes checked when prepared
  methods         Cleora and nine baselines behind one signature; the baselines
                  are the authors' code or a standard library (PecanPy + gensim,
                  ProNE, NetMF, FastRP, RandNE, karateclub) at pinned versions
  embed_worker    one method on one graph in a fresh process: time, peak memory
  tasks           the protocol: node classification (10% labels or the official
                  split) and link ranking (edges for the embedding, for the
                  probes, for validation and for test are disjoint; 500 shared
                  negatives per positive; retrieval among all nodes)
  probes          logistic regression, linear SVM, MLP and k-NN on nodes; dot
                  product, cosine, bilinear, MLP and linear models on pairs.
                  Every hyper-parameter, feature scaling included, is chosen on
                  validation data
  runner          resumable jobs, one JSON file each; failures are recorded
  configurations  what a table row is: Cleora fed as one reflexive column or as
                  two columns, left or symmetric, with the library's iteration
                  count or a fixed one
  selection       "tuned" iteration counts, chosen on validation from a sweep
  iteration_fit   leave-one-graph-out comparison of iteration rules
  report, display aggregation, variant selection on validation, ranks, tables

Scripts: setup.sh (three pinned environments, four pinned checkouts),
prepare_datasets.py, run_benchmark.py, select_iterations.py, make_report.py,
make_figures.py (UMAP), kernel_speed.py, and replicate.sh, which runs all of
it and can skip any part.

The logic that decides what the tables say (variant selection, ranks, tuned
iterations, regret) is unit-tested on fabricated job files.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
- Two processes that draw the same link split at once no longer race for
  one temporary file.
- The five largest graphs get one hour per run instead of six (replicate.sh).
- Ranks and wins: a graph on which only one method has a result counts for
  nobody.
- "n/a (size)" instead of "n/a (dense)": Walklets is limited by its
  Python-level graph, not by a dense matrix.
- pycleora benchmark: the hint for multi-label datasets pointed to the
  script this suite replaced.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The wrapper ran DeepWalk's walks (p = q = 1) through SparseOTF, which
computes second-order transition probabilities for every step - O(degree)
work that first-order walks do not need, and that PecanPy itself warns
about. On dense graphs it made DeepWalk very slow (it exceeded the time
limit on every one of the five largest graphs, at both dimensionalities).

DeepWalk now uses FirstOrderUnweighted, or PreCompFirstOrder when the edges
are weighted. The walks are sampled from the same distribution; all DeepWalk
results are recomputed with this engine. node2vec keeps SparseOTF.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
… dimensions

- A setup from scratch on a second machine showed that the FastRP reference
  code imports gensim, which the pinned requirements of the main environment
  did not list (the first machine had it by accident).
- On the five largest graphs the baselines run at the dimensionality of their
  papers only, with three hours per run; the figures fall back to those
  vectors there.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The baselines were reported at the better of 128 and 1024 dimensions on
validation, and Cleora at the best of its four families: every column chose
among several results. Now every column of the main tables is a single
configuration at 1024 dimensions, the same for every graph.

- Baselines: their authors' settings at 1024 dimensions.
- Cleora, one column and Cleora, two columns: the iteration count the library
  suggests for the task, left normalisation for classification and symmetric
  for link ranking - fixed for all graphs by the mean validation score over
  the graphs (new table families_validation). The two columns are the same
  method fed in two ways and are not ranked against each other.
- What choosing iterations and family on validation adds moves out of the
  main tables into tables of its own (tuned_nc, tuned_lp).
- The 3.2 default at 1024 dimensions only; no 128-dimensional runs.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
README: what the library does and the three choices that matter (how the
graph is given, the normalisation, the number of iterations), with examples
that are executed by the test suite; the benchmark tables, generated from
the result files by benchmarks/make_report.py; UMAP figures; measured kernel
speed against 3.2.1 on x86-64 and Apple silicon; and a list, also generated,
of every graph on which Cleora is not the best method.

benchmarks/README.md: datasets, methods and their exact settings, splits,
probes, metrics, the one-configuration-per-method rule, how the iteration
rule was derived and what it costs, the timing protocol, replication.

benchmarks/RESULTS.md: all 54 tables. benchmarks/results.csv.gz: every
aggregated number. benchmarks/figures: one UMAP figure per dataset.

In short, at 1024 dimensions, one configuration per method for all graphs:
link ranking with dot product or cosine - Cleora is the best method on 14 of
26 graphs, mean rank 1.58 against 3.27 for the best baseline; node
classification with logistic regression - best on 10 of 17, mean rank 2.00
against 3.35 (ahead on single-label graphs, behind on the four multi-label
ones); embedding time 75 ms to 10 s where the baselines need 0.7 s to 100
minutes.

CHANGELOG: new entries for this work; numbers that came from the removed
benchmark script are gone, the speed table is measured with
benchmarks/kernel_speed.py. COMPARISON.md: the row of unmeasured embedding
times for 17 libraries is removed.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The last runs are in: Coauthor Physics (GraRep), the link runs of
ogbn-products and LiveJournal, the tuned configurations of Reddit.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
One generated sentence under the headline table: with the number of
iterations and the normalisation chosen on validation data, on how many
graphs Cleora is ahead of every baseline. It is marked as outside the
comparison, because the baselines run at their authors' settings.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Removed: linear SVM and k-NN (node classification), the bilinear form, the
MLP over concatenated vectors and the linear SVM on x*y (link ranking).
Kept: logistic regression and an MLP on node vectors - embeddings as
features of a downstream model; dot product and cosine - what a
nearest-neighbour index computes; logistic regression on x*y - a dot
product with one learned weight per dimension.

The kept probes are unchanged. Run again on another machine (Cora and
CiteSeer; Cleora and RandNE), they give the stored dot product and cosine
MRRs exactly and the stored scores of the trained probes to within 0.001.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
- Retrieval among all nodes is measured on the official OGB link splits too
  (ogbl-collab, ogbl-ddi, ogbl-ppa): every distinct official test pair is
  queried from both of its ends. Until now these three graphs had the
  sampled-candidate protocol only.
- The report is organised by the three uses of the vectors (display.SCENARIOS):
  similarity search among all nodes, node classification, link ranking
  against sampled candidates. Headline, list of losses and rank tables are
  built from the same rows; the retrieval tables give every method the better
  of dot product and cosine on validation, as the link tables do.
- "Cleora, tuned" is reported for the two scores the sweep validates.
- Dead code removed (report.HEADLINE, two unused imports, one unused local).

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
randusr and others added 7 commits November 19, 2026 13:43
The link task of ogbl-collab, ogbl-ddi and ogbl-ppa was run again for every
method (retrieval among all nodes); all other numbers are the stored ones.
The methodology says which probes were removed and how Cleora ranked on them.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
SparseMatrix::entry_factor / row_factor say how an entry of the left or the
symmetric Markov matrix is put together from the stored weights; the kernel
used to spell that out itself. split_tasks cuts a buffer into the pieces of
the balanced tasks. No behaviour change: preparation for a second kernel that
needs both.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
embed() now runs its rounds - propagate, normalise rows - on the embedding
matrix kept as blocks of 16 columns, one cache line per entity and block,
and handles one block after the other. The row-major kernel reads a whole
source row per matrix entry (4 KB at 1024 dimensions) from all over a matrix
far larger than the caches, and runs at the speed of the memory bus; a block
is a slab that fits the caches or nearly does, and only the 8 bytes of an
entry are streamed per block.

The result is bit-identical to the row-major loop (same operations in the
same order per coordinate; the normalisation factor of a row is applied when
its lines are read), for any number of threads; a test pins it on a matrix
with hub rows, an empty row and dimensions that are no multiple of 16.

Per round, R-MAT graph with 0.6M nodes and 12M edges, 1024 dimensions:
  Intel Core Ultra 9 275HX  24 threads  1.50 s -> 0.61 s
                             8 threads  1.63 s -> 0.88 s
                             1 thread   5.38 s -> 5.01 s
  2.1M nodes, 40M edges     24 threads  6.17 s -> 4.06 s
  230k nodes, 30M edges     24 threads  2.16 s -> 0.66 s
Memory is unchanged: two matrices.

The initial vectors are written straight into the blocks (Initial), so
embedding::embed takes the seed instead of a matrix. Loops with a residual
or a convergence test, and the in-place propagation of a caller's array,
stay on the row-major kernel.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
embed() returns unit vectors: their dot product is the cosine and says how
alike two entities are. Which links exist depends on more than that - a well
connected entity is in more of them - and methods whose vectors are not
normalised carry that in the length of their vectors.

scale_by_popularity(graph, embeddings, exponent) multiplies every vector by
degree ** exponent, so that the dot product is the cosine times
(degree_i * degree_j) ** exponent. The default exponent, 1/16, is the one
with the best mean validation score over 21 graphs of the benchmark (next
commits); 14 of them find held-out links better with the prior than without.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
- The similarity configurations of Cleora (automatic and fixed iteration
  counts) apply scale_by_popularity at the library's default exponent. The
  rule that holds for every method - dot product or cosine, whichever
  validates better on the graph - then decides between the prior and none.
  Classification configurations are unchanged.
- The iteration count chosen from the sweep for the link task is run as the
  similarity configuration (configurations.for_purpose); the sweep itself
  validates by cosine and is not affected.
- popularity_exponent.py measures candidate exponents on validation edges
  (never test edges); the report shows the table, replicate.sh runs it on the
  graphs of tiers S and M.
- tasks.validation_scores; runner.link_split and runner.embed are public
  names now that a second script uses them.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The in-place propagation of an existing array reads whole rows; embed() works
through the matrix in blocks of columns. Printing both shows what the blocks
gain on a given graph and machine.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Cleora's link task was run again on all 26 graphs (the four families and the
tuned configurations, now with the popularity prior) and its embedding times
were measured again on the idle machine with the new kernel; every baseline
number is the stored one.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants