For contributors

The two orderings a net carries

rcsr_index is where a net is shown; complexity_rank is when it was ingested. They are different columns and conflating them is the known failure mode.

Every ingested net carries two numbers that both look like "position in a list", and they answer two unrelated questions:

• rcsr_index — where does this net belong in the RCSR symbol ordering a reader expects to browse? A presentation order.
• complexity_rank — in what order were the nets processed at ingest, simplest first? A processing order. A listing will sort by it on request, but it is never a claim about the net itself.

They are not correlated and they are not interchangeable. pcu carries complexity_rank 1 — one vertex, three edges, the simplest thing in the set — and rcsr_index 1536, because p is a long way into the alphabet. aab is the mirror image: rcsr_index 1, complexity_rank 2131. A listing can be asked to sort by either (see below); what neither number will survive is being read as the other. Reporting a complexity_rank as a net's position in the RCSR index, or an rcsr_index as a statement about how complicated a net is, is the mistake this page exists to stop.

Both numbers are now on the pcu row. They were not always: pcu is one of the five seeded demonstration rows the loader will not overwrite, and it used to be skipped entirely, which left it with NULL in both columns. It is now enriched instead — the ingest writes the two orderings and their provenance onto a pre-existing row and touches nothing else — so a page describing pcu as an example of a blank index is describing a state that no longer exists.

The licence position, which is now split

The dominant fact about this collection, stated before anything below depends on it. There are 3077 net rows. 2443 are published under CC BY 4.0 and 634 are held, unpublished, and answer 403. The split is not arbitrary and it is not about the nets — it is about who owns the source the values came from:

A record at least one of whose values came from MOF+ is the project's own data. MOF+ is this group's own service, so those nets are ours to license, and the project licenses its data CC BY 4.0. That is a decision recorded in CONTEXT §6, not a grant discovered on disk, and the row says so in its own licence_source:

MOF++ project decision, CONTEXT §6 (Licensing): MOF+ is this group's own service, so nets retrieved from it are the project's own data to license, and the project licenses its data CC BY 4.0. No third-party grant is claimed.

The last sentence is load-bearing. The MOF+ reader's own finding — that no machine-readable licence grant is stated anywhere in the client or the retrieval cache — is still true and is still carried in the ingest report beside the decision. The decision answers a different question from the one that note asks.

A record sourced only from the RCSR archive is somebody else's data and stays held. rcsr.anu.edu.au is a third party, the archive file states no terms, and nobody has confirmed those terms in writing. There is therefore no licence to record: licence holds the constant unverified (deliberately not a licence identifier), licence_source is NULL, published is false. The project's own CC BY 4.0 decision covers the project's own data and says nothing about somebody else's archive.

This is an open blocker, and it is the only one left. What resolves it is one thing and one thing only: a written confirmation of the RCSR archive's terms — from rcsr.anu.edu.au, from the archive's maintainers, or from an upstream statement that can be cited. Nothing in the software will decide it, no default will be applied, and no index, key or checksum will be re-licensed by inference. When it arrives, the fix is a licence and a licence_source on 634 rows and mofpp apply-licence; the identifiers are already minted and stable, so every held URI resolves to a full record the moment it lands.

A held record is not hidden and not a 404. It is absent from every listing, and its own URI answers 403 with its identity, its provenance and both of these orderings intact, plus the list of what is withheld (nets.held_dict, and web.held_record for why that status code). The client-side detail is in the API reference; the position in Licensing.

One wart, recorded rather than smoothed over: the five seeded demonstration rows are published as CC-BY-4.0 with licence_source NULL. That is the seed's own position, from before any of this existed. The licence pass reads only rows with a non-null source, so it never reaches them — deliberately, because it must not relicense a row whose values it did not write. They are five published rows whose licence names no basis, and the edit that would fix it is in seed.py, not in the loader.

Presentation order — rcsr_index

This is the column anything that lists nets in RCSR order reads. It is persisted at ingest and never recomputed downstream: no request path, no template and no export re-derives it. A page that wants RCSR order does ORDER BY rcsr_index and a net whose column is NULL sorts after the block (see below).

Ascending rcsr_index is what a net listing does by default — the HTML search at /data/nets and the REST listing GET /api/v1/nets alike. There is exactly one ORDER BY for a list of nets, nets.order_clause in services/nets.py, and every surface calls it, so they cannot come to disagree.

The three orderings a listing can be asked for

nets.ORDERINGS is ("rcsr", "name", "complexity") and nets.DEFAULT_ORDER is rcsr. A caller chooses with order= — the Order select on the search form (templates/nets_search.html, read by web.nets_page) and the order query parameter on the REST listing (rest.search_nets).

Three properties of that clause are deliberate and worth knowing before you depend on them:

• Nulls are pinned, not left to the database. SQLite sorts NULLs first and PostgreSQL sorts them last, so the clause states the placement itself: indexed nets first in index order, then unindexed nets by name. The same is done for a missing complexity_rank.
• Name is the tie-break inside every ordering, so pagination is stable: a non-deterministic tail silently duplicates and drops rows between pages.
• An unrecognised order= is not an error. nets.parse_order returns the default, and the surface names the ordering it actually used back to the reader — in the results sentence on the page and in the order field of the JSON envelope.

The coordination-sequence search is the exception: nets.search_cs takes no order and always calls order_clause() with its default, so the select on the form does not apply to a cs query.

It is derived, not authoritative

This is the part that must not be forgotten. rcsr_index is our arithmetic, not RCSR's. The rule, quoted from mofpp.services.rcsr.RCSR_INDEX_RULE, which is where it is defined once so that this page can quote it rather than restate it and drift:

rcsr_index is derived, not authoritative: the archive carries no ordinal. Every symbol in the archive is sorted into ascending plain ASCII lexicographic order and given its 1-based position in that ordering. Because - (0x2D) sorts before every letter, a suffixed variant falls immediately after its base symbol and before the next base (pcu < pcu-b < pcv). The provenance recorded on every record is derived.

Concretely, over the 2930 symbols in RCSRnets-2019-06-01.arc: pcn is 1535, pcu is 1536, then pcu-g 1537, pcu-g-e 1538, pcu-h 1539. Variants group under their base for free, which is the behaviour a reader browsing an index wants, and it falls out of plain byte comparison without any suffix handling.

Every record read from the archive has rcsr_index_provenance set to the string derived (rcsr.RCSR_INDEX_PROVENANCE). That column exists precisely so nobody has to guess later whether the number was read or computed. In the database that is 2930 rows — every symbol in the archive, with none left behind: the four seeded rows whose names are archive symbols (pcu, pyr, qom, tbo) are enriched with the index rather than skipped. The other 147 rows carry NULL: the 146 ingested MOF+-only nets and one seeded row (pcu-c), none of which has a symbol to derive an index from.

The value is served as well as stored: every net representation carries rcsr_index_provenance and a sentence spelling out what the number is (nets.RCSR_INDEX_NOTES), and the same sentence is a template global, so the page and the JSON cannot come to describe it differently.

A blank index is not one fact but three, and the served record says which: alongside a null rcsr_index goes an rcsr_index_absence of

• outside-archive — the name is not one of the 2930 symbols, so there is no position to hold. All 147 blanks in the database today are this, raf and ycs-z among them.
• unrecorded — the name is an archive symbol, so the net does have a place in the ordering and this row simply carries no number. No row is in this state. It was the state of the four seeded archive symbols before enrichment, and it stays a defined answer because a future load that skips a row would put one back into it.
• unknown — the local archive could not be read, so the two cannot be told apart.

None of them is a licence to derive an index at request time; the column stays the only authority for the number and nets.rcsr_index_absence only explains its absence.

Why it had to be derived

The local Systre archive stores no ordinal, and its physical block order is not the order to use. The first blocks in the file are mcm, tts, tth, sql, hxl, hcb, kgm, fsz — neither alphabetical nor RCSR's published listing. That is molsys archive accretion order: the sequence in which nets happened to be added to the file. Presenting nets in it would be presenting an artefact of somebody else's file history as though it were a chemical index.

RCSR's own listing order is not on this machine either. There is no 3dall.txt, no .cgd dump and no other RCSR export here — only the .arc, which is a canonical-key lookup table carrying key, version, id, checksum and two empty fields, and no cell, coordinates, spacegroup, transitivity, coordination sequence or vertex symbol. So the choice was between a documented derivation and no ordering at all, and the derivation was taken with its provenance stamped on every row.

What a future contributor should do

If you obtain RCSR's real published listing order — from an upstream export, a 3dall.txt, or the site itself — replace this column with it and flip rcsr_index_provenance from derived to authoritative. That is the intended end state. Nothing downstream computes the index, so replacing the column is a data migration and not a code change; the provenance string is how a reader tells which one they are looking at. Until then, do not cite rcsr_index as though it were RCSR's numbering, because it is not.

Nets with no RCSR symbol

A net that exists only in the MOF+ source has no symbol in the archive, so it gets rcsr_index NULL — never 0, never an invented tail number — and sorts after the whole RCSR block. The order within that tail is quoted here verbatim from mofpp.services.reconcile.SECONDARY_ORDER_RULE:

A net with no RCSR symbol has rcsr_index NULL and is placed after the whole RCSR block. Within that tail, records are ordered by ascending plain ASCII-lexicographic net name — the same byte-for-byte, case-sensitive comparison rcsr.sort_symbols applies to the archive symbols, with no case folding, normalisation or suffix handling. Names are unique across the union, so the order is total, and it depends on nothing but the names themselves: it is identical across runs, processes and platforms.

147 nets are in that tail today, not the fifteen a sampled cache once showed. It opens acs-b, afw-c, ant-c, bad, bae and closes xfv-z, ycr-z, ycs-z, yct-z, yct-zs. They are the MOF+-only nets: MOF+ deposit names such as eea_vc3_1-1_A and sln_112_topoFF-sq-trig that were never going to be RCSR symbols, and suffixed variants such as dia-c6 and srs-c4 that the 2019 archive does not carry.

Ingestion order — complexity_rank

complexity_rank says which nets to ingest, reconcile and check first, on the principle that the simplest ones fail fastest and most legibly: if the loader is broken, you want to find out on a one-vertex net, not two hours in on a net with 3264 vertices. It is a processing order, and that is the whole of its meaning.

A listing will sort by it if a reader asks for order=complexity, which is a genuinely useful way to see the small nets first. That does not promote it into a statement about a net: it is not a difficulty score, not a curation or quality ranking, and rank 1 does not mean "most important". Anything that reads it as an assessment of the net rather than of the ingest run is reading it wrong, which is why the results sentence on the page that offers the ordering calls it ingestion complexity rather than complexity.

The key is a chain of levels and the first available level wins:

1. transitivity — the quadruple pqrs. A regular net (1 1 1 1) is the simplest thing there is, so this level dominates whenever it is known.
2. vertex_and_edge_classes — number of distinct vertices, then number of distinct edges. (These are p and q by another name: a caller that holds only the first two members of the quadruple should pass them here rather than pretend to a full transitivity.)
3. cell_contents — number of vertices in the unit cell, then edge count.
4. max_coordination — the largest coordination number in the net.
5. The symbol, lexicographically — the deterministic final tie-break. This is not a level in the sense the other four are. A level is a claim about how complicated a net is; a symbol is not. It is appended to the key produced at every level, so two nets a level cannot separate are still totally ordered, and it is therefore never "first available" on its own.

A level is available only when every one of its values is present. Level 3 with an unknown edge count would have to compare a known vertex count against nothing, and inventing the missing half is exactly the fabrication this data set cannot afford. A half-known level is skipped, not patched.

Every rank carries a complexity_basis: the level that was used, the values it was computed from, and the resulting key. A rank can therefore be audited from the stored row alone, without re-reading either source or re-running the ranking.

Levels 1, 2 and 4 are used by nothing in this collection, and that is a fact about the sources rather than about the ranking: no source here supplies a transitivity, and every record that supplies a coordination number also supplies the cell contents that level 3 asks for first.

Nets with no key at all

A net for which levels 1 to 4 are all unavailable has no key. It is ranked last and counted — never ordered by its name as though its name said something about its complexity, and never given a guessed level.

Of the 3077 nets in the union, 2443 rank at level 3 (cell_contents) and 634 are unkeyed — the RCSR-only block, and all of it, because the archive supplies no transitivity, no cell contents and no coordination number. The keyed nets hold ranks 1 to 2443 and the unkeyed hold 2444 to 3077, in the tie-break order, and the count is reported rather than absorbed.

Which source supplies what

Two sources, and each is the only one on this machine that measures what it owns. The archive is the canonical-key authority and carries no crystallography at all; the .mfpx deposit is the only source of a cell, a coordinate or an edge. Every row records this per field in field_sources, so the table below is a summary of what is on the rows, not a substitute for reading them.

Where both sources offer the same field with different values, identity and canonical-key fields go to RCSR and crystallographic and geometric fields go to MOF+; the losing value is kept in the reconciler's report and named in field_sources, never merged into the winner. A conflict on a field with no agreed authority is refused rather than resolved by a default. On this data no conflict arises at all: the two offerings overlap only on source_id, which is equal by construction for a matched net.

The Systre key, and why it is the whole of an RCSR-only record

systre_key and systre_key_checksum are on 2930 rows — every archive symbol — carried verbatim from the .arc and attributed to rcsr in field_sources. They are NULL on the 147 rows with no RCSR symbol: an unknown key, not an empty one, because Systre is not installed here and nothing on this machine derives, extends, repairs or verifies a key. The checksum stored beside it is the archive's own, copied from the same block so that the copy can be audited against the upstream file later; it is not computed or checked here.

The key is now served, as a systre block on the full record and on the HTML page (nets.systre), carrying the key, the checksum, the provenance of each, and — where the key is blank — an absence of outside-archive with a sentence saying that neither source we read states a key for that net. List rows do not carry it: a 500-row page of Systre keys is not a listing.

On an RCSR-only record it is the only substantive datum there is. Such a row has no cell, no vertices, no coordination sequence and no catenation, because the archive carries none of those; what it has is a canonical key, a checksum, a derived index and a name. That is also the sharp edge of the licence position: those records are held, so the one thing they hold is the one thing withheld — and it is withheld for the right reason, since the key is RCSR archive text and the RCSR archive's terms are exactly what nobody has confirmed. The held 403 says this in its own systre block rather than staying silent about it, and systre_key is named in the record's withheld list.

Coordination sequences are served with their provenance

The same discipline, applied to the sequences: a full record carries a coordination_sequences block whose state says how far the values below it can be trusted — corroborated (2412 records), reassigned (7), not-recorded (653), unattributed (5, the seeded rows), and two states nothing is in today, uncorroborated and withheld. Without it a reassigned record and a corroborated one look identical, which asserts that the source supplied an assignment it did not supply. What the states mean is the subject of the next section.

catenated is three-valued, and NULL means unknown

catenated is nullable, and the three states are true, false and NULL — not two. NULL is unknown, never "no". In the database: 62 true, 2377 false, 638 NULL (the 634 RCSR-only rows, plus four seeded rows the loader does not overwrite).

The RCSR archive carries no catenation field, so no source says whether such a net is interpenetrated, and the column is nullable precisely so that absence can be stated instead of guessed. A listing asked for include_catenated=false cannot certify those rows un-catenated and does not return them, but it has not judged them catenated either; the envelope counts them separately.

The true/false values are not a measurement. They come from the legacy client's own rule — the name contains -c — which is why every one of them is attributed mofplus:name-heuristic rather than mofplus. A name heuristic is not a computed property of the topology, and the row says so rather than presenting the guess as a datum MOF+ sent. A sentence to that effect (nets.catenated_note) travels with the value everywhere it is shown — record, list row and page — because false on its own reads as a finding.

Coordination sequences and the class-mapping hazard

The coordination sequences are the source's own, byte for byte. Nothing here computes one into a record, extends it, repairs it, or replaces it with a walked one. But which vertex class each supplied sequence describes is a separate question from what the sequences are, and getting it wrong serves a reader a real sequence attached to the wrong vertex — which is worse than serving nothing, because it looks right.

The cheap invariant is cs[k][0] == coordination_number: the first term of a coordination sequence is the coordination number of the vertex it describes. It fires first and it refuses outright any record whose sequences cannot belong to the classes they are stored against.

That check is blind whenever two vertex classes share a coordination number. If classes 0 and 1 are both 3-coordinated, a sequence swapped between them still satisfies cs[k][0] == coordination_number on both, and nothing in the arithmetic notices. Measured over the complete cache: 5537 vertex classes across 2443 records, and 1079 of those records — 44% — have two or more classes of equal coordination number, so on nearly half the collection the cheap check cannot see a permutation at all.

The topology check exists to catch exactly that. Under check_against_topology, which every production path passes, the reader walks the periodic net once per class from one atom of that class (services.topology.coordination_sequence) and matches the walk against the sequences the source supplied. A class keeps its own positional sequence whenever that sequence matches, so a correct record is left byte-identical.

What that found over the full cache: 7 records, 14 classes, whose stored positional mapping disagrees with the walk. All seven were resolved, each a clean bijection — the multiset of sequences the source stored is exactly the multiset the walk asks for, so the correction is a permutation of the source's own values and nothing is synthesised:

Six of those seven were invisible while the cache was a sample; only ycs-z was in it. On ycs-z, 72 of 96 atoms were being served a sequence their own connectivity contradicts.

The resolution is recorded, not silently applied. Such a row carries field_sources["cs"] = "mofplus:cs-class-mapping-resolved-by-topology" instead of a flat mofplus, plus cs_topology_mismatch naming the classes that disagreed and cs_class_mapping giving the mapping. Every checked row — 2419 of them — also carries cs_corroboration, so that "nothing disagreed" and "nothing was checked" are different states of the record rather than the same silence.

A class whose sequence could not be corroborated is withheld, not guessed. Where the walk matches no supplied sequence, or matches two that differ, that class's cs is left null, the class is named in cs_withheld, and the row carries cs_unresolved in field_sources. No record in the collection is in that state today (0 withheld), and the path is exercised by a test rather than left to be discovered.

Two limits, stated so nobody assumes otherwise. A permutation across different coordination numbers is still refused outright by the cheap invariant — the record becomes a reported failure rather than a resolved one, which is loud but is a case the resolver could handle and does not. And read_cache() still defaults check_against_topology=False; every production path passes True, but a caller reading the cache directly gets the source's positional mapping.

Separately, and not a mapping question: 19 records store no coordination sequence at all, among them raf, rag, cuz and bph. Vertex.cs is not nullable, so those vertices carry [] — absent, never a walked or invented sequence. And 13 vertex symbols across 12 records (dnd, dni-d, dno, dnt, gea, nzn, rht, rht-x, twf, uro, urq, urs) arrived cut off by the source at exactly 512 characters. They are stored verbatim with a note saying the string names fewer ring sizes than the vertex has angles, and must not be read as whole vertex symbols.

Fields that are NULL on every ingested record

None of these is an omission to be filled in later by inference. Each is a statement about what the sources contain. Counts are over the 3072 rows the ingest wrote; the five seeded rows keep their own values.

And three that are NULL on exactly the 634 RCSR-only rows, for one reason each:

None of these appears in any row's field_sources dict, because a field with no source has no provenance to record. A NULL here means absent, which is a different claim from zero and a very different claim from inferred.

What is actually loaded

Stated here so nobody re-derives them from the reports. The MOF+ cache is complete — all 2443 nets the MOF+ listing names were retrieved, 0 failures — so the sampling cap that shaped every figure on earlier versions of this page, and the 2105 records that were "pending" under it, are history. Nothing is capped, sampled or pending.

• The union is 3077 nets — 2296 in both sources, 634 RCSR-only, 147 MOF+-only. 3072 rows were written by the ingest; five names (pcu, pyr, qom, tbo, pcu-c) already existed as seeded demonstration rows and were enriched, not overwritten — they received the two orderings and their provenance and nothing else. Ingested rows are distinguishable from seeded ones by exactly one predicate: Net.source IS NOT NULL.
• Every union member has a row. 0 names in the union with no row anywhere, 0 rows in the table the union does not hold.
• "RCSR-only" is a fact about the sources now, not an artefact of a cap. All 634 are confirmed RCSR-only: MOF+ does not list the symbol at all. The pending category — a symbol MOF+ lists but the cache had not reached — is empty, and the split is still recomputed against the full MOF+ listing on every run rather than stored as a constant (reconcile.RCSR_ONLY_SPLIT_RULE), so it would report a gap again if one appeared.
• 2443 rows carry geometry and 634 do not. Geometry follows the MOF+ deposit, not the overlap: every MOF+-backed row has a cell, vertices and edges — 123316 vertex rows and 251497 edge rows in total — and every RCSR-only row has none, because the archive carries none. (2438 of those deposits were written by the run; five are the seeded rows' own, which enrichment does not touch.)
• 2443 rows are published and 634 are held, on the split set out at the top of this page. What a reader sees today is a site listing 2443 nets, with 634 identifiers that resolve with 403.
• The name match between the two sources is uncorroborated, not corroborated. The archive carries no coordination sequence — a keyword census of all 2930 blocks finds only key, version, id, checksum, ref and desc — so there is no shared field to compare the MOF+ sequences against. All 2292 ingested overlaps are recorded uncorroborable in field_sources.name_match, which is counted as a check that could not be performed rather than one that passed; the remaining 780 ingested rows are single-source, where there is nothing to corroborate.

Attribution

The RCSR symbols, canonical Systre keys and checksums come from the Reticular Chemistry Structure Resource:

M. O'Keeffe, M. A. Peskov, S. J. Ramsden and O. M. Yaghi, Acc. Chem. Res. 2008, 41, 1782–1789. https://rcsr.anu.edu.au

The local copy is the Systre archive RCSRnets-2019-06-01.arc. The rcsr_index derivation described on this page is ours, not RCSR's, and must not be cited as RCSR numbering. Records built from that archive alone remain unpublished; see the licence position above.