Testing Methodology

Principle

Every new code must be covered by tests. Pure logic is covered by unit tests, interactions between components are covered by integration tests.

Test Layers

Layer 1: Unit tests (pure logic, no I/O)

In-module #[cfg(test)] mod tests blocks. Fast, deterministic, no external dependencies.

What to test:

• Serialization roundtrips (CBOR encode/decode for all GossipMessage variants)
• Key construction and ordering (db::keys)
• Deterministic ID computation (compute_msg_id, dm_chat_id)
• Canonical string building, hex parsing (api::utils)
• Cryptographic primitives (crypto::keccak256, crypto::parse_addr20)
• Data structure logic (InboxBatcher merging/flushing, PeerCache, MerkleTree)

Locations:

• crates/node/src/types.rs -- gossip roundtrips, inbox batcher, peer cache
• crates/node/src/sync/merkle.rs -- Merkle tree operations
• crates/node/src/handlers/context.rs -- XOR distance
• crates/node/src/handlers/mpsc/utils.rs -- unique ID generation
• crates/db/src/keys.rs -- key format and ordering
• crates/db/src/messages.rs -- msg_id computation, text preview
• crates/node/src/sync/protocol.rs -- SyncRequest/SyncResponse CBOR roundtrips
• crates/api/src/utils.rs -- canonical signing, dm_chat_id, hex helpers
• crates/crypto/src/lib.rs -- keccak256, address parsing, ECDSA verify_sig_recover

Layer 2: DB integration tests (RocksDB with tempdir)

Tests that open a real RocksDB instance in a temporary directory. Verify that read/write operations work correctly end-to-end through the storage layer.

What to test:

• put_message + range roundtrip
• Message idempotency (same msg_id written twice)
• Pagination (after_key / limit)
• Time range filtering (from_ts, to_ts, combined)
• Read progress (set, get, monotonic increase)
• Inbox upsert + list (ordering by activity time, pagination)
• Member add, remove, list, membership check
• Control messages (msg_type > 0)
• Sync helpers: for_each_msg_id, get_message_cbor_by_id, get_messages_cbor_batch (incl. byte limit), get_bucket_msg_ids

Location: crates/db/src/lib.rs (#[cfg(test)] mod db_tests)

Helper: temp_db() creates a ChatDb backed by TempDir.

Layer 3: In-process integration tests (multiple nodes)

Full nodes running in a single tokio runtime, communicating via real libp2p TCP connections and gossipsub.

What to test:

• Node startup and graceful shutdown
• Peer discovery and connection via bootnodes
• Message sending (DM stored locally via async DB writer)
• Message propagation between nodes via gossip
• Message ordering by timestamp
• ListUserChats (inbox populated after DM send)
• ReadChatMessage (read progress stored via async DB writer)
• Merkle-tree sync per domain:
  • test_sync_messages -- 500 DMs, exercises multi-bucket drill-down and chunked FetchAndPush
  • test_sync_members -- group with admin + member, verifies role preservation across sync
  • test_sync_identity -- two identity blobs, verifies blob content after sync

Location: crates/node/tests/integration.rs

Infrastructure:

• run_node(AppConfig) returns NodeHandle with cmd_tx, db, listen_addrs
• Each test node uses a random keypair, port 0, temp DB directory
• two_connected_nodes() helper spins up a pair with gossipsub mesh
• gossipsub requires >= 1 peer, so even "local" tests use two nodes
• Tests send Command variants via cmd_tx and assert on DB state or oneshot responses
• Gossip propagation tests sleep 2-3s for mesh formation + message delivery

Running Tests

cargo test -p node                                     # All tests (unit + integration, production code only)
cargo test -p node --features test-support             # Same + clock-skew E2E tests via MockClock
cargo test -p node --lib                               # Unit tests only (fast)
cargo test -p node --test integration                  # Integration tests only
cargo test -p db                                       # DB unit + integration tests
cargo test -p node -- test_name                        # Single test by name
cargo test -p node -- --nocapture                      # With stdout visible

test-support cargo feature

Some integration tests need to bring a node up in a non-default way -- a controllable HLC clock, in-memory transport stubs, fault injection wrappers, anything else that doesn't make sense in production. Rather than letting those helpers leak into the prod build, the crate exposes them through node::test_support, a module gated behind the test-support cargo feature.

How to use it:

• Run gated tests with cargo test -p node --features test-support. Without the flag the gated tests are silently skipped (the rest of the suite still runs).
• Put helpers that wrap or replace production entry points under crates/node/src/test_support.rs. Re-export them with pub fns. Production binaries are compiled without the feature, so anything in that module is invisible to release builds.
• Mark new tests that depend on the module with #[cfg(feature = "test-support")] (either on the test fn or on a containing mod).

First user of the mechanism is run_node_with_clock, which threads a custom Arc<dyn Clock> into run_node for clock-skew E2E tests under tests/integration.rs::clock_skew_e2e. Future helpers (network fault injection, swappable storage, deterministic randomness) can land in the same module without changing the feature gate.

Clock Abstraction in Tests

types::clock::Clock is the time source HLC depends on. Production uses SystemClock (wraps SystemTime::now). Tests use MockClock, an AtomicU64-backed clock with set(ms) and advance(delta_ms) so a test can rewind into the past, jump forward, or simulate per-node drift.

Integration tests that need a controllable clock build nodes via node::test_support::run_node_with_clock(config, clock) (see the test-support feature section above). Unit tests in crates/node/src/hlc_state.rs use MockClock directly for the HLC algorithm tests (drift bound, overflow handling, monotonicity under concurrent stamping).

Writing New Tests

For new pure logic (types, computations, parsing)

Add #[cfg(test)] mod tests in the same file. Test edge cases, roundtrips, and error paths. No I/O, no sleeps.

For new DB operations

Add a test in crates/db/src/lib.rs::db_tests using temp_db(). Write data, read it back, verify.

For new gossip message variants

1. Add a roundtrip test in crates/node/src/types.rs::gossip::tests
2. If the variant affects cross-node behavior, add an integration test in crates/node/tests/integration.rs

For new HTTP API endpoints

1. Add the Command variant and handler
2. Add an integration test that sends the command via cmd_tx and verifies the response and/or DB state

For new protocol interactions (sync, query/response)

Add integration tests with two+ nodes verifying the full round-trip.

For entirely new subsystems

If the new functionality does not fit any of the categories above (e.g. a new transport layer, a new storage backend, a standalone utility crate):

1. Create a dedicated test file or #[cfg(test)] mod tests block within the new module
2. If the subsystem interacts with other components, add integration tests in crates/node/tests/ (one file per subsystem, e.g. crates/node/tests/new_subsystem.rs)
3. Follow the same layering principle: pure logic in unit tests, cross-component interactions in integration tests

Test Conventions

• Tests go at the end of the module (clippy: "items after test module")
• Use #[cfg(test)] to avoid compiling test code in release builds
• Integration tests use #[tokio::test] (async runtime required)
• Prefer assert_eq! with descriptive messages over bare assert!
• Clean up resources: hold TempDir handles, call shutdown().await
• Do not hardcode ports -- always use port 0 for OS assignment