p2p-mes Protocol Documentation

p2p-mes is a peer-to-peer messenger built on libp2p (GossipSub for messaging, Kademlia for discovery), RocksDB storage, and ECDSA / Keccak256 signature-based authentication. Every node stores all data and reconciles with its peers through Merkle-tree anti-entropy synchronization.

This site is the protocol reference: it describes the wire formats, the cryptography, and everything a third party needs to build an interoperable client. It is generated from the Markdown files maintained alongside the code, so the documentation and the implementation never drift apart.

Where to start

• Building a client? Read Building a Client and the API Reference. The ~13 HTTP endpoints are the entire client-facing surface -- you do not need the gossip or sync internals to build an interoperable client. For keys, encrypted DMs, attachments, and running MLS groups on this protocol, follow the step-by-step E2EE Cookbook.
• Implementing or operating a node? Read the Protocol Specification and Internals sections, which describe the node-to-node mechanics.

How this documentation is organized

• Philosophy & Design -- the principles behind the protocol: why state is minimized, why every node replicates everything, and the two-layer (transport vs. client) split.
• Protocol Specification -- the normative wire reference: the gossip message set and its CBOR encoding, the anti-entropy sync state machine, and the cryptographic primitives.
• Client Implementation -- a practical, end-to-end guide to building a client: request signing, the message lifecycle, pagination, and the opaque Layer-2 fields used for end-to-end encryption.
• HTTP API -- the REST surface, with an interactive reference generated from the server's OpenAPI specification.
• Internals -- architecture, storage layout, shared types, retention, and testing -- aimed at contributors to the node itself.

Conventions

Fixed identifier lengths are load-bearing throughout the protocol: user addresses are 20 bytes, while chat and message identifiers are 32 bytes. Multi-byte integers embedded in storage keys are big-endian. All persistent data is syncable between nodes.

Design Philosophy

p2p-mes makes a small number of deliberate, sometimes uncompromising choices. Together they explain almost every "why" behind the wire formats, the storage layout, and the API surface described elsewhere on this site. Read this page first: much of the rest of the specification is the mechanical consequence of the five principles below.

Minimal state, maximal derivation

The protocol stores data only when there is no alternative. Anything that can be computed from inputs, or derived from data already stored, is never written down.

Two examples that recur throughout the design:

• A direct-message conversation has no stored membership and no stored identifier. Its chat_id is derived deterministically from the two participant addresses (see Cryptography & Authentication), so access control falls out of the math: only the two parties can compute it.
• Unread counts are never stored. They are derived at read time by comparing a chat's latest sequence number against the reader's stored read progress.

Every candidate for persistence is challenged with one question: can we avoid storing it? Less stored state means less to synchronize, less to keep consistent, and fewer ways for two nodes to disagree.

Performance is a feature

Throughput and latency are treated as correctness properties, not tuning knobs bolted on at the end.

• HTTP writes are fire-and-forget: a request is validated, signed gossip is published, a write is queued to an asynchronous DB writer, and the client gets its response before the write commits. The event loop never blocks on disk.
• All persistence flows through a single asynchronous DB writer, which is also the single source of truth for metrics and synchronization state. One path means one place to reason about ordering.
• Storage keys are laid out for the access pattern -- time-ordered message scans, reverse-time inbox listing, prefix bucket scans for sync -- so the hot reads are sequential range iterations rather than random lookups.

Where readability and speed conflict, the rule is to measure first; but once a path is proven hot, it is optimized without apology.

Full replication over sharding

Every node is a full replica: it stores all messages, members, and identity records, and can answer any query on its own. There is no ownership, no routing by key, no "ask the responsible node."

This is a deliberate trade. An earlier design sharded data by XOR distance with a replication factor; it was removed in favor of full replication because the consistency story is dramatically simpler. With every node holding everything, agreement reduces to a single question -- do two nodes hold the same set of records? -- which is answered efficiently by Merkle-tree anti-entropy sync across three independent domains: messages, members, and identity.

The cost is storage and write amplification; the benefit is that any node can serve any client, synchronization is uniform, and there is no rebalancing. Because of this choice, all persistent data must be syncable -- any new stored field has to be covered by an anti-entropy domain, or it will silently diverge between nodes.

Two layers: a dumb transport, a smart client

The node deliberately understands as little as possible. It is split into two layers that behave identically for direct messages and groups.

Layer 1 -- node-enforced. These behaviors are built into every node and cannot be bypassed: signed message delivery with deduplication, inbox maintenance, group membership with add-wins CRDT semantics, authentication and authorization on every request, anti-entropy sync, and retention.

Layer 2 -- client-defined, opaque to the node. A few fields are stored and relayed verbatim, never interpreted:

• msg_type (u8) -- the client assigns the meaning (for example: attachment metadata, MLS handshake); the node treats every value as valid and opaque.
• control (byte string) -- an opaque payload carried alongside a message.
• the per-DM identity blob -- stored last-write-wins, never inspected.

The consequence is the central architectural bet: semantics live in the client, not the network. A minimal client can send plaintext using Layer 1 alone. A privacy-focused client can build end-to-end encryption, key exchange, or any other protocol entirely on Layer 2 -- the network neither needs to understand it nor is able to interfere with it. See Building a Client.

What "interoperable" means

The interoperability contract is Layer 1: any client that signs requests correctly can exchange plaintext messages, manage groups, track read progress, and publish identity with any other client, through any node. That is the guarantee the word "interoperable" carries here.

Layer 2 is deliberately not interoperable by default. The msg_type values, the structure of control payloads, the client formats inside text, and any encryption scheme are defined by the client, not the protocol. Independently built clients interoperate in plaintext (Layer 1) plus whatever documented Layer 2 profiles they both implement. Three profiles are documented today in Building a Client: the self-signed identity key bundle (key distribution over untrusted nodes), DM encryption (a prefixed envelope in text), and attachments (control msg_type = 10). Group E2EE runs (D)MLS on top of the same identity bundle; its transport binding is documented in the E2EE Cookbook.

Trust model

The protocol is precise about what a node can and cannot prove, and a client should assume nothing beyond it.

What the node enforces. Every state-changing operation is signed. The node recovers the author's address from the ECDSA signature (see Cryptography & Authentication) and checks authorization -- group admin rights, membership for group sends -- before accepting a write. So authorship and authority are verifiable: a node can prove who requested a change and that they were allowed to make it. Message identity and deduplication use a BLAKE3 content hash. A Hybrid Logical Clock timestamps every conflict-bearing operation and rejects timestamps from peers that exceed local wall-clock by more than a fixed drift bound, so a single misclocked or malicious peer cannot drag the cluster's logical time forward.

What the node does not provide. It does not read the meaning of Layer 2 payloads, so confidentiality from the node operator is not a transport guarantee -- it is delegated to the client via Layer 2 encryption. The network also assumes a population of honest full replicas: anti-entropy converges what honest nodes hold, but the protocol does not by itself defend against a Byzantine replica that selectively withholds or serves stale records. A formal adversary model is still being developed; today's guarantees are authenticity and authorization -- not confidentiality from the operator (delegated to clients) and not Byzantine fault tolerance. Clients that need stronger properties should layer them on top.

Privacy: what the node operator sees

Confidentiality in p2p-mes is content-only, and only when a client adds Layer 2 encryption. Everything a node needs to route, store, and synchronize is plaintext to the operator of any node the data reaches -- which, under full replication, is every node:

• Who talks to whom. Sender and group members are plaintext. A DM chat_id is blake3(domain || min(a,b) || max(a,b)), so an operator who suspects a pair of addresses can confirm they are conversing.
• When. HLC and origin_wall_ts stamps expose timing and activity patterns.
• How much. Message sizes, group sizes, and frequency are observable.

Message bodies can be hidden with Layer 2 E2EE, but the social graph and metadata cannot -- they are inherent to a fully replicated store. A client should say this plainly to its users: p2p-mes protects message contents from the operator (with E2EE), not the fact that, when, or with whom you communicate.

Gossip Protocol

Overview

Nodes communicate via GossipSub (libp2p) using CBOR-encoded messages. Two topics are used:

• p2p-mes/commands -- outgoing commands and broadcasts
• p2p-mes/responses -- query responses

Message deduplication is handled at GossipSub level using BLAKE3 hash of message data as message_id.

Audience: node implementers. This page documents the node-to-node gossip layer. Client applications never speak gossip directly -- they use the HTTP API (Building a Client, API Reference). Read on to understand how nodes propagate and reconcile data, or to build a node.

GossipMessage Enum

All gossip traffic is a single CBOR-encoded enum:

#![allow(unused)]
fn main() {
enum GossipMessage {
    PutMessage(PutMessage),             // Store a new message
    InboxFanout(InboxFanout),           // Disabled (full replication)
    BatchedInboxFanout(BatchedInboxFanout), // Disabled (full replication)
    Query(Query),                       // Request data
    QueryResponse(QueryResponse),       // Respond to query
    Ack(Ack),                          // Deprecated
    ReadProgress(ReadProgress),         // Mark-as-read
    ReadProgressAck(ReadProgressAck),   // Mark-as-read acknowledgment
    MembershipOp(MembershipOp),        // Membership change
    MembershipOpBatch(Vec<MembershipOp>), // Batch of ops
    PutIdentity(PutIdentity),          // User identity propagation (HLC LWW)
    MessageOp(MessageOpRecord),        // Edit / delete an existing message
}
}

Encoding/decoding:

#![allow(unused)]
fn main() {
let bytes = gossip_msg.encode();          // serde_cbor::to_vec
let msg = GossipMessage::decode(&bytes);  // serde_cbor::from_slice
}

CBOR Serialization Format

GossipMessage uses serde's internally-tagged representation. Each variant serializes as a CBOR map with one key (the variant name) whose value is the variant's payload:

PutMessage example:
  {"PutMessage": {"msg_id": <bytes32>, "chat_id": <bytes32>, ...}}

MembershipOp example:
  {"MembershipOp": {"chat_id": <bytes32>, "target": <bytes20>, "sig": <bytes65>,
                     "role": 0, "op_type": 0, "hlc": 111669149696005}}
  // `hlc` is a packed HlcTimestamp: 48 bits physical_ms + 16 bits logical.

MembershipOpBatch example:
  {"MembershipOpBatch": [<MembershipOp>, <MembershipOp>, ...]}

CBOR type mapping:

Note: serde_cbor is the serialization library. Fields with #[serde(default)] are omitted when at default value on some paths -- implementors should handle both presence and absence.

Byte arrays are CBOR arrays, not byte strings. No serde_bytes annotation is used, so [u8; N] and Vec<u8> fields encode as CBOR arrays of u8 integers (major type 4), never byte strings (major type 2). The <bytes32> notation in examples above is shorthand for such an array. See decoding msg_cbor in the Client Guide for a worked decoder.

Message Types

PutMessage

Initiated by HTTP API. Broadcasts a new chat message to the network.

#![allow(unused)]
fn main() {
struct PutMessage {
    msg_id: [u8; 32],           // BLAKE3(chat_id || sender || hlc_packed_be || text)
    chat_id: [u8; 32],          // Chat identifier
    kind: ChatKind,             // Dm { peer } | Group { title } | Channel { title }
    sender: [u8; 20],           // Sender's address
    members: Option<Vec<[u8; 20]>>, // Chat members (for inbox fanout)
    text: String,               // Message text (empty for control messages)
    hlc: HlcTimestamp,          // Server-stamped HLC (storage/CRDT/sync)
    origin_wall_ts: u64,        // Frozen originator wall-clock (UI display)
    origin: String,             // PeerId of originating node
    needs_ack: bool,            // Always false (fire-and-forget mode)
    msg_type: u8,               // client-defined; 0 = regular text
                                // (registry in CLIENT_GUIDE.md)
    control: Option<Vec<u8>>,   // protocol-structure payload (CBOR, opaque)
}
}

Node-enforced fields (node validates and uses these):

• msg_id -- computed by node, used for dedup and Merkle tree
• chat_id -- used for routing, storage, membership checks
• sender -- verified against auth signature
• members -- used for inbox fanout (DM only; groups use members CF)
• hlc -- stamped server-side by the originating API node's HlcState; drives the messages CF key, retention cutoff comparisons, and inbox last_ts. Receiver-side gossip handler feeds the value through HlcState::receive and drops the message on drift violation
• origin_wall_ts -- frozen sender wall-clock; UI display only, never participates in distributed logic
• origin -- used for gossip routing
• kind -- set once by the originating API node and stored verbatim. For DMs, Dm.peer is the original recipient (the {peer} path parameter of the send request) -- it is not rewritten per reader; only user_inbox entries get a viewer-relative peer (see TYPES.md)

Client-opaque fields (node stores and relays without interpretation):

• msg_type: u8 -- client-defined, node does not switch on this value
• control: Option<Vec<u8>> -- opaque payload for client-to-client protocols
• text: String -- fully opaque to the node (size-capped at the API only)

All msg_type values are a client convention, not a protocol requirement -- any u8 value is valid. 0 = plain message (set by the plain message endpoints); the control-type registry (10/20/21 and friends) lives in CLIENT_GUIDE.md.

Flow:

1. HTTP handler creates PutMessage with computed msg_id
2. Publishes to p2p-mes/commands
3. All nodes receive, each stores via DbOp::PutMessage
4. Dedup via seen_msg CF prevents double storage
5. On store success, process_db_op updates user inboxes locally

Query / QueryResponse

Request-reply pattern over gossip for data retrieval.

#![allow(unused)]
fn main() {
struct Query {
    query_id: [u8; 16],        // Correlation ID
    kind: QueryKind,
    requester: String,          // PeerId of requester
}

enum QueryKind {
    GetChatRange {
        user: [u8; 20],
        chat_id: [u8; 32],
        from_ts: u64,
        to_ts: Option<u64>,
        after_key: Option<Vec<u8>>,
        limit: usize,
        #[serde(default)]
        reverse: bool,          // false = oldest-first; true = newest-first
    },
    ListUserChats {
        user: [u8; 20],
        limit: usize,
        after_key: Option<Vec<u8>>,
    },
}
}

reverse is tagged #[serde(default)]: peers predating the field send no key and decode it as false, so the CBOR wire format stays backward-compatible. Under full replication this query path is dormant (every node answers GetChatRange from its local store), but the field is threaded through so the sharded path stays correct if it is ever re-enabled.

#![allow(unused)]
fn main() {
struct QueryResponse {
    query_id: [u8; 16],        // Matches Query.query_id
    kind: QueryResponseKind,
}

enum QueryResponseKind {
    GetChatRange(GetChatRangeResponseRaw),
    ListUserChats(ListUserChatsResponseRaw),
}
}

Flow:

1. MPSC handler publishes Query to p2p-mes/commands
2. Stores PendingQuery with oneshot channel for response
3. Any node with data publishes QueryResponse to p2p-mes/responses
4. First response wins, sent back to HTTP client
5. Timeout: 30 seconds, cleaned up by ack_timeout_check timer

ReadProgress / ReadProgressAck

Broadcasts read progress across nodes.

#![allow(unused)]
fn main() {
struct ReadProgress {
    progress_id: [u8; 16],     // Correlation ID
    user: [u8; 20],
    chat_id: [u8; 32],
    seq: u32,                   // Mark all messages up to this seq as read
    origin: String,             // PeerId of origin node
}

struct ReadProgressAck {
    progress_id: [u8; 16],
    from: String,               // PeerId of acknowledging node
}
}

Each node stores read progress in user_read_progress CF. Update is monotonic: only advances if seq > current.

MembershipOp

Protocol-level membership change. Travels via gossip, updates members CF only (never messages CF).

#![allow(unused)]
fn main() {
struct MembershipOp {
    chat_id: [u8; 32],          // Target group chat
    target: [u8; 20],           // User being added/removed/created-as/promoted
    sig: Vec<u8>,               // ECDSA signature (65 bytes)
    role: u8,                   // 0=participant, 1=admin (2=owner is derived, never sent)
    op_type: MembershipOpType,  // see enum below
    hlc: HlcTimestamp,          // Originator's HLC stamp (packed u64)
    title: Option<String>,      // Create only: group title (chat_id preimage)
    nonce: Option<Vec<u8>>,     // Create only: 16-byte creation nonce (chat_id preimage)
}

enum MembershipOpType {
    Add = 0,
    Remove = 1,
    Create = 2,             // signer becomes the owner (role=2)
    TransferOwnership = 3,  // target = new owner; old owner demoted to admin
    DeleteGroup = 4,        // target unused; tombstones all members
}
}

Client signature deliberately excludes hlc -- clients have no access to node-level HLC state. The canonical signature message is keccak256(chat_id || target || op_type_byte) exactly as before. The node stamps hlc server-side via its HlcState immediately before publishing, mirroring how the legacy ts: u64 was set by now_millis(). See md/TIME_AND_CONSISTENCY.md for the rationale.

Flow:

1. HTTP API POST /groups/{chat_id}/ops verifies ECDSA signature
2. For Create ops: API verifies chat_id == blake3(domain || signer || nonce || title) (title validated: <= 128 bytes UTF-8, no control characters; nonce must be exactly 16 bytes)
3. MPSC handler stamps each op with ctx.hlc.stamp() (one stamp per op)
4. MPSC handler checks role-based authorization via DB. For Create it independently re-verifies the chat_id binding from step 2 (the command channel is reachable without the HTTP layer)
5. Duplicate Create rejected if group already has members
6. All valid ops published as one GossipMessage::MembershipOpBatch. Create ops carry title + nonce so receivers can re-verify
7. Gossip handler feeds incoming hlc into ctx.hlc.receive(), dropping the op if it exceeds local now by more than the configured max-drift bound (default 5 minutes). Then re-verifies sig + auth (don't trust other nodes); for Create it also recomputes the chat_id from (signer, nonce, title) -- a relaying node cannot forge the title or fabricate a Create for a foreign chat_id
8. Stored via DbOp::MembershipOp { hlc, ... }. The DB writer routes to add_member_synced / remove_member_synced, which perform the CRDT merge with removed_at semantics (see STORAGE.md). TransferOwnership and DeleteGroup route to their own atomic DbOps (DbOp::TransferOwnership, DbOp::DeleteGroup) so the multi-record updates commit in a single WriteBatch each. Verified Creates additionally queue DbOp::SetGroupCreateMeta, which records the immutable creation trio (title, creator, nonce) in chats_meta (write-once)

Anti-entropy sync ships the full MemberInfo state (role, added_at, optional removed_at) per record via SyncMemberRecord. Receivers apply via DbOp::ApplyMemberRecord, which calls apply_member_record_synced for a monotonic CRDT merge that preserves tombstones either side might have observed independently. See SYNC.md.

Role model and authorization rules:

Three roles: participant (0), admin (1), owner (2). Exactly one owner per group -- Create makes the signer the owner. Both MPSC and gossip handlers enforce the same rules independently (don't trust other nodes):

DeleteGroup tombstones every active member (removed_at = hlc in one WriteBatch) and clears each member's inbox entry, so the chat disappears from /conversations for everyone on every node. Stored messages are left untouched -- the retention GC removes them on its own schedule.

Group creation via compound endpoint: Group creation is done via POST /groups/{chat_id}/ops with a Create op. The client generates a random 16-byte nonce, picks an optional title, computes chat_id = blake3(domain || signer || nonce || title), and signs the Create op. The API verifies the derivation. This eliminates the security gap where the old POST /groups endpoint generated chat_id server-side and the client couldn't sign it.

Group creation metadata (title, creator, nonce): A verified Create materializes the immutable creation trio into chats_meta via DbOp::SetGroupCreateMeta (write-once -- later attempts are no-ops). Because the title and nonce are part of the chat_id preimage and the creator's signature covers the chat_id, the trio is self-verifying: any node can recompute the hash and reject a forged copy regardless of who delivered it. Consequences:

• The title is immutable for the lifetime of the group (rename would change the chat_id). A future mutable-metadata layer can be added on top without breaking this anchor.
• The trio travels over two channels: the gossip Create op (live path) and the members-sync piggyback on the owner record (anti-entropy path, see SYNC.md). No separate propagation round is needed.
• Inbox entries (user_inbox) and /conversations responses expose the title through ChatKind::Group { title }; the DB hydrates it from chats_meta at inbox-write time, so messages themselves never carry the title.

LeaveGroup reuses MembershipOp(Remove): DELETE /groups/{chat_id}/membership publishes a standard MembershipOp with op_type=Remove and target=sender. No new gossip variant was needed.

Duplicate Create protection: Both MPSC handler and gossip handler reject Create ops when list_members(chat_id, limit=1) returns any results. This prevents group hijacking: once a group is created and has members, no subsequent Create op can overwrite ownership.

MembershipOpBatch: GossipMessage::MembershipOpBatch(Vec<MembershipOp>) carries all ops from a single compound call in one gossip message. The receiver processes ops in order within the batch, ensuring atomic delivery (e.g. Create+Add arrives together, preventing split-brain where Add arrives before Create).

Authorization inside a batch uses an in-memory virtual-state overlay (see handlers/membership_batch.rs): a Create populates the overlay with the new owner, and a subsequent Add in the same batch consults the overlay before falling back to the on-disk members CF. Without this overlay an [Create, Add, Add] batch would fail because the async DB writer is FIFO and has not committed the Create by the time the Add is authorized. TransferOwnership swaps the two roles in the overlay, enabling [Create, Transfer, Remove(self)] -- "create a group, hand it over, walk away" -- in one request.

The overlay also tracks chats deleted within the batch: after a DeleteGroup op, every subsequent op for the same chat_id in that batch is rejected (the deletion is terminal for the request).

Individual MembershipOp messages are still supported for backward compatibility and single-op flows (e.g. LeaveGroup).

Two delivery channels for compound membership:

One HTTP call to POST /groups/{chat_id}/ops produces:

• 1 GossipMessage::MembershipOpBatch message (all membership ops)
• M GossipMessage::PutMessage messages (client data: MLS Welcome/Commit)

Membership ops are processed before accompanying messages.

sequenceDiagram
    participant C as Client
    participant API as HTTP API
    participant MPSC as MPSC Handler
    participant G as GossipSub
    participant DB as DB Writer

    C->>API: POST /groups/{chat_id}/ops
    API->>API: Verify ECDSA sig + nonce/title binding (Create only)
    API->>MPSC: Command::MembershipOp

    MPSC->>MPSC: Role-based authorization + duplicate Create guard
    MPSC->>G: MembershipOpBatch (all ops)
    MPSC->>DB: DbOp::MembershipOp (per op)

    loop Each accompanying message
        MPSC->>G: PutMessage
        MPSC->>DB: DbOp::PutMessage
    end

    Note over DB: process_db_op updates inboxes locally
    MPSC-->>C: 200 OK

Membership Operations

Membership operations use a dedicated MembershipOp gossip variant (not embedded in PutMessage with msg_type routing).

Operation Types (op_type)

MembershipPayload (control field)

#![allow(unused)]
fn main() {
struct MembershipPayload {
    target: [u8; 20],   // User being added/removed
    sig: Vec<u8>,        // ECDSA signature (65 bytes)
    role: u8,            // 0=participant, 1=admin (default 0)
}
}

Admin Signature Format

Canonical 53-byte message: chat_id[32] || target[20] || op_type[1]

op_type values: Add=0, Remove=1, Create=2, TransferOwnership=3, DeleteGroup=4. For DeleteGroup target carries no protocol meaning; clients conventionally sign their own address.

Signature: sign(keccak256(canonical_message)) -- standard Ethereum ECDSA with recovery (r[32] || s[32] || v[1]).

verify_membership_sig in crates/crypto recovers the signer address for caller to verify against admin.

group_chat_id Computation

#![allow(unused)]
fn main() {
fn group_chat_id(creator: [u8; 20], nonce: &[u8; 16], title: &str) -> [u8; 32] {
    BLAKE3("p2p-mes:chat:group:v2:" || creator[20] || nonce[16] || title_utf8)
}
}

• nonce is 16 bytes of crypto-random data generated client-side; the fixed length keeps the preimage unambiguous without length prefixes and guarantees unique chat_ids under concurrent creation by the same creator.
• title is the group's creation name, hashed as raw UTF-8 bytes with no Unicode normalization -- the client must sign over a chat_id computed from the exact bytes it submits. An unnamed group contributes zero title bytes.
• Because the creator signs the chat_id and the chat_id commits to the title, the title is authenticated end-to-end and immutable.

The trio (creator, nonce, title) is stored in chats_meta on every node (see STORAGE.md) so it can be re-served to peers that missed the Create -- each receiver re-verifies the hash before accepting it.

CRDT add-wins Semantics

Membership uses add-wins conflict resolution:

• Each member entry has added_at timestamp for ordering
• If both add and remove exist for the same user at the same timestamp, add wins
• add_member_crdt only skips if existing added_at is strictly greater
• Removed members (key deleted) can always be re-added

This ensures convergence across nodes: all nodes applying the same set of membership messages in any order will reach the same state.

PutIdentity

Propagates a user identity blob to all nodes. Uses last-write-wins semantics under HLC.

#![allow(unused)]
fn main() {
struct PutIdentity {
    user: [u8; 20],     // User address
    blob: Vec<u8>,      // Opaque identity blob (max 1024 bytes)
    hlc: HlcTimestamp,  // Server-side HLC stamp of the originating write
    origin: String,     // PeerId of the originating node
}
}

Flow:

1. HTTP PUT /identity reaches the MPSC handler, which stamps hlc via the node's HlcState and sends DbOp::PutIdentity to the async DB writer
2. DB writer applies HLC-LWW, updates seen_identity sync index, notifies Merkle tree
3. MPSC handler publishes PutIdentity to p2p-mes/commands (real-time gossip)
4. All nodes receive; gossip handler first calls HlcState::receive(msg.hlc) -- the incoming HLC is rejected if it exceeds local now by more than DEFAULT_MAX_DRIFT_MS, otherwise the local clock advances -- then routes through DbOp::PutIdentity (same HLC-LWW pipeline)
5. If incoming.hlc > stored.hlc (or no stored value): overwrite with [hlc_packed:u64be:8][blob]
6. If incoming.hlc <= stored.hlc: silently discard (stale delivery)

Last-write-wins: hlc is stamped server-side by the originating API node's HlcState; clients never specify time. The client signature scheme is unchanged.

Merkle tree sync: Identity is included in the anti-entropy protocol (domain Identity). Nodes that were offline during gossip will receive the data via Merkle-tree sync; the receiver routes each SyncIdentityRecord through the same HLC-LWW pipeline, so resurrecting an older blob is impossible.

MessageOp

Edits or deletes an already-stored message. The design never mutates the synced set: an operation is an append-only record, and the message row clients read is a materialized projection of (original message + all its ops). Because the record set only grows and the projection is computed by an order-independent rule, a node that is behind can never resurrect an old version.

#![allow(unused)]
fn main() {
struct MessageOpRecord {
    target: [u8; 32],      // msg_id of the message being changed
    chat_id: [u8; 32],     // chat the target belongs to
    editor: [u8; 20],      // must equal the target's original sender
    op_kind: u8,           // 0 = Edit, 1 = Delete
    new_text: String,      // replacement text (empty for Delete)
    hlc: HlcTimestamp,     // server-stamped by the originating node
    sig: Vec<u8>,          // 65-byte client ECDSA signature
}
}

The record is used verbatim as both the gossip payload and the anti-entropy wire record -- there is no separate encoding, so the two paths can never drift.

Conflict resolution (the fold). Applying operations to a message is an order-independent rule, so every node converges regardless of delivery order:

• an operation applies only if editor == target.sender, op_kind is known, and msg.hlc < op.hlc <= msg.hlc + 48h (the edit window);
• a Delete is terminal -- once a message is deleted, no later Edit revives it;
• otherwise an Edit wins only if its hlc beats the edit currently in effect (ties broken by op_id bytes).

All comparisons use stamps carried inside the records, never a local clock, so the verdict is identical on every node.

Signature -- delete only. A delete carries a client ECDSA signature over keccak256(chat_id || target || op_kind || blake3("")); every node independently recovers the signer and requires it to equal editor. As with MembershipOp, the hlc is not signed (the node stamps it server-side).

An edit carries no signature (sig is empty). This is the same asymmetry as plain messages, and for the same reason: an edit writes new content, and content authenticity is the recipient's job, verified end-to-end on Layer 2 (the ciphertext is a sign-then-encrypt envelope -- see CLIENT_GUIDE.md). The node only relays it; a forged edit yields an envelope the recipient cannot verify and rejects. At the node, an edit is authorized exactly like a send: the originating node checks editor == target.sender against the authenticated request (X-Sig).

A delete is different in kind and is the one operation the node must authorize itself. It produces the absence of content: a recipient sees only a deleted: true stub and cannot tell "the author deleted this" from "a node erased it without permission" -- there is nothing left to authenticate end-to-end. So the delete is signed, the signature travels with the operation, and every node re-verifies it (a peer is never trusted to have checked). Blake3-hashing the empty text keeps the payload fixed-length and identical in shape to a would-be signed edit, so the scheme stays general even though only deletes are signed today.

Flow:

1. HTTP PATCH/DELETE /messages/{msg_id} reaches the MPSC handler, which loads the target, checks authorship and the 48h window, stamps hlc, routes DbOp::MessageOp to the DB writer, then publishes MessageOp to p2p-mes/commands (live path)
2. The DB writer records the op in CF message_ops, updates the seen_op sync index and the ops_by_chat feed, and -- if the target is stored locally and the op wins the fold -- rewrites the message row in place, all in one WriteBatch
3. Gossip receivers feed hlc through HlcState::receive, re-verify the signature, then route the same DbOp::MessageOp. Dedup by op_id makes double delivery (gossip + sync) idempotent
4. If the op arrives before the message it targets (normal during anti-entropy backfill), it simply waits in message_ops; the later-arriving message folds it in on write via fold_ops_for_message

Two delivery channels. Gossip is the live path (online nodes apply within milliseconds); the MessageOps anti-entropy domain is the backstop for nodes that were offline when the op was published. Neither alone suffices -- gossip is fast but lossy, sync is reliable but runs only every few sync ticks -- and double delivery is safe because both converge on one DbOp::MessageOp behind the seen_op dedup. See SYNC.md for the domain and RETENTION.md for how aged ops are reclaimed.

Materialized row. After an Edit the message's text is replaced and edited_at is set; after a Delete text is cleared and deleted is set, but the row survives as a stub (same key, same msg_id) so pagination stays stable and clients can evict cached copies. Readers never fold ops themselves -- they see a plain list of messages.

Deprecated and disabled variants

These variants remain in the GossipMessage enum for CBOR backward compatibility but are not part of the active protocol. New node implementations neither send nor process them, and clients never see them.

InboxFanout / BatchedInboxFanout (disabled)

Disabled under full replication. Because every node stores all messages, each node updates user inboxes locally inside process_db_op right after writing PutMessage, so explicit gossip fanout is unnecessary. If sharding is ever re-introduced (where the node storing a message may differ from the node owning a user's inbox), these would need to be re-enabled.

#![allow(unused)]
fn main() {
struct InboxFanout {
    users: Vec<[u8; 20]>,       // All chat members (targets for inbox update)
    chat_id: [u8; 32],
    kind: ChatKind,
    last_sender: [u8; 20],
    last_ts: u64,
    last_seq: u32,
    last_msg_id: [u8; 32],
}

struct BatchedInboxFanout {
    items: Vec<InboxFanout>,    // Up to 100 items per batch
}
}

Ack (deprecated)

Legacy acknowledgment for PutMessage. No longer used -- all messages are fire-and-forget (needs_ack = false).

#![allow(unused)]
fn main() {
struct Ack {
    msg_id: [u8; 32],
    from: String,
    response: PutMessageResponseRaw,
}
}

CBOR Compatibility Rules

When modifying gossip message types:

• Add new fields with #[serde(default)] -- safe, backward compatible
• Never remove fields -- old nodes will fail to deserialize
• Never change enum tag names or numbering
• Never change field types
• New enum variants are safe -- unknown variants cause deserialization errors, but senders should be updated first

msg_id Computation

#![allow(unused)]
fn main() {
fn compute_msg_id(chat: &[u8], sender: &[u8], hlc: HlcTimestamp, text: &str) -> [u8; 32] {
    BLAKE3(chat || sender || hlc.to_packed().to_be_bytes() || text.as_bytes())
}
}

Same inputs always produce the same msg_id, enabling idempotent storage across nodes. The packed HLC is hashed as 8 big-endian bytes -- same shape as the legacy ts: u64, so the resulting digest stays 32 bytes.

msg_id is a stable identity, not a live content hash. It is computed once from the original text and never recomputed. Once a message is edited its stored text no longer hashes to its msg_id, and that is intentional: the id stays fixed so dedup, the messages Merkle tree, and every cross-node reference keep working. The node has never re-derived msg_id from stored content (messages carry no client signature; trust sits at the API edge), so nothing depends on the former msg_id == BLAKE3(current text) coincidence. Edits and deletes travel as MessageOp records in their own sync domain instead of perturbing the messages tree.

op_id Computation

#![allow(unused)]
fn main() {
fn compute_op_id(
    target: &[u8; 32], editor: &[u8; 20], op_kind: u8,
    hlc: HlcTimestamp, new_text: &str,
) -> [u8; 32] {
    BLAKE3("p2p-mes:msgop:v1:" || target || editor || [op_kind]
           || hlc.to_packed().to_be_bytes() || new_text.as_bytes())
}
}

The deterministic id of a MessageOp, used for dedup (seen_op) and as the MessageOps Merkle leaf. The signature is deliberately excluded: it is not guaranteed byte-identical across signers of the same payload, and hashing it in would let re-signing mint a second id for one logical operation.

DM chat_id Computation

#![allow(unused)]
fn main() {
fn dm_chat_id(a: [u8; 20], b: [u8; 20]) -> [u8; 32] {
    let (lo, hi) = if a <= b { (a, b) } else { (b, a) };
    BLAKE3("p2p-mes:chat:dm:v1:" || lo || hi)
}
}

Deterministic: both participants compute the same chat_id. No membership storage needed for DMs.

Identify-Based API Discovery

Nodes advertise their public HTTP API endpoint to peers through the libp2p identify protocol -- not through gossip. Identify metadata is exchanged automatically on every connection, so the announcement costs no extra network traffic and needs no dedicated message type.

Wire format. A node with public_api_url set in its config embeds the URL into the identify agent_version string as a whitespace-delimited token:

p2p-mes/1.0.0 api=https://node1.example.com:8080

Nodes without a public_api_url send a plain p2p-mes/1.0.0.

Receiving side. On each identify Received event the node parses the peer's agent_version (crates/node/src/handlers/kad/identify.rs):

• A well-formed api=<url> token (http/https scheme, <= 256 bytes) is recorded in the in-memory API registry (PeerId -> URL)
• A missing or malformed token removes any previous registry entry for that peer (the peer may have restarted without a public_api_url)
• The entry is evicted when the last connection to the peer closes

Presence in the registry therefore means "connected right now" -- no TTL or liveness probing is needed. The registry is ephemeral state: it is never persisted, never synced, and rebuilds itself from identify exchanges within seconds of a restart.

Trust. The URL is self-reported by the peer over an authenticated libp2p connection (the transport proves the sender owns its PeerId). Consistent with the trust model (see PHILOSOPHY.md), a dishonest node could advertise an arbitrary URL; the network assumes honest replicas.

Client surface. The registry backs the public HTTP endpoint GET /network/nodes (see API.md), which clients use for bootstrap and failover instead of hardcoding node URLs.

Sync Protocol

Overview

Anti-entropy synchronization between peers using Merkle trees over four data domains: messages, members, identity, and message-ops (edit/delete). Ensures eventual consistency: if two nodes have different data sets, the sync protocol detects and resolves the difference.

Transport: libp2p request_response::Behaviour over /p2p-mes/sync/1.0.0 Serialization: Length-prefixed CBOR (4-byte big-endian length + CBOR payload) Max message size: 16 MB

Merkle Tree Structure

Level 0 (Root):    1 node         root = BLAKE3(level1[0..256])
Level 1:         256 nodes        level1[i] = BLAKE3(leaves[i*256 .. i*256+256])
Level 2 (Leaves): 65536 buckets   leaf[i] = XOR of all msg_ids in bucket i

Bucket Assignment

bucket_index = first 2 bytes of msg_id as big-endian u16

Each leaf is an XOR accumulator: commutative and associative, so insertion order does not matter.

Memory Footprint

Fixed ~2.1 MB regardless of message count:

• 65536 leaves x 32 bytes = 2 MB
• 256 L1 nodes x 32 bytes = 8 KB
• 1 root x 32 bytes = 32 bytes

Startup Rebuild (O(1) memory)

Each domain has its own seen-index CF (seen_msg, seen_member, seen_identity). At startup, all three trees are rebuilt by streaming scans:

#![allow(unused)]
fn main() {
// Messages
let mut tree_msgs = MerkleTree::new_empty_leaves();
db::messages::for_each_msg_id(&db, |id, _ts| tree_msgs.xor_leaf(&id));
tree_msgs.recompute_all();

// Members
let mut tree_members = MerkleTree::new_empty_leaves();
db::members::for_each_member_record_id(&db, |id| tree_members.xor_leaf(&id));
tree_members.recompute_all();

// Identity
let mut tree_identity = MerkleTree::new_empty_leaves();
db::identity::for_each_identity_record_id(&db, |id| tree_identity.xor_leaf(&id));
tree_identity.recompute_all();
}

No Vec allocation -- O(1) memory per tree.

Incremental Update

On each new message stored:

#![allow(unused)]
fn main() {
tree.insert(&msg_id);
// 1. XOR msg_id into leaf[bucket_index]
// 2. Recompute level1[bucket_index / 256] = BLAKE3(256 leaves)
// 3. Recompute root = BLAKE3(256 L1 nodes)
// Cost: 1 XOR + 2 BLAKE3 hashes
}

XOR Properties

• Commutative: A XOR B = B XOR A -- order doesn't matter
• Associative: (A XOR B) XOR C = A XOR (B XOR C) -- grouping doesn't matter
• Self-inverse: A XOR A = 0 -- double insert cancels out (important pitfall!)
• Identity: A XOR 0 = A

Sync Domains

Three independent Merkle trees track three data domains:

Each domain uses the same 5-step protocol and the same MerkleTree struct (65536 buckets). The SyncDomain enum in SyncRequest/SyncResponse distinguishes them on the wire.

Domain: Messages

Every successful DbOp::PutMessage write sends msg_id to merkle_msg_tx. Append-only -- msg_ids are never removed from the tree by the live write path; retention GC removes them out-of-band via xor_batch (see RETENTION.md).

Messages carry HLC: the messages CF key embeds hlc.to_packed().to_be_bytes() in the 8-byte slot that used to hold ts: u64, and msg_id itself is BLAKE3(chat || sender || hlc_packed_be || text). A separate origin_wall_ts: u64 field on MsgV1 is the frozen sender wall-clock used purely for UI display; it never participates in storage ordering or sync.

Includes: regular messages, control messages (msg_type != 0), accompanying messages from compound membership operations.

Domain: Members

DbOp::MembershipOp writes to members CF using add_member_synced / remove_member_synced, which atomically update the seen_member sync index and send MerkleUpdate to merkle_member_tx. Under HLC semantics, Remove never physically deletes the record -- it advances removed_at in place -- so every state transition is a MerkleUpdate::Replace { old, new }. MerkleUpdate::Remove is no longer emitted by the members pipeline (the variant is kept for future use).

DbOp::ApplyMemberRecord is the sync-side counterpart: it carries the full peer MemberInfo and routes through apply_member_record_synced, which performs the monotonic CRDT merge (max(added_at), max(removed_at), role from the dominant side) before emitting the matching MerkleUpdate. This is how anti-entropy converges on tombstoned state without resurrecting removed members.

Group-creation-meta piggyback. The wire record (SyncMemberRecord) has an optional group_meta field carrying the group's immutable creation trio {title, creator, nonce}:

• Sender side (get_member_records_batch): the trio is attached only to the owner record (role=2) of a chat whose creation meta is locally known and titled -- exactly one carry per chat per batch, so member-heavy chats never duplicate the title N times.
• Receiver side (store_synced_members): the carry is self-verifying -- the receiver recomputes chat_id = blake3(domain || creator || nonce || title) and silently drops the carry on mismatch (the member record itself is still applied). A verified carry queues DbOp::SetGroupCreateMeta (write-once into chats_meta).
• Why this converges: a node is missing the title exactly when it missed the group's Create op, which also means it lacks the owner member record -- so the members Merkle roots diverge and the next sync round delivers the owner record together with the carry.
• group_meta is not part of the member record_id: the Merkle tree hashes membership state only, and the carry is opportunistic transport. Including it would change every existing record_id and force a full network resync.

Domain: Identity

DbOp::PutIdentity writes to identity CF using put_identity, which atomically updates the seen_identity sync index and sends MerkleUpdate to merkle_identity_tx. LWW semantics under HLC -- an incoming write is applied only if incoming.hlc > stored.hlc. Updates replace the old record_id via MerkleUpdate::Replace. The gossip PutIdentity handler additionally calls HlcState::receive on the incoming HLC before forwarding, so the local clock stays in line with the network and out-of-bound drift is rejected at the edge.

Domain: MessageOps

DbOp::MessageOp writes to CF message_ops via apply_message_op, which atomically records the operation, updates the seen_op sync index and the ops_by_chat feed, materializes the change into the target message row when present, and -- on a genuinely new op (seen_op dedup) -- sends the raw op_id to merkle_op_tx. Append-only like Messages: an operation is immutable once published, so this is a plain Insert, never a Replace. Retention GC removes aged ops out-of-band via xor_batch (see RETENTION.md), the same way it trims messages.

The stored MessageOpRecord is the sync wire record unchanged -- no re-encode step -- so get_op_records_batch streams the CBOR straight from the primary CF. On receipt, store_synced_ops (in crates/node/src/sync/handler.rs) independently re-checks each record before it reaches the DB writer:

• id match: record_op_id(record) must equal the op_id the diff asked for, else the peer answered with something else;
• retention: ops with hlc.physical_ms <= cutoff_ts are dropped (soft-filter, below);
• signature (delete only): a delete's client signature is re-verified from scratch and the recovered address must equal editor -- peers are never trusted to have checked. An edit carries no signature (its content authenticity is the recipient's Layer 2 concern), so it is accepted like a replicated plain message; a peer that altered the edit in flight is still caught by the id-match check above (the text is part of op_id). See PROTOCOL.md for why only deletes are signed.

Whether the editor may actually change the target message (authorship, edit window, delete-is-terminal) is decided later by the fold in db::message_ops, which needs the target row that this node may not hold yet. An op that arrives before its target simply waits in message_ops and is folded in when the message is stored.

Why append-only converges without resurrection. An edited message's stored text no longer hashes to its msg_id, so replicated message rows are shipped verbatim (DbOp::ApplySyncedMessage) rather than re-authored -- re-deriving the id would fork the message's identity and the messages trees could never converge. The edit/delete itself rides this separate append-only domain, so the messages tree is never perturbed by a mutation and the "deleted message never resurfaces" property falls out of the grow-only op set plus the order-independent fold.

What is NOT in any Merkle tree

• user_inbox CF entries (derived locally from PutMessage in process_db_op)
• chats_meta CF entries (updated as side effect of put_message; the group-creation trio inside it is transported by the members-domain piggyback described above, not diffed directly)
• user_read_progress CF entries (propagated via ReadProgress gossip)

Retention Soft-Filter

The Messages domain applies a time-based filter on both sides of the sync exchange so that aged-out records never propagate between peers even when local GC is out of phase.

Each side computes cutoff_ts = now_ms - RETENTION_WINDOW locally (retention::cutoff_ts_now()); nothing about retention is exchanged on the wire.

Responder side

SyncDomain::Messages dispatches to filtered DB helpers instead of the plain ones:

• Bucket IDs: get_bucket_msg_ids_filtered(db, bucket, cutoff_ts)
• Fetch payloads: get_messages_cbor_batch_filtered(db, ids, max_bytes, cutoff_ts)

Filtering reads the packed HLC from the seen_msg value (bytes 32..40) and extracts physical_ms via extract_hlc_physical_from_seen_value; the comparison against cutoff_ts stays a single millisecond test (no extra DB lookup).

SyncDomain::MessageOps applies the same soft-filter: both get_bucket_op_ids and get_op_records_batch take a cutoff_ts and drop ops whose hlc.physical_ms <= cutoff_ts, read from the seen_op value's trailing 8-byte stamp. Members and Identity domains are not filtered.

Receiver side

store_synced_messages in crates/node/src/sync/handler.rs decodes each incoming MsgV1, checks msg.hlc.physical_ms() > cutoff_ts, and silently drops aged ones before they reach DbOp::ApplySyncedMessage. Each rejection increments the sync_messages_rejected_total Prometheus counter.

store_synced_ops applies the mirror check for the MessageOps domain (record.hlc.physical_ms() > cutoff_ts), alongside the id-match and signature re-checks described above; drops increment sync_ops_rejected_total.

The receiver re-check defends against clock skew, mismatched RETENTION_WINDOW between nodes, and malicious peers that bypass the responder filter.

Why both sides

The responder filter alone is enough for well-behaved peers, but two-sided enforcement makes "deleted messages never resurface" a property of every node independently, not a property that requires trusting every peer in the mesh. See RETENTION.md for the full retention design.

Sync State Machine (5 Steps)

Initiated every sync_interval_secs (default 30s, configurable via AppConfig) with a random connected peer. Domains are selected round-robin over ALL_SYNC_DOMAINS: tick 0 = Messages, tick 1 = Members, tick 2 = Identity, tick 3 = MessageOps, tick 4 = Messages, etc. Each domain syncs roughly every 4 * sync_interval_secs.

Step 1: Root Exchange

Initiator --> RootExchange { root, msg_count }
Responder --> RootResult { root, msg_count, in_sync }

If in_sync == true: done, trees are identical.

Step 2: Level-1 Exchange

Initiator --> Level1Exchange { hashes: Vec<[u8; 32]> }   // 256 L1 hashes
Responder --> DifferingL1 { indices: Vec<u8>, hashes: Vec<[u8; 32]> }

Compares 256 L1 hashes. Returns indices where they differ, plus responder's hashes for those indices.

Step 3: Leaf Exchange

Initiator --> LeafExchange { l1_indices, hashes }
  // hashes = 256 leaves per l1_index, concatenated
Responder --> DifferingLeaves { buckets: Vec<u16> }
  // absolute bucket index = l1_idx * 256 + leaf_offset

Drills down into differing L1 nodes, comparing individual leaf buckets.

Step 4: Bucket IDs

Initiator --> BucketIds { buckets: Vec<(u16, Vec<[u8; 32]>)> }
  // For each differing bucket: all msg_ids in that bucket
Responder --> BucketDiff { a_missing, b_missing }
  // a_missing = IDs responder has but initiator doesn't
  // b_missing = IDs initiator has but responder doesn't

Uses HashSet for O(1) set difference computation. Reads msg_ids from CF seen_msg using 2-byte prefix iteration.

Input Validation (DoS Protection)

All responder handlers validate incoming vector sizes before processing. Oversized payloads from malicious peers are rejected with a synthetic RootResult { in_sync: true } that terminates the session.

Constants defined in crates/node/src/sync/handler.rs.

Step 5: Fetch and Push

Initiator --> FetchAndPush { fetch: Vec<[u8; 32]>, push: Vec<(msg_id, cbor)> }
  // fetch = msg_ids initiator needs from responder
  // push  = full CBOR messages responder needs from initiator
Responder --> Messages { messages: Vec<(msg_id, cbor)>, has_more: bool }
  // messages = data initiator requested

Bidirectional data exchange:

• Initiator pushes messages that responder is missing
• Responder pushes messages that initiator is missing
• Both sides store via normal DbOp::PutMessage pipeline (dedup via seen_msg)

Chunking: if total CBOR bytes exceed 1 MB, has_more = true and initiator sends another FetchAndPush for remaining IDs.

Session Management

#![allow(unused)]
fn main() {
struct SyncManager {
    sessions_by_request: HashMap<OutboundRequestId, SyncSession>,
    peer_to_request: HashMap<PeerId, OutboundRequestId>,
    timeout_secs: u64,          // 60 seconds
}

struct SyncSession {
    peer: PeerId,
    domain: SyncDomain,
    started_at: Instant,
    state: SyncState,
    request_id: Option<OutboundRequestId>,
}

enum SyncState {
    WaitingForRoot,
    WaitingForL1,
    WaitingForLeaves { differing_l1: Vec<u8> },
    WaitingForBucketDiff { differing_buckets: Vec<u16> },
    WaitingForMessages { pending_fetch: Vec<[u8; 32]> },
    Complete,
}
}

• One session per peer at a time
• Sessions timeout after 60 seconds
• Cleanup runs before each sync tick (every 30s)

Wire Format

[4 bytes: big-endian u32 length][N bytes: CBOR payload]

Read:

#![allow(unused)]
fn main() {
let len = u32::from_be_bytes(read_exact(4));
if len > 16MB: error
let buf = read_exact(len);
let msg = serde_cbor::from_slice(&buf);
}

Write:

#![allow(unused)]
fn main() {
let data = serde_cbor::to_vec(&msg);
write_all((data.len() as u32).to_be_bytes());
write_all(data);
close();
}

Protocol Messages Reference

All request and response variants carry a domain: SyncDomain field (#[serde(default)] = Messages for backward compat with old nodes).

SyncRequest (Initiator -> Responder)

SyncResponse (Responder -> Initiator)

Key Pitfall: Double XOR Cancellation

Never send the same record_id to the Merkle tree twice from the same write path. Since A XOR A = 0, a double insert effectively removes the record from the tree.

Messages: Merkle update only happens when put_message returns seq > 0 (not a duplicate).

Members/Identity (mutable domains): Updates use MerkleUpdate::Replace { old, new } which XOR-cancels the old record_id and XOR-inserts the new one. The old record_id is recomputed from the current DB state before overwriting, so no separate reverse index is needed. Removes use MerkleUpdate::Remove(old).

The DB writer is the single source of truth for all three Merkle trees.

Cryptography

Overview

The project uses Ethereum-compatible cryptographic primitives for identity and authentication. Message integrity is handled by BLAKE3 hashing. The node's P2P identity uses secp256k1 keys.

Algorithms

Address Derivation (Ethereum-style)

Private Key (32 bytes, secp256k1)
  |
  v
Public Key (uncompressed, 65 bytes: 0x04 || X || Y)
  |
  v (drop first byte 0x04)
Keccak256(pubkey[1..65])  -->  32 bytes
  |
  v (take last 20 bytes)
Address = hash[12..32]    -->  20 bytes

This is identical to Ethereum address derivation. Addresses are displayed as 0x-prefixed hex strings (42 characters).

ECDSA Signature Verification

Implementation in crates/crypto/src/lib.rs:

#![allow(unused)]
fn main() {
fn verify_sig_recover(
    string_to_sign: &str,
    sig_hex: &str,          // 65 bytes: r[32] || s[32] || v[1]
    claimed_addr_hex: &str, // 0x-prefixed 20-byte address
) -> Result<(), String>
}

Process

1. Compute msg_hash = Keccak256(string_to_sign.as_bytes())
2. Parse 65-byte signature: r || s || v
   • Normalize v: if v >= 27, subtract 27 (Ethereum convention)
3. Try ECDSA recovery with provided v, then with 1 - v (tolerance for incorrect recovery ID)
4. For each recovered public key:
   • Convert to uncompressed SEC1 format (65 bytes)
   • Compute addr = Keccak256(pubkey[1..])[12..32]
   • Compare with claimed address using constant-time comparison (subtle::ConstantTimeEq) to prevent timing side-channel attacks
5. Return Ok(()) if any attempt matches, otherwise Err

Signature Format

[r: 32 bytes][s: 32 bytes][v: 1 byte]
Total: 65 bytes, hex-encoded in X-Sig header (130 hex chars)

v values: 0 or 1 (or 27/28 in Ethereum convention -- both accepted).

Canonical String-to-Sign

Built by crates/api/src/utils.rs::canonical::build_string_to_sign:

p2p-mes-v1
METHOD:{METHOD}
PATH:{path}
QUERY:{canonical_query}
BODY:{canonical_body}
TS:{timestamp_ms}
NODE:{peer_id_base58}

Canonicalization

Query parameters:

1. Parse URL query string
2. Sort pairs by (key, value)
3. Percent-encode each key and value (NON_ALPHANUMERIC charset)
4. Join with &: key1=value1&key2=value2

JSON body:

1. Parse JSON
2. Flatten to dot-notation: {"a": {"b": 1}} -> a.b=1
3. Arrays use [] suffix: {"items": [1,2]} -> items[]=1&items[]=2
4. An array element that is itself an object (or a nested array) is not flattened further: the element becomes a single pair whose value is its compact JSON serialization -- no whitespace, object keys sorted alphabetically at every depth: {"ops": [{"b": 1, "a": "x"}]} -> ops[]={"a":"x","b":1}. The JSON string is then percent-encoded like any other value (step 6). POST /groups/{chat_id}/ops is the main endpoint this applies to; the group-ops entry in the published test-vectors.json gives a byte-exact reference
5. Sort pairs by (key, value)
6. Percent-encode and join

Form body: same as query params

Binary/other: raw={hex_of_body}

Empty body/query: empty string (no pairs)

Public Utility Functions

#![allow(unused)]
fn main() {
/// Keccak-256 hash of arbitrary bytes.
fn keccak256(bytes: &[u8]) -> [u8; 32]

/// Parse hex string (with or without 0x prefix) into 20-byte address.
/// Returns None for invalid hex or wrong length.
fn parse_addr20(s: &str) -> Option<[u8; 20]>
}

Message ID (msg_id)

Deterministic hash for idempotent message storage. The node computes it -- the hlc stamp is assigned server-side by the originating node, so clients never compute msg_id themselves (they receive it in the HTTP response):

#![allow(unused)]
fn main() {
fn compute_msg_id(chat: &[u8], sender: &[u8], hlc: HlcTimestamp, text: &str) -> [u8; 32] {
    BLAKE3(chat || sender || hlc.to_packed().to_be_bytes() || text.as_bytes())
}
}

hlc.to_packed() is a u64 (48 bits physical milliseconds + 16 bits logical) hashed as 8 big-endian bytes -- the same shape as the legacy ts: u64 it replaced, so the digest stays 32 bytes. Same inputs always produce the same msg_id across all nodes. See PROTOCOL.md and TYPES.md for HlcTimestamp.

DM Chat ID

Deterministic chat identifier for direct messages:

#![allow(unused)]
fn main() {
fn dm_chat_id(a: [u8; 20], b: [u8; 20]) -> [u8; 32] {
    let (lo, hi) = if a <= b { (a, b) } else { (b, a) };
    BLAKE3("p2p-mes:chat:dm:v1:" || lo || hi)
}
}

Both participants compute the same chat_id regardless of who sends first. No server coordination or membership storage needed.

Node Identity

Nodes use secp256k1 keypairs for libp2p identity:

• Private key: 32 bytes, specified in TOML config as hex
• PeerId: derived from public key, displayed as Base58
• CLI command to derive PeerId: cargo run -p node -- peer-id 0x<64hex>

Group Chat ID

#![allow(unused)]
fn main() {
fn group_chat_id(creator: [u8; 20], nonce: &[u8; 16], title: &str) -> [u8; 32] {
    BLAKE3("p2p-mes:chat:group:v2:" || creator || nonce || title_utf8)
}
}

The client generates a random 16-byte nonce, picks an optional group title, computes the chat_id, and includes both nonce and title in the Create op request. Every verifier (HTTP API, MPSC handler, gossip receivers, sync piggyback) recomputes the hash and rejects a mismatch.

The title is hashed as raw UTF-8 bytes with no Unicode normalization -- signer and verifier must agree on the exact byte sequence. An unnamed group hashes zero title bytes. Because the creator's membership signature covers the chat_id and the chat_id commits to the title, the title is authenticated transitively and is immutable for the lifetime of the group.

See PROTOCOL.md for the full compound creation flow.

Membership Signature Verification

#![allow(unused)]
fn main() {
fn verify_membership_sig(
    chat_id: &[u8; 32],
    target: &[u8; 20],
    op_type: u8,        // Add=0, Remove=1, Create=2
    sig_bytes: &[u8],   // 65 bytes: r[32] || s[32] || v[1]
) -> Result<[u8; 20], String>
}

Canonical 53-byte message: chat_id[32] || target[20] || op_type[1]

Hash: keccak256(canonical_message). Recover admin address from ECDSA signature. Tolerant to both raw v (0/1) and Ethereum v (27/28).

Known Limitation: ts/HLC is not cryptographically authoritative

The canonical message above deliberately excludes the HLC stamp (and the legacy ts: u64 it replaced). Clients have no access to node-level HLC state, so the originating API node stamps hlc on behalf of the client immediately after signature verification. This keeps the client signature compact and means clients never specify time.

Consequence: a peer that controls a gossip pipe could rewrite the hlc field on a relayed MembershipOp (or any PutMessage / PutIdentity) without invalidating the client signature, biasing CRDT decisions. The current trust model treats API nodes as trusted relays, so we accept this gap; CRDT-level merging is the only defence today.

Closing this gap is a separate workstream (node-level identity + keypair, sign every op at gossip publish), explicitly out of scope for the HLC migration. Until that lands, do not deploy nodes you do not trust to honestly forward HLC stamps.

Merkle Tree Hashing

See SYNC.md for details. Uses BLAKE3 for:

• Level-1 nodes: BLAKE3(256 leaf values concatenated)
• Root: BLAKE3(256 L1 values concatenated)
• Leaf values are XOR accumulators (not hashes)

Building a Client

This guide walks through everything needed to build an interoperable p2p-mes client: how to authenticate, how to derive chat identifiers, the message lifecycle, and how to layer end-to-end encryption on top. For the exact schema of every endpoint, see the API Overview and the generated API Reference; for the design rationale behind these choices, see Design Philosophy.

A client never talks to the peer-to-peer network directly. It speaks HTTP to a single node, which signs nothing on the client's behalf except the network clock (see the trust note below). Everything else -- authorship, chat membership, encryption -- is the client's responsibility.

Prerequisites

To talk to a node you need three things:

• An ECDSA secp256k1 keypair. The user's identity (their address) is derived from the public key exactly as in Ethereum.
• The node's HTTP API base URL (for example http://localhost:3000).
• The node's PeerId (Base58). Every request is bound to a specific node, so the client must know which node it is addressing.

Only the first is a real secret you must provision. The other two are discoverable at runtime: GET /node/info returns any node's PeerId, and GET /network/nodes returns the URLs of other live nodes (next section).

Node discovery and failover

Do not hardcode a single node URL. Nodes are full replicas that can be added or retired at any time, so a client should treat any one URL only as an entry point and learn the rest of the network from it.

Two endpoints support this, and neither requires signed headers -- they work before you have signed anything:

• GET /node/info -- the node's own peer_id (needed for the X-Node header of every signed request) and its advertised api_url.
• GET /network/nodes -- a catalog of live API-serving nodes: { "nodes": [ { "peer_id": ..., "api_url": ... }, ... ] }. The answering node includes itself. Every listed node was connected to the answering node at response time, so the entries are live.

The recommended bootstrap flow:

1. Ship the client with one or more seed URLs (any known nodes).
2. On startup, call GET /network/nodes on a seed and cache the returned catalog locally.
3. Pick a node, fetch its peer_id via GET /node/info (or from the catalog entry), and use that value as X-Node in signed requests.
4. If the node stops responding, switch to another cached entry and refresh the catalog from it.

Two caveats. First, the catalog is the answering node's view of its direct connections, not a global census -- different nodes may return slightly different (overlapping) lists; any of them is enough for failover. Second, remember that every signed request embeds the target node's PeerId: after switching nodes, sign subsequent requests with the new node's peer_id in X-Node.

Identity and addresses

A user is identified by a 20-byte address derived from their public key:

address = keccak256(uncompressed_pubkey[1..])[12..32]   // last 20 bytes

This is identical to Ethereum address derivation. Addresses appear in the API as 0x-prefixed hex (42 characters). See Cryptography & Authentication for the full derivation.

Authentication: signing every request

Every endpoint requires a signature, except the two discovery endpoints (GET /node/info, GET /network/nodes) described above. The node verifies it by recovering the signer's address from the signature and comparing it to the X-User header -- so there is no session, token, or password. Each request is signed independently.

Required headers

What you sign

You do not sign the raw request bytes. You sign a canonical string built from the request, so the signature is stable regardless of JSON key order or whitespace. The string is:

p2p-mes-v1
METHOD:{UPPERCASE_METHOD}
PATH:{path}
QUERY:{canonical_query}
BODY:{canonical_body}
TS:{timestamp_ms}
NODE:{node_peer_id_base58}

canonical_query and canonical_body are produced the same way:

1. Reduce the input to a list of (key, value) pairs.
   • Query string: parse as URL-encoded pairs.
   • JSON body: flatten to dot notation -- {"a":{"b":1}} becomes a.b=1; arrays use a [] suffix -- {"t":[1,2]} becomes t[]=1, t[]=2. An array element that is itself an object is not flattened into dotted keys: the whole element becomes the pair's value, serialized as compact JSON with object keys sorted alphabetically at every depth -- {"ops":[{"b":1,"a":"x"}]} becomes the single pair ops[]={"a":"x","b":1} (see Signing group operations).
   • Empty body or query: the result is the empty string.
2. Sort the pairs by key, then value.
3. Percent-encode every key and every value with the NON_ALPHANUMERIC set -- that is, everything except A-Z a-z 0-9 is escaped, including ., -, _, and ~. The = and & joiners are not escaped.
4. Join as key1=value1&key2=value2.

The full normative rules (including form bodies and binary payloads) are in Cryptography & Authentication.

How you sign

1. Build the canonical string above.
2. msg_hash = keccak256(utf8_bytes(canonical_string)).
3. Produce a recoverable ECDSA signature over msg_hash.
4. Serialize as r[32] || s[32] || v[1] and hex-encode into X-Sig. The recovery byte v may be 0/1 or the Ethereum-style 27/28; both are accepted.

Worked example

Sending {"text":"Hello, world!"} as a DM. The canonical body is text=Hello%2C%20world%21 (comma, space, and ! are escaped). With no query string, the string to sign is:

p2p-mes-v1
METHOD:POST
PATH:/dialogs/0xabcdef1234567890abcdef1234567890abcdef12/messages
QUERY:
BODY:text=Hello%2C%20world%21
TS:1700000000000
NODE:12D3KooWExampleNodePeerId

Hash it with Keccak-256, sign, and send the signature in X-Sig.

Common pitfalls

• Signing raw bytes instead of the canonical string. Re-serialize through the canonicalization rules; do not hash the JSON you happened to send.
• Wrong timestamp unit. X-Ts is milliseconds, and the node rejects anything more than 30 seconds from its own clock.
• Wrong or missing X-Node. The node rejects requests addressed to a different PeerId.
• Aggressive percent-encoding. NON_ALPHANUMERIC escapes far more than a typical URL encoder; verify against the worked example.
• Dot-flattening objects inside arrays. {"ops":[{...}]} does not produce ops[].op_type=... pairs. Each element becomes one ops[] pair whose value is the element's compact JSON with alphabetically sorted keys. Getting this wrong rejects every POST /groups/{chat_id}/ops with 401 while flat-bodied requests keep working.

Reference test vectors

Signing is the easiest thing to get subtly wrong, so the site ships machine-readable vectors at test-vectors.json. Each entry pairs a request with its exact canonical_string, the Keccak-256 message_hash_keccak256, the resulting x_sig, and the full headers to send. They are produced from a fixed test key (0x1111...1111, address 0x19e7e376e7c213b7e7e7e46cc70a5dd086daff2a) using the same code the node verifies with, and a test re-checks every one.

To validate your client, replay a vector: rebuild the canonical string from its request, confirm it matches byte-for-byte, then hash, sign, and verify your signature recovers to the signer address. The POST /dialogs/{peer}/messages vector is the worked example above with its signature filled in. The POST /groups/{chat_id}/ops vector exercises an array-of-objects body -- replay it first if group requests return 401 while DMs work. Its per-op sig values are real signatures by the same test key, and its chat_id derives from the test address, the body's nonce and the body's title, so every layer of a group request can be checked against it.

Deriving chat identifiers

Chat IDs are 32 bytes and are computed by the client, not assigned by the server.

Direct messages -- derived from the two participant addresses, order- independent, so both parties compute the same value with no coordination:

chat_id = blake3("p2p-mes:chat:dm:v1:" || min(a,b) || max(a,b))

min/max are taken over the raw 20-byte addresses. No membership is stored for DMs: the ability to compute the ID is the access control.

Groups -- derived from the creator's address, a random 16-byte nonce the client generates at creation time, and the group title:

chat_id = blake3("p2p-mes:chat:group:v2:" || admin_address || nonce || title_utf8)

• nonce must be exactly 16 bytes.
• title is the group's name as raw UTF-8 bytes, no Unicode normalization -- hash the exact string you will send in the create request's title field. For an unnamed group, omit title from the request and hash zero title bytes.
• Both nonce and title are sent in the create operation, and every node re-verifies the derivation (the HTTP API rejects a mismatch with 400; gossip receivers drop forged ops).

Because your signature covers the chat_id and the chat_id commits to the title, the title is authenticated end-to-end -- and therefore immutable: renaming a group is not possible without creating a new one. The title comes back to you in GET /conversations (inside kind) and GET /groups/{chat_id}/members. See Cryptography & Authentication.

The message lifecycle

Sending is fire-and-forget. When you POST a message, the node validates and signs nothing further, publishes it to the gossip network, queues it for storage, and returns 200 before the write is durable. A success response means "accepted and broadcast by this node," not "durably replicated everywhere." Convergence across nodes happens asynchronously through anti-entropy sync.

Reading is served from the queried node's local store and returns immediately -- under full replication every node stores everything, so there is no network round-trip on read. A node that is still catching up simply returns its partial local view. Treat reads as eventually consistent; see Operational notes for client authors at the end of this guide for handling incompleteness, ordering, retries, and node selection.

Sending a direct message

POST /dialogs/0xPEER.../messages
X-User: 0xSENDER...
X-Ts: 1699900000000
X-Node: 12D3KooW...
X-Sig: 0x<130 hex chars>
Content-Type: application/json

{ "text": "Hello, world!" }

Response:

{
  "chat_id": "0x<32-byte hex>",
  "msg_id":  "0x<32-byte hex>",
  "ts": 1699900000000
}

Group messages are the same against POST /groups/{chat_id}/messages; the node rejects the send if the sender is not a member.

Reading history (and decoding messages)

GET /dialogs/{peer}/messages (and the group equivalent) accept from/to (millisecond bounds), limit (1-1000), a reverse flag, and an opaque after cursor. The response is a page of messages plus a next_after cursor:

{
  "items": [
    { "key": "0x<hex key>", "msg_cbor": "0x<hex-encoded CBOR>" }
  ],
  "next_after": "0x<opaque cursor>"
}

Things to note:

• Message bodies are returned as hex-encoded CBOR (msg_cbor). The client must hex-decode, then CBOR-decode, to obtain the message fields (sender, timestamp, text, msg_type, control). The message structure is documented in Gossip Protocol and Types.
• Cursors are opaque. Pass next_after back as after to fetch the next page; do not parse or construct cursors yourself. One identity is guaranteed: the cursor is the storage key of the page's last row, so every item's key is itself a valid after value -- that is what makes tail-polling possible after a cursorless page (next bullet).
• next_after comes back only on full pages -- and null does not mean "end of history". The node emits a cursor only when the page hit limit; a shorter or empty page carries next_after: null, meaning "end of this node's current local view":
  • items non-empty, next_after: null -- normal exhaustion: you have everything this node currently holds in the requested window. To keep following the tail, remember the key of the newest item and later re-poll forward (reverse omitted or false) with it as after -- or re-send the after you just used and dedup by key/msg_id.
  • items empty, next_after: null -- ambiguous. The window may be genuinely empty, or the node may not have the data yet: sends are fire-and-forget (200 returns before the write commits, so a message you just sent can be invisible for a few hundred milliseconds) and a node behind on sync serves its partial view. Never treat this page as proof of exhaustion: keep the cursor you used and retry when you have reason to expect data (you just got a 200, or /conversations shows unread > 0).
  • A non-null next_after promises nothing either: it means "this page was full", not "more rows exist" -- the next call may legitimately return an empty page.
• Direction is controlled by reverse. Omit it (or send false) for oldest-first paging from from. Send true for newest-first paging from to -- the natural chat UI: load the latest screen with ?reverse=true&limit=N, then keep scrolling up by feeding next_after back as after (still with reverse=true). Keep the flag stable across one pagination run; the after cursor is the same opaque key in both modes, only the direction of travel flips.

Deep-linking to one message: GET /messages/{msg_id}

To jump straight to a single message -- e.g. opening a chat from a push notification -- call GET /messages/{msg_id} instead of scanning a range. The response carries the same hex-encoded CBOR as a range item, so you reuse the decode path above:

{ "msg_id": "0x<32-byte hex>", "msg_cbor": "0x<hex-encoded CBOR>" }

The node authorizes you against the message: DM participants for DMs, current members for groups. If the id is unknown or you are not allowed to see it, you get 404 -- existence is never leaked to non-participants. From the decoded hlc/seq you can then page around the message with the range endpoint.

Reference: decoding msg_cbor

Each msg_cbor is the hex of a CBOR map. Decode in two steps: hex -> bytes, then CBOR -> fields. The keys and value types:

Gotcha: byte fields are CBOR arrays, not byte strings. msg_id, chat_id, sender (and control, when present) encode as CBOR arrays (major type 4) of u8 integers -- not CBOR byte strings (major type 2), because the wire format uses no serde_bytes annotation. A decoder that expects byte strings will fail to parse.

Gotcha: the kind tag is a text string. kind.t is the CBOR text string "0" (DM), "1" (group), or "2" (channel) -- not an integer. This falls out of serde's adjacently-tagged enum encoding and stays this way for compatibility with already-stored messages, so match on strings, not numbers.

Gotcha: kind.d.peer is not "the other side". For DMs, d.peer is the address of the original recipient, fixed by the sender at send time -- the same bytes for every reader. In a message you received, d.peer is therefore your own address. To find your interlocutor:

other = (d.peer == my_address) ? sender : d.peer

Do not confuse this with /conversations, where kind.peer is viewer-relative (the node rewrites each user's inbox entry to hold the other participant). A client that builds its chat list from wire messages using d.peer alone files every incoming DM into a "dialog with itself" -- and read markers posted to that derived peer target a chat that does not exist.

Worked example. A DM with text = "Hello, world!", msg_type = 0, and no control payload encodes as the following msg_cbor (also checked by cargo test -p db, test msg_cbor_reference_vector, so it cannot drift):

aa66736368656d6101666d73675f69649820111111111111111111111111111111111111111111111111111111111111111167636861745f69649820182218221822182218221822182218221822182218221822182218221822182218221822182218221822182218221822182218221822182218221822182218226673656e646572941833183318331833183318331833183318331833183318331833183318331833183318331833183363686c631b018bcfe5680000006e6f726967696e5f77616c6c5f74731b0000018bcfe56800637365710164746578746d48656c6c6f2c20776f726c6421686d73675f7479706500646b696e64a2617461306164a164706565729418441844184418441844184418441844184418441844184418441844184418441844184418441844

It decodes to:

• schema = 1
• msg_id = 0x1111...11 (32 bytes)
• chat_id = 0x2222...22 (32 bytes)
• sender = 0x3333...33 (20 bytes)
• hlc = 111411200000000000 (physical 1700000000000 ms, logical 0: 1700000000000 << 16)
• origin_wall_ts = 1700000000000
• seq = 1
• text = "Hello, world!"
• msg_type = 0
• kind = { "t": "0", "d": { "peer": 0x4444...44 } } -- a DM; note the string tag, and that 0x4444...44 is the recipient the sender addressed (sender is 0x3333...33), per the d.peer gotcha above

Reference decoder (Rust; any CBOR library works -- the keys are plain strings):

#![allow(unused)]
fn main() {
use serde::Deserialize;

#[derive(Deserialize)]
struct Message {
    schema: u8,
    msg_id: [u8; 32],
    chat_id: [u8; 32],
    sender: [u8; 20],
    hlc: u64,            // packed: (physical_ms << 16) | logical
    origin_wall_ts: u64,
    seq: u32,
    text: String,
    #[serde(default)]
    msg_type: u8,
    #[serde(default)]
    control: Option<Vec<u8>>,
    #[serde(default)]
    edited_at: Option<u64>,  // packed HLC; Some only if the message was edited
    #[serde(default)]
    deleted: bool,           // true only if deleted (then `text` is "")
    // `kind` omitted here; unknown CBOR keys are skipped by default.
}

let bytes = hex::decode(msg_cbor.trim_start_matches("0x"))?;
let msg: Message = serde_cbor::from_slice(&bytes)?;
}

Read progress and unread counts

Mark progress with POST /dialogs/{peer}/messages/read carrying the highest sequence number you have read:

{ "seq": 123 }

Success is a 200 with an empty body (the group variant behaves the same) -- do not expect JSON there.

Unread counts are not stored: GET /conversations derives each chat's unread by comparing its latest sequence against your stored read progress.

Editing and deleting messages

You can edit or delete your own messages after sending. Two endpoints do this, both signed with the standard auth headers like any other request:

• PATCH /messages/{msg_id} -- replace the text. Body: { "text": "<new text>" }.
• DELETE /messages/{msg_id} -- clear the text and tombstone the message. Body: { "sig": "0x<hex>" }.

Both return the affected id and the node's server-stamped operation time:

{ "msg_id": "0x<32-byte hex>", "op_ts": 1699900000000 }

op_ts is milliseconds and is assigned by the node -- you never send it.

Why delete is signed but edit is not

Edit and delete are authorized differently, and getting this asymmetry right is the whole trick:

• An edit writes new content. Its authenticity is verified end to end by the recipient on Layer 2 -- the sign-then-encrypt envelope -- exactly like a normal message. So an edit needs no operation signature: the PATCH body is just { "text": "<new text>" }, and authorship at the node rests on the normal request signature (X-Sig), which must recover to the message's original sender. Editing is just like sending.
• A delete produces the absence of content. The recipient is left with a deleted: true stub and has nothing to re-authenticate on Layer 2 -- it cannot cryptographically tell "the author deleted this" from "a node erased it without permission." So the node itself must authorize the delete, against a dedicated operation signature that every node re-verifies as the tombstone propagates over gossip.

The rule in one line: sign an operation only when it destroys content the recipient can no longer re-authenticate. A delete does, so sign it; an edit -- like a normal send -- does not, because Layer 2 covers the new bytes.

The delete operation signature

A DELETE therefore carries two distinct signatures, and they are easy to confuse:

1. The request signature in X-Sig, over the canonical string as for any request (the body's sig field is just another key that gets canonicalized).
2. An operation signature in the body's sig field, which authorizes the delete itself. Every node re-verifies it as the tombstone propagates over gossip, so it -- not the X-User header -- is what proves you authored the message.

The operation signature is taken over a canonical 97-byte payload:

payload = chat_id[32] || target_msg_id[32] || op_kind[1] || blake3(new_text)[32]

• chat_id -- the chat the message lives in. For a DM you derive it yourself (see Deriving chat identifiers); for a group it is the group chat_id.
• target_msg_id -- the 32-byte id of the message being deleted (the {msg_id} in the path).
• op_kind -- one byte; the payload's operation tag, 1 for a delete.
• blake3(new_text) -- 32-byte BLAKE3 hash of the replacement text. A delete has no replacement text, so new_text is the empty string "" and this is BLAKE3 of zero bytes.

Then sign exactly as a group op does: hash the payload with keccak256, produce a recoverable secp256k1 signature, serialize it r[32] || s[32] || v[1] (65 bytes), hex-encode with a 0x prefix, and put it in the body sig field. Do not include any timestamp in the payload -- the node stamps op_ts server-side, so the signed bytes stay stable.

The node accepts the delete only if the signature recovers to the original message's sender -- that is the author-only rule, enforced cryptographically rather than by trusting a header.

Worked example -- deleting message 0x1111...11 in chat 0x2222...22:

[ 0..32]  chat_id        0x2222...22
[32..64]  target_msg_id  0x1111...11
[64..65]  op_kind        0x01                 (delete)
[65..97]  blake3(text)   blake3("")           (BLAKE3 of the empty string, 32 bytes)

sig = 0x || secp256k1_recoverable( keccak256(payload) )   (130 hex chars)

The request body then carries only this sig.

Rules

• Author only. You can change only your own messages. On a delete this is the operation signature recovering to the original sender; on an edit it is the request signature (X-Sig) doing the same. Either way, you cannot touch anyone else's message.
• 48-hour window. An edit or delete is accepted only within 48 hours of the original message's send time. After that the message is frozen.
• Delete is terminal and network-wide. The text is cleared on every node, and a later edit of a deleted message is refused. There is no un-delete.

Rendering edited and deleted messages

Neither operation changes the message's msg_id or its position -- the row is rewritten in place, keeping the same id and seq. Detect the new state from the decoded msg_cbor (see decoding msg_cbor):

• Edited -- the optional edited_at field is now present (the packed HLC of the edit). text holds the new value. Show an "edited" marker and replace any cached copy of the text.
• Deleted -- the optional deleted field is true and text is "". The message stays in history as a stub with the same id. Render it as "message deleted" and drop any cached plaintext for that id -- including decrypted Layer 2 content.

Both fields are absent on an untouched message, so a decoder that ignores unknown keys keeps working on messages that were never changed.

Keeping caches fresh

An edit or delete rewrites the message in place without bumping the chat's last_ts or its unread count: the conversation is not reordered, and /conversations gives no signal that anything changed. A client that already holds a page in memory therefore will not notice the change on its own. For v1:

• Re-fetch the visible message window when you poll (you already poll for new messages) and reconcile each row by msg_id; or
• Re-fetch a single message with GET /messages/{msg_id} to pull its latest state -- for example before acting on a cached copy.

There is no "changes since" endpoint today; one may be added later, but until then re-reading is the mechanism.

Encrypted chats

Edit and delete are content-agnostic: the node treats text as opaque bytes. To edit an E2EE message, encrypt the new plaintext and send the fresh ciphertext envelope as text -- the flow is identical to a plaintext edit, and the new envelope is authenticated end to end on Layer 2 exactly like the original message. Delete behaves the same regardless of what was stored; its operation signature covers blake3(""), independent of the ciphertext it clears.

Groups

Group membership is managed through a single compound endpoint, POST /groups/{chat_id}/ops, which bundles one or more membership operations and optional accompanying messages (for example, encryption handshake data) in one call:

{
  "ops": [
    { "op_type": "create", "target": "0xADMIN...", "role": 2, "sig": "0x..." },
    { "op_type": "add",    "target": "0xMEMBER...", "role": 0, "sig": "0x..." }
  ],
  "messages": [],
  "nonce": "0x<16-byte hex>",
  "title": "My Group Chat"
}

nonce is required whenever the batch contains a create op. title names the group at creation: it is the same string you hashed into the chat_id (see Deriving chat identifiers), 1-128 UTF-8 bytes, no control characters, immutable afterwards; omit it for an unnamed group. List members with GET /groups/{chat_id}/members (returns the member list plus the group title; role is 0 = participant, 1 = admin, 2 = owner), and leave with DELETE /groups/{chat_id}/membership. The title also appears in every /conversations entry for the group, so a client never needs to store it separately.

For end-to-end-encrypted groups, note that the title is public network metadata (it is gossiped and stored in plaintext on every node). Privacy- sensitive clients should leave groups unnamed at the node level and carry the display name inside their encrypted Layer 2 payloads instead.

Roles

Creating a group makes you its owner (role 2) -- there is exactly one per group. Owners do everything; admins (role 1) may add participants and remove non-owners; participants only chat. Assigning or changing roles is owner-only, and role 2 can never be requested via add -- it moves only through transfer:

• { "op_type": "transfer", "target": "0xNEWOWNER..." } -- signed by the current owner; the target must already be a member. You become an admin, the target becomes the owner. This is the escape hatch for the owner, who cannot leave the group directly: transfer first, then leave (both ops can ride in one batch).
• { "op_type": "delete", "target": "0xSELF..." } -- signed by the owner; dissolves the group for everyone. The chat vanishes from every member's /conversations; old messages age out via retention rather than being wiped immediately.

The second signature

Group operations require two distinct signatures, and confusing them is the most common group-related bug:

1. The request signature in X-Sig, over the canonical string (as for any request).
2. A per-operation signature inside each op's sig field, over the raw binary message chat_id[32] || target[20] || op_type[1], hashed with Keccak-256. The op_type byte is 0 for add, 1 for remove, 2 for create, 3 for transfer, 4 for delete -- even though the JSON field spells it "add"/"remove"/"create"/"transfer"/"delete". For delete the target has no protocol meaning; sign your own address.

The per-op signature lets every node independently verify who authorized each membership change as it propagates over gossip. Leaving a group (DELETE .../membership) carries the same kind of per-op signature over chat_id || sender || 1 (a self-remove).

Signing group operations

POST /groups/{chat_id}/ops is the one endpoint whose body holds an array of objects, and its canonical form is where clients most often go wrong. Each element of ops (and of messages, when present) is not flattened into dotted keys. Instead it becomes a single pair: the key is ops[], the value is the element serialized as compact JSON -- no whitespace, object keys sorted alphabetically at every depth. The create+add request above therefore reduces to exactly four pairs:

nonce=0x<16-byte hex>
ops[]={"op_type":"add","role":0,"sig":"0x...","target":"0xMEMBER..."}
ops[]={"op_type":"create","role":2,"sig":"0x...","target":"0xADMIN..."}
title=My Group Chat

Then the usual steps apply: sort the pairs by (key, value) -- note this orders the ops[] pairs by their JSON strings, so here add comes before create regardless of their positions in the request -- percent-encode every key and value, and join with &. Two consequences worth spelling out:

• A plain JSON.stringify of the op object is not enough: it preserves insertion order, so sort the keys of every object recursively before serializing.
• The serialized values must match the sent body byte-for-byte: same hex casing, same 0x prefixes, integers without a decimal point.

The test vectors include a full group-ops request with the exact canonical string, so an implementation of this rule can be checked byte-for-byte before touching a live node.

Identity blobs

A user may publish one opaque identity blob (for example, a public-key bundle) with PUT /identity, and anyone may fetch it with GET /identity/{address}. The blob is base64 in JSON and capped at 1024 bytes; the node stores it last-write-wins and never inspects it.

{ "identity": "SGVsbG8gV29ybGQ=" }

Layer 2: end-to-end encryption

The node is a transport: it never reads message contents for meaning. Two channels exist, with a normative split of duties:

• POST .../messages (text) is the only transport for user content -- literal text or an encrypted envelope (ciphertext in base64 is still a string). Max 45056 bytes of UTF-8 (44 KiB), sized so any payload that fits the control cap also fits here once base64-encoded and prefixed. How a client marks its envelope formats inside text is a client-layer convention -- see text prefixes.
• POST .../messages/control (msg_type + control) carries protocol / service structures only -- attachment metadata, the MLS handshake. control is base64 in JSON (note: addresses, IDs, and signatures are hex, but control and identity blobs are base64), max 43692 base64 chars = 32 KiB decoded; msg_type (u8) says which structure it is. Clients MUST NOT claim new control types for user content.

A typical E2EE client performs its handshake over control messages, then sends ciphertext through text as ordinary messages, encrypting on the client and decrypting after the CBOR decode on read. Because the node cannot interpret any of this, two clients can agree on any scheme without node support.

Three Layer 2 conventions are documented in this guide and are normative for interoperability: the identity key bundle (key distribution), DM encryption (p2pmes:dm-e2ee-v1: envelopes in text), and attachments (control msg_type = 10). Group E2EE runs (D)MLS on top of the same identity bundle (control 20/21 for the handshake, p2pmes:mls-app-v1: in text for application messages). The sections below define the wire formats and vectors; for the implementation walkthrough -- key lifecycle, MLS-on-p2p-mes recipes, state recovery, pitfalls -- follow the E2EE Cookbook.

Text prefixes: identifying client formats

The node enforces exactly one thing about text: its size. Whether the string is literal text or an envelope of some client format is decided entirely by the client layer, and the recommended way to mark a format is a short prefix (slug) in front of the base64 payload:

<slug>:<base64 payload>          e.g.  p2pmes:dm-e2ee-v1:pGpl...

A prefix is how a client identifies formats it can handle: a known slug means "decode accordingly" (decrypt the envelope, render the card); an unknown one means a protocol-compatible but format-foreign client. The protocol does not regulate the slug namespace and nodes never inspect it -- but each profile in this guide pins its own prefix (that is what makes the profile interoperable across clients), and anything outside these profiles is free space.

Slugs used by the profiles in this guide and the demo client:

Recommended client rules:

• a known prefix -> decode accordingly;
• an unknown p2pmes:* prefix -> render as "unsupported message" (forward compatibility), never as raw base64;
• literal user text that starts with p2pmes: -> wrap it in p2pmes:text-v1: before sending, so it cannot be mistaken for an envelope;
• previews are the client's job: build conversation-list lines from your local message cache (decrypting first where needed) and never surface a raw envelope string in the UI.

Identity key bundle (idbundle-v2)

Every encryption feature below needs one thing from the network: a way to learn a peer's public encryption key given only their address. The identity blob (PUT /identity) is the distribution channel, but the transport does not authenticate blob contents -- the blob is opaque to nodes, and the PutIdentity gossip message carries no client signature, so any node can rewrite anyone's blob network-wide. The bundle fixes this end to end: it is self-signed by the account key, so a reader verifies it against the one thing it already knows authentically -- the peer's address.

The bundle also solves custody. The Ed25519 identity seed is derived deterministically from the account key, so a client holds exactly one long-term secret; identity, DM encryption, MLS keys, and any future derived key all recover from it.

Key derivation

seed(gen) = keccak256( sign65( account_key,
                keccak256("p2p-mes:id-seed:v1" || u32be(gen)) ) )

• sign65 is the recoverable ECDSA signature r[32] || s[32] || v[1] with v normalized to 0/1 (subtract 27 from Ethereum-style 27/28).
• Signing MUST be deterministic (RFC 6979, the default in secp256k1 libraries). A signer that randomizes k would derive a different seed every time and break recovery.
• gen is a rotation counter starting at 0. Rotating the identity key = bump gen, re-derive, republish the bundle. After device loss, recover by reading your own published bundle (it carries gen) or by trying gen = 0, 1, 2... until the derived key matches.

From the 32-byte seed, everything else is deterministic:

The Ed25519 -> X25519 conversion is the standard RFC 7748 birational map: u = (1 + y) / (1 - y) mod p on the public side (p = 2^255 - 19, y from the Ed25519 point encoding with the sign bit cleared).

Custody trade-off, stated plainly: deriving everything from the account key means one secret to protect and full recovery from it -- and it means compromise of the account key exposes recorded E2EE traffic, not just future impersonation. This is a deliberate choice for recoverability; the account key already controls the bundle (an attacker holding it can rotate your keys anyway), and message retention bounds how much recorded ciphertext exists.

Bundle format

IdentityBundleV2 = {
  "v":      2,
  "gen":    uint,              ; seed rotation counter
  "ts":     uint,              ; unix ms at publication (anti-replay)
  "ed_pub": bstr .size 32,     ; Ed25519 public key
  ? "kp":   bstr,              ; last-resort MLS KeyPackage (TLS-serialized)
  "sig":    bstr .size 65      ; recoverable secp256k1 signature, v in {0,1}
}

preimage = "p2p-mes:idbundle:v2" || u32be(gen) || u64be(ts)
           || ed_pub || (kp | empty)
sig      = sign65(account_key, keccak256(preimage))

Publish it base64-encoded via PUT /identity. Without kp the bundle is 133 bytes; with a typical KeyPackage (~400 bytes) it stays well inside the 1024-byte blob limit.

kp is a last-resort KeyPackage for (D)MLS (MLS_128_DHKEMX25519_AES128GCM_SHA256_Ed25519, signed inside by ed_pub): it lets others add you to groups while you are offline, and because its initKey is the deterministic X25519 above, a pending Welcome stays decryptable from the seed alone even after total state loss. The cost is a reusable init key -- seed compromise exposes recorded Welcomes. That trade is inherent to the deterministic design and is accepted here.

Verifying a bundle (reader side)

1. Fetch GET /identity/{address}. A 32-byte blob is a legacy raw Ed25519 key: unauthenticated -- treat it as trust-on-first-use and refuse E2EE features that need stronger guarantees.
2. CBOR-decode; require v == 2.
3. Rebuild preimage from the fields, keccak256 it, ecrecover the signature, and require the recovered address to equal the address you fetched. A node cannot forge this -- it does not hold the account key.
4. Pin (gen, ts) per peer and require monotonic growth: a bundle older than the pinned one is a replay (a malicious node serving a rotated-away key) -- keep the pinned one and warn.

What this buys: a malicious node can no longer substitute keys (the MITM on E2EE bootstrap is closed); it can only serve a stale bundle (caught by pinning), serve nothing (a visible availability failure), or delete the blob. The full authentication chain becomes: address -> secp256k1 bundle signature -> ed_pub -> (KeyPackage signature -> MLS keys) / (message signatures).

Reference test vector

Account keys are the docs' fixed test keys; never use them outside tests.

Alice account key = 0x11 * 32
  address     = 19e7e376e7c213b7e7e7e46cc70a5dd086daff2a
  seed_sig    = 43b00ef256fa15181de74cb6b9f6668853a21296176b62dde763fc73f6faabbe1437fdc1ce513eb3d6998704329914bcf655c5c16b4542a61198e7f7d7a1713101
  seed        = 6cccccc86d8b15e7ea7fc0a817235593f6631018d0e28e810677184dcee01dde
  ed25519_pub = 2d41d2645fed8100c458f8442301dd92e504d1e16bf258272d7fa9b8738bd45d
  x25519_pub  = fb1a6a847fbee0fb7f4119c602a8d0e6a9aa8e2cd785f0c580de3ec5b2e6f705

Bob account key = 0x22 * 32
  address     = 1563915e194d8cfba1943570603f7606a3115508
  seed_sig    = 631a18bd87f0af380c34fba2b451a31739c3d8e7ca7645986f239b963e21258f03f9b53e21845dfc56d16a28baa75c18038597100a8555a315b1555e42e4292a01
  seed        = 12846d92d13846ec688705784824ca4a4ebfe1b3d339efb872e27e27b096b51a
  ed25519_pub = 73c163c6727022f1d51334199bb2405bc85197e4eb00430de51cfcb3e8302fbe
  x25519_pub  = 0251e47e353278612e450513eee3667a2c8bbf2c3b80470fe5f5ae496b2dd540

Alice's bundle with gen = 0, ts = 1700000000000, kp omitted:

preimage    = 7032702d6d65733a696462756e646c653a7632000000000000018bcfe568002d41d2645fed8100c458f8442301dd92e504d1e16bf258272d7fa9b8738bd45d
keccak256   = 7e12d9e103fa16043608ce20fb211bf5b80b64334219c8149ab1dc3d24a2f6b1
sig         = f48b94afee153bf7eb729b822c298faa832b5f1c3d99def204be65927b0f1f706e1813c1dafb497656f93257b1181c3d21058e14a5512aa05d0e715fb70097df01
bundle CBOR = a56176026367656e006274731b0000018bcfe568006665645f70756258202d41d2645fed8100c458f8442301dd92e504d1e16bf258272d7fa9b8738bd45d637369675841f48b94afee153bf7eb729b822c298faa832b5f1c3d99def204be65927b0f1f706e1813c1dafb497656f93257b1181c3d21058e14a5512aa05d0e715fb70097df01

Direct message encryption (dm-e2ee-v1)

Stateless end-to-end encryption for direct messages, rooted in the identity bundle. Design goals, in order: nothing to lose (no session state, no ratchet -- every message is independently decryptable by the two static identities), full recovery from the account key alone, and end-to-end authenticity over nodes you do not trust. The deliberate non-goal is forward secrecy within a conversation: compromise of a party's account key exposes their recorded DM ciphertext. Groups do not use this profile -- they run (D)MLS, which provides FS/PCS where it matters most.

The convention

• A DM E2EE message is an ordinary message: text = "p2pmes:dm-e2ee-v1:" + base64(envelope) with the envelope below -- user content, encrypted or not, always rides the plain message endpoint. The prefix is pinned by this profile. A side benefit of the text carriage: msg_id commits to text, so the ciphertext is bound by the message id (unlike control payloads).
• Requires both parties to have published a verified idbundle-v2. Without one, fall back to plaintext and say so in the UI.

Envelope format

DmEnvelopeV1 = {
  "eph_pub": bstr .size 32,      ; fresh X25519 ephemeral, one per message
  "wraps": [+ KeyWrap],          ; content key wrapped per recipient
  "nonce":  bstr .size 12,       ; AES-GCM nonce for "ct"
  "ct":     bstr                 ; AES-256-GCM(content_key, nonce, SignedContent)
}

KeyWrap = {
  "addr":  bstr .size 20,        ; recipient address (peer, and yourself)
  "nonce": bstr .size 12,
  "key":   bstr .size 48         ; AES-256-GCM(kek, nonce, content_key)
}

SignedContent = {
  "m":   bstr,                   ; the inner content, CBOR, signed byte-exact
  "sig": bstr .size 64           ; Ed25519 over "p2p-mes:dm-sig:v1" || chat_id || m
}

InnerContent = {
  "t":  uint,                    ; content type: 0 = text, 10 = attachments,
  "ts": uint,                    ;   100-255 app-private
  ...                            ; t = 0: "text": tstr
}                                ; t = 10: "attachments" / "caption" as in the
                                 ;         attachments profile
                                 ; "ts" = sender wall-clock ms -- the SIGNED
                                 ;   display time

InnerContent is deliberately reusable: the same map -- without the SignedContent wrapper, since MLS authenticates its own senders -- is also the normative plaintext framing of MLS application messages (sent as p2pmes:mls-app-v1: + base64 in text); see the E2EE Cookbook.

Sending

1. Fetch and verify the peer's bundle; compute their static X25519 as toMontgomery(ed_pub). Your own keys come from your seed.
2. Build InnerContent with ts = now_ms, CBOR-encode it into m, and sign: sig = Ed25519_sign(seed, "p2p-mes:dm-sig:v1" || chat_id || m).
3. Generate a random 32-byte content_key and a fresh ephemeral X25519 keypair. Encrypt: ct = AES-256-GCM(content_key, nonce, SignedContent).
4. Wrap content_key twice -- for the peer and for yourself (so your own devices and history restores can read it): kek = HKDF-SHA256(ikm = X25519(eph_priv, recipient_static_pub), salt = empty, info = "p2p-mes:dm-wrap:v1"), then key = AES-256-GCM(kek, wrap_nonce, content_key).
5. Send "p2pmes:dm-e2ee-v1:" + base64(envelope) as text via the plain DM endpoint.

Receiving

1. A text starting with p2pmes:dm-e2ee-v1: is this profile: strip the prefix and base64-decode the rest into the envelope.
2. Find your wrap by addr, compute the same kek with your static X25519 private key against eph_pub, unwrap content_key, decrypt ct.
3. Verify sig over "p2p-mes:dm-sig:v1" || chat_id || m against the expected author's ed_pub -- the peer's verified bundle for incoming messages, your own for echoes of your messages. Reject on failure.
4. Decode m. Use its signed ts as the authoritative display time -- it is the one timestamp a relay cannot forge (hlc and origin_wall_ts are not client-signed). Flag duplicates (same sig seen before) as replays.
5. Dispatch on t: 0 render text; 10 follow the attachments profile's receive flow from step 2.

Why sign-then-encrypt

ECIES to public keys is sender-anonymous: anyone, including a node, can encrypt to two published bundles and inject a message that decrypts cleanly -- GCM authenticates bytes, not authorship. The inner Ed25519 signature binds author and chat: the chain is address -> bundle signature -> ed_pub -> message signature. Replaying an old envelope into another chat fails the chat_id binding; replaying into the same chat repeats a known sig and is flagged. This also closes, for E2EE DMs, the known relay-tampering gap on control described in the attachments section.

Size budget

The envelope costs 365 bytes over the inner plaintext (two wraps, ephemeral key, signature, framing), and the base64 carriage adds one third on top. text is capped at 45056 bytes (44 KiB), sized so that anything that would fit the 32 KiB control cap also fits here enveloped: after the prefix, base64, and envelope overhead an E2EE DM carries about 33 KB of inner plaintext -- a very long text or a large multi-attachment album. A payload that does not fit goes by reference through the attachments pipeline (upload the encrypted blob, send the metadata) -- never chunk one logical message across several messages.

Reference test vector

Alice -> Bob (identities from the bundle vector above), chat_id = blake3("p2p-mes:chat:dm:v1:" || min || max) = a91602ff4fbe6b4ff0555945932d5367db2b815cbcb6d05cdf3c399c6fa9e30f. Fixed values (test only): content_key = 0x44*32, eph_priv = 0x55*32, wrap nonces 0x66*12 (Bob) and 0x77*12 (Alice), message nonce 0x88*12, ts = 1700000000000, text "Hello, Bob!".

inner m     = a36174006274731b0000018bcfe5680064746578746b48656c6c6f2c20426f6221
msg sig     = 56cf1140361ca416222fd29ffb3a37b0a474e2ec22b3868ddab55af1f072dc15cd433d8d82a287f3e93e519d2677ff4e3155f2c0f155f3aa1450aa1841a15700
eph_pub     = 38ab664bd86f77d7e66bdd9ae0792913a94fd8b33a1260027e4b46c1f4884c67
kek (Bob)   = 9bef6660bff2a7b28246abb7010774affd36764e34f75baa58f268de69e955b3
wrap (Bob)  = e5c06987c117db0c8d917898e7c9a19cc72f90d479d9311df3bef577bab5fa58917ccd1466976ed483493679254d7dc2
kek (Alice) = 24d98109b6123a1984dca8f7c130175e90c9cdee27ddc3e5d28fdec9f5e32319
wrap (Alice)= 0fe06669ed36807974a548748cc839ddc9210f7bb2a124d09725300df6e2ebe0fb8699bb25817a6b0d2c5c5c14f170f8
ct          = d47ac0c2045d7fd2f79e7a3f3200c0be9ee16bf675a57603cf4270ef98b744171c28b47ba6c08da862a778541ea499e0e8fc46a250e144431c6a62ad39627c8d7e985d48f585cddbfaacf8eca1c25ee1ec70f54f37bbdacf47e6ac3e119fa21dd6fdda434356673c5233ed23765e38499dc1c2d8a8d6ec604a65e06d

Full envelope (398 bytes of CBOR; the wire text is "p2pmes:dm-e2ee-v1:" + base64(envelope) -- 18 + 532 = 550 chars for this vector):

a4676570685f707562582038ab664bd86f77d7e66bdd9ae0792913a94fd8b33a1260027e4b46c1f4884c6765777261707382a36461646472541563915e194d8cfba1943570603f7606a3115508656e6f6e63654c666666666666666666666666636b65795830e5c06987c117db0c8d917898e7c9a19cc72f90d479d9311df3bef577bab5fa58917ccd1466976ed483493679254d7dc2a364616464725419e7e376e7c213b7e7e7e46cc70a5dd086daff2a656e6f6e63654c777777777777777777777777636b657958300fe06669ed36807974a548748cc839ddc9210f7bb2a124d09725300df6e2ebe0fb8699bb25817a6b0d2c5c5c14f170f8656e6f6e63654c888888888888888888888888626374587cd47ac0c2045d7fd2f79e7a3f3200c0be9ee16bf675a57603cf4270ef98b744171c28b47ba6c08da862a778541ea499e0e8fc46a250e144431c6a62ad39627c8d7e985d48f585cddbfaacf8eca1c25ee1ec70f54f37bbdacf47e6ac3e119fa21dd6fdda434356673c5233ed23765e38499dc1c2d8a8d6ec604a65e06d

Attachments (files and media)

The replicated store carries only short text and the small control payload -- binary files must never enter it (every byte would be copied to every node and live for the whole retention window). Files are therefore shared by reference: the file is encrypted client-side, uploaded to ordinary HTTPS object storage (for example an S3-compatible bucket operated alongside the nodes), and the message carries only compact metadata -- the URL, the decryption key material, an integrity digest, plus an optional caption. The node relays that metadata as an opaque Layer 2 payload; the storage provider sees only ciphertext.

This convention is an adaptation of XMTP's remote-attachment content types (XIP-17 and XIP-50): the encryption scheme and the metadata fields are reused as-is; the wire encoding is CBOR instead of protobuf, and versioning rides on msg_type instead of XMTP's content-type identifiers.

The convention

An attachment message is a regular control message:

• POST /dialogs/{peer}/messages/control or POST /groups/{chat_id}/messages/control
• msg_type = 10 (attachments v1; a breaking format change would claim a new value)
• control = base64 of the CBOR metadata described below

Control msg_type values used by known conventions:

User content never claims a control type: encrypted DMs and MLS application messages ride text with a format prefix (see text prefixes).

Metadata format

The control payload is a CBOR map:

MultiRemoteAttachment = {
  "attachments": [+ RemoteAttachmentInfo],   ; one or more
  ? "caption": tstr                          ; message-level caption,
                                             ; recommended <= 1000 chars
}

RemoteAttachmentInfo = {
  "url":            tstr,             ; where the ciphertext lives
  "content_digest": bstr .size 32,    ; SHA-256 of the uploaded ciphertext
  "secret":         bstr .size 32,    ; HKDF input key material
  "salt":           bstr .size 32,    ; HKDF salt
  "nonce":          bstr .size 12,    ; AES-GCM nonce
  "scheme":         tstr,             ; MUST be "https://"
  ? "content_length": uint,           ; ciphertext size in bytes
  ? "filename":     tstr,             ; display hint (duplicates the envelope)
  ? "mime_type":    tstr,             ; display hint (duplicates the envelope)
}

Field names follow XIP-50's RemoteAttachmentInfo with three deviations: content_digest is 32 raw bytes rather than a hex string; mime_type is an extension (XIP-50 keeps the MIME type only inside the encrypted envelope, which makes it impossible to render a useful preview before downloading); and the message-level caption is an extension -- XIP-50 has no caption concept, and without it a caption would require a separate text message.

Encoding notes:

• Byte fields here are real CBOR byte strings (major type 2). This is the opposite of the node's msg_cbor quirk (arrays of integers, see the decoding gotcha above). The distinction is deliberate: control is client-defined end to end, so this profile picks the compact native encoding. On read, the control field itself arrives as a CBOR array of integers at the outer msg_cbor layer -- reassemble it into bytes, then CBOR-decode those bytes with this schema.
• Each attachment MUST use freshly generated secret, salt, and nonce. Never reuse key material across attachments (XIP-50 requires per-file keys).
• The decrypted payload MUST NOT itself be a MultiRemoteAttachment -- no recursion (XIP-50 rule).

The encrypted envelope

What you encrypt and upload is not the raw file but a small CBOR envelope (the analog of XMTP's XIP-15 attachment type), so the storage provider learns neither the file name nor its type:

Attachment = {
  "filename":  tstr,
  "mime_type": tstr,
  "data":      bstr    ; the file bytes
}

Encryption

The scheme is exactly XMTP's remote-attachment encryption, so existing implementations translate directly:

1. Generate secret (32 random bytes), salt (32 random bytes), and nonce (12 random bytes) -- fresh per attachment.
2. key = HKDF-SHA256(ikm = secret, salt = salt, info = empty, length = 32). The info parameter is empty (zero bytes), matching the XMTP implementation.
3. ciphertext = AES-256-GCM(key, nonce, plaintext = envelope CBOR) with no additional authenticated data. The 16-byte GCM tag is appended to the ciphertext -- the default behavior of WebCrypto, Rust's aes-gcm, and Python's cryptography.
4. content_digest = SHA-256(ciphertext) -- over the exact bytes uploaded.

Sending

1. Build and encrypt the Attachment envelope as above.
2. Upload the ciphertext to your storage. Use hex(content_digest) as the object key: uploads become idempotent, identical files deduplicate, and the key leaks nothing. How you authenticate the upload (presigned PUT, operator-issued credentials) is a deployment concern, not part of the protocol.
3. Repeat per attachment, fresh key material each time.
4. Assemble MultiRemoteAttachment, CBOR-encode it, base64 it, and send it as a control message with msg_type = 10.

Receiving

1. Decode msg_cbor; on msg_type == 10, reassemble the control bytes and CBOR-decode the metadata.
2. Render the caption (if present) and a placeholder from filename / mime_type / content_length. This is also where chat previews come from: use the caption for your conversation-list line. Do not fetch automatically: the URL is sender-controlled, and a download reveals your IP address to whoever operates the storage (the XIP-50 threat model). Fetch on user action, or auto-fetch only from storage hosts your deployment trusts.
3. Before fetching, require scheme == "https://" and reject anything else. While fetching, enforce a size cap -- treat content_length as a hint, not a promise.
4. Verify SHA-256(downloaded bytes) == content_digest; abort on mismatch.
5. Derive the key, decrypt (the GCM tag authenticates the content), CBOR-decode the envelope, and render according to mime_type. Treat filename as a display string only -- never let it choose a filesystem path or an execution decision.

A client that does not understand msg_type = 10 sees an ordinary control message with empty text and falls back to its generic "unsupported message" state -- the same fallback rule XIP-50 prescribes.

Reference test vector

Fixed inputs (fixed key material is for tests only):

secret   = 0x11 repeated 32 times
salt     = 0x22 repeated 32 times
nonce    = 0x33 repeated 12 times
envelope = { "filename": "hello.txt", "mime_type": "text/plain",
             "data": "Hello, world!" (13 bytes) }

Expected outputs (all hex):

envelope CBOR  (60 bytes):
a36866696c656e616d656968656c6c6f2e747874696d696d655f747970656a746578742f706c61696e64646174614d48656c6c6f2c20776f726c6421

derived key:
6b8121baa9b21c516029bf185397a0f193ff7297a22ae6292f92259961a461a0

ciphertext with appended GCM tag (76 bytes):
ffbb7ec2d2424f497d2ba1ba33416f51f529e652d16747816a4273d489715ed4dd0f05f7d52ffff3d112a99ab9d30969791c41ff33e20b165c3285cd9ee40a9e16ae5bd562d3cfcc186a7838

content_digest:
f0775ac2e1fd2096683a1ac3a338d56e92c051647ad0f4297c2477d6d964f277

With url = "https://files.example.com/" + hex(content_digest), all optional attachment fields present, and no caption, the full one-attachment MultiRemoteAttachment encodes to 332 bytes of CBOR:

a16b6174746163686d656e747381a96375726c785a68747470733a2f2f66696c65732e6578616d706c652e636f6d2f663037373561633265316664323039363638336131616333613333386435366539326330353136343761643066343239376332343737643664393634663237376e636f6e74656e745f6469676573745820f0775ac2e1fd2096683a1ac3a338d56e92c051647ad0f4297c2477d6d964f27766736563726574582011111111111111111111111111111111111111111111111111111111111111116473616c7458202222222222222222222222222222222222222222222222222222222222222222656e6f6e63654c33333333333333333333333366736368656d656868747470733a2f2f6e636f6e74656e745f6c656e677468184c6866696c656e616d656968656c6c6f2e747874696d696d655f747970656a746578742f706c61696e

and the control field of the request body is its base64:

oWthdHRhY2htZW50c4GpY3VybHhaaHR0cHM6Ly9maWxlcy5leGFtcGxlLmNvbS9mMDc3NWFjMmUxZmQyMDk2NjgzYTFhYzNhMzM4ZDU2ZTkyYzA1MTY0N2FkMGY0Mjk3YzI0NzdkNmQ5NjRmMjc3bmNvbnRlbnRfZGlnZXN0WCDwd1rC4f0glmg6GsOjONVuksBRZHrQ9Cl8JHfW2WTyd2ZzZWNyZXRYIBERERERERERERERERERERERERERERERERERERERERERZHNhbHRYICIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiZW5vbmNlTDMzMzMzMzMzMzMzM2ZzY2hlbWVoaHR0cHM6Ly9uY29udGVudF9sZW5ndGgYTGhmaWxlbmFtZWloZWxsby50eHRpbWltZV90eXBlanRleHQvcGxhaW4=

Size budget

DM and group control payloads are both capped at 32 KiB (decoded). One RemoteAttachmentInfo with a 90-character URL and all optional fields is ~330 bytes, and a caption shares the same budget (CBOR counts its UTF-8 bytes), so the cap fits dozens of attachments plus a full-length caption. Keep albums to ~10 per message for UX, not because of the budget. The text cap (45056 bytes) is deliberately sized so the same metadata also fits inside the encrypted carriages, where it rides base64-encoded in text instead.

There is no protocol-level limit on the blob itself -- cap uploads by storage policy and enforce a download cap client-side.

Storage requirements

Anything that serves plain unauthenticated HTTP GET works. For the typical S3-compatible setup:

• Objects must be readable without credentials by anyone holding the URL. Privacy comes from the unguessable content-addressed key plus the encryption -- the same capability-by-knowledge pattern as DM chat IDs. Do not embed presigned GET URLs in messages: they expire (7 days maximum on S3, far short of the 30-day message retention) and their length blows the DM control budget.
• Set the bucket lifecycle to at least the message retention window (30 days) so links do not die before the messages that carry them. Links that outlive retention are a privacy consideration for the operator.
• Web clients need CORS allowing GET on the bucket.

Privacy and integrity caveats

• In plaintext chats, node operators can decrypt your attachments. The secret travels in control, which every node stores -- and the caption, filenames, and URLs are readable there directly. The encryption protects against the storage provider and anyone who merely obtains the URL -- not against nodes. That is exactly the trust level of plaintext text. For confidentiality from node operators, carry the same fields inside an encrypted envelope instead of the plaintext control: in DMs, send dm-e2ee-v1 with inner t = 10; in (D)MLS groups, put the MultiRemoteAttachment CBOR inside an MLS application message. The blob encryption itself is identical in all three carriages.
• msg_id does not cover control (it commits to chat || sender || hlc || text only), so attachment metadata is not bound by the message id -- the same known relay-tampering gap as hlc rewriting (see Ordering and timestamps). The encrypted carriages close it: the dm-e2ee-v1 envelope signs the inner content, and MLS authenticates its application messages.

Current limitations

• No node-side previews. /conversations carries no content preview field at all -- by design, the node cannot read a caption buried in an opaque payload (or a ciphertext envelope), so previews are the client's job. Render conversation-list lines from your local message cache; an attachment's caption becomes visible only after fetching and decoding the message.

Encoding conventions

Error handling

Errors are returned with a conventional HTTP status and a JSON body:

{ "error": "forbidden" }

A 400 arrives in one of two shapes, depending on what failed:

• Declared field validation (the size/range rules on body and query fields, e.g. text 1-45056 bytes, limit 1-1000) returns error: "validation_error" plus a structured fields map naming each offending field and why:

  {
    "error": "validation_error",
    "fields": {
      "text": { "msg": "text must be 1..=45056 bytes of UTF-8 (44 KiB)",
                "value": "" }
    }
  }
  

• Everything else is the flat shape with no fields: malformed path parameters ({"error": "bad hex"}, {"error": "wrong id length"}), malformed cursors, and semantic input errors ("unknown op_type", "nonce required for create op", "identity blob exceeds 1024 bytes").

Handle both: fields is present only when error is "validation_error".

Operational notes for client authors

This section is the honest current state of the protocol from a client's point of view: what works today, the sharp edges, and how to cope with them. Several items here are limitations the protocol intends to address; they are called out so you can design around them now.

Real-time delivery and background

There is no push, WebSocket, or SSE today. The only way to learn of new messages is to poll GET /conversations (cheap: reverse-time, carries unread counts) and then GET .../messages for chats that changed. Poll on a backoff while foregrounded. Background delivery on iOS/Android does not work -- there is no push gateway (APNs/FCM), so a backgrounded app will not receive messages until it next polls. Do not emulate typing/presence with normal messages: they would replicate and persist for the whole retention window. Real-time transport and push are the largest planned additions.

Which node, and trusting it

A client speaks HTTP to a single node it does not run (a phone cannot hold a full replica of the whole network). There is no node discovery, health, or failover endpoint yet: pin a node URL (or a short operator-provided list) and handle transport errors with retry/backoff. You trust that node's operator with all of your metadata -- see Privacy.

What 200 means; delivery state

A 200 means the node accepted and broadcast the message, not that it is durably stored or delivered (the write is queued after the response). There are no delivery receipts, and the only read signal is coarse per-chat read progress. Build optimistic UI and reconcile by reading back; do not present "delivered" as a guarantee.

Idempotency and retries

The node computes msg_id from its own HLC, so resending the same text after a timeout produces a different msg_id -- a duplicate, not a dedup. There is no client idempotency key and no anti-replay nonce yet. Until there is: prefer waiting for the response (its body carries the real msg_id) before retrying; if you must retry blind, dedup on the client by (sender, text, approximate time) and reconcile when the real msg_id arrives.

Ordering and timestamps

Each message carries two times: hlc (the network-consistent stamp that defines storage and sync order) and origin_wall_ts (the sender's wall clock, for display). Sort by hlc for stable, cross-node-consistent ordering. Show origin_wall_ts as the human time, but treat it as untrusted -- the sender sets it and nothing validates it, so clamp obvious outliers and never use it for ordering. Note a known gap: a malicious relay can rewrite hlc without invalidating the client signature, so hlc is not cryptographically authoritative; prefer a node you trust until this is closed.

seq, read progress, and switching nodes

seq is assigned locally by each node (last_seq + 1 at write time), not globally. Because nodes can apply the same messages in different orders (gossip vs. anti-entropy sync), the seq of one message can differ between nodes -- and so can pagination cursors, which embed the storage key. Consequences:

• Use msg_id (deterministic BLAKE3) as the stable, cross-node message identity.
• Treat seq, cursors, and read progress (which is keyed by seq) as node-relative. Marking read on node A does not map cleanly onto node B.
• For consistent unread/read state, keep one identity pinned to one node until globally consistent sequencing lands.

Pagination and the message tail

GET .../messages pages in both directions: forward (oldest-first from from) and, with reverse=true, newest-first from to -- the "load the latest screen, then scroll up" pattern (see Reading history above). Use /conversations (last_ts, unread) as the cheap signal that a chat has a new tail. Since reads never block on the network, a node that is behind returns a partial page with no "is this complete?" flag -- show a soft "syncing" state rather than implying the history is final, and apply the next_after contract from Reading history: a cursorless page marks the end of the node's current view, never proof that the history is exhausted.

Clock skew and X-Ts

X-Ts must be within +/- 30 s of the node's clock or the request is rejected (401). There is no server-time endpoint yet, so sync the device clock (NTP); if you see unexplained 401s, suspect clock drift before signature bugs.

Key custody and recovery

Every request is signed by the user's secp256k1 key, so the key is the identity. Store it in the platform secure store (iOS Secure Enclave / Android Keystore). Note that recoverable ECDSA with hardware-backed keys is fiddly: you must recover the recovery byte v (the node tolerates a wrong v by trying both). There is no key backup or recovery -- losing the key permanently loses the identity. Design enrollment and backup UX accordingly.

Multi-device

One keypair is one identity. Multi-device is not specified: sharing the private key across devices authenticates fine, but read progress is node-relative (see above) and any Layer 2 session state is yours to coordinate. Treat the protocol as single-device today.

Identity blobs and key trust

PUT /identity is signed, so the receiving node can attest who published a blob, and the blob propagates network-wide (last-write-wins by HLC). But the transport does not authenticate blob contents to readers -- gossip replication carries no client signature, so a malicious node can rewrite any address's blob network-wide. Do not trust a bare key fished out of a blob. The identity key bundle is the documented answer: contents self-signed by the account key, verified by the reader via ecrecover against the peer's address, with (gen, ts) pinning against stale-bundle replay. Treat legacy raw 32-byte blobs as trust-on-first-use only, and refuse E2EE bootstrap on them unless the user explicitly accepts the risk.

Groups: rekey is not atomic

A compound ops call can bundle membership changes with accompanying messages (e.g. an MLS Commit), but there is no atomicity between them: a Remove can apply while the rekey message is lost, leaving the group in a broken crypto state. Until atomic membership+rekey exists, detect the gap (an expected rekey never arrives) and recover by re-issuing it.

Text length and large messages

text is validated as 1-1000 Unicode scalar values (Rust chars), not bytes -- an emoji counts as one or more chars, and percent-encoding during canonicalization does not change the count. There is no server-side chunking: messages longer than 1000 must be split client-side, and reassembly is your Layer 2 concern.

Feature scope today

• No first-class media. The replicated store carries only text plus the opaque control payload -- binary files never enter it. Files are shared by reference instead: encrypted blobs on external HTTPS storage, metadata in control. See Attachments for the convention, including the encryption scheme.
• Edit and delete are supported for your own messages within 48 hours of sending -- see Editing and deleting messages. Beyond that window messages are append-only and leave only via the retention window.
• channel chats are reserved -- no channel create/post/subscribe endpoints exist yet.

E2EE Cookbook

This page is the practical companion to Building a Client. The guide defines the normative wire formats (bundle, DM envelope, attachments) and ships the test vectors; this page explains how to implement them correctly: which keys exist, in what order to derive and verify things, how to prepare MLS for this protocol, what to persist, and how to recover when something is lost. If this page and the guide ever disagree, the guide wins.

The intended reader is a client developer who is comfortable calling crypto libraries but is not a cryptographer. Every constant, byte order, and domain string is pinned exactly, and each recipe ends with checkpoint values -- if your intermediate output does not match a checkpoint byte-for-byte, stop and fix that step before moving on.

1. The mental model

A client holds exactly one long-term secret: the secp256k1 account key. It already authenticates every HTTP request; everything below is derived from it deterministically, so a user who restores the account key on a new device recovers their entire cryptographic identity.

secp256k1 account key                 (the ONE secret; signs HTTP requests,
  |                                    membership ops, and the bundle)
  |  deterministic RFC 6979 signature of a fixed string
  v
Ed25519 seed (32 bytes)               (never leaves the device; cached)
  |
  +-- Ed25519 keypair                 -> bundle "ed_pub", DM message
  |                                      signatures, MLS signature key
  +-- X25519 static keypair           -> DM key wraps, MLS initKey
  |     pub  = toMontgomery(ed_pub)      (same point, two encodings)
  |     priv = clamp(SHA512(seed)[0..32])
  +-- MLS leaf encryption key         -> X25519(SHA256(seed || "mls-enc-key"))
  +-- state backup key                -> HKDF-SHA256(seed,
                                           info="p2p-mes:state-backup:v1")

Confidentiality is decided by the carriage, not by the payload format:

The rule that picks the carriage: text is the only transport for user content -- literal or encrypted, a ciphertext envelope is still a string -- while control (msg_type + payload) carries only protocol / service structures (attachment metadata, MLS handshake). A client marks its envelope formats with a prefix inside text (see text prefixes); the node interprets neither field and enforces only size caps.

Attachments are the same MultiRemoteAttachment CBOR in all three cases -- sent as plain control (msg_type = 10), as dm-e2ee inner content (t = 10), or inside an MLS application message. The blob encryption never changes; only where the metadata rides does.

2. What to store on the device

The seed is "hot" (in app memory) while the account key can stay behind the platform's biometric gate: derivation needs one signature at first launch, after which the app never touches the account key except to sign HTTP requests.

3. Libraries

ts-mls pins its peer dependencies to exact versions (no semver ranges): on 1.6.2 that is @noble/curves@2.0.1, @noble/ciphers@2.1.1, and four @hpke/* packages. If your app (or another dependency) wants any other @noble/curves, the package manager reports a peer conflict. Align your direct dependency to the exact pinned version, or resolve it with your package manager's overrides/resolutions -- do not force-install two copies of a crypto library.

4. Recipe: derive your keys

Steps (full spec: Identity key bundle):

// 1. One deterministic signature of a fixed message. gen starts at 0 and
//    only changes when the user rotates their identity key.
const msgHash = keccak256(utf8("p2p-mes:id-seed:v1") || u32be(gen));
const sig65   = secpSignRecoverable(accountKey, msgHash); // r||s||v, v in {0,1}
// If your library returns v = 27/28, subtract 27 BEFORE hashing.

// 2. The seed is the hash of the signature bytes.
const seed = keccak256(sig65);                            // 32 bytes

// 3. Everything else is derived, never stored as a second secret.
const edPriv  = seed;                                     // Ed25519 private key
const edPub   = ed25519.getPublicKey(seed);
const xPub    = ed25519.utils.toMontgomery(edPub);        // X25519 public
const xPriv   = ed25519.utils.toMontgomerySecret(seed);   // X25519 private

Checkpoints for accountKey = 0x11 * 32, gen = 0:

seed   = 6cccccc86d8b15e7ea7fc0a817235593f6631018d0e28e810677184dcee01dde
ed_pub = 2d41d2645fed8100c458f8442301dd92e504d1e16bf258272d7fa9b8738bd45d
x_pub  = fb1a6a847fbee0fb7f4119c602a8d0e6a9aa8e2cd785f0c580de3ec5b2e6f705

Pitfalls, in the order people hit them:

1. Randomized signatures. A signer that adds entropy (some hardware wallets, some HSM APIs) derives a different seed every call. Test: derive twice, compare. If unequal, your signer is not RFC 6979.
2. v = 27/28 not normalized. The seed then differs across libraries.
3. Hashing the hex string instead of the bytes. keccak256 inputs above are raw bytes: the UTF-8 of the domain string, the 4-byte big-endian gen, the 65 signature bytes.
4. Using the account key directly for DH. Never: the secp256k1 key signs; all encryption flows through the derived 25519 keys.

5. Recipe: publish and verify bundles

Publish once at first launch and on every rotation (gen bump):

const ts = Date.now();
const preimage = utf8("p2p-mes:idbundle:v2") || u32be(gen) || u64be(ts)
              || edPub || (kpBytes ?? empty);
const bundle = cbor({ v: 2, gen, ts, ed_pub: edPub,
                      ...(kpBytes && { kp: kpBytes }),
                      sig: secpSignRecoverable(accountKey, keccak256(preimage)) });
await PUT("/identity", { identity: base64(bundle) });

Verify every bundle you read, before using any key from it:

function verifyBundle(addr: bytes20, blob: bytes): Bundle | "legacy" | null {
  if (blob.length === 32) return "legacy";          // bare Ed25519, unauthenticated
  const b = cborDecode(blob);
  if (b.v !== 2) return null;
  const preimage = utf8("p2p-mes:idbundle:v2") || u32be(b.gen) || u64be(b.ts)
                || b.ed_pub || (b.kp ?? empty);
  if (ecrecover(b.sig, keccak256(preimage)) !== addr) return null;  // forged
  const pin = pins.get(addr);
  if (pin && (b.gen < pin.gen || (b.gen === pin.gen && b.ts < pin.ts)))
    return null;                                    // stale replay -- keep the pin
  pins.set(addr, { gen: b.gen, ts: b.ts, ed_pub: b.ed_pub });
  return b;
}

What to do on each outcome:

Checkpoint: Alice's bundle for gen = 0, ts = 1700000000000, no kp, hashes to 7e12d9e1...24a2f6b1 and recovers to 0x19e7e376e7c213b7e7e7e46cc70a5dd086daff2a (full vector in the guide).

6. Recipe: encrypted DMs

Full wire format: dm-e2ee-v1. The shape to remember: sign, then encrypt, then wrap the key twice.

async function sendDm(peerAddr: bytes20, content: InnerContent) {
  const peer = verifyBundle(peerAddr, await GET(`/identity/${hex(peerAddr)}`));
  if (peer === "legacy" || peer === null) throw new NoE2ee();

  const chatId = blake3(utf8("p2p-mes:chat:dm:v1:")
                || min(myAddr, peerAddr) || max(myAddr, peerAddr));

  const m   = cbor({ t: content.t, ts: Date.now(), ...content.fields });
  const sig = ed25519.sign(utf8("p2p-mes:dm-sig:v1") || chatId || m, seed);

  const contentKey = random(32);
  const eph        = x25519.keygen();
  const ct = aesGcmEncrypt(contentKey, random(12), cbor({ m, sig }));

  const wraps = [ [peerAddr, toMontgomery(peer.ed_pub)],
                  [myAddr,   xPub] ]                      // self-wrap: your
    .map(([addr, staticPub]) => {                         // own history stays
      const kek = hkdfSha256(x25519.dh(eph.priv, staticPub),
                             /*salt*/ empty, /*info*/ utf8("p2p-mes:dm-wrap:v1"));
      return { addr, nonce: random(12),
               key: aesGcmEncrypt(kek, nonce, contentKey) };
    });

  await POST(`/dialogs/${hex(peerAddr)}/messages`, {
    text: "p2pmes:dm-e2ee-v1:"
        + base64(cbor({ eph_pub: eph.pub, wraps, nonce: ct.nonce, ct: ct.bytes })),
  });
}

Receiving (after the standard msg_cbor decode):

1. text does not start with p2pmes:dm-e2ee-v1: -> not this profile. Otherwise strip the prefix, base64-decode the rest, and CBOR-decode the envelope.
2. Find the wrap where addr == myAddr; derive the same kek from your static X25519 private and eph_pub; unwrap content_key; decrypt ct.
3. Determine the expected author from the decoded message's sender field (in a DM it is either the peer or you). Verify sig over "p2p-mes:dm-sig:v1" || chat_id || m against that party's verified bundle key -- your own ed_pub for echoes of your messages.
4. Reject on any failure. On success, check the replay set (hash of sig); duplicates are replays, drop them. Display using the signed inner ts.
5. Dispatch on t: 0 text, 10 attachments (continue in the attachments receive flow), 100..255 your app's private types.

Pitfalls:

• Nonce reuse is fatal in GCM. Every random(12) above is fresh: per-wrap and per-message. Never derive nonces from counters you might reset.
• The info strings are load-bearing. "p2p-mes:dm-wrap:v1" (HKDF) and "p2p-mes:dm-sig:v1" (signature context) must match byte-for-byte.
• Sign m, the CBOR bytes -- not the decoded object. Serialize once, sign those bytes, ship those bytes. Re-encoding on the other side may produce different bytes and a false verification failure.
• Don't skip verification for your own echoes. A malicious node can forge sender just as easily.

Checkpoint (vector in the guide): with the fixed test inputs, Bob's kek is 9bef6660...69e955b3 and the full envelope is 398 bytes.

7. Recipe: attachments in each carriage

The attachment pipeline (encrypt -> upload -> metadata) is fully specified in the guide's Attachments section, with its own vector. Composition rules:

• Plaintext chat: send the MultiRemoteAttachment CBOR as control with msg_type = 10. Operators can read the metadata and decrypt blobs -- same trust level as plaintext text.
• Encrypted DM: put the same fields into the dm-e2ee inner content with t = 10 ({ t: 10, ts, attachments: [...], caption? }). Operators see nothing.
• MLS group: put the same fields into the application-message framing with t = 10 -- { t: 10, ts, attachments: [...], caption? }, the same InnerContent map as the DM carriage (normative, see 8.3) -- and send the encrypted result through the plain group endpoint with the p2pmes:mls-app-v1: prefix. Operators see nothing.

The plaintext case is deliberately unchanged by the text-carriage migration: MultiRemoteAttachment is a protocol structure -- its schema is fixed by the protocol, a client only renders it -- so it legitimately rides the control channel. This section is the reference example of what control is for; user content never claims a control type.

The blob-side work (fresh secret/salt/nonce per file, HKDF -> AES-256-GCM, SHA-256 digest, content-addressed upload) is identical in all three -- write it once. On receive, always run the security checklist in order: scheme check -> size cap while streaming -> digest verify -> decrypt -> render.

8. Recipe: MLS for p2p-mes

RFC 9420 explains MLS itself; read it (or its many summaries) separately. This section is only about the choices that make MLS run on p2p-mes and the operational rules that keep it healthy.

8.1 Fixed choices

Capabilities hold enum names, not numeric ids. In ts-mls the capability lists carry string keys ("mls10", the ciphersuite name, "basic"); numeric registry ids are looked up at encode time. Passing numbers (versions: [1]) encodes garbage silently: signatures still verify, because every party encodes the same garbage, but any TLS decode of a leaf node, ratchet tree, or commit that embeds such capabilities returns nothing. This one mistake is what historically looked like "ts-mls cannot round-trip its own encodings" -- see interop status.

Verification rule: when you receive a KeyPackage (from a bundle or a Welcome), check the chain before trusting it: the bundle signature proves ed_pub belongs to the address; the KeyPackage must be signed by that same ed_pub; the credential inside must spell that same address. Any mismatch -> reject the member.

8.2 Deterministic keys and the KeyPackage

MLS normally wants fresh, single-use KeyPackages published to a delivery service. p2p-mes deliberately deviates: keys are deterministic from the seed, and one last-resort KeyPackage lives in the identity bundle. That buys offline adds (invite a user who has never been online at the same time as you) and Welcome recovery from the bare seed, at the accepted cost that a seed compromise exposes recorded Welcomes.

Construction (mirror this exactly):

const kp = await generateKeyPackageWithKey(credential, capabilities,
             defaultLifetime, [], { signKey: seed, publicKey: edPub }, cs, []);
// Override the random HPKE keys with the deterministic ones:
kp.publicPackage.initKey                 = toMontgomery(edPub);
kp.privatePackage.initPrivateKey         = toMontgomerySecret(seed);
const enc = x25519Keypair(sha256(seed || utf8("mls-enc-key"))); // leaf key,
kp.publicPackage.leafNode.hpkePublicKey  = enc.pub;             // MUST differ
kp.privatePackage.hpkePrivateKey         = enc.priv;            // from initKey
// The keys changed, so both signatures must be redone, leaf first:
kp.publicPackage.leafNode = await signLeafNodeKeyPackage(leafTbs, seed, cs.signature);
kp.publicPackage          = await signKeyPackage(kpTbs, seed, cs.signature);

Serialize publicPackage (TLS encoding) into the bundle's kp field. Both sides can rebuild the same KeyPackage from the same seed, which is what makes state-loss recovery possible.

8.3 Transport mapping

The MLS handshake rides group control messages (POST /groups/{chat_id}/messages/control, payloads base64) -- these are protocol structures, which is what control is for:

Application messages are the group's user content, so they are not a control type: send the TLS-encoded MLSMessage (mls_private_message) through the plain group endpoint, prefixed and base64-encoded in text -- no CBOR wrapper around the MLS bytes:

await POST(`/groups/${hex(chatId)}/messages`, {
  text: "p2pmes:mls-app-v1:" + base64(encodeMlsMessage(privateMessage)),
});

All these payloads (welcome, tree, commit, and the application MLSMessage) are standard TLS presentation-language encodings (RFC 9420 wire structures) -- nothing library-specific. In ts-mls terms: commit is encodeMlsMessage(result.commit) (what createCommit returns in commit already is a complete MLSMessage -- do not wrap it again), welcome is encodeMlsMessage({version: "mls10", wireformat: "mls_welcome", welcome: result.welcome}), and tree is encodeRatchetTree(state.ratchetTree) / decodeRatchetTree. Note the tree codec is not re-exported from the ts-mls package root -- deep-import it from ts-mls/dist/src/ratchetTree.js.

The ratchet tree rides with the Welcome instead of relying on the optional ratchet_tree GroupInfo extension, so a joiner never depends on the committer having embedded it. If welcome + tree outgrows the 32 KiB control cap (roughly 150+ members), send {"ref": RemoteAttachmentInfo} instead: upload the CBOR as an encrypted blob via the attachments pipeline and let the joiner fetch it.

Application-message framing (normative). The plaintext handed to MLS (createApplicationMessage) is the same InnerContent CBOR map as dm-e2ee-v1:

plaintext = InnerContent = {
  "t":  uint,          ; 0 = text ("text": tstr)
  "ts": uint,          ; sender wall-clock ms, authenticated display time
  ...                  ; 10 = attachments ("attachments" / "caption")
}                      ; 100-255 = app-private

There is no SignedContent wrapper and no extra signature here: MLS already authenticates the sender (leaf signature key) and binds the message to the group and epoch. ts rides inside the AEAD, so it is the one display time a relay cannot forge -- same rationale as in the DM profile (hlc/origin_wall_ts are not client-authenticated). This framing is what makes two independent clients read each other's groups: a byte-compatible MLS layer is not enough if the plaintext layout differs, so do not invent a private one.

Membership change = one compound call. Never send the protocol-level op and the MLS messages separately; bundle them so they propagate together:

// Adding Bob: fetch + verify his bundle, take kp from it, then:
const { newState, welcomeBytes, commitBytes, ratchetTree } =
  await addMlsMember(state, bobKeyPackage);

await POST(`/groups/${hex(chatId)}/ops`, {
  ops: [ { op_type: "add", target: hex(bobAddr), role: 0, sig: opSig } ],
  messages: [
    { msg_type: 21, control: b64(cbor({ commit: commitBytes })), recipients: [] },
    { msg_type: 20, control: b64(cbor({ welcome: welcomeBytes,
                                        tree: encodeRatchetTree(ratchetTree) })),
      recipients: [hex(bobAddr)] },
  ],
});
persist(newState);   // AFTER the POST succeeded

recipients: [] fans out to all members; the Welcome targets only the joiner's inbox. Removal is the same shape with a remove op and a Commit carrying the remove proposal. There is still no atomicity between the ops and the messages (see the guide's operational notes) -- which is why reconciliation below exists.

Two earlier carriages are superseded: an early demo used msg_type 11/12 for Welcome/Commit and shipped application ciphertext hex-encoded in text; a later revision moved dm-e2ee envelopes and MLS application messages into control as msg_type 11 and 22. Today user content -- including ciphertext -- rides text with a format prefix, and control carries protocol structures only (10, 20, 21). Do not reuse 11, 12, or 22.

8.4 Interop status (read before mixing stacks)

The bstr payloads in 8.3 are plain RFC 9420 TLS encodings, and on the pinned ts-mls 1.6.2 they are verified to round-trip: MLSMessage for Welcome and Commit decodes and re-encodes byte-identically, a member that only ever saw the wire bytes processes the Commit, and a joiner joins from wire-decoded welcome + tree alone. Nothing in the convention is ts-mls-specific; a non-ts-mls client (e.g. openmls) interoperates by producing the same TLS structures.

Two legacy traps, kept here because both fail confusingly:

• "TLS decode is broken" is a capabilities bug, not a library bug. Earlier revisions of this section claimed only the Welcome round-trips in TLS and prescribed library-internal CBOR for commit and tree. The real cause was capabilities built with numeric enum ids (see 8.1): they encode silently into garbage that still signature-verifies, and then every TLS decode of a leaf, tree, or commit returns nothing. Use string enum names and TLS round-trips fine.
• An early internal demo used msg_type 11/12 for Welcome/Commit, with the commit bytes as library-internal CBOR of the ts-mls object. That format is superseded and must not be implemented: 11, 12, and 22 are unassigned today (11 and 22 briefly carried dm-e2ee envelopes and MLS application ciphertext as control types before user content moved to text). The MLS handshake speaks 20/21 with the TLS payloads above, application messages ride text with the p2pmes:mls-app-v1: prefix, and there are no wire-compatibility obligations to the old formats.

8.5 The receiving loop

For each group, poll GET /groups/{chat_id}/messages and process in HLC order (that is the order the range endpoint returns):

1. Deduplicate by msg_id (nodes may deliver duplicates after sync).
2. Dispatch: control msg_type = 20 -> if it is addressed to you and you are not yet in the group, join (joinGroup with the bundled tree); control msg_type = 21 -> processMessage, then persist the new state before acking anything to the UI; a text starting with p2pmes:mls-app-v1: -> strip the prefix, base64-decode, decrypt as an application message, then CBOR-decode the plaintext as InnerContent (see 8.3) and dispatch on its t exactly as in the DM receive flow (0 text, 10 attachments).
3. An application message or commit from an epoch ahead of yours means you missed a commit: buffer it, keep reading history forward -- the missing commit is earlier in the range. If you reach the tail and the gap remains (retention ate it), go to recovery.
4. An epoch behind yours: a duplicate or a competing commit that lost the race; drop it. Two concurrent commits on one epoch are resolved by order: the first one in HLC order wins, the second fails to apply -- its author must rebase (re-issue on the new epoch).

8.6 State: persist, backup, restore

MLS state is the one thing that is not derivable from the seed: it ratchets with every message. Rules:

• Persist the serialized ClientState after every successful operation (send or receive). Treat it like a database, not a cache.
• encodeGroupState does not include your config. In ts-mls, ClientState = GroupState & { clientConfig }, and the state codec covers only the GroupState part. After decodeGroupState, reattach your clientConfig ({ ...decoded, clientConfig }) before first use; a bare decoded state crashes on its first send/receive (Cannot read properties of undefined (reading 'paddingConfig')). Keep the config next to your snapshots -- it never travels inside them.
• Encrypted backup (recommended): after every commit (debounced), encrypt a state snapshot with the backup key (HKDF-SHA256(seed, info = "p2p-mes:state-backup:v1"), AES-256-GCM, fresh nonce) and upload it via the attachments pipeline (content-addressed blob). Keep the latest (digest, url) pointer locally and, optionally, as a self-addressed encrypted DM so it survives the device.

Restore paths, in order of preference:

1. Snapshot + replay: fetch and decrypt the latest snapshot, then process all group messages with HLC after it. Complete recovery.
2. Welcome replay: your Welcome (msg_type 20) is still in the stored history for retention-window days, and its decryption key derives from the seed. Re-join from it, then replay every later commit in order.
3. Re-add: history no longer contains what you need. Ask any admin (encrypted DM) to remove and re-add you -- new epoch, forward-only. History before the re-add stays unreadable; that is forward secrecy working as intended, not a bug to fix.

8.7 Reconciliation: membership list vs. MLS roster

The node's member list (GET /groups/{chat_id}/members) and the MLS roster can diverge because ops and MLS messages are not atomic. Run this check on every group open (cheap: one HTTP call + local compare):

Convention: repairs are idempotent -- a stale repair Commit simply fails the epoch check and is dropped, so multiple admins racing to fix the same divergence is safe.

8.8 Do / Don't

• Do verify the full chain (bundle sig -> ed_pub -> KeyPackage sig -> credential address) for every KeyPackage you accept.
• Do persist state after every MLS operation, before updating the UI.
• Do send membership ops and their MLS messages in one compound call.
• Don't encrypt application messages to a roster the members list says is stale (removed member still in your tree) -- reconcile first.
• Don't process MLS messages out of HLC order.
• Don't put user content on the control channel -- ciphertext is still user content and rides text with a format prefix.
• Don't reuse control msg_type slots 10/20/21 (or the retired 11/12/22) for anything else; keep app-private control types in 100-255 and mark app-private text formats with your own prefix.
• Don't ship the account key or the seed into your MLS library's storage; hand it only the derived keys it needs.

9. Consolidated checkpoints

All from the guide's vectors (fixed test keys 0x11*32 / 0x22*32):

Also replay the signing vectors in test-vectors.json first -- E2EE bugs are unreachable while request signing is broken.

10. Troubleshooting

HTTP API

Overview

Framework: axum Swagger UI: /swagger-ui OpenAPI JSON: /api-docs/openapi.json Default bind: http://localhost:3000 (configurable via listen_api in TOML config) TCP backlog: 4096 (explicit, for high-RPS scenarios) TCP_NODELAY: enabled (disables Nagle's algorithm for low-latency responses) Keepalive idle timeout: 30 s (header_read_timeout -- closes idle connections cleanly) Concurrency limit: 2400 in-flight requests (tower ConcurrencyLimitLayer, derived from Little's law for 100-node deployment at 600 K peak RPS); public discovery endpoints have a separate 256-slot limit so unauthenticated calls cannot occupy the main budget HTTP server: manual accept loop with hyper_util::server::conn::auto::Builder (not axum::serve) to expose hyper's HTTP/1.1 keepalive tuning

Two-Layer Design

The API exposes both layers of the node architecture:

Layer 1 endpoints (node-enforced):

• Message sending/reading: POST/GET /dialogs/{peer}/messages, POST/GET /groups/{chat_id}/messages
• Single message by id: GET /messages/{msg_id}; edit/delete own message: PATCH/DELETE /messages/{msg_id}
• Inbox: GET /conversations
• Group administration: POST /groups/{chat_id}/ops, DELETE /groups/{chat_id}/membership
• Group members: GET /groups/{chat_id}/members
• Read progress: POST /dialogs/{peer}/messages/read, POST /groups/{chat_id}/messages/read
• Identity: PUT /identity, GET /identity/{address}

Layer 2 fields (client-opaque, passed through):

• msg_type (u8) in message send body -- node stores as-is
• control (base64 string -> Vec) in DM/group control endpoints -- opaque blob
• Identity stored in dedicated identity CF via PUT/GET /identity endpoints (see below)

Channel semantics (normative): POST .../messages (text) is the only transport for user content -- literal text or a client-format envelope (E2EE ciphertext in base64 is still a string). POST .../messages/control (msg_type + control) carries protocol / service structures only (attachments metadata, MLS handshake). Clients MUST NOT introduce new control msg_types for user content. See CLIENT_GUIDE.md for the text-prefix convention that marks client formats inside text.

A client can use Layer 1 alone for plaintext messaging. Layer 2 enables any client-side protocol (E2EE, key exchange, etc.) via opaque fields. See ARCHITECTURE.md for the full two-layer description.

Authentication

All endpoints require ECDSA signature-based authentication via headers -- except the two public discovery endpoints (GET /node/info, GET /network/nodes), which are unauthenticated: clients call them during bootstrap before they have anything to sign, and infrastructure health checks cannot produce signatures.

Signature Verification

1. Build canonical string-to-sign:

   p2p-mes-v1
   METHOD:{METHOD}
   PATH:{path}
   QUERY:{canonical_query}
   BODY:{canonical_body}
   TS:{ts_ms}
   NODE:{node_id}
   

2. Canonicalization rules:

   • Query params: URL-decoded, sorted by key then value, percent-encoded
   • JSON body: flattened to key=value pairs (nested objects use dot notation, arrays use []; an object inside an array is not flattened -- it becomes one pair holding its compact JSON with alphabetically sorted keys, see CRYPTO.md), sorted, percent-encoded
   • Form body: parsed, sorted, percent-encoded
   • Binary body: raw={hex}

3. Compute msg_hash = Keccak256(string_to_sign)

4. Recover public key from ECDSA signature (tries both recovery IDs v=0 and v=1)

5. Derive address: Keccak256(pubkey_uncompressed[1..])[12..32]

6. Compare derived address with X-User claim

Verification Checks

• X-Sig-Version must be "p2p-mes-v1"
• |now - X-Ts| <= 30 seconds
• X-Node must match this node's PeerId
• Recovered address must match X-User

Endpoints

POST /dialogs/{peer}/messages

Send a direct message.

Path params:

• peer -- recipient's address (hex 0x-prefixed, 20 bytes)

Request body:

{
  "text": "Hello, world!"       // 1-45056 bytes of UTF-8 (44 KiB), byte-counted
}

Response 200:

{
  "chat_id": "0x...",           // 32 bytes hex
  "msg_id": "0x...",            // 32 bytes hex
  "ts": 1699900000000           // Originator wall-clock at send time (= MsgV1.origin_wall_ts)
}

Notes:

• chat_id is computed deterministically: BLAKE3("p2p-mes:chat:dm:v1:" || min(sender, peer) || max(sender, peer))
• Message is published to gossip immediately, response returns before DB commit
• msg_type = 0 (regular text message)
• text is the only transport for user content, encrypted or not; the 44 KiB cap is sized so a 32 KiB payload fits after base64 (+33%) plus a client format prefix. The cap counts bytes, not chars

POST /dialogs/{peer}/messages/control

Send a control message carrying a protocol / service structure (attachments metadata, MLS handshake).

Path params:

• peer -- recipient's address (hex 0x-prefixed)

Request body:

{
  "msg_type": 10,               // 1-255 (0 is reserved for regular text)
  "control": "pGplbmNyeXB0aW9u"  // base64-encoded CBOR payload; max 32 KiB (<= 43692 base64 chars)
}

Response 200:

{
  "chat_id": "0x...",
  "msg_id": "0x...",
  "ts": 1699900000000           // Originator wall-clock at send time
}

Notes:

• text is set to empty string for control messages
• The control payload is opaque to the node -- passed through as-is
• msg_type is client-defined opaque u8 (node does not interpret it); known conventions: 10 = attachments, 20 = MLS Welcome, 21 = MLS Commit
• Control is not a transport for user content -- encrypted user messages ride text on the plain endpoint (see CLIENT_GUIDE.md)

GET /conversations

List user's chats (inbox) with unread count.

Query params:

• limit -- max results (1-1000, default 50, capped at 500)
• after -- pagination cursor (hex-encoded raw key from previous response)

Response 200:

{
  "items": [
    {
      "chat_id": "0x...",
      "kind": { "type": "dm", "peer": "0x..." },
      "last_ts": 1699900000000,
      "last_sender": "0x...",
      "unread": 3,
      "cursor": "0x..."
    }
  ],
  "next_after": "0x..."         // null when the page was not full (no continuation cursor)
}

Notes:

• Results sorted by most recent first (reverse timestamp in RocksDB key)
• kind is one of: {"type": "dm", "peer": "0x..."}, {"type": "group", "title": "..."}, {"type": "channel", "title": "..."}. channel is a reserved type: there are no channel create/post/subscribe endpoints yet, so clients currently encounter only dm and group
• kind.title for groups is the immutable creation title (see POST /groups/{chat_id}/ops); null for unnamed groups. The node hydrates it from chats_meta when writing the inbox entry, so it is present regardless of which node stored the triggering message
• For dm chats, kind.peer here is viewer-relative: the node rewrites each user's inbox entry to hold the other participant, so the caller always sees their interlocutor. This is deliberately different from the kind embedded in msg_cbor (wire messages), where d.peer is the original recipient fixed by the sender -- see the notes on GET /dialogs/{peer}/messages
• unread = last_seq - last_read_seq (from user_read_progress CF)
• The response carries no content preview by design: chat-list lines are the client's job, rendered from the local message cache (an E2EE envelope is opaque to the node anyway; see CLIENT_GUIDE.md)

GET /dialogs/{peer}/messages

Get DM message history with a specific peer.

Path params:

• peer -- peer's address (hex 0x-prefixed)

Query params:

• from -- start timestamp in ms (default 0)
• to -- end timestamp in ms (optional, default unlimited)
• after -- pagination cursor (hex-encoded RocksDB key)
• limit -- max results (1-1000, default 100)
• reverse -- scan direction (default false). false walks oldest -> newest starting at from; true walks newest -> oldest starting at to, i.e. "load the chat tail, then scroll up". Keep this flag stable across one pagination sequence -- after is the same opaque cursor in both modes, only the direction of travel flips

Response 200:

{
  "items": [
    {
      "key": "0x...",           // Hex-encoded RocksDB key (for pagination)
      "msg_cbor": "0x..."       // Hex-encoded CBOR MsgV1
    }
  ],
  "next_after": "0x..."         // null when the page was not full (no continuation cursor)
}

Notes:

• chat_id is derived from (user, peer) -- no membership check needed for DMs
• Client must decode CBOR msg_cbor to get message fields (sender, text, hlc, origin_wall_ts, seq, msg_type, control, kind). origin_wall_ts is the frozen sender wall-clock for UI display; hlc is the network-consistent stamp used for storage ordering and sync. Two optional fields signal a client-side edit or delete: edited_at (packed HLC, present once the message has been edited) and deleted (true on a deleted stub, whose text is empty) -- both absent on untouched messages; see PATCH/DELETE /messages/{msg_id}
• Inside msg_cbor, kind.t is a CBOR text string ("0" dm / "1" group / "2" channel, not an integer), and for DMs kind.d.peer is the original recipient fixed by the sender -- in an incoming message it equals the reader's own address. Interlocutor = d.peer == me ? sender : d.peer. This differs from /conversations, whose kind.peer is viewer-relative. See the decoding reference in CLIENT_GUIDE.md
• next_after contract: the cursor is emitted only when the page hit limit; a shorter or empty page returns next_after: null, which means "end of this node's current view", not "end of history". Writes are async (200 before commit) and nodes converge via sync, so an empty page with a null cursor is ambiguous -- keep the after you used and retry later instead of concluding the history is exhausted. The cursor is the storage key of the page's last row, so any item's key is a valid after value (tail-polling can resume from the newest item received). A non-null cursor only means "the page was full" -- the next call may return an empty page
• Forward (reverse=false) returns messages oldest-first (ascending HLC, which is monotonic across sends); reverse (reverse=true) returns them newest-first, with next_after pointing at the oldest row of the page so the next call walks further back
• Under full replication the queried node serves this from its local store and responds immediately -- there is no 30 s network wait (that path only applies to the disabled sharded mode). A node that has not finished syncing returns its partial local view, and there is no completeness signal, so treat history as eventually consistent
• skip_membership_check = true for DM routes

POST /dialogs/{peer}/messages/read

Mark messages as read up to a given sequence number.

Path params:

• peer -- peer's address (hex 0x-prefixed)

Request body:

{
  "seq": 123                    // Sequence number (>= 1)
}

Response 200: empty body

Notes:

• Marks all messages with seq <= given_seq as read
• Broadcasts ReadProgress via gossip to update other nodes
• Read progress is monotonic: only advances, never goes backward

DELETE /groups/{chat_id}/membership

Leave a group (self-remove).

Path params:

• chat_id -- group chat identifier (hex 0x-prefixed, 32 bytes)

Request body:

{
  "sig": "0xabcdef..."            // ECDSA sig over keccak256(chat_id || sender || 1)
}

Response 200: empty body

Error 403: {"error": "owner cannot leave group"}

Notes:

• The group owner cannot leave -- they must transfer ownership first (op_type: "transfer" via POST /groups/{chat_id}/ops). Admins and participants leave freely
• Signature is required for gossip propagation verification
• Inbox entry for the chat is cleared so the group disappears from /conversations
• Publishes MembershipOp(Remove, target=self) via gossip for cross-node propagation; remote nodes receive the gossip and clear their local inbox copy for this user too

Group Messages

All group message endpoints require the caller to be a current member of the group (verified via is_member check in the MPSC handler).

POST /groups/{chat_id}/messages

Send a text message to a group.

Path params:

• chat_id -- group chat identifier (hex 0x-prefixed, 32 bytes)

Request body:

{
  "text": "Hello group!"           // 1-45056 bytes of UTF-8 (44 KiB), byte-counted
}

Response 200:

{
  "chat_id": "0x...",
  "msg_id": "0x...",
  "ts": 1699900000000           // Originator wall-clock at send time
}

Notes:

• text is the only transport for user content, encrypted or not (e.g. MLS application messages ride here with a client prefix); the 44 KiB cap counts bytes and fits a 32 KiB payload in base64 plus a prefix
• The group title is fixed at creation (title in POST /groups/{chat_id}/ops, part of the chat_id derivation) and reported by /conversations and /members; it does not travel with individual messages
• Membership check happens in put_message MPSC handler
• Non-members receive error: "not a group member"
• Inbox upsert for all members happens locally in process_db_op after message storage

POST /groups/{chat_id}/messages/control

Send a control message carrying a protocol / service structure to a group (attachments metadata, MLS Welcome/Commit). Not a transport for user content -- encrypted group messages ride text on the plain endpoint.

Path params:

• chat_id -- group chat identifier (hex 0x-prefixed, 32 bytes)

Request body:

{
  "msg_type": 20,
  "control": "pGplbmNyeXB0aW9u"    // base64-encoded CBOR; max 32 KiB (<= 43692 base64 chars)
}

Response 200:

{
  "chat_id": "0x...",
  "msg_id": "0x...",
  "ts": 1699900000000           // Originator wall-clock at send time
}

GET /groups/{chat_id}/messages

Get group message history (paginated).

Path params:

• chat_id -- group chat identifier (hex 0x-prefixed, 32 bytes)

Query params:

• from -- start timestamp in ms (default 0)
• to -- end timestamp in ms (optional)
• after -- pagination cursor (hex-encoded RocksDB key)
• limit -- max results (1-1000, default 100)
• reverse -- scan direction (default false). Same semantics as the DM endpoint: false = oldest-first from from, true = newest-first from to (chat tail, then scroll up)

Response 200:

{
  "items": [
    {
      "key": "0x...",
      "msg_cbor": "0x..."
    }
  ],
  "next_after": "0x..."
}

Notes:

• skip_membership_check = false -- explicit membership verification
• Non-members receive empty result (not an error)
• Same next_after contract and msg_cbor decoding notes as GET /dialogs/{peer}/messages above

POST /groups/{chat_id}/messages/read

Mark group messages as read up to a given sequence number.

Path params:

• chat_id -- group chat identifier (hex 0x-prefixed, 32 bytes)

Request body:

{
  "seq": 123
}

Response 200: empty body

Notes:

• skip_membership_check = false -- non-members receive error
• Broadcasts ReadProgress via gossip

GET /groups/{chat_id}/members

List group members with roles plus the group's creation title. Pure local read from members CF + chats_meta -- no gossip roundtrip needed.

Path params:

• chat_id -- group chat identifier (hex 0x-prefixed, 32 bytes)

Response 200:

{
  "members": [
    {
      "address": "0x1234...",
      "role": 1
    },
    {
      "address": "0x5678...",
      "role": 0
    }
  ],
  "title": "My Group Chat"
}

Notes:

• Only current members can view the list
• Role values: 0 = participant, 1 = admin, 2 = owner (exactly one per group)
• title is the group's immutable creation title; null for unnamed groups (and for groups whose creation meta has not reached this node yet)
• Non-members receive error: "not a group member"

POST /groups/{chat_id}/ops

Compound membership operation: create group, add/remove members with optional accompanying messages (MLS Welcome/Commit). This is the only group management endpoint.

Path params:

• chat_id -- group chat identifier (hex 0x-prefixed, 32 bytes)

Request body:

{
  "ops": [
    {
      "target": "0x1234...5678",
      "sig": "0xabcdef...",
      "role": 0,
      "op_type": "add"
    }
  ],
  "messages": [
    {
      "text": "",
      "msg_type": 20,
      "control": "pGplbmNyeXB0aW9u",
      "recipients": ["0x1234...5678"]
    }
  ],
  "nonce": "0xabcdef1234567890abcdef1234567890",
  "title": "My Group Chat"
}

Response 200:

{
  "ops_processed": 1,
  "messages_sent": 1
}

Authorization:

• ECDSA signature recovery per operation (API-level crypto check)
• For create ops: nonce field is required (exactly 16 bytes); API verifies chat_id == blake3(domain || signer || nonce || title) where title defaults to the empty string when omitted
• Role-based check in MPSC handler (DB access):
  • create: no prior state needed; signer becomes the owner (role=2)
  • add: signer must be admin or owner. Assigning role: 1 (admin) or changing an existing member's role requires the owner; the owner's own role cannot be modified via add; role: 2 is never assignable via add
  • remove: signer must be admin/owner, OR signer == target for self-remove. The owner cannot be removed (by anyone, including self) -- transfer ownership first
  • transfer: signer must be the owner; target (the new owner) must be an active member; self-transfer is rejected. Old owner becomes admin, target becomes owner
  • delete: signer must be the owner. Deletes the whole group

Behavior:

• Membership ops processed first, then accompanying messages
• Inbox entries (including for creator on Create) are updated locally by process_db_op after accompanying message storage
• Duplicate Create rejected if group already has members
• Batch ops published as one MembershipOpBatch gossip message
• Within a batch, later ops see the effect of earlier ones (virtual overlay), so [create, add, add] or [create, transfer, remove(self)] work in a single request
• After a delete op, any further op for the same chat in the same batch is rejected
• delete tombstones all members and clears every member's inbox entry; stored messages are untouched (retention GC removes them later)
• Each op independently verifiable via its own ECDSA signature
• No atomicity guarantee between ops and messages
• ops array must not be empty; messages, nonce and title are optional (nonce becomes required when the batch contains a create op)
• Accompanying messages obey the same size caps as the dedicated message endpoints: text max 45056 bytes, control max 43692 base64 chars (32 KiB decoded); control is base64, like on the control endpoints
• title sets the group's name at creation. It participates in the chat_id derivation (blake3(domain || creator || nonce || title)), so the exact same string must be used when computing the chat_id client-side. Immutable after creation. Constraints: 1..=128 bytes UTF-8, no control characters, only valid alongside a create op. Omit for an unnamed group
• The stored title is returned in GET /conversations (inside kind) and GET /groups/{chat_id}/members
• op_type values: "add", "remove", "create", "transfer", "delete"
• role values: 0 = participant (default), 1 = admin (2 = owner is set only by create/transfer, never accepted in the request)

Error responses for /groups/{chat_id}/ops:

GET /messages/{msg_id}

Fetch a single message by its msg_id. Enables deep-linking straight to one message (e.g. from a push notification) without pulling a whole range.

Path params:

• msg_id -- message identifier (hex 0x-prefixed, 32 bytes)

Response 200:

{
  "msg_id": "0x...",            // Echoed back (hex, 32 bytes)
  "msg_cbor": "0x..."           // Hex-encoded CBOR MsgV1 (same shape as range items)
}

Notes:

• Pure local read -- under full replication every node holds all messages, so there is no gossip roundtrip
• Authorization is enforced node-side from the decoded message: for DMs the caller must be one of the two participants (sender or peer); for groups/channels the caller must be a current member
• A 404 is returned both when the id is unknown and when the caller may not see the message, so message existence is never leaked to non-participants
• The msg_cbor payload is byte-identical to a range item's msg_cbor, so clients reuse their existing decode path
• A message that was edited or deleted carries the optional edited_at / deleted fields inside the decoded msg_cbor (see the decode note under GET /dialogs/{peer}/messages) -- this is how a client detects the edit/delete state. Edits and deletes are issued via PATCH /messages/{msg_id} and DELETE /messages/{msg_id}

Error responses:

PATCH /messages/{msg_id}

Edit one of your own messages. text is replaced in place -- the message keeps the same msg_id and the same position/seq in history, so nothing reorders and pagination cursors stay valid.

Path params:

• msg_id -- message identifier (hex 0x-prefixed, 32 bytes)

Request body:

{
  "text": "corrected text"      // new content, 1-45056 bytes of UTF-8 (44 KiB), byte-counted (same cap as send endpoints)
}

Response 200:

{
  "msg_id": "0x...",            // 32 bytes hex, unchanged
  "op_ts": 1699900000000        // Server-stamped wall-clock component of the operation HLC; surface as "edited at"
}

No operation signature:

• An edit carries no body sig. Authorship at the node rests on the request-level X-Sig auth headers -- the caller must be the message's original sender -- mirroring how a plain message send is authorized (also unsigned)
• The new text is content whose authenticity each recipient verifies end-to-end on Layer 2: the ciphertext is a sign-then-encrypt envelope, checked exactly like a plain message, so no separate operation signature is needed
• Contrast with DELETE /messages/{msg_id}, which does carry an operation sig: a delete leaves only an unauthenticatable deleted: true stub, so the node must authorize it explicitly. Edits are unsigned, deletes are signed

Notes:

• Only the message's original author may edit it
• 48-hour edit window: the edit is accepted only within 48 h of the original message, measured from HLC stamps so all nodes agree on the cutoff
• Deletion is terminal -- once a message is deleted it can no longer be edited
• On success a packed-HLC edited_at field appears on the message and surfaces in the decoded msg_cbor on GET endpoints (see GET /messages/{msg_id})
• The edit propagates to online nodes via gossip and to offline nodes via anti-entropy sync; duplicate delivery is idempotent
• Edits do not reorder the chat or change unread counts (the inbox is left untouched)

Error responses:

DELETE /messages/{msg_id}

Delete one of your own messages. The message becomes a stub: it stays in history at the same msg_id and position, but its text is cleared and a deleted: true flag is set, so pagination stays stable.

Path params:

• msg_id -- message identifier (hex 0x-prefixed, 32 bytes)

Request body:

{
  "sig": "0x..."                // 65-byte ECDSA operation signature (see below), hex
}

Response 200:

{
  "msg_id": "0x...",            // 32 bytes hex
  "op_ts": 1699900000000        // Server-stamped wall-clock component of the operation HLC
}

Operation signature:

• Unlike an edit, a delete requires a body sig -- an operation signature separate from the auth headers. X-Sig authenticates the HTTP request; sig authorizes the delete, must recover to the original message's sender address, and every node re-verifies it on receipt
• The client signs keccak256(canonical_payload) where canonical_payload = chat_id[32] || target_msg_id[32] || op_kind[1] || blake3(new_text)[32] (97 bytes). op_kind = 1 for delete, and since there is no new text blake3("") (BLAKE3 of the empty string) fills the last 32 bytes
• Standard Ethereum ECDSA recovery (r[32] || s[32] || v[1], v in {27,28} or {0,1})
• chat_id is not in the URL -- the node looks it up from the stored message, but the client must feed it into the payload (computed for DMs, path param for groups)
• The HLC / timestamp is not signed -- the server stamps it, mirroring membership ops

Why a delete is signed but an edit is not:

• A delete erases content, leaving only a deleted: true stub. No recipient can authenticate that stub end-to-end -- they cannot tell "the author deleted this" from "a node erased it without permission." So the node itself must authorize the delete against this signature, and every node re-verifies it
• An edit writes new content instead, whose authenticity the recipient checks end-to-end on Layer 2 (a sign-then-encrypt envelope, like any plain message), so it needs only the request-level X-Sig auth and no operation sig. Hence the asymmetry: edits are unsigned, deletes are signed

Notes:

• Only the message's original author may delete it, and the same 48-hour window (measured from HLC stamps) applies
• Deletion is terminal -- a deleted message cannot be revived or re-edited
• The stub keeps the same msg_id and stays in GET responses, so pagination stays stable; clients should render "message deleted" and evict any cached text. The deleted: true flag surfaces in the decoded msg_cbor (see GET /messages/{msg_id})
• Propagates to online nodes via gossip and to offline nodes via anti-entropy sync; duplicate delivery is idempotent
• Deletes do not reorder the chat or change unread counts (the inbox is left untouched)

Error responses:

PUT /identity

Store caller's opaque identity blob. Overwrites any previous value by HLC last-write-wins (no client-visible versioning). The identity is propagated network-wide: the write is gossiped to online peers and reconciled across all nodes by Merkle-tree anti-entropy sync (domain Identity). See PROTOCOL.md (PutIdentity) and SYNC.md.

Request body:

{
  "identity": "SGVsbG8gV29ybGQ="   // base64-encoded blob, max 1024 bytes raw
}

Response 200:

{}

Error responses:

Notes:

• The blob is opaque to the node -- it does not parse or validate the contents
• Key in RocksDB identity CF = caller's 20-byte address
• Routed through DbOp::PutIdentity (async DB writer: HLC last-write-wins, sync-index update, Merkle notify), then published as a PutIdentity gossip message

GET /identity/{address}

Retrieve a previously stored identity blob by user address. Any authenticated user can read any identity.

Path params:

• address -- target user address (hex 0x-prefixed, 20 bytes)

Response 200:

{
  "identity": "SGVsbG8gV29ybGQ="   // base64-encoded blob
}

Response 404: No identity stored for this address.

Notes:

• Direct synchronous read from identity CF
• No membership or ownership check -- any authenticated caller can query any address

Discovery Endpoints (no authentication)

Two public endpoints let clients discover the network without hardcoding node URLs. Neither requires signature headers. See Building a Client for the bootstrap/failover flow.

GET /node/info

Returns the answering node's identity.

Response 200:

{
  "peer_id": "16Uiu2HAm4TfDHnyCsHwyCTDDbAV3g4Cc32DhSXoPzhbfLGN6HpQP",
  "api_url": "https://node1.example.com:8080"
}

Notes:

• peer_id -- this node's libp2p PeerId (base58). Clients need it for the X-Node header when signing requests to this node
• api_url -- the node's advertised API base URL (public_api_url in its config), null if the node does not advertise one
• Served directly from in-process state; no node roundtrip

GET /network/nodes

Returns the catalog of live API-serving nodes known to the answering node, including itself when it advertises a public_api_url.

Response 200:

{
  "nodes": [
    {
      "peer_id": "16Uiu2HAm4TfDHnyCsHwyCTDDbAV3g4Cc32DhSXoPzhbfLGN6HpQP",
      "api_url": "https://node1.example.com:8080"
    },
    {
      "peer_id": "16Uiu2HAmKQKBgQD3g4Cc32DhSXoPzhbfLGN6HpQP4TfDHnyCsHwy",
      "api_url": "https://node2.example.com:8080"
    }
  ]
}

Notes:

• Entries come from the in-memory API registry populated by libp2p identify announcements of currently connected peers (see PROTOCOL.md, "Identify-Based API Discovery") -- every listed endpoint was live at response time
• Nodes that do not set public_api_url (relay-only or private nodes) never appear
• The list is a point-in-time snapshot of the answering node's direct connections, not a global network census; querying any node yields enough live endpoints for failover
• Order is not significant

Error Responses

Two body shapes exist; clients must handle both.

Flat error (any status): returned for everything except declared field validation -- authorization failures, not-found, malformed path parameters (bad hex, wrong id length), malformed cursors, and semantic input errors (unknown op_type, nonce required for create op, identity blob exceeds 1024 bytes). No fields map:

{
  "error": "forbidden"
}

Validation error (400 only): returned when a declared size/range rule on a body or query field fails (text 1-45056 bytes, limit 1-1000, seq >= 1, control payload size, ...). error is always the literal "validation_error" and fields names each offending field:

{
  "error": "validation_error",
  "fields": {
    "text": {
      "msg": "text must be 1..=45056 bytes of UTF-8 (44 KiB)",
      "value": ""
    }
  }
}

Dispatch rule: fields is present if and only if error equals "validation_error". A 400 alone does not imply a fields map.

Internal Architecture

HTTP Request
  |
  v
HTTP Metrics Middleware (optional, if expose_metrics=true)
  |
  v
CORS Layer (allow all origins/methods/headers)
  |
  v
Signature Middleware (auth.rs)
  |
  v
Route Handler
  |
  v
Command::* --> MPSC channel (4096 buffer) --> Event Loop
  |
  v
oneshot channel <-- Response from handler
  |
  v
HTTP Response

Each handler creates a Command variant, sends it via MPSC, and waits on a oneshot channel for the response.

API Reference

This is the complete HTTP API, generated from the server's OpenAPI specification (via utoipa) and rendered with Scalar. The specification is regenerated on every docs build, so it always matches the running node. For request signing, the message lifecycle, and end-to-end examples, read Building a Client first.

The interactive explorer below loads its viewer from a CDN, so it needs network access to render. Offline, use the raw specification at openapi.json, or the Swagger UI served by any running node at /swagger-ui.

Open the interactive API reference in full screen →

Endpoints at a glance

Every endpoint requires the authentication headers described in Building a Client and summarized in API Overview. See Cryptography & Authentication for the signing algorithm.

Architecture

Overview

P2P messenger built on libp2p (GossipSub + Kademlia) with RocksDB storage, axum HTTP API, and ECDSA/Keccak256 signature-based authentication.

Every node is a full replica -- stores all messages, serves all queries. Consistency between nodes is maintained by Merkle-tree anti-entropy sync protocol.

Two-Layer Architecture

The node provides two independent layers. Both work identically for DM and group chats.

Layer 1 -- Node-Enforced (always active)

These behaviors are built into every node and cannot be bypassed:

• Message delivery: PutMessage gossip + DbOp, dedup via seen_msg CF
• Inbox update: Local inbox upsert in process_db_op after PutMessage write, user_inbox CF (gossip-based InboxFanout disabled under full replication)
• Group administration: MembershipOp gossip (Create, Add, Remove), members CF with CRDT add-wins semantics
• Authentication: ECDSA signature verification on every HTTP request
• Authorization: admin role checks for membership ops, is_member checks for group messages
• Sync: Merkle-tree anti-entropy for messages, members, identity, and message_ops (edit/delete) CFs (four independent trees, round-robin sync tick)
• Retention: background GC deletes messages older than RETENTION_WINDOW from CFs messages and seen_msg, XOR-cancels them from the Merkle tree, and applies a soft-filter on both sides of the sync exchange so aged records never resurface from peers (see RETENTION.md)

Layer 2 -- Client-Optional (opaque to node)

The node stores and relays these fields without interpretation:

• msg_type: u8 in PutMessage -- node never interprets this value. Client defines meaning (0 = plain message; protocol structures use 10/20/21 -- registry in CLIENT_GUIDE.md). Any u8 value is valid
• control: Option<Vec<u8>> in PutMessage -- opaque payload, stored and relayed without inspection
• identity: Option<Vec<u8>> in ChatKind::Dm -- stored in chats_meta via last-write-wins, max 128 bytes

A client can operate without Layer 2 entirely -- plaintext messaging works via Layer 1 alone. Or a client can build any E2EE protocol on top: identity blobs distribute keys, control messages carry protocol structures (attachments metadata, MLS handshake), and encrypted user content rides text as prefixed envelopes (DM E2EE and MLS groups already work this way -- see CLIENT_GUIDE.md and E2EE.md).

See PROTOCOL.md for wire format details of each layer.

Crate Structure

Cargo.toml (workspace)
  members: [crates/node]
  path deps: crates/api, crates/db, crates/crypto, crates/types

crates/node/    -- Binary entry point: swarm, event loop, handlers, sync
crates/api/     -- axum HTTP server, auth middleware, DTOs, Command enum
crates/db/      -- RocksDB wrapper: messages, inbox, members, read progress
crates/types/   -- Shared domain primitives: ChatKind, fixed-length IDs
crates/crypto/  -- ECDSA recovery, Keccak256, Ethereum-style address derivation

Dependency Graph

node --> api --> types
 |       |
 |       +--> crypto
 |
 +--> db --> types
 |
 +--> types
 +--> crypto

Event Loop

Single tokio::select! loop in crates/node/src/lib.rs::run_node() handling:

The retention GC runs as a separate background task spawned next to the event loop (tokio::spawn(retention::run_gc_loop(...)) in run_node). It shares the same shutdown_token so graceful shutdown also stops the GC. The GC takes the Merkle write lock directly per chunk; it does not flow through process_db_op.

Data Flow: Sending a Message

Client --> HTTP POST /dialogs/{peer}/messages
  |
  v
Auth middleware (ECDSA signature verification)
  |
  v
Command::PutMessage --> MPSC channel --> Event loop
  |
  +--> 1. Compute msg_id (BLAKE3)
  +--> 2. Publish GossipMessage::PutMessage to topic "p2p-mes/commands"
  +--> 3. Send DbOp::PutMessage to async DB writer
  +--> 4. process_db_op stores message + upserts inbox for all members
  +--> 5. Respond to HTTP client immediately (fire-and-forget)

All other nodes receive gossip:
  +--> Store message via DbOp::PutMessage (dedup via seen_msg)
  +--> process_db_op upserts inbox locally (no separate InboxFanout)

Data Flow: Reading Messages

Client --> HTTP GET /dialogs/{peer}/messages
  |
  v
Auth middleware --> Command::GetChatRange --> MPSC
  |
  v
Handler checks local DB first:
  - If data exists locally: respond from DB
  - If not: publish Query via gossip, wait for QueryResponse (30s timeout)

Data Flow: Compound Membership Operation

Client --> HTTP POST /groups/{chat_id}/ops
  |         Body: { ops: [...], messages: [...], nonce: "...", title: "..." }
  v
Auth middleware (ECDSA sig verification)
  |
  v
Binding verification (for Create ops):
  chat_id == blake3(domain || signer || nonce || title)
  |
  v
Command::MembershipOp --> MPSC channel --> Event loop
  |
  +--> 1. Verify ops: admin sig, role checks, Create binding re-check,
  |       duplicate Create guard
  +--> 2. Publish GossipMessage::MembershipOpBatch (all ops in one message;
  |       Create ops carry title + nonce for remote re-verification)
  +--> 3. Send DbOp::MembershipOp for each op (local write to members CF);
  |       Create also sends DbOp::SetGroupCreateMeta (title/creator/nonce
  |       into chats_meta, write-once)
  +--> 4. For each accompanying message:
  |       a. Compute msg_id, publish PutMessage gossip
  |       b. Send DbOp::PutMessage (process_db_op upserts inbox locally)
  +--> 5. Respond to HTTP client

One HTTP call produces two categories of gossip traffic:

• 1 MembershipOpBatch message (membership ops, written to members CF)
• M PutMessage messages (client data such as MLS Welcome/Commit, written to messages CF)

Membership ops are processed before accompanying messages. Inbox updates happen locally in process_db_op after each PutMessage is stored.

Handler Architecture

Two handler families share HandlerContext:

HandlerContext {
  db: Arc<ChatDb>,
  swarm: &mut Swarm<MyBehaviour>,
  pending_queries: &mut HashMap<[u8; 16], PendingQuery>,
  peer_cache: &PeerCache,
  local_peer_id: PeerId,
  local_peer_id_str: Arc<str>,   // cached PeerId string, avoids Base58 encode per request
  db_write_tx: DbOpSender,
  inbox_batch_tx: Option<InboxBatchSender>,
}

• MPSC handlers (handlers/mpsc/): process HTTP API commands, publish gossip, respond to client
  • leave_group -- owner prevention check (admins and participants may leave; the owner must transfer ownership first), publishes MembershipOp(Remove, self) via gossip, then queues DbOp::DeleteInboxEntry so the chat disappears from /conversations
  • put_message -- publishes PutMessage gossip + queues DbOp::PutMessage; inbox upsert handled by process_db_op; is_member gate rejects non-members for Group/Channel chats
  • message_op -- edit/delete: loads the target message, checks authorship (editor == sender) + the 48h edit window, stamps the HLC, queues DbOp::MessageOp, then publishes MessageOp gossip. A delete additionally carries a client signature the handler verifies (it needs the target's chat_id, absent from the URL) and every node re-verifies -- erasure leaves a deleted stub no recipient can authenticate, so the node is the last line of defence. An edit is unsigned: it writes new content the recipient checks on Layer 2, so authorship rests on the request-level X-Sig, like a plain send. A node offline at publish time converges via the MessageOps sync domain
  • membership_op -- compound membership: publishes one MembershipOpBatch gossip message; duplicate Create protection; role-based authorization (owner/admin/participant, see PROTOCOL.md); inbox upsert handled by process_db_op after accompanying PutMessage writes; per Remove op queues DbOp::DeleteInboxEntry for the target; TransferOwnership and DeleteGroup dispatch dedicated atomic DbOps; authorization within the batch uses a virtual-state overlay so later ops see the effect of earlier ones (e.g. [Create, Add, Add] or [Create, Transfer, Remove(self)]); after a DeleteGroup op the chat is terminal for the rest of the batch
  • get_group_members -- pure local read from members CF, no gossip roundtrip; returns (address, role) pairs; is_member gate rejects non-members
• Gossip handlers (handlers/gossip/): process incoming GossipSub messages, write to DB
  • Includes membership_op handler with independent sig verification and duplicate Create protection; for Remove ops queues DbOp::DeleteInboxEntry for the target
  • message_op -- feeds the incoming HLC through HlcState::receive; for a delete, re-verifies the client signature (recovered signer must equal editor), never trusting the relaying peer; an edit is unsigned and relayed like a plain message. Then queues DbOp::MessageOp; authorship/window/terminal-delete are settled by the fold in db::message_ops against the stored row
  • MembershipOpBatch handler processes ops in order with a shared BatchMembers overlay (handlers/membership_batch.rs) so authorization for later ops can see the effect of earlier ones without waiting for the FIFO DB writer to flush; the overlay also tracks chats deleted within the batch (deleted_chats) and rejects any op that follows a DeleteGroup for the same chat_id

Async DB Writer

All DB writes are non-blocking. DbOp variants are sent via unbounded channel to spawn_db_writer() which processes them in spawn_blocking (blocking thread pool).

DbOpSender is a thin wrapper around UnboundedSender<DbOp> that increments the db_writer_queue_depth Prometheus gauge on every successful send. The matching decrement happens inside spawn_db_writer when the op is received. This gives real-time observability into writer backpressure without adding bounded-channel complexity.

HTTP handler / Gossip handler
  |
  v
db_write_tx.send(DbOp::PutMessage { ... })  // non-blocking, increments queue depth gauge
  |
  v
spawn_db_writer task (background):
  +--> decrement queue depth gauge
  +--> process_db_op() in spawn_blocking
  +--> on success: notify Merkle tree via MerkleSenders
  |    (msg_tx for messages, member_tx for members, identity_tx for identity)
  +--> increment chat_messages_stored_total metric
  +--> upsert inbox for all chat members (local, no gossip)

The DB writer is the single source of truth for metrics, Merkle tree updates, and inbox upserts. All paths (gossip, HTTP API, sync) converge in process_db_op.

HLC State

Per-node Hybrid Logical Clock that timestamps every CRDT-bearing op (members, identity, messages). Constructed once in run_node as Arc<HlcState> and shared with every HandlerContext cloned in the event loop. Three operations matter:

• stamp() -- called by MPSC handlers (mpsc/membership_op.rs, mpsc/leave_group.rs, mpsc/set_identity.rs, mpsc/put_message.rs) whenever the originating API node needs an HLC for an outgoing op. Wait-free under uncontended load; cost is comparable to the legacy now_millis() call it replaced.
• receive(remote_hlc) -- called by gossip handlers (gossip/membership_op.rs, gossip/put_identity.rs, gossip/put_message.rs) on every inbound op. Advances local state so the network's HLC stays monotonic across nodes. Rejects values that exceed local wall-clock by more than DEFAULT_MAX_DRIFT_MS (5 min) so a misclocked peer cannot drag the cluster forward.
• current() (test-only) -- snapshot used by integration tests.

State is in-memory only. On node restart it re-initialises to (now, 0) and self-corrects to network consensus via the first incoming gossip receive().

Construction is hidden inside run_node. Tests that need a controllable clock go through node::test_support::run_node_with_clock which lives behind the test-support cargo feature; production binaries never compile that module.

Inbox Batching (disabled)

InboxBatcher infrastructure is preserved in the codebase but disabled under full replication. With every node storing all messages, inbox upserts happen locally inside process_db_op after writing PutMessage, eliminating the need for gossip-based InboxFanout entirely.

If sharding is re-introduced, the batcher should be re-enabled:

• Uses HashSet for O(1) user deduplication
• Flushes when pending >= 100 items OR >= 200ms elapsed
• Produces single BatchedInboxFanout gossip message per flush

Replication Model

Full replication (since Phase 3): every node stores everything.

am_i_responsible() always returns true. The original XOR-distance sharding logic is preserved in comments for potential future re-enablement.

libp2p Stack

Transport: TCP + QUIC (both enabled via .with_tcp() + .with_quic()).

MyBehaviour {
  ping:       ping::Behaviour,
  identify:   identify::Behaviour,             -- peer info + API endpoint announcement
  autonat:    autonat::Behaviour,
  kademlia:   kad::Behaviour<MemoryStore>,     -- peer discovery only
  gossipsub:  gossipsub::Behaviour,            -- message propagation
  sync_rr:    request_response::Behaviour,     -- point-to-point sync
}

API Node Discovery

The event loop owns an in-memory API registry (HashMap<PeerId, String>, next to PeerCache): the advertised HTTP API base URLs of currently connected peers.

• Fill: on identify Received, handlers::kad::handle_identify parses the peer's agent_version for an api=<url> token (the peer's public_api_url config value) and records it
• Evict: when the last connection to a peer closes, the entry is removed -- registry membership means "connected right now"
• Serve: Command::GetNetworkNodes snapshots the registry for the public GET /network/nodes endpoint (client bootstrap/failover); handlers see it as a read-only reference in HandlerContext

The registry is deliberately ephemeral (never persisted, never synced): it rebuilds itself from identify exchanges within seconds of a restart, which follows the "minimal state, maximal derivation" principle. Wire format and trust notes: PROTOCOL.md, "Identify-Based API Discovery".

GossipSub Configuration

Connection Settings

File Map

crates/node/src/
  lib.rs                        -- run_node(), NodeHandle, MyBehaviour, event loop
  main.rs                       -- Thin CLI wrapper: args, tracing, Ctrl+C
  config.rs                     -- TOML config loading
  cli.rs                        -- CLI argument parsing (clap)
  keys.rs                       -- PeerId derivation from private key
  types.rs                      -- GossipMessage, DbOp, InboxBatcher, PeerCache
  metrics.rs                    -- Prometheus metrics
  retention.rs                  -- Background GC: RETENTION_WINDOW, gc_cycle, run_gc_loop

crates/node/tests/
  integration.rs                -- In-process multi-node integration tests

crates/node/src/handlers/
  mod.rs                        -- Handler dispatching
  context.rs                    -- HandlerContext, am_i_responsible()

crates/node/src/handlers/mpsc/
  mod.rs                        -- MPSC dispatcher
  put_message.rs                -- Send message handler (group fanout via list_members)
  membership_op.rs              -- Compound membership handler: batch gossip publish, role-based authorization, transfer/delete ops
  leave_group.rs                -- Leave group with owner prevention
  list_user_chats.rs            -- List conversations handler
  get_chat_range.rs             -- Get message history handler
  read_chat_message.rs          -- Mark-as-read handler (skip_membership_check flag)
  get_group_members.rs          -- List group members with roles (pure local read)
  utils.rs                      -- Helper functions

crates/node/src/handlers/gossip/
  mod.rs                        -- Gossip dispatcher
  put_message.rs                -- Store incoming message
  membership_op.rs              -- Membership change handler with independent sig verification
  inbox_fanout.rs               -- Update user inboxes
  query.rs                      -- Handle Query/QueryResponse
  read_progress.rs              -- Handle ReadProgress
  ack.rs                        -- Handle Ack (deprecated)

crates/node/src/handlers/kad/
  mod.rs                        -- Kad event dispatcher
  identify.rs                   -- Identify protocol event handler

crates/node/src/sync/
  mod.rs                        -- Module aggregation
  protocol.rs                   -- SyncRequest/SyncResponse, SyncCodec
  session.rs                    -- SyncSession, SyncManager, SyncState
  handler.rs                    -- Responder + Initiator logic
  merkle.rs                     -- MerkleTree (65536 buckets, XOR accumulators)

crates/api/src/
  lib.rs                        -- Module aggregation
  server.rs                     -- HTTP routes, OpenAPI, listen_api()
  command.rs                    -- Command enum, raw response types
  dto.rs                        -- HTTP DTOs with validation
  auth.rs                       -- Signature middleware
  utils.rs                      -- Hex conversion, dm_chat_id, ApiError, canonical signing
  metrics.rs                    -- HTTP metrics middleware

crates/db/src/
  lib.rs                        -- Re-exports
  store.rs                      -- RocksDB init, CF configuration
  types.rs                      -- MsgV1, ChatMeta, InboxEntry, query types
  keys.rs                       -- RocksDB key builders
  messages.rs                   -- Message storage, range reads, sync helpers
  chats.rs                      -- Inbox management, read progress
  members.rs                    -- Group membership, member sync helpers
  identity.rs                   -- Identity storage (LWW), identity sync helpers
  errors.rs                     -- ChatDbError

crates/types/src/
  lib.rs                        -- ChatKind, ID length constants

crates/crypto/src/
  lib.rs                        -- ECDSA, Keccak256, address derivation

Storage (RocksDB)

Overview

Engine: RocksDB via rust-rocksdb Wrapper: ChatDb struct in crates/db/src/store.rs Compression: Zstd (all column families) Default path: ./chatdb-data

Column Families

11 CFs plus default:

messages

Purpose: Chat messages storage Key: [chat_id:32][hlc_packed:u64 BE][seq:u32 BE] = 44 bytes Value: MsgV1 (CBOR)

• Prefix extractor: 32 bytes (chat_id) -- enables efficient prefix_same_as_start iteration
• Bloom filter: 10 bits/key (~1% FPR)
• Memtable prefix bloom: 0.1 ratio
• The 8-byte slot holds HlcTimestamp::to_packed().to_be_bytes() (48 bits physical_ms + 16 bits logical). Big-endian packed HLC preserves the chronological lex order RocksDB depends on: physical dominates, logical breaks ties within a single millisecond.
• The byte layout is identical to the legacy ts: u64 slot, so the retention soft-filter and key-prefix scans keep working unchanged.

seen_msg

Purpose: Message deduplication + msg_id-to-key lookup for sync + HLC physical_ms source for retention Key: [msg_id:32] = 32 bytes Value: message key (44 bytes) -- points to CF messages entry

• Prefix extractor: 2 bytes (first 2 bytes of msg_id = Merkle bucket index)
• Bloom filter: 10 bits/key
• whole_key_filtering: true -- enables point lookups using full 32-byte key
• Memtable prefix bloom: 0.1 ratio
• Triple access pattern: point lookups (dedup), prefix scans (sync bucket enumeration), and HLC read-out (retention soft-filter)
• extract_hlc_physical_from_seen_value reads the packed HLC from bytes 32..40 of the value and returns its upper 48 bits as wall-clock milliseconds. Retention compares this against cutoff_ts directly. Legacy entries with empty values are treated as physical_ms = 0 (cleared on the first retention GC pass without a migration).

chats_meta

Purpose: Aggregated chat metadata Key: [chat_id:32] = 32 bytes Value: ChatMeta (CBOR)

#![allow(unused)]
fn main() {
struct ChatMeta {
    last_ts: u64,
    last_seq: u32,
    last_msg_id: [u8; 32],
    members: Vec<[u8; 20]>,
    last_msg_ts: u64,
    last_announced_ms: u64,
    title: Option<String>,        // Group creation title (immutable)
    creator: Option<[u8; 20]>,    // Group creator address (immutable)
    create_nonce: Option<Vec<u8>>, // 16-byte creation nonce (immutable)
}
}

• title / creator / create_nonce: the group's immutable creation trio, written once by DbOp::SetGroupCreateMeta (set_group_create_meta, guarded by creator.is_some()). All three participate in the group chat_id derivation (blake3(domain || creator || nonce || title)), so the trio is self-verifying against the chat_id: nodes re-serve it to peers via the members-sync piggyback (see SYNC.md) without any trust in the sender. None on all three for DM chats; title alone is None for unnamed groups
• The trio adds bytes to a value that is already read-modify-written on every put_message (seq allocation), so it costs no extra I/O on the hot path
• Inbox writes hydrate an empty group/channel title from this CF (upsert_inbox_for_members) -- one memtable-hot point read per upsert -- so user_inbox entries and /conversations always carry the title, whichever path (local write, gossip, sync) stored the message
• Bloom filter: 10 bits/key
• No prefix extractor (point lookups only)

user_inbox

Purpose: User's chat list sorted by most recent first Key: [user:20][rev_ts:u64 BE][chat_id:32] = 60 bytes Value: InboxEntry (CBOR)

• rev_ts = u64::MAX - last_ts -- reverse sort for "newest first" iteration
• Prefix extractor: 20 bytes (user_id)
• Bloom filter: 10 bits/key
• Memtable prefix bloom: 0.1 ratio
• Entries are point-deleted by delete_user_inbox_entry(user, chat_id) after MembershipOp::Remove or LeaveGroup (one prefix scan to recover rev_ts, then a WriteBatch delete)

#![allow(unused)]
fn main() {
struct InboxEntry {
    chat_id: [u8; 32],
    kind: ChatKind,
    last_ts: u64,
    last_msg_id: [u8; 32],
    last_sender: [u8; 20],
    last_seq: u32,
}
// No content preview is stored -- previews are derived client-side.
// Entries written by older builds may carry a legacy last_text_preview
// field; serde ignores it on decode.
}

members

Purpose: PRIMARY persistent store for group/channel membership. DMs are implicit (not stored here). Key: [chat_id:32][user:20] = 52 bytes Value: MemberInfo (CBOR)

#![allow(unused)]
fn main() {
struct MemberInfo {
    role: u8,                              // 0 = participant, 1 = admin, 2 = owner
    added_at: HlcTimestamp,                // HLC of the latest Add
    removed_at: Option<HlcTimestamp>,      // HLC of the latest Remove, if any
}
}

Exactly one owner (role=2) exists per group: Create writes the signer with role=2; TransferOwnership atomically demotes the old owner to admin and promotes the new one in a single WriteBatch.

• Prefix extractor: SliceTransform::create_fixed_prefix(CHAT_ID_LEN) -- enables bloom-filtered prefix scans for list_members / list_members_with_info
• Bloom filter: 10 bits/key
• Memtable prefix bloom: 0.1 ratio
• ensure_dm_members removed -- DM membership is implicit via chat_id hash

CRDT model: Each (chat, user) record carries both added_at and removed_at. The record is active when removed_at is absent or strictly less than added_at; otherwise it is tombstoned. Remove never physically deletes the row -- it monotonically advances removed_at. This keeps the sync record_id (which embeds both timestamps) stable long enough for Merkle anti-entropy to converge, eliminating the "removed member resurrected from a partition" bug.

Merge rule (used by apply_member_record_synced for incoming sync records): added_at = max(local, remote), removed_at = max(local, remote), role taken from the side with the dominant added_at (local wins on tie). Monotonic in all three fields.

Sync model: Members CF is synced via Merkle-tree anti-entropy (domain Members). Each write through add_member_synced, remove_member_synced, or apply_member_record_synced atomically updates the seen_member sync index and sends MerkleUpdate::Replace or Insert to the members Merkle tree. Real-time propagation still uses GossipSub MembershipOp / MembershipOpBatch. See SYNC.md for details.

Operations:

user_read_progress

Purpose: Last read sequence number per user per chat Key: [user:20][chat_id:32] = 52 bytes Value: u32 (big-endian, 4 bytes)

• Prefix extractor: 20 bytes (user_id)
• Bloom filter: 10 bits/key
• Memtable prefix bloom: 0.1 ratio
• Update is monotonic: only writes if new_seq > current_seq

identity

Purpose: Opaque user identity blob (e.g. public keys for E2EE key exchange) Key: [user_id:20] = 20 bytes Value: [hlc_packed:u64be:8][blob:N] -- 8-byte packed HLC stamp prefix followed by raw blob (max 1024 bytes)

• No prefix extractor (point lookups only by exact 20-byte key)
• Bloom filter: 10 bits/key
• Compression: Zstd
• Overwrites previous value on PUT (no versioning, no history)
• The 8-byte prefix is the packed HlcTimestamp (48 bits physical_ms + 16 bits logical) used for last-write-wins: an incoming blob is stored only if incoming_hlc > stored_hlc. The byte layout matches the legacy ts: u64 slot, so no value-size change.
• GET /identity/{address} strips the 8-byte HLC prefix before returning the blob to clients
• Synced via Merkle-tree anti-entropy (domain Identity). Writes through DbOp::PutIdentity atomically update the seen_identity sync index. Real-time propagation via GossipMessage::PutIdentity

seen_member

Purpose: Sync index for members (record_id -> primary key lookup) Key: [record_id:32] = 32 bytes (BLAKE3 of chat_id, user_id, role, added_at_packed_be, removed_at_or_zero_packed_be) Value: [chat_id:32][user_id:20] = 52 bytes (pointer to CF members)

• Prefix extractor: 2 bytes (Merkle bucket index, same as seen_msg)
• Bloom filter: 10 bits/key, whole_key_filtering: true
• Dual access: point lookups (dedup) + prefix scans (sync bucket enumeration)

seen_identity

Purpose: Sync index for identity (record_id -> primary key lookup) Key: [record_id:32] = 32 bytes (BLAKE3 of user, hlc_packed_be, blob) Value: [user_id:20] = 20 bytes (pointer to CF identity)

• Prefix extractor: 2 bytes (Merkle bucket index)
• Bloom filter: 10 bits/key, whole_key_filtering: true
• Dual access: point lookups (dedup) + prefix scans (sync bucket enumeration)

message_ops

Purpose: Edit/delete operations over stored messages (append-only) Key: [target_msg_id:32][op_id:32] = 64 bytes Value: MessageOpRecord (CBOR)

#![allow(unused)]
fn main() {
struct MessageOpRecord {
    target: [u8; 32],   // msg_id being changed
    chat_id: [u8; 32],  // chat the target belongs to
    editor: [u8; 20],   // must equal the target's original sender
    op_kind: u8,        // 0 = Edit, 1 = Delete
    new_text: String,   // replacement text (empty for Delete)
    hlc: HlcTimestamp,  // server-stamped by the originating node
    sig: Vec<u8>,       // 65-byte client sig for a Delete; empty for an Edit
}
}

Only a delete is signed (and re-verified by every node): erasure produces a deleted stub that no recipient can authenticate end-to-end, so the node is the last line of defence. An edit writes new content whose authenticity the recipient checks on Layer 2, so it carries no signature -- the same trust model as a plain message. See PROTOCOL.md.

• Prefix extractor: 32 bytes (target_msg_id) -- all ops targeting one message are a single bloom-filtered prefix scan (fold_ops_for_message), used when a message arrives after ops that target it
• whole_key_filtering: true, bloom 10 bits/key
• op_id closes the key rather than the op's HLC: two nodes can independently stamp the same packed HLC, and an HLC-keyed layout would let one op silently overwrite another while its id lingered in the sync index. The fold is order-independent, so the key encodes no ordering
• The stored CBOR is also the sync wire record (no re-encode)

seen_op

Purpose: Sync index for message ops (record_id -> pointers) + HLC source for retention Key: [op_id:32] = 32 bytes (BLAKE3 over the operation, domain-separated) Value: [target:32][chat_id:32][op_hlc_packed:u64be:8] = 72 bytes

• Prefix extractor: 2 bytes (Merkle bucket index, same as seen_msg)
• Bloom filter: 10 bits/key, whole_key_filtering: true
• The value carries everything needed to rebuild the other two keys (primary = target || op_id, feed = chat || hlc || target) and to apply the retention filter, so neither sync nor GC decodes the CBOR record -- mirrors how seen_msg embeds the messages key

ops_by_chat

Purpose: Per-chat change feed (written from day one; consumed by a future changes endpoint) Key: [chat_id:32][op_hlc_packed:u64be:8][target_msg_id:32] = 72 bytes Value: [op_kind:1] = 1 byte

• Prefix extractor: 32 bytes (chat_id)
• Bloom filter: 10 bits/key
• Big-endian HLC right after chat_id makes "everything that changed after cursor X" a single forward prefix scan; target_msg_id closes the key because a packed HLC is unique only within one node. Nothing reads this CF yet -- it exists so the planned GET .../messages/changes?since=<hlc> endpoint needs no backfill

Key Construction

All keys use big-endian encoding for correct lexicographic sort:

#![allow(unused)]
fn main() {
fn key_messages(chat: &[u8; 32], hlc: HlcTimestamp, seq: u32) -> Vec<u8> {
    // [chat_id:32][hlc_packed_be:8][seq:4]
}

fn key_user_inbox(user: &[u8; 20], rev_ts: u64, chat: &[u8; 32]) -> Vec<u8> {
    // [user:20][rev_ts:8][chat_id:32]
}

fn key_members(chat: &[u8; 32], user: &[u8; 20]) -> Vec<u8> {
    // [chat_id:32][user:20]
}

fn key_user_read_progress(user: &[u8; 20], chat: &[u8; 32]) -> Vec<u8> {
    // [user:20][chat_id:32]
}

fn key_message_ops(target: &[u8; 32], op_id: &[u8; 32]) -> Vec<u8> {
    // [target_msg_id:32][op_id:32]
}

fn key_ops_by_chat(chat: &[u8; 32], op_hlc: HlcTimestamp, target: &[u8; 32]) -> Vec<u8> {
    // [chat_id:32][op_hlc_packed_be:8][target_msg_id:32]
}
}

RocksDB Configuration

parallelism:        num_cpus
compression:        Zstd (all CFs)
block_cache_mb:     512 (default)
memtable_mb:        512 (default)
bloom_filter:       10.0 bits/key (all CFs)
log_files_kept:     5
write_sync:         false (WAL still provides crash safety)

Core Operations

put_message

Atomic write (WriteBatch) of 3 entries:

1. CF messages: message CBOR
2. CF seen_msg: msg_id -> message key (for dedup + sync lookup)
3. CF chats_meta: updated ChatMeta (incremented seq)

Idempotent: checks seen_msg first, returns (msg_id, 0, "") if duplicate.

range (message range query)

Cursor-based pagination using after_key_b64, in either direction (reverse):

• Forward (reverse = false): seeks the window low bound (or just after the cursor -- decode base64 key, increment seq, carrying into the packed-HLC field on u32::MAX) and iterates Direction::Forward to the high bound, oldest-first
• Reverse (reverse = true): seeks the window high bound (or just before the cursor -- decrement seq, borrowing from the packed-HLC field on 0) and iterates Direction::Reverse (RocksDB seek_for_prev) down to the low bound, newest-first -- the chat-tail + scroll-up access pattern
• Both directions run under prefix_same_as_start (the 32-byte chat_id prefix extractor bounds the scan to one chat) and enforce the opposite [from_ts, to_ts] bound explicitly; the cursor step is shared via step_message_cursor
• Returns RangePage { items, next_after_key_b64 }; in reverse mode the cursor points at the page's oldest row so the next call walks further back

upsert_inbox_for_members

For each user in the members list:

• Reads existing InboxEntry for this (user, chat_id)
• Deletes old entry (old rev_ts key)
• Writes new entry with updated rev_ts
• Ensures user always has exactly one inbox entry per chat

list_user_chats

Prefix iteration on user_inbox CF with user_id prefix:

• Reads entries in reverse chronological order (rev_ts sorting)
• Joins with user_read_progress to compute unread count
• Supports cursor-based pagination

Sync Helpers

Each domain has the same three sync helpers (mirrored API):

#![allow(unused)]
fn main() {
// Messages (crates/db/src/messages.rs)
fn for_each_msg_id(db, callback: FnMut([u8; 32], u64))      // Merkle rebuild, yields (msg_id, hlc.physical_ms)
fn get_bucket_msg_ids(db, bucket: u16) -> Vec<[u8; 32]>     // Sync step 4
fn get_messages_cbor_batch(db, ids, max_bytes) -> (Vec, has_more) // Sync step 5

// Members (crates/db/src/members.rs)
fn for_each_member_record_id(db, callback)
fn get_bucket_member_ids(db, bucket: u16) -> Vec<[u8; 32]>
fn get_member_records_batch(db, ids, max_bytes) -> (Vec, has_more)

// Identity (crates/db/src/identity.rs)
fn for_each_identity_record_id(db, callback)
fn get_bucket_identity_ids(db, bucket: u16) -> Vec<[u8; 32]>
fn get_identity_records_batch(db, ids, max_bytes) -> (Vec, has_more)

// Message ops (crates/db/src/message_ops.rs) -- bucket/fetch take a
// cutoff_ts for the retention soft-filter (pass 0 to disable)
fn for_each_op_record_id(db, callback)
fn get_bucket_op_ids(db, bucket: u16, cutoff_ts) -> Vec<[u8; 32]>
fn get_op_records_batch(db, ids, max_bytes, cutoff_ts) -> (Vec, has_more)
}

The *_batch functions use multi_get_cf to batch both lookup steps (seen index -> primary CF) into two bulk RocksDB calls instead of N individual gets. This reduces per-key syscall overhead, especially during sync step 5 where hundreds of records may be fetched at once.

Retention Helpers

Used by the background GC cycle (crates/node/src/retention.rs) and by the sync soft-filter. All live in crates/db/src/messages.rs except for_each_chat_id which is in crates/db/src/chats.rs. See RETENTION.md for the overall design.

#![allow(unused)]
fn main() {
// Read-side helpers
fn extract_hlc_physical_from_seen_value(value: &[u8]) -> Option<u64>
fn for_each_chat_id(db, callback: FnMut(&[u8; 32]))

// Filtered variants of the sync helpers, used by the sync responder
fn get_bucket_msg_ids_filtered(db, bucket, cutoff_ts) -> Vec<[u8; 32]>
fn get_messages_cbor_batch_filtered(db, ids, max_bytes, cutoff_ts) -> (Vec, has_more)

// Write-side helpers, used by the GC cycle
fn range_delete_old_messages(db, chat_id, cutoff_ts) -> Result<()>
fn scan_seen_msg_aged(db, cutoff_ts, limit) -> Result<Vec<[u8; 32]>>
fn delete_seen_msg_batch(db, msg_ids) -> Result<()>

// Message-ops GC (crates/db/src/message_ops.rs). The scan returns each
// op with its seen_op value so the delete can rebuild all three keys
// (message_ops, seen_op, ops_by_chat) without re-reading.
fn scan_seen_op_aged(db, cutoff_ts, limit) -> Result<Vec<([u8; 32], Vec<u8>)>>
fn delete_op_batch(db, entries) -> Result<()>
}

extract_hlc_physical_from_seen_value reads the packed HlcTimestamp from bytes 32..40 and returns its physical_ms (upper 48 bits) -- a plain wall-clock millisecond value the retention cutoff can compare against directly.

range_delete_old_messages issues a single delete_range_cf on CF messages over [chat_id, HlcTimestamp::ZERO, 0] .. [chat_id, HlcTimestamp::from_parts(cutoff_ts + 1, 0), 0). The packed HLC puts physical_ms in the upper 48 bits, so every stored message with hlc.physical_ms <= cutoff_ts -- regardless of its logical counter or seq -- falls strictly below the end key. RocksDB processes this as a tombstone; physical reclamation happens during compaction.

Async Write Pipeline

Handler --> db_write_tx.send(DbOp) --> spawn_db_writer:
  |         (unbounded channel)
  tokio::spawn_blocking(|| process_db_op(db, op, merkle_senders))
  |
  +--> DbOp::PutMessage       --> put_message()          --> merkle.msg_tx.blocking_send(msg_id)
  +--> DbOp::UpsertInbox      --> upsert_inbox_for_members()
  +--> DbOp::SetReadProgress  --> set_user_read_progress()
  +--> DbOp::MembershipOp     --> add_member_synced / remove_member_synced (HLC CRDT, op_type 0/1/2)
  |                                --> merkle.member_tx.blocking_send(MerkleUpdate)
  +--> DbOp::ApplyMemberRecord --> apply_member_record_synced (merge full peer state from sync)
  |                                --> merkle.member_tx.blocking_send(MerkleUpdate)
  +--> DbOp::TransferOwnership --> transfer_ownership_synced()  // atomic two-record role swap,
  |                                --> 2x merkle.member_tx      // one WriteBatch
  +--> DbOp::DeleteGroup      --> delete_group_synced()         // bulk tombstone all members,
  |                                --> Nx merkle.member_tx      // then per-user inbox cleanup
  |                                --> Nx delete_user_inbox_entry()
  +--> DbOp::PutIdentity      --> put_identity()
  |                                --> merkle.identity_tx.blocking_send(MerkleUpdate)
  +--> DbOp::ApplySyncedMessage --> put_message_replicated()   // peer row verbatim (id + edited_at
  |                                --> merkle.msg_tx            // + deleted preserved); NOT re-authored
  +--> DbOp::MessageOp        --> apply_message_op()           // record op + rewrite target row,
  |                                --> merkle.op_tx             // one WriteBatch; Insert (append-only)
  +--> DbOp::DeleteInboxEntry --> delete_user_inbox_entry()  // queued after MembershipOp::Remove
                                                                // and LeaveGroup so FIFO ordering
                                                                // applies membership removal first

Channel design:

• db_write_tx is unbounded -- handlers never block on DB write submission.
• Merkle channels (msg_tx, member_tx, identity_tx) are bounded (8192 items) -- provides backpressure if the select! loop falls behind on Merkle updates. blocking_send is safe because process_db_op runs inside spawn_blocking. The RocksDB write completes before the Merkle send, so data is never lost even if the channel is temporarily full.

HTTP responses return before DB commits. This is safe because:

• Reads can go through gossip Query/QueryResponse to other nodes
• Duplicates are handled by seen_msg idempotency
• Merkle tree is updated after successful write, not before

Important Constraints

• Fixed ID lengths (20-byte user, 32-byte chat/msg) are critical for RocksDB prefix extractors and key layout. Changing them requires a migration plan.
• WriteBatch ensures atomicity of message + seen_msg + chats_meta updates.
• write_sync = false for performance. WAL still provides crash safety.

Types Reference

crates/types -- Shared Domain Primitives

ID Types

#![allow(unused)]
fn main() {
const CHAT_ID_LEN: usize = 32;
const USER_ID_LEN: usize = 20;
const SENDER_ID_LEN: usize = 20;
const MSG_ID_LEN: usize = 32;

type ChatId = [u8; 32];
type UserId = [u8; 20];
type SenderId = [u8; 20];
type MsgId = [u8; 32];
}

ChatKind

#![allow(unused)]
fn main() {
#[serde(tag = "t", content = "d")]
enum ChatKind {
    #[serde(rename = "0")]
    Dm { peer: UserId },          // Direct message (peer semantics: see below)
    #[serde(rename = "1")]
    Group { title: Option<String> },
    #[serde(rename = "2")]
    Channel { title: Option<String> },
}
}

Methods: type_id() -> u8, is_dm() -> bool Default: Dm { peer: [0u8; 20] } (backward compatibility)

CBOR representation: {"t": "0", "d": {"peer": [...]}} -- compact tagged format.

Two wire-level notes:

• The tag t is a CBOR text string ("0"/"1"/"2"), never an integer -- a consequence of serde's adjacently-tagged encoding, kept as-is for compatibility with already-stored data.
• Dm.peer semantics depend on where the value lives. In wire messages (MsgV1.kind, gossip PutMessage.kind) it is the original recipient as fixed by the sender -- identical bytes for every reader, so in a received message it equals the reader's own address (interlocutor = peer == me ? sender : peer). In user_inbox entries (and therefore in the /conversations API) the node rewrites peer per inbox owner to be the other participant (upsert_inbox_for_members in crates/db), so there it is always viewer-relative.

Note: User identity blobs (e.g. public keys) are stored in the dedicated identity CF (see STORAGE.md), not inside ChatKind.

Message Types

There are no msg_type constants in code -- the field is a client-defined opaque u8 (0 = plain message, set by the plain message endpoints). Known control conventions (attachments 10, MLS Welcome/Commit 20/21, retired 11/12/22, app-private 100-255) are documented in the client guide registry. User content never claims a control type -- it rides text, optionally marked with a client format prefix.

HlcTimestamp

#![allow(unused)]
fn main() {
#[serde(transparent)]
struct HlcTimestamp(u64);  // packed: 48 bits physical_ms + 16 bits logical
}

Hybrid Logical Clock timestamp used by CRDT-like domains (members, identity) and as a network-wide ordering key for messages. Packed into a single u64:

• Upper 48 bits: physical_ms -- milliseconds since UNIX epoch
• Lower 16 bits: logical -- per-node counter for events within one ms

Methods: from_parts(phys, log), from_packed(u64), to_packed() -> u64, physical_ms() -> u64, logical() -> u16, next_logical() -> HlcTimestamp. Ord is derived from the packed u64, which gives lexicographic (physical, logical) ordering -- the property CRDTs rely on for last-writer-wins decisions.

Constants: MAX_PHYSICAL_MS (48 bits set, year ~10889), MAX_LOGICAL = u16::MAX, HlcTimestamp::ZERO (sentinel for absent optional HLC fields in record-id derivation).

Serde: #[serde(transparent)] -- wire encoding is identical to a raw u64. The 8-byte slot for the legacy ts: u64 in seen_msg value layout, messages CF key, and identity CF value is reused without size change. Big-endian encoding of the packed value preserves HLC ordering under RocksDB's lexicographic key comparison.

Clock

#![allow(unused)]
fn main() {
trait Clock: Send + Sync + 'static {
    fn now_ms(&self) -> u64;
}

struct SystemClock;             // wraps SystemTime::now()
struct MockClock { ... }        // AtomicU64-backed, test-controllable
}

Time-source abstraction. Production uses SystemClock; tests use MockClock with set(ms) and advance(delta_ms). The abstraction exists so HLC's clock-skew behaviour is reproducible in the test suite.

Shared via Arc<dyn Clock> so multiple owners (HLC state, retention loop) can read the same source without further synchronisation.

crates/db -- Storage Types

MsgV1 (stored in CF messages)

#![allow(unused)]
fn main() {
struct MsgV1 {
    schema: u8,                   // Always 1
    msg_id: MsgId,
    chat_id: ChatId,
    sender: SenderId,
    hlc: HlcTimestamp,            // Server-stamped HLC (storage/CRDT/sync)
    origin_wall_ts: u64,          // Frozen originator wall-clock (UI display)
    seq: u32,                     // Monotonic sequence within chat
    text: String,                 // current text (post-edit; empty if deleted)
    msg_type: u8,                 // client-defined; 0 = regular text
    control: Option<Vec<u8>>,     // CBOR payload for control messages
    kind: ChatKind,
    edited_at: Option<HlcTimestamp>, // HLC of the winning Edit (absent = never edited)
    deleted: bool,                // true once deleted (stub row, text cleared)
}
}

edited_at and deleted are the materialized projection of any MessageOp records targeting this message (see PROTOCOL.md). Both are #[serde(default, skip_serializing_if = ...)], so an untouched message serializes byte-for-byte as before -- adding edit/delete support changed no existing wire vector. msg_id is not recomputed on edit: it stays the message's stable identity even though the stored text no longer hashes to it.

ChatMeta (stored in CF chats_meta)

#![allow(unused)]
fn main() {
struct ChatMeta {
    last_ts: u64,
    last_seq: u32,
    last_msg_id: MsgId,
    members: Vec<UserId>,
    last_msg_ts: u64,
    last_announced_ms: u64,
    title: Option<String>,          // Group creation title (immutable)
    creator: Option<UserId>,        // Group creator (immutable)
    create_nonce: Option<Vec<u8>>,  // 16-byte creation nonce (immutable)
}
}

The title / creator / create_nonce trio is the group's immutable creation metadata, written once via set_group_create_meta and verifiable against the chat_id (blake3(domain || creator || nonce || title)). All three are None for DMs; title alone is None for unnamed groups. See STORAGE.md and SYNC.md.

InboxEntry (stored in CF user_inbox)

#![allow(unused)]
fn main() {
struct InboxEntry {
    chat_id: ChatId,
    kind: ChatKind,
    last_ts: u64,
    last_msg_id: MsgId,
    last_sender: SenderId,
    last_seq: u32,
}
// No content preview -- previews are derived client-side.
}

MemberInfo (stored in CF members)

#![allow(unused)]
fn main() {
struct MemberInfo {
    role: u8,                              // 0 = participant, 1 = admin, 2 = owner
    added_at: HlcTimestamp,                // HLC of the latest Add
    removed_at: Option<HlcTimestamp>,      // HLC of the latest Remove
}

impl MemberInfo {
    fn is_active(&self) -> bool {
        match self.removed_at {
            None => true,
            Some(r) => self.added_at > r,
        }
    }
}
}

Public reads (get_member_info, list_members, is_member) filter to is_active(). Sync helpers iterate raw seen_member and include tombstones so Merkle anti-entropy can carry removed_at between peers. See STORAGE.md for the full CRDT model.

MessageOpRecord (stored in CF message_ops)

#![allow(unused)]
fn main() {
struct MessageOpRecord {
    target: MsgId,           // message being changed
    chat_id: ChatId,         // chat the target belongs to
    editor: UserId,          // must equal the target's original sender
    op_kind: u8,             // OP_KIND_EDIT = 0, OP_KIND_DELETE = 1
    new_text: String,        // replacement text (empty for Delete)
    hlc: HlcTimestamp,       // server-stamped by the originating node
    sig: Vec<u8>,            // client sig for a Delete; empty for an Edit
}
}

An append-only edit/delete operation. The same struct is the stored record, the gossip payload, and the anti-entropy wire record -- no re-encode. op_kind == OP_KIND_DELETE is terminal. Only a delete is signed (and re-verified by every node): erasure yields a deleted stub that cannot be authenticated end-to-end, so the node authorizes it. An edit is unsigned -- its content is checked by the recipient on Layer 2, like a plain message. The message row clients read is the materialized fold of a message and all its ops; the fold rule (op_applies_to) and apply_message_op live in crates/db/src/message_ops.rs. See PROTOCOL.md and SYNC.md.

Query/Result Types

#![allow(unused)]
fn main() {
struct PutMsg<'a> {
    chat: &'a ChatId,
    sender: &'a [u8; 20],
    hlc: HlcTimestamp,        // Stored in CF `messages` key + msg_id hash
    origin_wall_ts: u64,      // Frozen sender wall-clock for UI display
    text: &'a str,
    kind: ChatKind,
    msg_type: u8,
    control: Option<&'a [u8]>,
}

struct RangeQuery<'a> {
    chat: &'a ChatId,
    from_ts: u64,
    to_ts: Option<u64>,
    after_key_b64: Option<String>,
    limit: usize,
    reverse: bool,            // false = oldest-first from `from_ts`;
                              // true = newest-first from `to_ts`
}

struct RangePage {
    items: Vec<(String, Vec<u8>)>,        // (key_b64, msg_cbor)
    next_after_key_b64: Option<String>,
}

struct ListChatsQuery<'a> {
    user: &'a [u8; 20],
    limit: Option<usize>,
    after_cursor_b64: Option<String>,
}

struct ListChatsPage {
    items: Vec<InboxEntryWithCursor>,
    next_after_cursor_b64: Option<String>,
}
}

crates/api -- HTTP Types

Command Enum

#![allow(unused)]
fn main() {
enum Command {
    PutMessage {
        chat_id: [u8; 32],
        kind: ChatKind,
        sender: [u8; 20],
        members: Option<Vec<[u8; 20]>>,
        text: String,
        // HLC + origin_wall_ts are stamped server-side in the MPSC
        // handler; clients never specify time.
        msg_type: u8,
        control: Option<Vec<u8>>,
        resp: oneshot::Sender<Result<PutMessageResponseRaw, String>>,
    },
    ListUserChats {
        user: [u8; 20],
        limit: usize,
        after_key: Option<Vec<u8>>,
        resp: oneshot::Sender<Result<ListUserChatsResponseRaw, String>>,
    },
    GetChatRange {
        user: [u8; 20],
        chat_id: [u8; 32],
        from_ts: u64,
        to_ts: Option<u64>,
        after_key: Option<Vec<u8>>,
        limit: usize,
        reverse: bool,        // false = oldest-first; true = newest-first
        skip_membership_check: bool,
        resp: oneshot::Sender<Result<GetChatRangeResponseRaw, String>>,
    },
    GetMessageById {          // Single message by id; node authorizes caller
        user: [u8; 20],
        msg_id: [u8; 32],
        resp: oneshot::Sender<Result<Option<Vec<u8>>, String>>,  // None = 404
    },
    ReadChatMessage {
        user: [u8; 20],
        chat_id: [u8; 32],
        seq: u32,
        resp: oneshot::Sender<Result<(), String>>,
    },
    GetNetworkNodes {         // Catalog of connected API-serving peers
        resp: oneshot::Sender<Result<Vec<NetworkNodeRaw>, String>>,
    },
    MessageOp {               // Edit/delete own message; node loads target,
        target: [u8; 32],     // checks authorship + 48h window, stamps HLC,
        editor: [u8; 20],     // verifies sig, then routes DbOp + gossip
        op_kind: u8,          // 0 = Edit, 1 = Delete
        new_text: String,
        sig: Vec<u8>,
        resp: oneshot::Sender<Result<MessageOpResponseRaw, String>>,
    },
    // Plus: MembershipOp, LeaveGroup, GetGroupMembers, SetIdentity,
    // GetIdentity -- see crates/api/src/command.rs
}
}

Raw Response Types

#![allow(unused)]
fn main() {
struct PutMessageResponseRaw {
    chat_id: [u8; 32],
    msg_id: [u8; 32],
    origin_wall_ts: u64,    // HTTP DTO surfaces this as `ts` for backward compat
}

struct InboxChatRaw {
    chat_id: [u8; 32],
    kind: ChatKind,
    last_ts: u64,
    last_sender: [u8; 20],
    unread: u32,
    cursor_key: Vec<u8>,
    source: String,
}

struct ListUserChatsResponseRaw {
    items: Vec<InboxChatRaw>,
    next_after_key: Option<Vec<u8>>,
}

struct ChatMessageRaw {
    key_raw: Vec<u8>,
    msg_cbor: Vec<u8>,
}

struct MessageOpResponseRaw {
    msg_id: [u8; 32],
    op_ts: u64,             // wall-clock component of the op's HLC ("edited at")
}

struct GetChatRangeResponseRaw {
    items: Vec<ChatMessageRaw>,
    next_after_key: Option<Vec<u8>>,
}

struct NetworkNodeRaw {
    peer_id: String,    // base58 PeerId
    api_url: String,    // advertised HTTP API base URL
}

struct GroupMembersRaw {
    members: Vec<([u8; 20], u8)>,   // (address, role) pairs
    title: Option<String>,          // group creation title from chats_meta
}
}

DTO Types (JSON serialization)

#![allow(unused)]
fn main() {
#[serde(tag = "type")]
enum ChatKindDto {
    #[serde(rename = "dm")]
    Dm { peer: String },
    #[serde(rename = "group")]
    Group { title: Option<String> },
    #[serde(rename = "channel")]
    Channel { title: Option<String> },
}

struct PostDirectMessageBody { text: String }               // validate: 1-45056 bytes
struct PostDirectMessageResponse { chat_id, msg_id, ts }

struct PostControlDirectMessageBody { msg_type: u8, control: String }
// validate: msg_type 1-255, control base64 1-43692 chars (32 KiB decoded)
struct PostControlMessageResponse { chat_id, msg_id, ts }

struct ListUserChatsQueryParams { limit: Option<usize>, after: Option<String> }
// validate: limit 1-1000
struct InboxChat { chat_id, kind, last_ts, last_sender, unread, cursor }
struct ListUserChatsResponse { items: Vec<InboxChat>, next_after: Option<String> }

struct GetChatRangeQueryParams { from, to, after, limit, reverse: Option<bool> }
// validate: limit 1-1000; reverse omitted => false (oldest-first)
struct ChatMessage { key: String, msg_cbor: String }
struct GetChatRangeResponse { items: Vec<ChatMessage>, next_after }
struct GetMessageResponse { msg_id: String, msg_cbor: String }  // GET /messages/{msg_id}

struct ReadChatMessageBody { seq: u32 }                     // validate: >= 1

struct NodeInfoResponse { peer_id: String, api_url: Option<String> }  // GET /node/info
struct NetworkNodeDto { peer_id: String, api_url: String }
struct NetworkNodesResponse { nodes: Vec<NetworkNodeDto> }  // GET /network/nodes

struct CompoundMembershipRequest { ops, messages, nonce: Option<String>, title: Option<String> }
// title: create-only, 1-128 UTF-8 bytes, no control chars; part of the
// chat_id preimage (see PROTOCOL.md)
struct GroupMember { address: String, role: u8 }
struct GroupMembersResponse { members: Vec<GroupMember>, title: Option<String> }
}

crates/node -- Node Types

GossipMessage

See PROTOCOL.md for full specification.

Key structs (abbreviated; full spec in PROTOCOL.md):

#![allow(unused)]
fn main() {
struct PutIdentity {
    user: [u8; 20],     // User address
    blob: Vec<u8>,      // Opaque identity blob (max 1024 bytes)
    hlc: HlcTimestamp,  // Server-stamped HLC, used for LWW
    origin: String,     // PeerId of the originating node
}

// Edit/delete an existing message. Carries `MessageOpRecord` verbatim
// (see crates/db), so gossip and anti-entropy ship identical bytes.
// GossipMessage::MessageOp(MessageOpRecord)
}

DbOp

#![allow(unused)]
fn main() {
enum DbOp {
    // HLC drives storage/CRDT/sync; origin_wall_ts is frozen sender wall-clock for UI.
    PutMessage { chat_id, sender, text, hlc: HlcTimestamp, origin_wall_ts: u64, kind, msg_type, control },
    UpsertInbox { chat_id, kind, users, all_members, ts, seq, msg_id, sender },
    SetReadProgress { user, chat_id, seq },
    // Discrete Add/Remove/Create from API or gossip. `hlc` is server-stamped.
    MembershipOp { chat_id, target, role, op_type, hlc: HlcTimestamp, recovered_signer },
    // Full peer state from anti-entropy sync. CRDT-merged with local record.
    ApplyMemberRecord { chat_id, target, info: MemberInfo },
    // Immutable group creation trio -> chats_meta (write-once). Senders
    // MUST have verified chat_id == blake3(domain||creator||nonce||title).
    SetGroupCreateMeta { chat_id, title: Option<String>, creator, nonce: Vec<u8> },
    // HLC-LWW on identity blob (max 1024 bytes).
    PutIdentity { user, hlc: HlcTimestamp, blob },
    // Message row replicated verbatim from a peer during sync (id +
    // edited_at + deleted preserved). Live authoring uses PutMessage.
    ApplySyncedMessage { msg: Box<MsgV1> },
    // Record an edit/delete op + materialize it into the target row,
    // one WriteBatch. From HTTP, gossip, and sync alike.
    MessageOp { record: MessageOpRecord },
    DeleteInboxEntry { user, chat_id },
    // Atomic two-record role swap: old owner -> admin, new owner -> owner.
    TransferOwnership { chat_id, current_owner, new_owner, hlc: HlcTimestamp },
    // Bulk-tombstone all members + per-user inbox cleanup. Messages left to GC.
    DeleteGroup { chat_id, hlc: HlcTimestamp },
}

/// Wrapper around UnboundedSender<DbOp> that increments
/// db_writer_queue_depth gauge on every successful send.
struct DbOpSender { inner: mpsc::UnboundedSender<DbOp> }

enum MerkleUpdate { Insert([u8; 32]), Replace { old, new }, Remove([u8; 32]) }

struct MerkleSenders {
    msg_tx: UnboundedSender<[u8; 32]>,          // messages (append-only)
    member_tx: UnboundedSender<MerkleUpdate>,    // members (mutable)
    identity_tx: UnboundedSender<MerkleUpdate>,  // identity (mutable)
    op_tx: UnboundedSender<[u8; 32]>,            // message ops (append-only)
}
}

Sync Types

See SYNC.md for SyncDomain, SyncRequest, SyncResponse, SyncState, SyncSession.

Retention Types

See RETENTION.md for the full GC design.

#![allow(unused)]
fn main() {
struct CycleStats {
    removed_count: usize,    // msg_ids removed from seen_msg and the Merkle tree
    chats_processed: usize,  // chats range-deleted in `messages`
    hit_limit: bool,         // true if GC_BATCH_LIMIT was reached
}
}

Constants in crates/node/src/retention.rs: RETENTION_WINDOW, GC_INTERVAL, GC_SHORT_INTERVAL, GC_BATCH_LIMIT, GC_CHUNK_SIZE.

Functions: cutoff_ts_at(now) -> u64, cutoff_ts_now() -> u64, gc_cycle(db, merkle_msgs, now) -> CycleStats, run_gc_loop(db, merkle_msgs, cancel).

HlcState (per-node HLC)

#![allow(unused)]
fn main() {
struct HlcState {
    state: AtomicU64,          // packed HlcTimestamp
    clock: Arc<dyn Clock>,
    max_drift_ms: u64,
}

enum HlcError {
    DriftTooLarge { incoming_ms: u64, local_ms: u64, max_drift_ms: u64 },
}

const DEFAULT_MAX_DRIFT_MS: u64 = 5 * 60 * 1000;  // 5 minutes
}

Per-node Hybrid Logical Clock state. Generates outgoing HLC stamps (stamp()) and merges incoming HLC values from gossip (receive()). Lives in-memory only -- on restart, the state resets to (clock.now_ms(), 0) and self-corrects via the first incoming gossip.

API:

• HlcState::new(clock) -- default 5-minute drift bound
• HlcState::with_max_drift(clock, max_drift_ms) -- custom bound (tests)
• stamp() -> HlcTimestamp -- always strictly greater than every prior stamp from this state; wait-free in the uncontended case
• receive(remote) -> Result<HlcTimestamp, HlcError> -- advances local state to dominate both. Returns DriftTooLarge and leaves state unchanged if remote.physical_ms > now + max_drift_ms

Shared via Arc<HlcState> across handlers without further synchronisation -- the single AtomicU64 serialises all updates.

NodeHandle

#![allow(unused)]
fn main() {
struct NodeHandle {
    cmd_tx: mpsc::Sender<Command>,
    local_peer_id: PeerId,
    db: Arc<ChatDb>,
    listen_addrs: Arc<RwLock<Vec<Multiaddr>>>,
}
}

Methods: shutdown(self) -- cancel token + await task. Returned by run_node(AppConfig).

HandlerContext

#![allow(unused)]
fn main() {
struct HandlerContext<'a> {
    db: Arc<ChatDb>,
    swarm: &'a mut Swarm<MyBehaviour>,
    pending_queries: &'a mut HashMap<[u8; 16], PendingQuery>,
    peer_cache: &'a PeerCache,
    api_registry: &'a HashMap<PeerId, String>,  // connected peers' advertised API URLs
    local_peer_id: PeerId,
    local_peer_id_str: Arc<str>,   // cached PeerId string (avoids Base58 per request)
    db_write_tx: DbOpSender,
    inbox_batch_tx: Option<InboxBatchSender>,
    hlc: Arc<HlcState>,
}
}

Configuration

#![allow(unused)]
fn main() {
struct AppConfig {
    keypair: Keypair,               // secp256k1
    listen: Multiaddr,              // e.g. /ip4/0.0.0.0/tcp/4001
    bootnodes: Option<Vec<Multiaddr>>,
    listen_api: Option<SocketAddr>, // e.g. 0.0.0.0:3000
    db_path: Option<String>,
    expose_metrics: bool,
    metrics_listen: Option<SocketAddr>,
    sync_interval_secs: u64,        // Merkle sync tick (default 30)
    public_api_url: Option<String>, // advertised API base URL (identify announcement)
}
}

TOML config file format (loaded via config::load_config):

private_key = "0x..."
listen = "/ip4/0.0.0.0/tcp/4001"
bootnodes = ["/ip4/.../tcp/4001/p2p/..."]
listen_api = "0.0.0.0:3000"
db_path = "./chatdb-data"
expose_metrics = true
metrics_listen = "0.0.0.0:9090"
# Publicly reachable base URL of this node's HTTP API. Advertised to
# peers via libp2p identify; omit on relay-only / private nodes.
public_api_url = "https://node1.example.com:8080"

public_api_url is validated on load (http(s):// scheme, no whitespace, non-empty host, <= 256 bytes; trailing / is stripped).

Retention and Garbage Collection

Overview

Each node bounds disk growth by deleting messages older than a fixed RETENTION_WINDOW. Retention is time-based only -- no read-progress or per-chat policy. The rule is symmetric across all nodes, requires no network coordination, and applies uniformly to every chat and every message type.

Rule

A message is eligible for deletion when its HLC stamp's wall-clock component falls past the cutoff:

hlc.physical_ms <= cutoff_ts
cutoff_ts = now_ms - RETENTION_WINDOW

Each node computes cutoff_ts locally from its own clock. No cutoff is persisted in the DB or exchanged over the wire. Storing the HLC in packed form (physical_ms in the upper 48 bits, logical in the lower 16) means the comparison is still a plain millisecond test -- identical in shape to the legacy ts: u64 rule.

Parameters

Defined as constants in crates/node/src/retention.rs:

RETENTION_WINDOW is hard-coded. Changing it requires recompilation and coordinated rollout across the network (see "Clock skew and divergence" below for why mismatched windows are safe but degrade sync convergence).

GC Cycle

retention::gc_cycle(db, merkle_msgs, now) is the unit of work; the background loop retention::run_gc_loop schedules it.

One cycle does:

1. cutoff_ts = cutoff_ts_at(now). Updates retention_cutoff_ts_ms Prometheus gauge.

2. Range-delete in CF messages for every chat enumerated from chats_meta: delete_range_cf([chat_id, HlcTimestamp::ZERO, 0], [chat_id, HlcTimestamp::from_parts(cutoff_ts + 1, 0), 0)). The packed HLC puts physical_ms in the upper 48 bits, so every message with hlc.physical_ms <= cutoff_ts -- regardless of logical or seq -- falls strictly below the end key.

3. Chunked scan of CF seen_msg: collect up to GC_CHUNK_SIZE msg_ids whose embedded hlc.physical_ms is <= cutoff_ts. For each chunk:

   • delete_seen_msg_batch removes them from seen_msg.
   • MerkleTree::xor_batch cancels their contribution from the in-memory messages tree (one write lock per chunk).

4. Repeat step 3 until either the scan returns less than the chunk size (nothing more to remove) or removed_count >= GC_BATCH_LIMIT.

5. Update counters: gc_messages_deleted_total, gc_chats_processed_total, and gc_cycle_duration_seconds.

6. Message-ops drain (gc_message_ops): the same chunked scan over CF seen_op, removing aged edit/delete operations from message_ops, seen_op, and ops_by_chat (via delete_op_batch) and XOR-cancelling them from the message-ops Merkle tree. scan_seen_op_aged returns each op together with its seen_op value so the delete rebuilds all three keys without re-reading. The count folds into gc_messages_deleted_total and is reported separately as CycleStats.ops_removed.

The cycle returns CycleStats { removed_count, ops_removed, chats_processed, hit_limit }. If hit_limit is true, the loop schedules the next cycle after GC_SHORT_INTERVAL instead of the steady-state GC_INTERVAL -- so backlogs drain quickly without forcing a single cycle to do unbounded work.

Why removing an op is safe. An operation may only target a message younger than the 48-hour edit window, while RETENTION_WINDOW is 30 days. By the time an op ages out, its target has aged out too: removing the op can never revert a materialized row (the row is the state; the op was only the transport), and no peer can still be serving the original message for the op to conflict with. Orphan ops whose target was never seen locally simply expire on their own stamp.

Why GC writes via direct Arc, not the DB writer pipeline

Most paths (gossip, HTTP API, sync) converge in process_db_op so a single thread owns Merkle mutations. The GC cycle bypasses that channel and takes the Merkle write lock directly, in chunks of GC_CHUNK_SIZE. This keeps the worst-case put_message latency during GC bounded by one chunk (~100 microseconds for 1 000 ids), because the write lock is released between chunks and the DB-writer/sync consumers can interleave.

Backward-compatibility for legacy seen_msg entries

Older seen_msg entries pre-date the change that stores the messages-CF key as the value (44-byte payload including the 8-byte packed HLC). Legacy entries with empty values are treated as physical_ms = 0 by extract_hlc_physical_from_seen_value, which means they are always picked up on the next GC cycle and naturally reclaimed -- no migration script.

Sync Soft-Filter

The retention boundary is enforced independently on both sides of the sync protocol. This is the property that makes "deleted messages never resurface from peers".

Responder side

When answering Merkle sync requests for SyncDomain::Messages:

• get_bucket_msg_ids_filtered(db, bucket, cutoff_ts) is used instead of the plain helper. msg_ids whose stored HLC has physical_ms <= cutoff_ts are stripped from the response before the wire send.
• get_messages_cbor_batch_filtered(db, ids, max_bytes, cutoff_ts) applies the same filter when serving FetchAndPush payloads.

The filter reads the packed HLC from the seen_msg value and pulls out physical_ms (no extra DB lookup) -- O(scanned bucket size).

Receiver side

In sync::handler::store_synced_messages:

• For every (msg_id, cbor) pair received via FetchAndPush, decode MsgV1, compare msg.hlc.physical_ms() with the local cutoff_ts_now().
• If physical_ms <= cutoff_ts: drop the message, increment sync_messages_rejected_total, never enqueue DbOp::PutMessage.

The receiver re-checks independently to defend against:

• Clock skew: peer's cutoff sits a few seconds behind ours.
• Mismatched parameters: peer compiled with a different RETENTION_WINDOW.
• Malicious peer: deliberately re-pushes aged-out records.

Why filtering applies to Messages and MessageOps

Both message-shaped domains are time-bounded and filtered symmetrically. SyncDomain::MessageOps uses the same soft-filter on op.hlc.physical_ms (responder: get_bucket_op_ids / get_op_records_batch take cutoff_ts; receiver: store_synced_ops re-checks and drops aged ops, counting them in sync_ops_rejected_total).

SyncDomain::Members and SyncDomain::Identity are mutable CRDT-like domains where records are inherently small and naturally bounded by membership and identity churn. No retention is currently applied to those domains; their sync helpers stay un-filtered.

Merkle Tree Implications

The in-memory Merkle tree for messages is bit-identical to the serialised state of seen_msg:

• On startup, the tree is rebuilt from for_each_msg_id (yields (msg_id, physical_ms) pairs -- the physical component is currently ignored at startup but is available for future per-cutoff trees).
• On PutMessage, tree.insert(msg_id) is called once.
• On retention GC, tree.xor_batch(removed_chunk) cancels the removed msg_ids in one pass per chunk. XOR is its own inverse, so the same primitive used for insert is reused for removal (xor_batch is the symmetric batch form of xor_leaf plus a single recompute_all over the affected level-1 subtrees).

This means after GC the tree root does change. Peers that have not yet GC'd see a different root in the next sync tick; the soft-filter on both sides ensures the discrepancy resolves to "no aged messages were transmitted" rather than "the deleter re-downloads them".

Storage Layout Changes

seen_msg value semantics

The value of CF seen_msg is the 44-byte messages-CF key ([chat_id:32][hlc_packed:u64 BE][seq:u32 BE]); retention uses the extract_hlc_physical_from_seen_value helper to pull the upper 48 bits (physical_ms) out of bytes 32..40. No extra DB lookup is needed -- the read path was already in place for the legacy ts: u64 slot; the HLC migration only changed the interpretation of those 8 bytes.

Physical deletion

range_delete_old_messages(db, chat_id, cutoff_ts) calls delete_range_cf on CF messages from [chat_id, HlcTimestamp::ZERO, 0] to [chat_id, HlcTimestamp::from_parts(cutoff_ts + 1, 0), 0). RocksDB processes this as a tombstone; physical reclamation happens during compaction.

delete_seen_msg_batch(db, msg_ids) removes the matching seen_msg entries in a single WriteBatch.

for_each_chat_id(db, callback) enumerates chats_meta keys so the cycle can issue one range-delete per chat. Empty chats are no-ops in RocksDB and contribute negligible overhead.

Metrics

All under the p2pmes_ Prometheus namespace:

Only the receiver side of the sync filter produces a metric. The responder side is silent because the underlying DB helpers return only the post-filter result; instrumenting it would require either changing the return shape or doing a double scan.

Clock Skew and Divergence

Each node computes cutoff_ts independently. Skew between nodes leads to a transient window where:

• Node A (clock 10s fast) considers a message expired.
• Node B (clock right) still considers it fresh.

What happens in this window:

• Sync responder filters by local cutoff: A drops the message from bucket responses; B includes it.
• Sync receiver filters by local cutoff: A rejects the message if B sends it; B accepts it if A sends it (A never will).
• Eventually consistent: once both clocks reach the message's ts + RETENTION_WINDOW, both nodes agree it is expired and drop it.

No node is ever forced to keep a message it considers expired, and no node is ever forced to discard a message it considers fresh.

A node with a wildly wrong clock simply removes its own data sooner or later than the rest of the network; it cannot poison its peers because the cutoff is never transmitted -- everyone applies their own.

Edge Cases

• Chats with all-aged history: range-delete reclaims all entries, chats_meta survives with stale last_seq / last_ts -- this is intentional (those fields are monotonic; user_inbox semantics rely on them).
• Concurrent PutMessage during GC: the DB writer and GC both operate on RocksDB which is thread-safe; the Merkle tree is serialised by its RwLock. PutMessage may briefly block on the Merkle lock during a chunk apply (GC_CHUNK_SIZE bounds this).
• GC during sync session: bucket responses prepared before the cycle may include msg_ids that GC has since removed; subsequent FetchAndPush will fail to find the payload (None from seen_msg) and the message is simply skipped. Sync converges on the next tick.
• Node restart mid-cycle: nothing persists about a "current cycle"; the next start-up rebuilds the Merkle tree from current seen_msg, and the next GC tick resumes work. No coordination needed.

See Also

• docs/SYNC.md -- Merkle protocol, where the soft-filter sits.
• docs/STORAGE.md -- seen_msg value layout, retention helpers.
• docs/ARCHITECTURE.md -- background-task layout.
• crates/node/src/retention.rs -- implementation and unit tests.

Testing Methodology

Principle

Every new code must be covered by tests. Pure logic is covered by unit tests, interactions between components are covered by integration tests.

Test Layers

Layer 1: Unit tests (pure logic, no I/O)

In-module #[cfg(test)] mod tests blocks. Fast, deterministic, no external dependencies.

What to test:

• Serialization roundtrips (CBOR encode/decode for all GossipMessage variants)
• Key construction and ordering (db::keys)
• Deterministic ID computation (compute_msg_id, dm_chat_id)
• Canonical string building, hex parsing (api::utils)
• Cryptographic primitives (crypto::keccak256, crypto::parse_addr20)
• Data structure logic (InboxBatcher merging/flushing, PeerCache, MerkleTree)

Locations:

• crates/node/src/types.rs -- gossip roundtrips, inbox batcher, peer cache
• crates/node/src/sync/merkle.rs -- Merkle tree operations
• crates/node/src/handlers/context.rs -- XOR distance
• crates/node/src/handlers/mpsc/utils.rs -- unique ID generation
• crates/db/src/keys.rs -- key format and ordering
• crates/db/src/messages.rs -- msg_id computation
• crates/node/src/sync/protocol.rs -- SyncRequest/SyncResponse CBOR roundtrips
• crates/api/src/utils.rs -- canonical signing, dm_chat_id, hex helpers
• crates/crypto/src/lib.rs -- keccak256, address parsing, ECDSA verify_sig_recover

Layer 2: DB integration tests (RocksDB with tempdir)

Tests that open a real RocksDB instance in a temporary directory. Verify that read/write operations work correctly end-to-end through the storage layer.

What to test:

• put_message + range roundtrip
• Message idempotency (same msg_id written twice)
• Pagination (after_key / limit)
• Time range filtering (from_ts, to_ts, combined)
• Read progress (set, get, monotonic increase)
• Inbox upsert + list (ordering by activity time, pagination)
• Member add, remove, list, membership check
• Control messages (msg_type > 0)
• Sync helpers: for_each_msg_id, get_message_cbor_by_id, get_messages_cbor_batch (incl. byte limit), get_bucket_msg_ids

Location: crates/db/src/lib.rs (#[cfg(test)] mod db_tests)

Helper: temp_db() creates a ChatDb backed by TempDir.

Layer 3: In-process integration tests (multiple nodes)

Full nodes running in a single tokio runtime, communicating via real libp2p TCP connections and gossipsub.

What to test:

• Node startup and graceful shutdown
• Peer discovery and connection via bootnodes
• Message sending (DM stored locally via async DB writer)
• Message propagation between nodes via gossip
• Message ordering by timestamp
• ListUserChats (inbox populated after DM send)
• ReadChatMessage (read progress stored via async DB writer)
• Merkle-tree sync per domain:
  • test_sync_messages -- 500 DMs, exercises multi-bucket drill-down and chunked FetchAndPush
  • test_sync_members -- group with admin + member, verifies role preservation across sync
  • test_sync_identity -- two identity blobs, verifies blob content after sync

Location: crates/node/tests/integration.rs

Infrastructure:

• run_node(AppConfig) returns NodeHandle with cmd_tx, db, listen_addrs
• Each test node uses a random keypair, port 0, temp DB directory
• two_connected_nodes() helper spins up a pair with gossipsub mesh
• gossipsub requires >= 1 peer, so even "local" tests use two nodes
• Tests send Command variants via cmd_tx and assert on DB state or oneshot responses
• Gossip propagation tests sleep 2-3s for mesh formation + message delivery

Running Tests

cargo test -p node                                     # All tests (unit + integration, production code only)
cargo test -p node --features test-support             # Same + clock-skew E2E tests via MockClock
cargo test -p node --lib                               # Unit tests only (fast)
cargo test -p node --test integration                  # Integration tests only
cargo test -p db                                       # DB unit + integration tests
cargo test -p node -- test_name                        # Single test by name
cargo test -p node -- --nocapture                      # With stdout visible

test-support cargo feature

Some integration tests need to bring a node up in a non-default way -- a controllable HLC clock, in-memory transport stubs, fault injection wrappers, anything else that doesn't make sense in production. Rather than letting those helpers leak into the prod build, the crate exposes them through node::test_support, a module gated behind the test-support cargo feature.

How to use it:

• Run gated tests with cargo test -p node --features test-support. Without the flag the gated tests are silently skipped (the rest of the suite still runs).
• Put helpers that wrap or replace production entry points under crates/node/src/test_support.rs. Re-export them with pub fns. Production binaries are compiled without the feature, so anything in that module is invisible to release builds.
• Mark new tests that depend on the module with #[cfg(feature = "test-support")] (either on the test fn or on a containing mod).

First user of the mechanism is run_node_with_clock, which threads a custom Arc<dyn Clock> into run_node for clock-skew E2E tests under tests/integration.rs::clock_skew_e2e. Future helpers (network fault injection, swappable storage, deterministic randomness) can land in the same module without changing the feature gate.

Clock Abstraction in Tests

types::clock::Clock is the time source HLC depends on. Production uses SystemClock (wraps SystemTime::now). Tests use MockClock, an AtomicU64-backed clock with set(ms) and advance(delta_ms) so a test can rewind into the past, jump forward, or simulate per-node drift.

Integration tests that need a controllable clock build nodes via node::test_support::run_node_with_clock(config, clock) (see the test-support feature section above). Unit tests in crates/node/src/hlc_state.rs use MockClock directly for the HLC algorithm tests (drift bound, overflow handling, monotonicity under concurrent stamping).

Writing New Tests

For new pure logic (types, computations, parsing)

Add #[cfg(test)] mod tests in the same file. Test edge cases, roundtrips, and error paths. No I/O, no sleeps.

For new DB operations

Add a test in crates/db/src/lib.rs::db_tests using temp_db(). Write data, read it back, verify.

For new gossip message variants

1. Add a roundtrip test in crates/node/src/types.rs::gossip::tests
2. If the variant affects cross-node behavior, add an integration test in crates/node/tests/integration.rs

For new HTTP API endpoints

1. Add the Command variant and handler
2. Add an integration test that sends the command via cmd_tx and verifies the response and/or DB state

For new protocol interactions (sync, query/response)

Add integration tests with two+ nodes verifying the full round-trip.

For entirely new subsystems

If the new functionality does not fit any of the categories above (e.g. a new transport layer, a new storage backend, a standalone utility crate):

1. Create a dedicated test file or #[cfg(test)] mod tests block within the new module
2. If the subsystem interacts with other components, add integration tests in crates/node/tests/ (one file per subsystem, e.g. crates/node/tests/new_subsystem.rs)
3. Follow the same layering principle: pure logic in unit tests, cross-component interactions in integration tests

Test Conventions

• Tests go at the end of the module (clippy: "items after test module")
• Use #[cfg(test)] to avoid compiling test code in release builds
• Integration tests use #[tokio::test] (async runtime required)
• Prefer assert_eq! with descriptive messages over bare assert!
• Clean up resources: hold TempDir handles, call shutdown().await
• Do not hardcode ports -- always use port 0 for OS assignment