eMPP (multi-protocol OP_RETURN envelope)
Protocol and consensus12.4.15+

eMPP (multi-protocol OP_RETURN envelope)

The multi-push OP_RETURN envelope the application protocols ride on.

eMPP ("eCash Multi-Protocol Payload") is a transport, not an application protocol: one OP_RETURN OP_RESERVED output carries one or more data pushes, and each push is a segment that starts with a 4-byte LOKAD id naming the protocol that owns the rest of the bytes. The node's empp module splits every such output into segments and stores one row per segment in SQLite. It does not interpret payloads. The other modules (SLP, ALP, RANK, BLOB, POKR) read the segments carrying their own LOKAD id from that table while they index a block. A payload can therefore arrive either in its protocol's native OP_RETURN form or wrapped in eMPP, and both forms give the same indexed state.

Status in this node #

ItemValue
Module nameempp (src/modules/empp/empp_module.cpp:79)
Module version / schema1.1.0 / schema 2, data version 2 (SCHEMA_VERSION, DATA_VERSION in src/modules/empp/empp_module.h). 12.4.6 added output_sats, moved the output value to the first segment and stores LOKAD-only segments; a database of an older version is rebuilt from the blocks at startup (see Upgrade below).
EnabledAlways. RegisterBuiltinModules registers it unconditionally, first, before every other module (src/modules/builtin_modules.cpp:31). No command-line flag turns it off.
Data file<datadir>/modules/empp.sqlite (src/modules/module_registry.cpp:196, src/modules/empp/empp_module.cpp:87)
MempoolNot indexed. eMPP has no mempool hook: only confirmed blocks produce rows.
Order dependencyCModuleRegistry::ConnectBlock calls modules in registration order (src/modules/module_registry.cpp:215-221), so eMPP rows for a block exist before SLP/RANK/POKR/BLOB/ALP run. DisconnectBlock runs in reverse order (src/modules/module_registry.cpp:233-239), so eMPP rows are removed last.

Envelope #

Text
scriptPubKey = OP_RETURN (0x6a)  OP_RESERVED (0x50)  <push 1>  <push 2>  ...  <push n>
push i body  = <lokad_id: 4 bytes> <protocol payload: 0..N bytes>
  • Any output of any transaction is scanned, including the coinbase and outputs other than vout 0 (CEmppModule::ProcessBlock). This differs from the eCash spec, see the last section.
  • A cheap pre-filter checks the first two bytes only (HasEmppHeader, src/modules/empp/empp_decoder.cpp:101-108). The strict decoder runs only if they match.
  • Policy (not eMPP): a transaction with more than 3 OP_RETURN outputs is non-standard (multi-op-return, src/policy/policy.cpp:115), and the default relay limit for an OP_RETURN scriptPubKey is 8005 bytes (MAX_OP_RETURN_RELAY, src/script/standard.h:85).

Byte layout #

Layout of the whole scriptPubKey:

OffsetSizeTypeEndiannessFieldNotes
01opcoden/aOP_RETURN = 0x6arequired
11opcoden/aOP_RESERVED = 0x50eMPP marker, required
2varpushsee belowsegment 1at least one segment is required
...varpushsee belowsegment 2..nrepeated until the end of the script

How each segment push is encoded (strict, canonical pushes only):

Opcode byteLength fieldAllowed lengthsNotes
0x01..0x4bnone (opcode is the length)1..75direct push
0x4c OP_PUSHDATA11 byte76..255lengths below 0x4c are rejected (non-canonical)
0x4d OP_PUSHDATA22 bytes, little-endian256..65535lengths below 0x100 are rejected
0x4e OP_PUSHDATA44 bytes, little-endian65536..1048576lengths below 0x10000 or above 1 MiB are rejected
0x00, 0x4f, 0x51..0x60, anything elsen/an/arejected: the whole envelope is invalid

Body of one segment (after the push header is removed):

OffsetSizeTypeEndiannessFieldNotes
04bytesbyte order as writtenlokad_idstored exactly as in the script, e.g. 53 4c 50 00 for SLP
4push length āˆ’ 4bytesprotocol-definedpayloadmay be empty (a 4-byte push is a valid segment)

Versioning: eMPP has no version byte. Versioning belongs to each protocol inside its payload.

Known LOKAD ids consumed by modules of this node:

LOKAD bytesASCIIConsumerDefined at
53 4c 50 00SLP\0SLP v1src/modules/slp/slp_types.h:15
53 4c 50 32SLP2ALPsrc/modules/alp/alp_types.h:15
52 41 4e 4bRANKRANKsrc/modules/rank/rank_module.cpp:35
42 4c 4f 42BLOBBLOBsrc/modules/blob/blob_module.cpp:31
50 4f 4b 52POKRPOKRsrc/modules/pokr/pokr_module.cpp:29

Segments with any other LOKAD id are still stored and visible through the REST API. No module consumes them.

Validation rules #

All rules are in empp::DecodeScript (src/modules/empp/empp_decoder.cpp). A failure rejects the whole output. Segments decoded before the failure are not kept.

  1. The script must be non-empty and start with OP_RETURN, else not_op_return (src/modules/empp/empp_decoder.cpp:114).
  2. Byte 1 must be OP_RESERVED (0x50), else missing_reserved (src/modules/empp/empp_decoder.cpp:121).
  3. Forbidden opcodes after the marker: OP_0 (0x00), OP_1NEGATE (0x4f), OP_1..OP_16 (0x51..0x60) (src/modules/empp/empp_decoder.cpp:46). Any other non-push opcode is also rejected (src/modules/empp/empp_decoder.cpp:74-75). Error bad_push.
  4. OP_PUSHDATA1 needs its length byte, and the length must be ≄ 0x4c (src/modules/empp/empp_decoder.cpp:54, :57).
  5. OP_PUSHDATA2 needs 2 length bytes (LE), and the length must be ≄ 0x100 (src/modules/empp/empp_decoder.cpp:59, :62).
  6. OP_PUSHDATA4 needs 4 length bytes (LE). The length must be ≄ 0x10000 and ≤ MAX_EMPP_PUSH_LEN = 1 MiB (src/modules/empp/empp_decoder.cpp:26, :64, :71, :72).
  7. Global cap: no push may exceed 1 MiB (src/modules/empp/empp_decoder.cpp:81).
  8. The push data must fit in the remaining script (no truncation) (src/modules/empp/empp_decoder.cpp:86).
  9. Every push must be at least 4 bytes (the LOKAD id), else short_segment (src/modules/empp/empp_decoder.cpp:138).
  10. At least one segment is required (OP_RETURN OP_RESERVED alone is no_segments) (src/modules/empp/empp_decoder.cpp:149).
  11. Indexer: outputs that fail decoding are skipped silently. No row is written and no error is recorded (CEmppModule::ProcessBlock).

Unit tests for every rule above: src/test/empp_decoder_tests.cpp (26 cases, for example rejects_pushdata4_oversize_dos_guard, rejects_mid_envelope_short_segment, decodes_lokad_only_segment_empty_payload_allowed).

Indexing #

Database <datadir>/modules/empp.sqlite, created in CEmppModule::CreateSchema.

Table empp_segments, one row per segment:

ColumnTypeNotes
txidTEXT NOT NULLlowercase hex txid (GetId().GetHex())
voutINTEGER NOT NULLoutput index
seg_idxINTEGER NOT NULL0-based position of the segment in the envelope
lokad_idBLOB NOT NULLthe 4 raw bytes
payloadBLOB NOT NULLsegment bytes after the LOKAD id; a LOKAD-only segment stores a zero-length blob
block_heightINTEGER NOT NULL DEFAULT -1height of the confirming block
block_timeINTEGER NOT NULL DEFAULT 0block header time (unix seconds)
satsINTEGER NOT NULL DEFAULT 0burn credited to this segment: the value of the OP_RETURN output (nValue / SATOSHI) on the first segment of the output (seg_idx 0), 0 on every other segment. The sats of the segments of one output add up to the output value once.
output_satsINTEGER NOT NULL DEFAULT 0value of the OP_RETURN output, the same on every segment of the output (12.4.6)

Primary key (txid, vout, seg_idx). Indexes: idx_empp_lokad_height (lokad_id, block_height DESC) and idx_empp_height (block_height DESC, txid).

Table empp_meta (key TEXT PRIMARY KEY, value TEXT NOT NULL) stores tip_height and tip_hash, the last block indexed, and data_version (2).

Connect: one SQLite transaction per block. The insert is an UPSERT on the primary key, so re-connecting a block changes nothing. The result of every insert is checked: if one fails, the error is logged, the whole block is rolled back and the tip stays on the parent (never a partial block). The tip meta is updated in the same transaction.

Disconnect (reorg): DELETE FROM empp_segments WHERE block_height = ?, then the tip moves back to the parent block, in one transaction (CEmppModule::DisconnectBlock). Connecting N blocks then disconnecting the last K leaves exactly the rows of the first Nāˆ’K blocks (empp_module_tests/connect_disconnect_symmetry_all_depths).

Startup reconciliation (shared by all modules, src/modules/module_registry.cpp:47-176): if the module's tip is ahead of the chain, or its tip hash is not on the active chain, or it is more than kGapReindexThreshold = 200,000 blocks behind (src/modules/module_registry.h:166), ResetForReindex deletes every segment and the tip meta and runs VACUUM. The module is then replayed from genesis. A smaller gap on the active chain is replayed without a wipe.

Upgrade (12.4.6). Rows written before data version 2 miss every LOKAD-only segment and carry the output value on every segment; the missing rows can only come back from the blocks. At Init, a database that has indexed state but no data_version 2 is emptied (ResetForReindex) and its tip is set to the genesis block (height 0, genesis hash). The startup reconcile then sees a module behind the chain whose hash is on the active chain and replays the chain into it (gap replay up to 200,000 blocks, full reindex beyond), eMPP first in each block. The log line is eMPP: indexed data version 1 < 2: rebuilding the index from the blocks. A new database is only stamped with data_version 2. Tests: empp_module_tests/upgrade_from_v1_rebuilds_from_genesis, new_database_is_stamped_without_tip, feature_module_indexers.py (upgrade phase).

Consumer API (in-process, not REST): SegmentsForBlock(height, lokad) returns the segments of one block for one LOKAD id, ordered by txid, vout, seg_idx, each with sats and output_sats (EmppRow). Called by the SLP, RANK, BLOB and POKR modules. ALP does not use it: it parses output 0 of each transaction itself, with the Chronik pushdata rules (see ALP envelope).

One output burns its value once (rule for consumers). The eMPP spec says nothing about the output's value; this is the node's accounting rule. Within one protocol, a consumer credits an output's value at most once per (txid, vout): RANK and POKR take the first decodable segment of their LOKAD in the output (their records are keyed by (txid, vout)) and weigh it with output_sats; BLOB credits output_sats to the first chunk of the output that it stores and 0 to the others. Two different protocols in one output each credit the value once (they are independent indexes). Before 12.4.6 every segment carried the full value, so a POKR output with two segments counted its burn twice and a two-chunk BLOB output doubled total_burn_sats.

REST API #

Routes are registered by CEmppModule::RegisterRoutes (src/modules/empp/empp_module.cpp:326-340) through AddModuleRoute, which uses the default TTL 0 (src/api/api_server.cpp:345-348). The responses are therefore not cached by the API response cache. All responses are application/json.

Method and pathParametersResponse
GET /api/v1/empp/statsnone{segments, distinctLokads, uniqueTxs, tipHeight, byLokad:[{lokad, lokadHex, count}]}. byLokad holds the top 50 LOKAD ids by segment count.
GET /api/v1/empp/recentlimit (default 50, clamped 1..200), before_height (optional, strict <){events:[{txid, vout, segIdx, lokad, lokadHex, payloadSize, blockHeight, blockTime, sats, outputSats}]}, newest first (block_height DESC, txid)
GET /api/v1/empp/by-lokad/{hex8}path: exactly 8 hex chars (4 bytes, either case); limit (default 50, clamped 1..200){lokad, lokadHex, events:[{txid, vout, segIdx, payloadSize, blockHeight, blockTime, sats, outputSats}]}. 400 missing_lokad without the path segment, 400 bad_lokad for a bad hex value.
GET /api/v1/empp/tx/{txid}path: txid (lowercase hex, compared as text){txid, segments:[{vout, segIdx, lokad, lokadHex, payloadHex, payloadSize, blockHeight, blockTime, sats, outputSats}]}. 404 not_found when no segment is indexed; 400 missing_txid without the path segment.

Field formats: lokadHex is 8 lowercase hex characters without separators (e.g. 534c5000). lokad is the 4 bytes as ASCII when every byte is in 0x20..0x7e, otherwise 0x followed by the hex value. NUL is not printable, so SLP shows as 0x534c5000 (LokadAscii). sats is the burn credited to the segment (output value on segment 0, else 0), outputSats the output value.

These routes are not described in /api/v1/openapi.json (src/api/openapi_handler.cpp has no eMPP entry).

Dashboard #

The "eMPP" tab (data-tab="empp", data-module="empp", hidden when /api/v1/modules does not list the module) has three sub-tabs:

RouteSub-tabCalls
/dashboard/emppHub: stat cards, "What is eMPP?", latest 10 segmentsempp/stats, empp/recent?limit=10
/dashboard/empp/recentRecent: table with 25/50/100 rowsempp/recent?limit=N
/dashboard/empp/bylokadBy LOKAD: free-text filterempp/by-lokad/<hex8>?limit=50

Other tabs show a "via eMPP" badge on rows whose API record has source: "empp" (sourceBadge in share/www/dashboard/js/dashboard.js).

The tab reads the API fields as documented above: events of empp/recent and empp/by-lokad, tipHeight for "Last Indexed", payloadSize for the byte column. Friendly labels (SLPv1, ALP (SLP2), RANK, BLOB, POKR) are keyed on the 8-hex lokadHex. The By LOKAD box accepts 8 hex chars (separators ,, :, spaces and a 0x prefix are allowed: 53,4c,50,00) or a 4-character name where \0 is a NUL byte (RANK, SLP\0), turns it into the 8 hex chars and calls /empp/by-lokad/<hex8>; anything else gets an inline message and no request (lokadToHex). Before 12.4.6 the tab read segments, lastIndexedHeight and payloadLen, keyed labels on 53,4c,50,00, and sent ?lokad=, so it showed no segment, a "Last Indexed" of 0, and By LOKAD always got HTTP 400. Tests: feature_module_indexers.py (dashboard harness).

Limits and edge cases #

  • Several eMPP outputs in one transaction are indexed independently (key includes vout).
  • Several segments with the same LOKAD id in one output are all kept (distinct seg_idx). Whether a consumer accepts more than one is up to that consumer.
  • Empty payload (lokad-only segment): stored, as a zero-length blob (the spec: the payload after the LOKAD is "an arbitrary sequence of bytes"; only a push of the empty string is invalid). Before 12.4.6 the empty payload was bound as a NULL pointer, which SQLite stores as NULL, the NOT NULL constraint failed, the unchecked insert lost the row silently while the other segments of the output were kept (empp_module_tests/lokad_only_segment_is_indexed).
  • The REST handlers run holding m_db_mutex (12.4.6). CSqliteWrapper does not serialise a bind, step and reset sequence on its own: it has one connection and hands every caller the same cached statement. Before the fix, /empp/stats reported segments of a half-written block, and concurrent /empp/tx calls returned each other's rows (module_db_concurrency_tests).
  • empp/tx/{txid} compares the txid as text: an uppercase txid finds nothing.
  • block_height DEFAULT -1 is never used: rows are written only for confirmed blocks.
  • Any LOKAD id is stored, so a junk 4-byte prefix counts in distinctLokads.

Differences from upstream / eCash spec #

Official text: https://github.com/Bitcoin-ABC/bitcoin-abc/blob/master/doc/standards/empp.md. Same envelope: "OP_RETURN (0x6a), OP_RESERVED (0x50), <pushop0> … <pushopN>", "One push op containing the entire payload for each fragment", a payload that "should start with a 4 byte LOKAD ID … Followed by an arbitrary sequence of bytes", and "Any pushop pushing the empty string is invalid". Where this node differs from the spec (open, needs an owner decision because Lotus policy allows several OP_RETURN outputs; no behaviour changed in 12.4.6):

  • The spec: "A valid eMPP tx must have an OP_RETURN output at the first (vout = 0) position. Any OP_RETURN at a position after that should be ignored". The node indexes every output.
  • The spec: "Unlike data in scriptSig, the opcode doesn't have to be encoded minimally". Our decoder rejects non-minimal pushes (rules 4 to 6) and OP_0/OP_1NEGATE/OP_1..16.
  • The LOKAD id is a "should" in the spec; the decoder rejects a push shorter than 4 bytes (rule 9).
  • A single push is capped at 1 MiB, a local limit (Lotus consensus allows OP_RETURN scripts of about 99,990 bytes, src/script/standard.h:79).
  • eCash chronik indexes eMPP per protocol. Our node also keeps a raw, protocol-agnostic segment table and exposes it over REST.

Worked example #

Unverified: at the time of writing, the public node lotus-127.givelotus.org answered 503 node_warming_up ("Rebuilding module indices from genesis") on /api/v1/empp/stats, so no mainnet eMPP transaction could be fetched. The example below is built from the unit test decodes_multi_segment_preserves_order (src/test/empp_decoder_tests.cpp:199): three segments SLP\0 + 01, RANK + 02 03, and BLOB with an empty payload.

Text
6a 50 05 534c500001 06 52414e4b0203 04 424c4f42
OffsetBytesMeaning
06aOP_RETURN
150OP_RESERVED, eMPP marker
205direct push, 5 bytes: segment 0
353 4c 50 00LOKAD SLP\0, so SLP v1
701payload (1 byte)
806direct push, 6 bytes: segment 1
952 41 4e 4bLOKAD RANK, so RANK
1302 03payload (2 bytes)
1504direct push, 4 bytes: segment 2
1642 4c 4f 42LOKAD BLOB, empty payload

If this script were vout 1 of a confirmed transaction, GET /api/v1/empp/tx/<txid> would return:

Text
{"txid":"<txid>","segments":[
 {"vout":1,"segIdx":0,"lokad":"0x534c5000","lokadHex":"534c5000","payloadHex":"01","payloadSize":1,"blockHeight":H,"blockTime":T,"sats":V,"outputSats":V},
 {"vout":1,"segIdx":1,"lokad":"RANK","lokadHex":"52414e4b","payloadHex":"0203","payloadSize":2,"blockHeight":H,"blockTime":T,"sats":0,"outputSats":V},
 {"vout":1,"segIdx":2,"lokad":"BLOB","lokadHex":"424c4f42","payloadHex":"","payloadSize":0,"blockHeight":H,"blockTime":T,"sats":0,"outputSats":V}]}

Segment 2 (BLOB, empty payload) is listed. Segment 0 carries "sats":V (the output value V), segments 1 and 2 "sats":0, all three "outputSats":V.

Source files #

  • src/modules/empp/empp_types.h: LOKAD id type, segment and message structs
  • src/modules/empp/empp_decoder.h, src/modules/empp/empp_decoder.cpp: header pre-filter and strict decoder
  • src/modules/empp/empp_module.h, src/modules/empp/empp_module.cpp: SQLite indexer, SegmentsForBlock, REST routes
  • src/modules/builtin_modules.cpp: registration order (eMPP first)
  • src/modules/module_registry.cpp, src/modules/module_registry.h: dispatch order, startup reconciliation
  • src/test/empp_decoder_tests.cpp: decoder unit tests
  • share/www/dashboard/index.html, share/www/dashboard/js/dashboard.js: eMPP tab
Source doc/protocols/empp.md in the lotusd repository