> ## Documentation Index
> Fetch the complete documentation index at: https://totem-cb8b3887.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Methodology & next steps

> How this firmware was analyzed, the tools used, and how to close the remaining gaps.

## Approach

The firmware findings come from **static analysis** of public artifacts: the v5.0.2 and
v5.0.3 firmware images and the companion Android app. No network service was attacked; the
releases API and S3 objects are served openly. The BLE protocol was additionally checked
end to end with the `totemctl` client (`cmd/totemctl` in this repository) against the
author's own device on firmware 4.1.3 and 5.0.3. That test found the
[transmit-mode](/protocols/ble#transmit-modes) behaviour on macOS, which static analysis
alone had missed.

### Tools

| Tool                              | Use                                                                |
| --------------------------------- | ------------------------------------------------------------------ |
| `esptool`                         | parse the ESP-IDF image header, segments, and app descriptor       |
| Python + `struct`                 | parse headers, carve segments, read the `esp_app_desc_t`           |
| `capstone` 6.0 (`CS_ARCH_XTENSA`) | disassemble the native Xtensa code                                 |
| `rizin` / `rabin2`                | validate the rebuilt ELF                                           |
| `strings` + `grep`                | recover the qstr symbol table and string constants                 |
| `hermes-dec`                      | decompile the Android app's Hermes bytecode (for the download URL) |

### Steps

<Steps>
  <Step title="Parse the image">
    Confirm ESP32, 7 segments, entry point, and read the app descriptor
    (IDF v5.4, build date, ELF hash).
  </Step>

  <Step title="Rebuild an ELF">
    Map the 7 segments to their load addresses with correct R/W/X perms in an
    `EM_XTENSA` ELF for Ghidra/IDA/rizin.
  </Step>

  <Step title="Identify the runtime">
    Native strings reveal MicroPython v1.25.0 — the app is Python, frozen as bytecode.
  </Step>

  <Step title="Recover the symbol table">
    Extract the qstr pools and rodata strings: the application's method, attribute and
    class names and every log line. The first v5.0.2 pass counted 2,479 identifiers and
    11,857 strings by string scanning. Walking the pools directly gives 4,086 qstrs for
    v5.0.2 (228 core + 972 const + 2,886 frozen) and 4,162 for v5.0.3 (228 + 975 + 2,959).
  </Step>

  <Step title="Reconstruct behavior">
    Map modules, classes, protocol categories, and subsystem behavior from those symbols
    and the `{}`-style (Python `str.format`) log messages.
  </Step>
</Steps>

## Why not a full source decompile?

The application is **frozen MicroPython bytecode**. Unlike CPython (which has mature
decompilers such as `uncompyle6`/`decompyle3`), MicroPython has **no mature
bytecode-to-source decompiler**. The best available is *bytecode disassembly* (opcodes
plus qstr references), not clean Python source.

However, the frozen format keeps **every identifier as an interned string** (qstr) and
retains all string literals. That symbol table plus the log messages reconstruct each
module's behavior at close to source fidelity — which is how this documentation was
built.

## Closing the remaining gaps

Three items were graded "needs bytecode disassembly" in the
[protocol completeness matrix](/protocols/overview):

1. The complete per-category `cmd_id` enumeration (`TOTEM_MSG_MAP`).
2. The exact binding of each `struct` format to a specific message.
3. The field order of `CHUNK_HDR_FMT`.

The frozen-bytecode disassembly has now been carried out and all three are **recovered**.
The values below are literals present in the binary — every `struct`-format string appears
verbatim in the DROM rodata — and the method that produced them is recorded afterward.

<Note>
  The [`/protocols/overview`](/protocols/overview) completeness matrix now marks these three
  as **Recovered** and reproduces the full decoded map; this page records how they were obtained.
</Note>

### `TOTEM_MSG_MAP` — confirmed

Defined in `espnow_conn_v2.py`, built by a `BUILD_MAP 13` / `STORE_NAME TOTEM_MSG_MAP`
opcode sequence (fully decoded). The map is **flat**, keyed by a 2-byte `bytes` value
`(cat_id, cmd_id)` (byte 0 = `cat_id`, byte 1 = `cmd_id`); **every** value is a payload
`struct`-format string.

<Warning>
  An earlier pass decoded the map's qstr-immediate values with `>>2` and reported
  "handler/type names" (`disconn_animation`, `dev_total_lightsleep_ms`, `dev_info`,
  `device_power`, `disabled`). Those were **decode artifacts and are wrong**. On ESP32
  MicroPython a qstr-immediate is tagged by `(o & 7) == 2` (REPR\_A), and the qstr number is
  `o >> 3`, not `o >> 2`. Under the correct `o >> 3` decode every entry resolves to a
  `struct` format string — the values below.
</Warning>

| `(cat_id, cmd_id)`             | `struct` format      | Size |
| ------------------------------ | -------------------- | ---- |
| `(0, 0)` / `(0, 1)` / `(0, 2)` | `<BBffbbhbbb6B`      | 23 B |
| `(1, 0)`                       | `<BBB7bHbb`          | 14 B |
| `(1, 2)`                       | `<BBiffHii`          | 24 B |
| `(1, 5)`                       | `<BB9B8bhh4B`        | 27 B |
| `(1, 6)`                       | `<BBbbB`             | 5 B  |
| `(1, 7)`                       | `<3Bbb`              | 5 B  |
| `(2, 0)`                       | `<BB6BffbbHbbbhhBBi` | 33 B |
| `(3, 1)` / `(3, 2)`            | `<BBHH4b4BHHBb`      | 20 B |
| `(7, 0)`                       | `<BBffbbbHbbH`       | 19 B |
| `(7, 1)`                       | `<BBHbffb`           | 14 B |

### `struct` format → message bindings — confirmed (for listed)

Each payload format begins with `BB` = the echoed `cat_id`/`cmd_id` header:

| `(cat_id, cmd_id)` | `struct` format | Size |
| ------------------ | --------------- | ---- |
| `(1, 0)`           | `<BBB7bHbb`     | 14 B |
| `(1, 2)`           | `<BBiffHii`     | 24 B |
| `(1, 6)`           | `<BBbbB`        | 5 B  |
| `(1, 7)`           | `<3Bbb`         | 5 B  |
| `(7, 1)`           | `<BBHbffb`      | 14 B |

### Transfer header — format & field order confirmed

`CHUNK_HDR_FMT = <HBBBiHiB`, `CHUNK_HDR_SZ = 16` bytes — packed by `f_ble/chunking.py`'s
`gen_transfer_buff` via `struct.pack_into` at buffer offset 2. The format string, field order,
**and field names** are **confirmed** from the live `pack_into` opcode stream: `gen_transfer_buff`
calls `struct.pack_into('<HBBBiHiB', buff, 2, self.file_id, self.status_id, self.action_id, self.file_type_id, self.byte_pos, self.chunk_no, self.file_size, 0)`
(`LOAD_ATTR` order in the bytecode). The first pass also bound the *name* `CHUNK_HDR_FMT` to
this format, because `f_ble/file_upload.py` imports `CHUNK_HDR_FMT` and `CHUNK_HDR_SZ` from
`chunking`. **That inference was wrong.** v5.0.2's `chunking` never defines those names, so
the import could not have succeeded. v5.0.3 defines them explicitly, as the separate 12-byte
**chunk** header: `CHUNK_HDR_FMT = '<HHiH'`, `CHUNK_HDR_SZ = 12`, `UPLOAD_TO_APP = 3`. See
[chunking](/protocols/message-format#chunking). v5.0.3 also turns the last byte below into
a flags byte.

| `struct` code | Field                                                                         | Type      |
| ------------- | ----------------------------------------------------------------------------- | --------- |
| `H`           | `file_id`                                                                     | uint16 LE |
| `B`           | `status_id`                                                                   | uint8     |
| `B`           | `action_id`                                                                   | uint8     |
| `B`           | `file_type_id`                                                                | uint8     |
| `i`           | `byte_pos`                                                                    | int32 LE  |
| `H`           | `chunk_no`                                                                    | uint16 LE |
| `i`           | `file_size`                                                                   | int32 LE  |
| `B`           | literal 0 in v5.0.2; `pack_flags(is_last_chunk, is_origin_compass)` in v5.0.3 | uint8     |

### How it was done

<Steps>
  <Step title="Locate the frozen data">
    In the image (file offsets): `0xf3d3` points *into* the qstr identifier-string data,
    but mid-pool — the sorted run of identifier strings starts much earlier (around
    `0xe000`) and flows continuously through `0xf3d3` with no sort reset. The concrete
    anchors are the frozen qstr pool header at seg0 `0x5de0c` (`prev=0x3f4213d0`,
    `total_prev_len=1200`, `len=2886`) with its `qstrs[]` pointer array at `0x5de24`. The
    frozen module-name registry begins near `0x27b00`; the DROM segment maps file `0x18` →
    vaddr `0x3f400020`.
  </Step>

  <Step title="Parse the frozen module table">
    Reconstruct the MicroPython v1.25 (`.mpy` version 6.3) `mp_raw_code_t` tree for the two
    comms modules that actually carry these constants: `espnow_conn_v2.py` (home of
    `TOTEM_MSG_MAP` and the per-message `struct`-format registry) and `f_ble/chunking.py`
    (home of `CHUNK_HDR_FMT`).
  </Step>

  <Step title="Disassemble the bytecode">
    Walk the bytecode with a v1.25-matched reader. Reuse the opcode tables from `py/bc0.h`
    / `tools/mpy-tool.py`, but note `mpy-tool.py -d` itself only reads standalone `.mpy`
    blobs; frozen firmware modules (emitted as `proto_fun_*` / `fun_data_*` /
    `const_qstr_table_data_*` C structs, not `.mpy` blobs) require a custom frozen-aware
    reader. Small integers `-16..47` load inline as single-byte opcodes `0x80`–`0xBF`;
    larger ones via `LOAD_CONST_SMALL_INT` (`0x22`, signed var-int) — both embed the value
    in the bytecode. The `TOTEM_MSG_MAP` dict build and the `CHUNK_HDR_FMT` string literal
    are then readable.
  </Step>

  <Step title="Cross-check">
    Validate recovered `(cat_id, cmd_id)` values against the pairs already visible in the
    logs. `(0x01, 0x02)` and `(0x03, 0x01)` validate directly against `TOTEM_MSG_MAP`.
    `(0x06, 0x07)` is a Peer-Sync BLE control frame handled outside `TOTEM_MSG_MAP`, so
    `cat 6` is absent from the map — its absence is expected, not a failed cross-check.
  </Step>
</Steps>

An alternative dynamic approach — running the exact MicroPython build under an ESP32
emulator or on hardware and dumping `TOTEM_MSG_MAP` / `CHUNK_HDR_FMT` from a REPL — would
yield the same values faster if a device or matching build is available.

## Diffing firmware versions

The v5.0.3 image was processed with the same reconstruction and compared to v5.0.2 function
by function:

<Steps>
  <Step title="Carve and find anchors">
    Carve the 7 segments from the image header. Then locate the structures the
    reconstructor needs by their shape rather than by fixed addresses:

    * **qstr pools:** headers `{prev, total_prev_len, alloc, len, hashes*, lengths*,
      qstrs[]}`, validated by the NUL terminator at each declared length.
    * **Type objects:** `mp_type_type` is the only type object whose base points to itself;
      `str`, `bytes`, `tuple`, `float` and `int` point to it and carry their name qstr at +6.
    * **Frozen module table:** the longest run of pointers to `{qstr_table, obj_table,
      proto}` records whose first qstr ends in `.py`.

    On v5.0.2 this reproduces every hand-found address. On v5.0.3 it finds the frozen pool at
    seg0 `0x5f59c` (base 1203, 2,959 qstrs), the const pool at `0x22208` (975) and the module
    table at `0x3f4287b0`–`0x3f428924` (94 modules).
  </Step>

  <Step title="Reconstruct and disassemble">
    Re-serialize every frozen module as a standalone v6.3 `.mpy` and disassemble it with
    MicroPython's own `mpy-tool.py -d`: 96/96 modules for v5.0.2 and 94/94 for v5.0.3, with
    no unknown opcodes, truncated functions or dangling children.
  </Step>

  <Step title="Diff">
    Split each disassembly into functions, drop the byte-offset column, and compare
    added, removed and changed functions plus the qstr and constant tables. 74 modules are
    identical, 20 changed and 2 were removed. See
    [What changed in 5.0.3](/firmware/changes-5.0.3).
  </Step>
</Steps>

<Warning>
  **Core-qstr off-by-one in the first v5.0.2 pass.** The reconstructor looked up core qstr
  *g* at `qstrs + 4*(g-1)`. The core pool's `qstrs[0]` is the null qstr, so this read the
  name of qstr *g − 1*, the alphabetical predecessor. Every core name in the old disassembly
  was shifted by one: `keys` stood for `len`, `find` for `format`, `sep` for `set`,
  `values` for `write`, `from_bytes` for `get`, `step`/`split` for `stop`/`start`, and
  `Ellipsis` for `Exception`. Frozen and const-pool names were unaffected, and so were all
  constants, struct formats and control flow. The protocol findings on these pages were
  re-checked against the corrected disassembly.
</Warning>

## Artifacts

The reverse-engineering scratch (rebuilt ELF, extracted symbol/string tables, carved
segments) is kept out of git via `.gitignore`. Regenerate it from the image with the
steps above; this `docs/` tree is the durable record.