The walkthrough

From an empty container to a working screen→camera link, in the order that fails fastest. Each step says what to do, what you should see, and what to do when it does not.

The pipeline, end to end

greener = further along on this box

loading…

How the pieces fit. A frame is a colour-and-pattern grid carrying a Reed–Solomon protected payload, plus a 1-cell calibration ring so the receiver can colour-calibrate itself. A detector finds the four grid corners in a photo; the shared demodulator + FEC then reads the bits. This suite trains that detector and then uses it: dataset → model → S2C.

1 · Environment

Training needs torch and ultralytics; the rest of the stack is NumPy/OpenCV/FastAPI. The container installs the CPU build of torch on purpose — an 8 GB VPS will not fit the CUDA stack, and this workload trains fine on CPU at small model sizes.

checking…

Rule of thumb on 4 vCPU: yolo26n-pose @ 448 px, batch 8 ≈ 1.2 s per image-epoch. 300 images × 40 epochs ≈ 25 min.

2 · Dataset

Two complementary sources. Synthetic data covers the palette/grid space cheaply; real photographs cover what simulation gets wrong (moiré, screen glare, phone colour pipelines, awkward angles).

2a · Synthetic (start here)

Renders frames across every modulation and grid, pushes them through the camera-channel model (blur, noise, gamma, vignetting, highlights, JPEG), and writes YOLO-pose labels.

python -m optical_modem.suite.tasks synth \
  --out /data/datasets/synth-001 --train 300 --val 60 --seed 1

Or press Generate on the Dataset page — same command, as a background job.

2b · Real captures (the capture station)

Open Display on the laptop, start a session, then open Photograph on the phone and enter the 6-character code (or scan the QR). Both advance after every shot. Each photo is stored with a sidecar describing the frame, and auto-labelled with the classical localiser — or with the active detector once you have one, which is how the loop improves with use.

python -m optical_modem.suite.tasks captures \
  --data-dir /data --out /data/datasets/real-001 --verified-only

100 photos of a screen at different angles/lighting is a meaningful dataset. The labeller verifies ~70–80 % of handheld shots; --verified-only keeps those.

2c · Merge into one training set

python -m optical_modem.suite.tasks merge --out /data/datasets/train-001 \
  --source /data/datasets/synth-001 --source /data/datasets/real-001

Check the labels before training: the Dataset page shows annotated previews. A wrong keypoint order is invisible in the logs and ruining in the model.

3 · Train

Fine-tune a pose model. The dataset's data.yaml already carries kpt_shape: [4, 3] and flip_idx: [1, 0, 3, 2], so horizontal flips keep the keypoint semantics (0=TL, 1=TR, 2=BR, 3=BL).

python -m optical_modem.data.train_yolo \
  --data /data/datasets/train-001/data.yaml \
  --model yolo26n-pose.pt --epochs 40 --imgsz 448 --batch 8 \
  --device cpu --project /data/runs --name run-001

The run writes weights/best.pt and model_card.json (keypoint order, geometry, metrics) next to it. Watch the log on the Train page: the number that matters is pose mAP50-95, not box mAP.

Expectations. A first run on 300 synthetic images reaches box mAP50-95 ≈ 0.7–0.8 within 20 epochs. If it plateaus near zero, the labels are wrong or the dataset path in data.yaml does not resolve inside the container — check the Dataset page's preview.

4 · Verify

Two different questions, both cheap:

Detection quality — mean corner error as a fraction of the frame side, detection rate, end-to-end decode rate, ms/frame. The Verify page runs scripts/evaluate.py.
Does the byte path work — the loopback check in S2C renders a frame, simulates a camera, decodes it with the active model and compares the payload byte-for-byte.

Target: corner error well under 0.5 cell (at 448 px, a 48-cell frame with a 0.4-cell error is still decodable because the ring re-aligns it). Decode rate is the number to quote.

5 · Promote the model

"Promote" copies a run's best.pt into the model registry and makes it active. The active model is what the S2C receiver and the capture station's auto-labeller use, so this is the switch between experiments and production.

# via the UI: Train → promote,   or
curl -X POST -b cookie.txt -H 'content-type: application/json' \
  -d '{"run":"run-001"}' https://YOUR-DOMAIN/api/suite/models/promote

Registry layout: /data/models/<name>/weights/best.pt plus active.json. Dropping a hand-trained best.pt in there and activating it from Models works too.

6 · Use it: S2C

last

On the laptop, open S2C, type a message (or pick a file) and press Start transfer: frames play full-screen at the chosen rate with repeats. On the phone, open the same page, choose the transfer, press Receive and let it shoot continuously — it decodes each still, and the transfer completes when every chunk has arrived, in any order.

Throughput is roughly payload_per_frame × fps. A 1280×720 rgb8/rs32 frame carries ~700 B; at 3 fps that is ~2 kB/s. Raise fps for a short range / still phone, lower it (or raise repeats) for a shaky hand.

Keep it organised: projects

open projects →

Every dataset, run, model and transfer can belong to a project — created in development and flipped to finished when it works. The Projects page shows what each one owns and the live event feed (job started/finished, dataset ready, model promoted).

Pick the project once on the Dataset or Train page and it sticks for the session; the API accepts the same field ("project": "v0.3-hd"). Records live in the database (Postgres in Docker, SQLite without it) on the /data side of the container — so "what did this box do last week?" survives a rebuild. Job logs and the photo files stay on the volume; the database holds the metadata, never the images.

Optional finder pattern

On the S2C page, finder adds a high-contrast shape to the inner corners of every frame: ring, dice, bullseye, checker, diamond, cross, x, bracket, triangle, solid — or none.

It never overwrites data. Its symbols are removed from the header and payload before either is filled; the layout refuses to build if the regions would overlap.
It costs capacity, and the UI says how much next to the selector (≈10 % for the 5×5 shapes, ≈6 % for the 4×4 ones). The receiver reports a finder_score per decode, so you can see whether it is helping.
Use it when the phone is far away, the screen is dim, or the frame is rotated — the anchor makes scale/rotation estimation cheaper and more stable. Leave it none for maximum throughput at close range.

7 · When something is off

Symptom	Usual cause / fix
Detector finds nothing	Frames too small in the photo (fill more of the view), too much blur, or the model was trained on a different grid size than the one being displayed. Try classical mode in the S2C receive panel to see whether the problem is localisation or decoding.
Corners found, decode fails	Wrong modulation/ECC in the transfer settings vs. what the receiver expects, or the screen is scaling the frame (use 1:1 / native resolution on the display page).
Training loss flat	Labels missing for some images (the merge skips unlabelled pairs), `data.yaml` pointing at a path that does not exist in the container, or `kpt_shape` mismatch.
OOM / killed during training	Lower `--batch` and/or `--imgsz`; the compose file caps the container at 2 GB, raise it to 4 GB if the box has 8 GB.
Phone camera never opens	The page must be served over HTTPS (Coolify gives you that). On plain http the capture page falls back to the system camera app; the S2C receiver does the same.

Command-line equivalents

Everything the UI does is also a command, so you can script or reproduce it outside the browser:

# what is in this container
python -m optical_modem.suite.tasks info --data-dir /data
python -m optical_modem info
# dataset -> train -> verify -> promote
python -m optical_modem.suite.tasks synth --out /data/datasets/s1 --train 300 --val 60
python -m optical_modem.suite.tasks merge --out /data/datasets/t --source /data/datasets/s1
python -m optical_modem.data.train_yolo --data /data/datasets/t/data.yaml --epochs 40
python scripts/evaluate.py --weights /data/runs/run-001/weights/best.pt --out r.json
# S2C from a terminal (no browser)
python -m optical_modem send --file README.md --out frames/
python -m optical_modem receive photo.jpg