tinylink/docs/user_manual.md

# TinyLink User Manual

Welcome to TinyLink: the friendly lab middleware that never sleeps, rarely panics, and keeps your results moving even when downstream systems are having a bad day.

## What TinyLink Can Do

TinyLink sits between laboratory instruments and CLQMS, then handles the heavy lifting:

- Receives messages from instruments over three connector types:
  - `http-json` (HTTP endpoint)
  - `hl7-tcp` (TCP socket)
  - `astm-serial` (physical serial port/COM)
- Wraps incoming payloads as raw content, then normalizes them into one canonical JSON format via the translator pipeline.
- Stores raw and normalized data in SQLite for durability.
- Deduplicates repeated payloads using a hash key.
- Sends results to CLQMS with automatic retry and backoff.
- Moves non-deliverable payloads to dead letter for review.
- Exposes operational endpoints for health and metrics.

Think of it as a reliable translator + traffic controller for instrument data.

## Default Ports and Endpoints

By default (from `config/app.yaml`):

- Instrument config + health + metrics API: `4001` (from `host.port`)
- Instrument connectors are configured per instrument entity (`inst1`, `inst2`, ...)

Useful endpoints:

- `GET http://localhost:4001/health`
- `GET http://localhost:4001/health/ready`
- `GET http://localhost:4001/metrics`
- `GET http://localhost:4001/dashboard`
- `POST http://localhost:3001/messages` (for JSON instrument payloads)

Architecture reference:

- `docs/separated_architecture.md`

## Add a New Instrument (New "Inst")

TinyLink now uses a single file-based configuration. There is no `POST /instruments` write flow.

To add an instrument, keep one config file in:

```text
core/
  config/
    app.yaml
```

### Step 1: Edit `app.yaml`

Example:

```yaml
host:
  url: http://localhost:4000/api/results
  apikey: ""
  port: 4001

inst1:
  enabled: true
  connector:
    type: serial
    port: COM1
    baudRate: 9600
  config:
    location: lab-a
  translator:
    engine: overrides
    forceInstrumentId: true
    meta:
      profile: astm-default
```

### Optional: Use per-instrument `.map` translator files

If you want a shorter per-instrument translator config, set `translator` to the map name.

Example: `translator: inst1` loads `config/inst1.map`.

`config/app.yaml` example:

```yaml
inst1:
  enabled: true
  connector:
    type: serial
    port: COM1
  translator: inst1
```

`config/inst1.map` example:

```text
# HEADER
<VT>H|\^&|||WST^P1|||||{instrument_id}^System1||P|1|{sample_id}<CR>

SAMPLE_ID = O[3]
CHEMNUM = R[3]
RESULT = R[4]

# ORDER
O|1|{SAMPLE_ID}||{order_tests}||||||{specimen_type}||||{tube_type}||||||||||O|<CR>

# RESULT
<<
@for R
R|1|^^^{CHEMNUM}|{RESULT}<CR>
@end
>>

# TERMINATOR
L|1|N<FS><CR>
```

Notes:

- Preferred style is section-based: `# MESSAGE_NAME` plus one body line.
- Use `<<` and `>>` for multiline section bodies.
- `KEY = value` remains supported for backwards compatibility.
- `ALIAS = R[3]` (or `O[11]`, `P[2]`, `OBX[5.2]`) defines readable field aliases.
- You can reference selectors directly as placeholders: `{R[3]}`.
- Control tokens in message bodies are decoded: `<VT>`, `<FS>`, `<STX>`, `<ETX>`, `<CR>`, `<LF>`.
- `@for R ... @end` repeats a multiline block for each incoming record of type `R`.
- If `records` are not pre-parsed, TinyLink auto-extracts records from `raw_payload`/`meta.raw_payload` for selector usage.
- Placeholders use `{name}` and missing values default to empty strings.
- `translator.messages` controls output order. If omitted, messages render in map file order.

What it does:

- `host` contains upstream endpoint and API settings (`url`, `apikey`, `port`).
- Every top-level entity other than `host` is an instrument (`inst1`, `inst2`, ...).
- `connector` is embedded per instrument, so each instrument has its own connector type/settings.
- `translator` stays embedded per instrument, so each instrument can define mapping rules and metadata.

### Step 2: Restart TinyLink

TinyLink validates instrument files before startup. Restart after adding or editing files.
Use this preflight command before `npm start`:

```powershell
npm run instrument:check
```

### Step 3: Verify instrument is loaded

You can still use read-only endpoints:

```powershell
curl http://localhost:4001/instruments
curl http://localhost:4001/instruments/inst1
```

If your instrument does not appear, run `npm run instrument:check` and fix reported errors.

### Matching rules (important)

TinyLink picks one instrument by matching connector + `instruments[].match` rules.

- 0 matches: payload is dropped (`no matching instrument config`).
- More than 1 match: payload is dropped (`ambiguous instrument match`).
- Exactly 1 match: translator runs and message continues through queue + delivery.

TinyLink is strict because your audit trail deserves peace and quiet.

## Canonical Payload Shape

After wrapping and normalization, TinyLink expects this shape.

Required fields:

- `instrument_id`
- `sample_id`
- `result_time` (ISO timestamp)
- `results[]` with at least one item containing:
  - `test_code`
  - `value`

Optional fields:

- `unit`, `flag`, `patient_id`, `operator_id`, `meta`

Example:

```json
{
  "instrument_id": "SYSMEX_XN1000",
  "sample_id": "SMP-20260326-001",
  "result_time": "2026-03-26T10:20:00Z",
  "results": [
    {
      "test_code": "WBC",
      "value": "8.2",
      "unit": "10^3/uL",
      "flag": "N"
    }
  ],
  "meta": {
    "source_protocol": "HL7",
    "connector": "hl7-tcp"
  }
}
```

## Delivery and Retry Behavior

TinyLink does not give up easily:

- Retries transient failures (timeouts, DNS/connection issues, HTTP 5xx).
- Uses backoff schedule: `30s -> 2m -> 10m -> 30m -> 2h -> 6h`.
- Maximum attempts: `10`.
- HTTP `400` and `422` are treated as non-retriable and moved to dead letter immediately.

## Health, Readiness, and Metrics

Use these to check system status:

- `GET /health`: connector status + queue counts (`pending`, `retrying`, `deadLetters`).
- `GET /health/ready`: confirms SQLite readiness.
- `GET /metrics`: Prometheus-style metrics, including attempts, latency, and success timestamp.

## Daily Operations (Quick Checklist)

- Start app: `npm start`
- Validate instrument files: `npm run instrument:check`
- Run migrations manually: `npm run migrate`
- Backup DB: `npm run maintenance -- backup`
- Vacuum DB: `npm run maintenance -- vacuum`
- Prune old delivery logs: `npm run maintenance -- prune --days=30`

## Troubleshooting New Instrument Onboarding

### Instrument returns `404`

- Confirm exact `instrument_id` spelling and casing.
- Verify the instrument exists in `config/app.yaml` as its own top-level key (`inst1`, `inst2`, ...).
- Verify that instrument has `connector.type`, connector settings, and `translator`.

### Data is not flowing

- Confirm instrument has `enabled: true`.
- Confirm connector matches incoming protocol.
- Confirm match rules fit connector metadata (`localPort`, `remoteAddress`, `remotePort`, or `comPort`).
- Check `/health` and `/health/ready`.

### Backlog or dead letters are growing

- Check `/metrics` for queue and delivery trends.
- Validate CLQMS URL/token/timeouts and downstream availability.

## Final Tip

If you are adding many instruments, standardize your naming (for example: `LAB1_XN1000`, `LAB1_COBAS_E411`) and keep connector mappings documented. Future-you will send present-you a thank-you card.