feat: initialize ingest-api with Koa and telemetry handling
- Create ingest-api with Koa framework for handling device telemetry. - Implement telemetry ingestion with MongoDB, Redis, and PostgreSQL. - Add API endpoints for telemetry and health checks. - Set up testing with Vitest for unit and end-to-end tests. - Configure Prettier and ESLint for code formatting and linting. - Create Docker Compose setup for PostgreSQL, MongoDB, and Redis services. - Establish contracts package for shared Zod schemas and TypeScript types. - Implement React frontend with device status and greeting functionalities. - Add VS Code setup documentation for onboarding developers.
This commit is contained in:
marco.chan committed
1 parent
534fa6dc51
commit
7923ba1b8d
70 files changed
+3085
-3434
No files matched your search
@@ -0,0 +1,93 @@
|
||||
# Architecture
|
||||
|
||||
Copy this layout for later device + dashboard projects.
|
||||
|
||||
## Lanes
|
||||
|
||||
- **Device lane:** `apps/ingest-api` (Koa) — firmware / gateways POST JSON.
|
||||
- **User lane:** `apps/crud-api` (Express) — React, humans, billing later.
|
||||
- **Shared types:** `packages/contracts` (Zod only). No Koa, Express, Mongo, or trest.
|
||||
|
||||
Frontend people can run `yarn workspaces focus web` and do not need the private trest registry.
|
||||
|
||||
## Data flow
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
subgraph devices [Device lane]
|
||||
D[Device / demo ping]
|
||||
I[Koa ingest-api :3001]
|
||||
D -->|POST /telemetry JSON| I
|
||||
end
|
||||
|
||||
subgraph stores [Stores]
|
||||
M[(MongoDB raw telemetry)]
|
||||
R[(Redis hot key TTL)]
|
||||
P[(PostgreSQL device_current_state)]
|
||||
A[(PostgreSQL analytics — not in this repo)]
|
||||
end
|
||||
|
||||
subgraph humans [User lane]
|
||||
W[React web :5173]
|
||||
C[Express crud-api :3000]
|
||||
W -->|axios GET /greeting GET /devices/:id/status| C
|
||||
end
|
||||
|
||||
I -->|insert event| M
|
||||
I -->|SET device:id EX 15s| R
|
||||
I -->|UPSERT current row| P
|
||||
M -.->|ETL cron later| A
|
||||
C -->|SELECT snapshot| P
|
||||
C -->|GET device:id| R
|
||||
```
|
||||
|
||||
Near-real-time product path (Postgres + Express is enough at this scale):
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant D as Device
|
||||
participant I as ingest-api Koa
|
||||
participant PG as Postgres
|
||||
participant C as crud-api Express
|
||||
participant W as React
|
||||
D->>I: POST /telemetry
|
||||
I->>PG: UPSERT device_current_state
|
||||
W->>C: GET /devices/dev-1/status poll 1s
|
||||
C->>PG: SELECT by device_id
|
||||
C->>W: lastSeen payload source postgres
|
||||
```
|
||||
|
||||
Redis is compared on the Devices page. It is not required for the UI to look live.
|
||||
|
||||
## Demo dual-write (do not copy blindly)
|
||||
|
||||
Ingest writes Mongo first, then Redis and Postgres best-effort. A 200 can still mean Redis or Postgres missed. Production should derive those from a queue. This handler is a teaching shortcut.
|
||||
|
||||
Redis TTL expiry means “no recent ping.” Redis restart looks like everyone is offline. Postgres still has `device_current_state`.
|
||||
|
||||
Ingest may only upsert that snapshot table. CRUD owns greeting / future users and orgs.
|
||||
|
||||
Hourly Mongo → Postgres analytics is the dotted ETL box. Not implemented.
|
||||
|
||||
## Ports
|
||||
|
||||
| Service | Port |
|
||||
| ---------- | ----- |
|
||||
| crud-api | 3000 |
|
||||
| ingest-api | 3001 |
|
||||
| web | 5173 |
|
||||
| postgres | 5432 |
|
||||
| mongo | 27017 |
|
||||
| redis | 6379 |
|
||||
|
||||
## Run
|
||||
|
||||
```bash
|
||||
docker compose up -d
|
||||
yarn install
|
||||
yarn dev:crud
|
||||
yarn dev:ingest
|
||||
yarn dev:web
|
||||
```
|
||||
|
||||
Open `http://127.0.0.1:5173/devices`, click **Ping dev-1**, watch Postgres and Redis columns. Wait 15s without pinging: Redis goes offline, Postgres keeps the last row.
|
||||
@@ -1,20 +1,18 @@
|
||||
# From Express + SQL to Monorepo with Nest Fastify, Vite React, and trest
|
||||
# From Express + SQL to Monorepo with Koa, Vite React, and trest
|
||||
|
||||
This guide is for developers who already know Express APIs and SQL, and are new to Yarn workspaces, NestJS, and private package usage.
|
||||
This guide is for developers who already know Express APIs and SQL, and are new to Yarn workspaces, Koa, and private package usage.
|
||||
|
||||
## 1) What You Are Building
|
||||
|
||||
Target architecture:
|
||||
|
||||
- Backend API: NestJS on Fastify
|
||||
- Backend API: Koa + trest
|
||||
- Frontend: Vite + React + Ant Design
|
||||
- Shared/private integration: trest in an isolated adapter package
|
||||
- Monorepo manager: Yarn workspaces
|
||||
|
||||
Why this shape:
|
||||
|
||||
- Keep your main backend on a scalable Fastify stack.
|
||||
- Evaluate trest without coupling everything to Koa ecosystem packages.
|
||||
- Keep the HTTP API on Koa so it matches trest's router.
|
||||
- Keep one repo and one lockfile for easier onboarding.
|
||||
|
||||
## 2) Mental Map (Old to New)
|
||||
@@ -159,32 +157,22 @@ Checkpoint:
|
||||
|
||||
- Vite app runs and Ant Design components render.
|
||||
|
||||
### Step F: Introduce trest Safely
|
||||
### Step F: Introduce trest on the API
|
||||
|
||||
Recommended approach:
|
||||
|
||||
- Keep trest usage in a dedicated package, for example packages/trest-adapter.
|
||||
- Expose only framework-agnostic functions from that adapter.
|
||||
- Avoid leaking Koa-specific types into app-wide contracts.
|
||||
|
||||
Starter commands:
|
||||
Install trest and its Koa peers in `apps/api`:
|
||||
|
||||
```bash
|
||||
mkdir -p packages/trest-adapter
|
||||
cd packages/trest-adapter
|
||||
yarn init -y
|
||||
yarn add @jig-software/trest koa @koa/router koa-body path-to-regexp zod bson
|
||||
yarn workspace api add @jig-software/trest koa @koa/router koa-body path-to-regexp zod bson
|
||||
```
|
||||
|
||||
Checkpoint:
|
||||
|
||||
- Importing trest from adapter works.
|
||||
- `GET /greeting?name=Marco` returns a JSON greeting from the Koa app.
|
||||
|
||||
### Step G: Wire API and Web
|
||||
|
||||
- Add one simple API endpoint in Nest.
|
||||
- Call endpoint from React page.
|
||||
- Render response using an Ant Design component.
|
||||
- Call `/greeting` from a React page.
|
||||
- Render the response using an Ant Design component.
|
||||
|
||||
Checkpoint:
|
||||
|
||||
@@ -197,8 +185,6 @@ marco-trest-test/
|
||||
apps/
|
||||
api/
|
||||
web/
|
||||
packages/
|
||||
trest-adapter/
|
||||
docs/
|
||||
package.json
|
||||
yarn.lock
|
||||
@@ -209,11 +195,10 @@ marco-trest-test/
|
||||
|
||||
Use these rules to avoid confusion:
|
||||
|
||||
1. Put backend runtime dependencies in apps/api.
|
||||
1. Put Koa, trest, and other backend runtime dependencies in apps/api.
|
||||
2. Put frontend runtime dependencies in apps/web.
|
||||
3. Put trest boundary dependencies in packages/trest-adapter.
|
||||
4. Keep root dependencies minimal and mostly tooling.
|
||||
5. Run installs from root unless targeting one workspace intentionally.
|
||||
3. Keep root dependencies minimal and mostly tooling.
|
||||
4. Run installs from root unless targeting one workspace intentionally.
|
||||
|
||||
## 8) Troubleshooting You Are Likely to Hit
|
||||
|
||||
|
||||
@@ -0,0 +1,149 @@
|
||||
# VS Code Setup Guide
|
||||
|
||||
This guide is for developers onboarding to this monorepo with Yarn workspaces, Nest API, and Vite web.
|
||||
|
||||
## Goals
|
||||
|
||||
- Open the workspace with correct TypeScript resolution.
|
||||
- Run API and web locally from VS Code.
|
||||
- Use one-click debug profiles.
|
||||
- Fix common red-squiggle and startup issues quickly.
|
||||
|
||||
## 1) Prerequisites
|
||||
|
||||
- Node.js installed
|
||||
- Corepack enabled
|
||||
- Yarn available
|
||||
|
||||
Quick checks:
|
||||
|
||||
```bash
|
||||
node -v
|
||||
corepack --version
|
||||
yarn -v
|
||||
```
|
||||
|
||||
## 2) Open the Correct Folder
|
||||
|
||||
Open the repository root folder in VS Code, not a subfolder.
|
||||
|
||||
Expected top-level folders:
|
||||
|
||||
- apps
|
||||
- packages
|
||||
- docs
|
||||
- .vscode
|
||||
|
||||
## 3) Install Dependencies from Root
|
||||
|
||||
Run in the root folder:
|
||||
|
||||
```bash
|
||||
yarn install
|
||||
```
|
||||
|
||||
Why: workspace metadata and PnP dependency graph are generated at root.
|
||||
|
||||
## 4) Configure Yarn SDK for VS Code
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
yarn dlx @yarnpkg/sdks vscode
|
||||
```
|
||||
|
||||
Then in VS Code:
|
||||
|
||||
1. Open Command Palette
|
||||
2. Run TypeScript: Select TypeScript Version
|
||||
3. Choose Use Workspace Version
|
||||
4. Run TypeScript: Restart TS Server
|
||||
|
||||
This fixes most false errors like cannot find module react or asset files.
|
||||
|
||||
## 5) Verify Workspace Settings
|
||||
|
||||
The workspace should contain these settings:
|
||||
|
||||
- typescript.tsdk points to .yarn/sdks/typescript/lib
|
||||
- typescript.enablePromptUseWorkspaceTsdk is true
|
||||
|
||||
If missing, rerun the SDK command in section 4.
|
||||
|
||||
## 6) Start and Debug from VS Code
|
||||
|
||||
Use Run and Debug with these profiles:
|
||||
|
||||
- API: Attach Nest (9229)
|
||||
- Web: Chrome (Vite 5173)
|
||||
- Full Stack Debug
|
||||
|
||||
They rely on background tasks:
|
||||
|
||||
- api:debug
|
||||
- web:dev
|
||||
|
||||
## 7) Manual Run Commands (Fallback)
|
||||
|
||||
From repo root:
|
||||
|
||||
```bash
|
||||
yarn workspace api start:dev
|
||||
yarn workspace web dev --host 0.0.0.0 --port 5173
|
||||
```
|
||||
|
||||
## 8) Common Issues and Fixes
|
||||
|
||||
### A) Package for api not found
|
||||
|
||||
Cause: workspace name changed but graph not refreshed.
|
||||
|
||||
Fix:
|
||||
|
||||
```bash
|
||||
yarn install
|
||||
yarn workspaces list --json
|
||||
```
|
||||
|
||||
### B) Cannot find module react, css, svg in editor
|
||||
|
||||
Cause: VS Code TypeScript server not using workspace SDK.
|
||||
|
||||
Fix:
|
||||
|
||||
1. Run yarn dlx @yarnpkg/sdks vscode
|
||||
2. Select Use Workspace Version
|
||||
3. Restart TS Server
|
||||
4. Reload window if needed
|
||||
|
||||
### C) Peer dependency warnings after installing trest stack
|
||||
|
||||
Cause: trest and Koa peers installed in wrong workspace, usually web.
|
||||
|
||||
Fix:
|
||||
|
||||
- Keep trest and Koa in apps/api
|
||||
- Keep web dependencies frontend-only
|
||||
|
||||
### D) Nest scaffold fails with schematics resolution
|
||||
|
||||
Use non-interactive fallback:
|
||||
|
||||
```bash
|
||||
CI=1 npx -y @nestjs/cli new apps/api --skip-git --package-manager yarn --strict
|
||||
```
|
||||
|
||||
## 9) Workspace Dependency Rules
|
||||
|
||||
- apps/api contains Koa, trest, and other backend runtime dependencies.
|
||||
- apps/web contains frontend runtime dependencies.
|
||||
- root should mostly contain workspace-level tooling.
|
||||
|
||||
## 10) Day-1 Validation Checklist
|
||||
|
||||
- yarn install completes from root
|
||||
- yarn workspaces list shows root, api, web
|
||||
- API starts locally
|
||||
- Web starts locally
|
||||
- Full Stack Debug launches successfully
|
||||
- No persistent false TypeScript module errors in editor
|
||||
@@ -0,0 +1,33 @@
|
||||
# Fresh Grad Checklist for VS Code Setup
|
||||
|
||||
Use this in order. Do not skip steps.
|
||||
|
||||
## 10-Minute Quick Path
|
||||
|
||||
1. Open repo root in VS Code
|
||||
2. Run yarn install at root
|
||||
3. Run yarn dlx @yarnpkg/sdks vscode
|
||||
4. Select TypeScript workspace version
|
||||
5. Restart TypeScript server
|
||||
6. Run API debug profile
|
||||
7. Run Web debug profile
|
||||
8. Open Full Stack Debug
|
||||
|
||||
## If Something Breaks
|
||||
|
||||
1. Rerun yarn install
|
||||
2. Check workspace names with yarn workspaces list --json
|
||||
3. Ensure api name in apps/api/package.json is correct
|
||||
4. Confirm .vscode settings include typescript.tsdk
|
||||
5. Reload VS Code window
|
||||
|
||||
## What to Ask for Help With
|
||||
|
||||
- Private registry token access issues
|
||||
- Workspace naming mismatches
|
||||
- Dependency added to wrong workspace
|
||||
- Debug profile not attaching on port 9229
|
||||
|
||||
## Golden Rule
|
||||
|
||||
When in doubt, run commands from repo root first, then target a specific workspace.
|
||||
Reference in new issue
Block a user