docs: add milestones, file targets, acceptance gates to all 10 ROADMAP priorities

- Each priority now has: milestones (M1-MN), concrete file target table
  (CREATE/MODIFY/DELETE per file), and binary acceptance gates
- Created docs/plans/0001-step-to-usd-implementation.md: full execution
  checklist for USD pipeline (Priorities 2, 4, 5) with:
  - Phase 1: dual-write USD beside GLB
  - Phase 2: partKey + three-layer material assignment model
  - Phase 3: seam/sharp payload to USD mesh primvars (index-space)
  - Phase 4: Blender render from USD
  - Phase 5: frontend ThreeDViewer partKey migration
  - Open questions decision table
  - Non-regression checklist

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
2026-03-11 15:10:45 +01:00
parent 208370628e
commit 5d912594dd
2 changed files with 609 additions and 154 deletions
+283 -154
View File
@@ -23,229 +23,357 @@
---
## 🗺 Open Work — Prioritized
## 🗺 Open Work — Reconciled Priorities
### Priority 1 — Tessellation Quality (Blocking visual quality)
This roadmap now treats the USD refactor as an implementation workstream, not as a blocked strategic idea.
**Goal:** Eliminate fan triangles and faceting on cylindrical surfaces (rings, bearings).
**Plan:** `plan.md` (6 tasks, GMSH Frontal-Delaunay as BRepMesh replacement)
The key architectural clarification from [docs/rfcs/0001-step-to-usd-workflow.md](/home/hartmut/Documents/Copilot/schaefflerautomat/docs/rfcs/0001-step-to-usd-workflow.md#L139) is:
| Task | File | What |
|---|---|---|
| T1 | `render-worker/Dockerfile` | `pip install gmsh>=4.15.0` |
| T2 | `export_step_to_gltf.py` | `--tessellation_engine occ\|gmsh` CLI arg |
| T3 | `export_step_to_gltf.py` | `_tessellate_with_gmsh()`: BREP → GMSH → Poly_Triangulation write-back |
| T4 | `admin.py` | `tessellation_engine` setting in SETTINGS_DEFAULTS + SettingsOut |
| T5 | `export_glb.py` | Read setting, pass `--tessellation_engine` to CLI |
| T6 | `Admin.tsx` | Dropdown: OCC vs. GMSH with description |
- USD becomes the canonical persisted scene asset
- the browser does not need to render USD directly
- the current 3D viewer workflow is preserved through a derived preview asset plus canonical `partKey`
**Risk:** GMSH Surface-Tag ↔ OCC Face mapping must be verified experimentally.
**Status:** Not started. `/implement` to begin.
That removes the old assumption that USD work must wait for a Three.js USD loader.
---
### Priority 1 — Pipeline Cleanup Foundation
### Priority 2 — Dead Code Deletion (Quick wins, no risk)
**Goal:** Reduce refactor risk by simplifying the current pipeline before introducing canonical scene concepts.
**Goal:** Remove code that is provably dead and confuses future readers.
This priority combines dead-code deletion and task decomposition because both are prerequisites for a controlled cut-over to USD.
| Item | Location | Why Dead |
|---|---|---|
| Pillow overlay block | `blender_render.py` lines 798851 | `transparent_bg=True` always, `else:` branch never runs |
| STL workflow | `admin.py`, `cad.py`, multiple tasks | Pipeline is GLB-only; `stl_quality`, `VALID_STL_QUALITIES`, `stl_size_bytes` all orphaned |
| `render_order_line_task` in step_tasks | `step_tasks.py` lines 7051050 | Duplicates `rendering/tasks.render_order_line_still_task` |
| `blender-renderer/` directory | repo root | Removed from docker-compose.yml already |
| `threejs-renderer/` directory | repo root | Migration 033 removed it from services |
| `flamenco/` directory | repo root | Migration 032 removed Flamenco |
| `renderproblems_tmp/` directory | repo root | Temp debugging screenshots, not code |
**Milestones:**
- M1: Dead code deleted — Pillow block, STL settings, orphaned directories
- M2: `step_tasks.py` decomposed into `backend/app/tasks/pipeline/` submodules
- M3: `blender_render.py` decomposed into `render-worker/scripts/_blender_*.py` submodules
**Estimated effort per agent:** 1 session
**Can run in parallel with Priority 3.**
**File targets:**
---
| Action | Path |
|---|---|
| DELETE lines 798851 | `render-worker/scripts/blender_render.py` (Pillow overlay, `else:` branch never runs) |
| DELETE | `blender-renderer/` directory |
| DELETE | `threejs-renderer/` directory |
| DELETE | `flamenco/` directory |
| DELETE | `renderproblems_tmp/` |
| DELETE lines 7051050 | `backend/app/tasks/step_tasks.py` (`render_order_line_task` — duplicates `rendering/tasks`) |
| REMOVE settings | `admin.py`: `VALID_STL_QUALITIES`, `stl_quality`, `generate-missing-stls` endpoint |
| REMOVE endpoint | `cad.py`: `POST /cad/{id}/generate-stl/{quality}` |
| CREATE | `backend/app/tasks/pipeline/extract.py` — metadata extraction (OCC parsing, < 2s) |
| CREATE | `backend/app/tasks/pipeline/thumbnail.py` — Blender thumbnail render |
| CREATE | `backend/app/tasks/pipeline/stills.py` — still render task |
| CREATE | `backend/app/tasks/pipeline/turntable.py` — turntable render task |
| THIN (< 80 lines) | `backend/app/tasks/step_tasks.py` — dispatch only |
| CREATE | `render-worker/scripts/_blender_gpu.py` |
| CREATE | `render-worker/scripts/_blender_import.py` |
| CREATE | `render-worker/scripts/_blender_materials.py` |
| CREATE | `render-worker/scripts/_blender_camera.py` |
| CREATE | `render-worker/scripts/_blender_scene.py` |
| THIN (< 80 lines) | `render-worker/scripts/blender_render.py` — entry point only |
### Priority 3 — Celery Task Decomposition (Maintainability, parallel-safe)
**Acceptance gates:**
- `grep -r "VALID_STL_QUALITIES\|stl_quality\|from PIL\|from Pillow" backend/ render-worker/` → 0 matches
- `wc -l backend/app/tasks/step_tasks.py` → < 100 lines
- `wc -l render-worker/scripts/blender_render.py` → < 80 lines
- Upload `81113-l_cut.stp`, trigger thumbnail → still renders correctly (no regression)
- `ls blender-renderer/ threejs-renderer/ flamenco/` → all return "No such file"
**Goal:** Split `step_tasks.py` (1,170 lines, 8+ pipeline steps) into focused modules.
### Priority 2 — USD Foundation Without Viewer Regression
**Target structure:**
```
backend/app/tasks/
├── step_tasks.py → keep only: process_step_file (thin dispatch)
├── pipeline/
│ ├── extract.py → extract_cad_metadata (OCC parsing, 0.1s)
│ ├── thumbnail.py → render_step_thumbnail (Blender call)
│ ├── stills.py → render_order_line_still_task (Cycles still renders)
│ └── turntable.py → render_order_line_turntable_task
```
**Goal:** Introduce canonical part identity and the three-layer material assignment model while keeping the current GLB-based browser UX working end-to-end.
Also: split `render-worker/scripts/blender_render.py` (853 lines) into sub-modules:
```
render-worker/scripts/
├── blender_render.py → entry point only (~80 lines)
├── _blender_gpu.py → GPU probe + activation
├── _blender_import.py → GLB import, rotation, smooth shading
├── _blender_materials.py → material library application + fallback
├── _blender_camera.py → auto camera from bbox, clip planes
└── _blender_scene.py → scene setup (Mode A vs Mode B)
```
**Milestones:**
- M1: `export_step_to_usd.py` produces valid USD with part hierarchy and `schaeffler:partKey` on every prim
- M2: `usd_master` MediaAsset type exists in DB and is stored after each export
- M3: `GET /api/cad/{id}/scene-manifest` returns partKey list with effective assignments
- M4: `PUT /api/cad/{id}/part-materials` accepts `{partKey → materialName}` map and persists it
- M5: Browser ThreeDViewer saves material overrides keyed by `partKey`, survives page reload
**Estimated effort:** 2 sessions
**Depends on:** Priority 2 (delete duplicate task first)
**File targets:**
---
| Action | Path |
|---|---|
| CREATE | `render-worker/scripts/export_step_to_usd.py` — XCAF → USD, hierarchy + metadata + partKey |
| ADD enum value | `backend/app/domains/media/models.py``usd_master` to `MediaAssetType` |
| CREATE migration | `backend/alembic/versions/060_usd_master_asset_type.py` |
| ADD JSONB columns | `backend/app/domains/products/models.py``CadFile.source_material_assignments`, `resolved_material_assignments`, `manual_material_overrides` |
| CREATE migration | `backend/alembic/versions/061_material_assignment_layers.py` |
| CREATE | `backend/app/services/part_key_service.py``generate_part_key(xcaf_label)`, `build_scene_manifest()` |
| CREATE | `backend/app/domains/products/schemas.py``SceneManifest`, `PartEntry` Pydantic models |
| ADD endpoint | `backend/app/api/routers/cad.py``GET /cad/{id}/scene-manifest` |
| MODIFY endpoint | `backend/app/api/routers/cad.py``GET/PUT /cad/{id}/part-materials` → partKey-keyed |
| ADD task | `backend/app/domains/pipeline/tasks/export_glb.py``generate_usd_master_task` (dual-writes beside GLB) |
| CREATE | `frontend/src/api/sceneManifest.ts``SceneManifest` interface, `fetchSceneManifest()` |
### Priority 4 — Render Job Tracking (Correctness bug)
**Open questions to decide before M1:**
- USD authoring library: `pxr` (OpenUSD full Python SDK) vs. plain USDA text templating vs. `usd-core` pip package
- seam/sharp payload encoding: custom primvars (`primvars:schaeffler:seamEdgeVertexPairs`) or a separate JSON sidecar
**Goal:** Fix the broken render job cancellation (synthetic `render-{line_id}` ID never matches real Celery task ID → `revoke()` is a no-op).
**Acceptance gates:**
- `python3 export_step_to_usd.py --step_path 81113-l_cut.stp` → valid `.usd` file, 25 part prims, each has `schaeffler:partKey` attribute
- `GET /api/cad/{id}/scene-manifest` returns `parts[]` array with `part_key`, `source_name`, `effective_material`, `is_unassigned`
- Click part in ThreeDViewer → assign material → reload page → material still assigned (persisted via `partKey`, not mesh name)
- CAD file with mismatched Excel names: UI shows `unmatched_source_rows` count > 0 and unassigned parts highlighted
- No regression: existing click/select/isolate/ghost/hide still works in browser
**What to build:**
- `RenderJobDocument` Pydantic schema stored in `order_lines.render_job_doc` (JSONB)
- Fields: `celery_task_id`, `state` FSM, `steps[]` with timing, `gpu_info`
- Migration: `alembic revision` — add `render_job_doc JSONB` to `order_lines`
- Update render tasks to write real `self.request.id` into the document
- Fix `orders.py` cancel endpoint to read `celery_task_id` from the document
**References:**
- RFC: `docs/rfcs/0001-step-to-usd-workflow.md`
- Execution checklist: `docs/plans/0001-step-to-usd-implementation.md`
**Files:**
- New: `backend/app/domains/rendering/job_document.py`
- New migration `06x_render_job_document.py`
- Modified: `render_order_line.py`, `render_thumbnail.py`, `orders.py`
### Priority 3 — Tessellation and Topology Quality
**Estimated effort:** 1 session
**Depends on:** Priority 3 (need clean task structure first)
**Goal:** Eliminate fan triangles on cylindrical surfaces (rings, bearings) and produce clean seams for UV unwrap.
---
**Milestones:**
- M1: GMSH 4.15+ installed in render-worker container
- M2: `export_step_to_gltf.py --tessellation_engine gmsh` produces fan-free GLB
- M3: `tessellation_engine` system setting wired through CLI → Admin UI dropdown
### Priority 5 — Structured Logging (Observability)
**File targets:**
**Goal:** Replace inconsistent `logger.info(f"...")` / `emit()` / `log_task_event()` mix with a `PipelineLogger` that writes to Python logging + Redis SSE + DB.
| Action | Path |
|---|---|
| ADD pip install | `render-worker/Dockerfile``gmsh>=4.15.0` |
| ADD arg + function | `render-worker/scripts/export_step_to_gltf.py``--tessellation_engine`, `_tessellate_with_gmsh()` |
| ADD setting | `backend/app/api/routers/admin.py``tessellation_engine` in `SETTINGS_DEFAULTS` + `SettingsOut` |
| MODIFY task | `backend/app/domains/pipeline/tasks/export_glb.py` — read setting, pass to CLI |
| ADD UI | `frontend/src/pages/Admin.tsx` — dropdown: OCC vs. GMSH |
**What to build:**
- `backend/app/core/pipeline_logger.py``PipelineLogger` class
- `step_start(step, context)``[STEP_NAME] starting — context`
- `step_done(step, duration_s, result)``[STEP_NAME] done in 0.34s`
- `step_error(step, error, exc)``[STEP_NAME] ERROR: ...`
- Optional new table: `pipeline_events(task_id, step_name, level, message, duration_s, context JSONB)`
- Migrate all task files to use PipelineLogger
*(Full task breakdown in `plan.md`)*
**Estimated effort:** 1-2 sessions
**Can start immediately, but has broad blast radius (touches all task files).**
**Acceptance gates:**
- `docker compose exec render-worker python3 -c "import gmsh; print(gmsh.__version__)"``4.15.x`
- `python3 export_step_to_gltf.py --step_path 81113-l_cut.stp --tessellation_engine gmsh` → no vertex with valence > 10 at cylinder seam edges (inspect via Blender Mesh Analysis overlay)
- Standard preset output size with GMSH ≤ 2× size with OCC at same deflection
- Sharp edge pairs still extracted and injected into GLB extras after GMSH tessellation
---
**Note:** Better tessellation directly benefits Priority 2 (USD seam/sharp payload) and Priority UV-unwrap work.
### Priority 6Tenant Isolation Completion (Security/correctness)
### Priority 4Viewer Migration to Canonical Part Identity
**Goal:** Make RLS actually work. Currently `build_tenant_db_dep()` yields `db` without calling `set_tenant_context()`, so all tenant isolation is silent no-op.
**Goal:** Move browser interactions from raw GLB mesh-name matching to canonical `partKey` without any UX regression.
**What to build:**
- `TenantContextMiddleware` in `backend/app/core/middleware.py`
- Extracts `tenant_id` from JWT, stores in `request.state`
- After DB session acquired: `SET LOCAL app.current_tenant_id = '...'`
- Celery tasks: add `set_tenant_context(db, tenant_id)` call at start of each task (Celery bypasses HTTP middleware)
- Role hierarchy migration: `admin``global_admin`, add `tenant_admin`
**Milestones:**
- M1: Preview GLB derivation embeds `partKey` as mesh `extras.partKey` on every selectable object
- M2: `ThreeDViewer` reads `partKey` from scene manifest + mesh extras on click, no longer uses raw `mesh.name`
- M3: `MaterialPanel` shows `partKey`, source name, assignment provenance; saves overrides by `partKey`
- M4: Unmatched source rows and unassigned parts surfaced in `MaterialPanel` reconciliation section
**Files:**
- New: `backend/app/core/middleware.py`
- Migration: `06x_role_hierarchy.py`
- Modified: `backend/app/main.py` (register middleware), `utils/auth.py`, all routers
**File targets:**
**Estimated effort:** 2 sessions
**Depends on:** Stable auth layer (Priority 3 dead code removal first)
| Action | Path |
|---|---|
| CREATE | `frontend/src/api/sceneManifest.ts``SceneManifest` + `PartEntry` interfaces |
| MODIFY | `frontend/src/components/cad/ThreeDViewer.tsx` — use `partKey` from scene manifest for selection, isolation, ghost |
| MODIFY | `frontend/src/components/cad/MaterialPanel.tsx` — show provenance, unmatched/unassigned sections |
| MODIFY | `frontend/src/api/cad.ts` — update `PartMaterialsMap` interface to `{ [partKey: string]: string }` |
| MODIFY | `backend/app/api/routers/cad.py``GET/PUT /cad/{id}/part-materials` keyed by partKey with provenance |
| ADD util | `render-worker/scripts/export_step_to_gltf.py` — embed `partKey` into mesh extras during GLB export |
---
**Acceptance gates:**
- `mesh.userData.partKey` exists on every mesh object in the Three.js scene after GLB load
- Select a part → DevTools shows the assignment payload contains `part_key: "ring_outer"`, not `mesh_name: "RingOuter_AF0"`
- Upload file with 3 unmatched Excel rows → MaterialPanel shows "3 unmatched source rows"
- After full page reload: all manual material assignments are restored correctly
- Isolation, hide, and ghost still work as before (no regression)
### Priority 7UV Unwrap Workflow (New feature, user-requested)
### Priority 5Canonical USD Export and Render Migration
**Goal:** Produce UV-unwrapped geometry GLBs with clean seams for downstream texture authoring.
**Depends on:** Priority 1 (GMSH tessellation must produce conforming seams first).
**Goal:** Switch Blender still/turntable renders to consume the canonical USD stage, retiring the production GLB as an intermediate render artifact.
**What to build:**
- New Blender script `_blender_uvunwrap.py` called from `export_gltf.py` after sharp edge marking
- UV unwrap via `bpy.ops.uv.unwrap(method='ANGLE_BASED', margin=0.001)`
- Seams are already set by `_apply_sharp_edges_from_occ()` (edge.seam=True)
- UV coordinates embedded in the production GLB
- Admin toggle: `uv_unwrap_enabled` in system_settings
**Milestones:**
- M1: `render-worker/scripts/import_usd.py` — Blender can import USD + restore seam/sharp from primvars
- M2: Still render from USD matches current production GLB render quality (side-by-side comparison)
- M3: Turntable render from USD works end-to-end
- M4: `generate_production_glb_task` bypassed; renders consume `usd_master` directly
**Estimated effort:** 1 session
**Depends on:** Priority 1 (GMSH) for clean seams
**File targets:**
---
| Action | Path |
|---|---|
| CREATE | `render-worker/scripts/export_step_to_usd.py` — STEP→USD exporter (seam/sharp payload on mesh prims) |
| CREATE | `render-worker/scripts/import_usd.py` — Blender USD import helper: reads `primvars:schaeffler:seamEdgeVertexPairs`, marks seam+sharp |
| MODIFY | `render-worker/scripts/blender_render.py` — accept `--usd_path` flag alongside `--glb_path` |
| MODIFY | `backend/app/services/render_blender.py` — pass `usd_master` asset path when available |
| MODIFY | `backend/app/domains/pipeline/tasks/export_glb.py` — retire `generate_gltf_production_task` once USD path validated |
| KEEP (compat) | `render-worker/scripts/export_gltf.py` — retained as fallback until USD path confirmed stable |
### Priority 8 — UI/UX Polish (from visual-audit-report.md)
**Acceptance gates:**
- `render_still --usd_path usd_master.usd` → PNG output visually identical to current production GLB render (diff tolerance < 5% SSIM)
- Blender log shows `[USD_IMPORT] 25 parts imported, 5044 seam/sharp edges restored` (not reconstructed by angle)
- `GET /api/media?asset_type=gltf_production` returns 0 new entries after switch (old records preserved)
- Turntable MP4 plays without texture or material pop artifacts
**Top actionable items from the audit:**
### Priority 6 — Admin and Product Surface Simplification
| Item | Where | Fix |
|---|---|---|
| Tooltip system | All settings pages | Add `title` or tooltip component to every input |
| Empty state messages | MediaBrowser, ProductLibrary | "No assets yet — upload a STEP file" |
| Notification batching | NotificationCenter | Group per-render noise into job summaries |
| Mobile navigation | Layout.tsx | Hamburger menu for viewport < 768px |
| Kanban rejection flow | OrderDetail | Drag-to-reject with reason field |
**Goal:** Remove the geometry-GLB vs production-GLB mental model from admin, product detail, and repair flows.
**Estimated effort:** 2 sessions
**Independent of all backend priorities.**
**Milestones:**
- M1: Admin tessellation settings collapsed from 4 knobs to `scene_*` + `preview_*` (2 or 3 knobs max)
- M2: Bulk actions renamed from `generate-missing-geometry-glbs``generate-missing-canonical-scenes`
- M3: ProductDetail shows single canonical scene status card, not dual-GLB
---
**File targets:**
### Priority 9 — Phase F: Hash-based Conversion Caching (Performance)
| Action | Path |
|---|---|
| MODIFY | `backend/app/api/routers/admin.py` — rename/collapse tessellation settings; new bulk action labels |
| CREATE migration | `backend/alembic/versions/06x_rename_tessellation_settings.py` — UPDATE system_settings SET key = 'scene_linear_deflection' WHERE key = 'gltf_production_linear_deflection' |
| MODIFY | `frontend/src/pages/Admin.tsx` — simplified tessellation panel: scene quality + optional preview override |
| MODIFY | `frontend/src/pages/ProductDetail.tsx` — single canonical scene card (status, regenerate button) |
| MODIFY | `backend/app/domains/media/models.py` — deprecate `gltf_geometry` / `gltf_production` (keep values, add `deprecated=True` metadata) |
**Goal:** Skip re-tessellation if STEP file hash hasn't changed. Cache geometry GLB by `sha256(step_file)`.
**Acceptance gates:**
- Admin Settings page has max 3 tessellation quality fields (down from 4)
- ProductDetail page: one "Canonical Scene" status section, no separate geometry/production rows
- `GET /api/admin/settings` no longer exposes `gltf_preview_linear_deflection` key (replaced by `scene_linear_deflection`)
**What to build:**
- `cad_files.step_hash` column (SHA256, nullable)
- Before GLB generation: compute hash, check if cached GLB exists for same hash
- If hit: copy cached GLB, skip OCC+GMSH, skip Blender
- Migration: `06x_step_hash_column.py`
### Priority 7 — Render Job Tracking and Structured Logging
**Estimated effort:** 1 session
**Depends on:** Priority 1 (GMSH) stable first
**Goal:** Fix broken render job cancellation (synthetic `render-{line_id}` ID never matches real Celery task ID) and establish structured per-step logging.
---
**Milestones:**
- M1: `RenderJobDocument` schema + migration; tasks write real `self.request.id` to DB
- M2: Cancel endpoint reads `celery_task_id` from job doc and calls `revoke()` — actually stops task
- M3: `PipelineLogger` integrated in all task files; every step emits `[STEP] start/done/error` with duration
### Priority 10 — USD Workflow RFC (Strategic, long-term)
**File targets:**
**Document:** `docs/rfcs/0001-step-to-usd-workflow.md`
**Status:** Proposed only — not planned for implementation yet.
| Action | Path |
|---|---|
| CREATE | `backend/app/domains/rendering/job_document.py``RenderJobDocument` Pydantic model, `update_step()`, `set_state()` |
| CREATE | `backend/app/core/pipeline_logger.py``PipelineLogger(step_start/done/error)` writing to logging + Redis SSE |
| CREATE migration | `backend/alembic/versions/062_render_job_document.py` — add `render_job_doc JSONB` to `order_lines` |
| MODIFY | `backend/app/domains/pipeline/tasks/render_order_line.py` — write `celery_task_id` + step events to job doc |
| MODIFY | `backend/app/domains/pipeline/tasks/render_thumbnail.py` — same |
| MODIFY | `backend/app/api/routers/orders.py` — cancel reads `render_job_doc.celery_task_id`, calls `celery.control.revoke()` |
**Summary:** Replace dual-GLB pipeline (geometry GLB → production GLB) with a single USD canonical scene. Three.js has no USD loader, so this requires either switching viewers or waiting for Three.js USD support.
**Acceptance gates:**
- Start a 60s render task → click Cancel → `celery inspect active` shows task is gone within 15s
- `GET /api/orders/{id}/lines/{line_id}` response includes `render_job_doc.steps[]` with per-step `duration_s`
- Worker log shows `[THUMBNAIL] done in 34.2s` format (not bare f-strings)
**Decision needed:** Is the viewer constraint acceptable? Deferred until Priority 14 are complete.
### Priority 8 — Tenant Isolation Completion
**Goal:** Make PostgreSQL RLS enforcement real. Currently `build_tenant_db_dep()` yields `db` without calling `SET LOCAL app.current_tenant_id`, making all tenant isolation a silent no-op.
**Milestones:**
- M1: `TenantContextMiddleware` registered; all HTTP requests set RLS context from JWT
- M2: All Celery tasks call `set_tenant_context(db, tenant_id)` at task start
- M3: `global_admin` + `tenant_admin` roles in DB; `require_admin()``require_global_admin()`
**File targets:**
| Action | Path |
|---|---|
| CREATE | `backend/app/core/middleware.py``TenantContextMiddleware(BaseHTTPMiddleware)` |
| MODIFY | `backend/app/main.py``app.add_middleware(TenantContextMiddleware)` |
| MODIFY | `backend/app/utils/auth.py``create_access_token()` embeds `tenant_id` in JWT claims |
| MODIFY | `backend/app/tasks/pipeline/thumbnail.py`, `extract.py`, `stills.py`, `turntable.py``set_tenant_context()` at start |
| CREATE migration | `backend/alembic/versions/063_role_hierarchy.py` — rename `admin``global_admin`, add `tenant_admin` |
| MODIFY | All routers using `require_admin()``require_global_admin()` |
**Acceptance gates:**
- Login as tenant A user → `GET /api/products` returns 0 results when tenant A has no products, even if tenant B has 50
- Verify via: `SELECT count(*) FROM products WHERE tenant_id != '<tenantA_id>'` returns 0 from within tenant A session
- Celery task logs show `[TENANT] context set: tenant_id=<uuid>` at start
- `GET /api/admin/users` returns 403 for `tenant_admin` role (only `global_admin` can list all users)
### Priority 9 — Hash-Based Scene Conversion Caching
**Goal:** Skip re-tessellation when the STEP file has not changed. Cache canonical scene + preview derivatives by `SHA256(step_file)`.
**Milestones:**
- M1: `cad_files.step_hash` column in DB; hash computed and stored on each export
- M2: Export task checks hash before processing — returns cached asset UUID on hit
- M3: Hash invalidated correctly when admin forces reprocess or deflection settings change
**File targets:**
| Action | Path |
|---|---|
| ADD column | `backend/app/domains/products/models.py``CadFile.step_hash: str \| None` |
| CREATE migration | `backend/alembic/versions/064_step_hash.py``ADD COLUMN step_hash VARCHAR(64)` |
| MODIFY | `backend/app/domains/pipeline/tasks/export_glb.py` (or future USD task) — hash check before subprocess call |
| ADD util | `backend/app/services/step_processor.py``compute_step_hash(file_path) -> str` |
**Acceptance gates:**
- Upload same STEP file twice → second task completes in < 2s (cache hit logged: `[CACHE] hash match, skipping tessellation`)
- Change deflection setting → force reprocess → new export runs fresh (hash same but settings changed, cache bypassed)
- `GET /api/cad/{id}` response includes `step_hash` field
### Priority 10 — UI/UX Polish
**Goal:** Address independent UI items from `visual-audit-report.md` that require no backend changes.
**Milestones:**
- M1: Tooltip/help text on every Admin settings input
- M2: Empty state messages in MediaBrowser, ProductLibrary, Orders
- M3: Notification batching — group per-render noise into job summaries
- M4: Mobile navigation — hamburger menu at < 768px
- M5: Kanban rejection flow — drag-to-reject with reason field
**File targets:**
| Action | Path |
|---|---|
| MODIFY | `frontend/src/pages/Admin.tsx``title` attributes on all inputs; help text below complex settings |
| MODIFY | `frontend/src/pages/MediaBrowser.tsx` — empty state: "No assets yet — upload a STEP file to get started" |
| MODIFY | `frontend/src/pages/ProductLibrary.tsx` — empty state |
| MODIFY | `frontend/src/pages/Orders.tsx` — empty state |
| MODIFY | `frontend/src/components/layout/Layout.tsx` — hamburger + slide-in nav for mobile |
| MODIFY | `frontend/src/pages/OrderDetail.tsx` — reject button with reason modal |
| CREATE | `frontend/src/components/shared/Tooltip.tsx` — reusable tooltip wrapper |
**Acceptance gates:**
- All Admin settings inputs have visible help text or tooltip (manual check: no input label without explanation)
- Mobile viewport (375px): no horizontal scroll, nav accessible via hamburger
- Submit a render → NotificationCenter shows one "Render complete (3 files)" summary, not 3 individual toasts
---
## Dependency Graph
```
Priority 1 (GMSH)
└── Priority 7 (UV Unwrap)
└── Priority 9 (Hash Cache)
Priority 1 (Pipeline Cleanup Foundation)
└── Priority 2 (USD Foundation Without Viewer Regression)
└── Priority 4 (Viewer Migration to Canonical Part Identity)
└── Priority 5 (Canonical USD Export and Render Migration)
└── Priority 6 (Admin and Product Surface Simplification)
└── Priority 9 (Hash-Based Scene Conversion Caching)
Priority 2 (Dead Code)
└── Priority 3 (Task Decomposition)
└── Priority 4 (Render Job Tracking)
└── Priority 6 (Tenant Isolation)
Priority 3 (Tessellation and Topology Quality)
└── Priority 5 (Canonical USD Export and Render Migration)
Priority 5 (Logging) — independent, can start anytime
Priority 8 (UI/UX) — independent, can start anytime
Priority 10 (USD) — deferred
Priority 7 (Render Job Tracking and Structured Logging) — can run in parallel
Priority 8 (Tenant Isolation Completion) — can run in parallel
Priority 10 (UI/UX Polish) — independent
```
---
## What To Do Next
**Option A — Fix visual quality first:**
`/implement` on `plan.md` (Priority 1: GMSH tessellation)
**Recommended execution path:**
1. Do Priority 1 first: clean up and split the current pipeline.
2. Start Priority 2 immediately after: add `partKey`, assignment-layer semantics, and scene manifest without changing the browser UX.
3. Run Priority 3 in parallel or immediately after, depending on whether current tessellation quality blocks scene-authoring confidence.
4. Use the implementation plan in `docs/plans/0001-step-to-usd-implementation.md` as the execution checklist for the USD workstream.
**Option B — Clean up dead code first (low risk, fast wins):**
→ Start Priority 2 dead code deletion, then Priority 3 task decomposition
**Parallel sprint option (2 agents):**
- Agent 1: Priority 1 (pipeline cleanup foundation)
- Agent 2: Priority 3 (tessellation and topology quality)
**Option C — Parallel sprint (2 agents):**
Agent 1: Priority 1 (GMSH) in worktree
Agent 2: Priority 2+3 (dead code + task split) in separate worktree
**Parallel sprint option (3 agents):**
- Agent 1: Priority 1
- Agent 2: Priority 3
- Agent 3: Priority 7 or Priority 8
**Option D — UI/UX sprint:**
→ Priority 8 audit items, completely independent of backend
**Do not defer anymore:**
- canonical `partKey`
- part-keyed browser material overrides
- scene manifest / preview contract
These are now considered implementation prerequisites for the long-term refactor, not optional strategy work.
---
@@ -256,5 +384,6 @@ Old planning files are kept for reference but superseded by this document:
- `PLAN_REFACTOR.md` — 1,173-line architectural plan (Phases 18 mapped to Priorities 28 above)
- `plan.md` — active GMSH implementation plan (Priority 1)
- `docs/rfcs/0001-step-to-usd-workflow.md` — USD RFC (Priority 10)
- `docs/plans/0001-step-to-usd-implementation.md` — actionable USD implementation plan
- `review-report.md` — latest code review results
- `visual-audit-report.md` — UX audit results
@@ -0,0 +1,326 @@
# Implementation Plan: STEP → USD Canonical Scene Workflow
> Execution checklist for ROADMAP.md Priorities 2, 4, and 5.
> RFC: `docs/rfcs/0001-step-to-usd-workflow.md`
> Date: 2026-03-11
---
## Prerequisites
- [ ] Priority 1 complete (step_tasks.py decomposed, blender_render.py decomposed)
- [ ] Decision: USD authoring library (see Open Questions below)
- [ ] Decision: seam/sharp payload encoding (primvars vs. JSON sidecar)
---
## Phase 1 — Dual-Write USD Beside GLB (Priority 2, M1M2)
**Goal:** Export a valid `usd_master` alongside the existing GLB pipeline without changing any browser behavior.
### Task 1.1 — `export_step_to_usd.py` scaffolding
**File:** `render-worker/scripts/export_step_to_usd.py`
**What it must produce:**
```
/Root — Stage root
/Root/Assembly — Top-level assembly prim (Xform)
/Root/Assembly/<AssemblyNode> — Per-component prim (Xform)
/Root/Assembly/<AssemblyNode>/<PartKey> — Leaf part prim (Xform)
/Root/Assembly/<AssemblyNode>/<PartKey>/Mesh — UsdGeomMesh
/Root/Looks/<MaterialName> — UsdShadeMaterial (placeholder binding)
```
**Required per-mesh prim attributes:**
| Attribute | Value source |
|---|---|
| `schaeffler:partKey` | `generate_part_key(xcaf_label_path)` |
| `schaeffler:sourceName` | XCAF `TDataStd_Name` attribute |
| `schaeffler:sourceColor` | XCAF embedded color (hex string) |
| `schaeffler:rawMaterialName` | from `CadFile.part_materials` if available |
| `schaeffler:tessellation:linearDeflectionMm` | CLI arg value |
| `schaeffler:tessellation:angularDeflectionRad` | CLI arg value |
| `primvars:schaeffler:seamEdgeVertexPairs` | OCC B-rep seam edges (index pairs in mesh-local space) |
| `primvars:schaeffler:sharpEdgeVertexPairs` | sharp edges from `_extract_sharp_edge_pairs()` |
**CLI interface:**
```bash
python3 export_step_to_usd.py \
--step_path /path/to/file.stp \
--output_path /path/to/output.usd \
[--linear_deflection 0.03] \
[--angular_deflection 0.05] \
[--color_map '{"Ring": "#4C9BE8"}'] \
[--tessellation_engine occ|gmsh]
```
**Acceptance gate:** `python3 export_step_to_usd.py --step_path 81113-l_cut.stp --output_path /tmp/test.usd`
- File exists, parseable
- 25 part prims with `schaeffler:partKey` attribute
- Part count matches `export_step_to_gltf.py` output for same file
### Task 1.2 — `usd_master` MediaAsset type
**File:** `backend/app/domains/media/models.py`
Add `usd_master = "usd_master"` to `MediaAssetType` enum.
**Migration:** `backend/alembic/versions/060_usd_master_asset_type.py`
```sql
ALTER TYPE mediaassettype ADD VALUE IF NOT EXISTS 'usd_master';
```
**Acceptance gate:** `GET /api/media?asset_type=usd_master` returns 200 (not 422 validation error).
### Task 1.3 — `generate_usd_master_task` Celery task
**File:** `backend/app/domains/pipeline/tasks/export_glb.py`
New task that:
1. Resolves STEP file path from `CadFile`
2. Calls `export_step_to_usd.py` subprocess
3. Stores resulting `.usd` as a `usd_master` `MediaAsset`
4. Does NOT touch existing GLB tasks (dual-write, no removal yet)
**Queue:** `step_processing` (fast, < 30s — tessellation only, no Blender)
**Called by:** `render_step_thumbnail` task after `render_step_thumbnail` succeeds, OR triggered independently via admin action.
**Acceptance gate:** After triggering task for a CadFile, `GET /api/cad/{id}/media` includes an asset with `asset_type: "usd_master"`.
---
## Phase 2 — Canonical Part Identity and Assignment Layers (Priority 2, M3M5)
### Task 2.1 — `part_key_service.py`
**File:** `backend/app/services/part_key_service.py`
```python
def generate_part_key(xcaf_label_path: str, source_name: str) -> str:
"""Deterministic slug from XCAF path + source name.
Format: lowercase alphanumeric + underscores, max 64 chars.
E.g.: 'ring_outer', 'ball_af0', 'cage_inner_ring'
For duplicate names: append _2, _3, etc.
For unnamed parts: 'part_{sha256(xcaf_path)[:8]}'
"""
...
def build_scene_manifest(cad_file: CadFile, usd_asset: MediaAsset) -> dict:
"""Read part metadata from USD or from CadFile.parsed_objects.
Returns dict matching SceneManifest Pydantic schema:
{
"cad_file_id": "...",
"parts": [
{
"part_key": "ring_outer",
"source_name": "RingOuter_AF0",
"prim_path": "/Root/Assembly/Bearing/RingOuter",
"effective_material": "SCHAEFFLER_010102_...",
"assignment_provenance": "manual|auto|default",
"is_unassigned": false
}
],
"unmatched_source_rows": [...],
"unassigned_parts": [...]
}
"""
...
```
### Task 2.2 — Three-layer material assignment model
**File:** `backend/app/domains/products/models.py`
Add three JSONB columns to `CadFile`:
```python
source_material_assignments: dict | None # from Excel / product import; keyed by source name
resolved_material_assignments: dict | None # auto-matched; keyed by partKey
manual_material_overrides: dict | None # browser-authored; keyed by partKey
```
**Migration:** `backend/alembic/versions/061_material_assignment_layers.py`
Keep existing `part_materials` column for backward compat during transition.
**Service helper:**
```python
def get_effective_assignments(cad_file: CadFile) -> dict:
"""Priority: manual_overrides > resolved > source color > 'unassigned'."""
...
```
### Task 2.3 — `GET /cad/{id}/scene-manifest` endpoint
**File:** `backend/app/api/routers/cad.py`
New endpoint returning `SceneManifest`. Calls `build_scene_manifest()` — reads from USD metadata if `usd_master` asset exists, otherwise reads from `CadFile.parsed_objects`.
**Acceptance gate:** Returns HTTP 200 with parts array; each part has `part_key`, `effective_material`, `is_unassigned`, `assignment_provenance`.
### Task 2.4 — Update `PUT /cad/{id}/part-materials`
**File:** `backend/app/api/routers/cad.py`
Accept `{ "part_key": "ring_outer", "material": "SCHAEFFLER_010102_..." }` body (or bulk map). Write to `manual_material_overrides` column (not the old `part_materials` column).
**Acceptance gate:** PUT with `partKey` → subsequent GET `/scene-manifest` shows that part's `assignment_provenance: "manual"`.
---
## Phase 3 — Seam/Sharp Payload to USD Mesh Prims (Priority 3 integration)
### Task 3.1 — Port seam derivation into USD exporter
**File:** `render-worker/scripts/export_step_to_usd.py`
After tessellation (OCC or GMSH), for each mesh face:
1. Identify seam edges from face/batch boundaries (STEPper approach: adjacent faces with different batch IDs)
2. Identify sharp edges from `_extract_sharp_edge_pairs()` (already implemented)
3. Convert both to **mesh-local vertex index pairs** (not world-space coordinates)
Write to USD mesh prim:
```python
mesh_prim.GetPrimvar("schaeffler:seamEdgeVertexPairs").Set(
Vt.Vec2iArray(seam_pairs), # [(vi0, vi1), ...]
)
mesh_prim.GetPrimvar("schaeffler:sharpEdgeVertexPairs").Set(
Vt.Vec2iArray(sharp_pairs),
)
```
**Why index-space:** Survives transforms cleanly; avoids KD-tree matching workaround in `_apply_sharp_edges_from_occ()`.
### Task 3.2 — `import_usd.py` Blender helper
**File:** `render-worker/scripts/import_usd.py`
```python
def import_usd_and_restore_topology(usd_path: str) -> list:
"""Import USD stage into Blender, restore seam/sharp from primvars.
Returns list of imported mesh objects.
"""
bpy.ops.wm.usd_import(filepath=usd_path)
for obj in bpy.context.scene.objects:
if obj.type != 'MESH':
continue
# Read custom attributes set by USD importer
seam_pairs = obj.get("schaeffler_seamEdgeVertexPairs") or []
sharp_pairs = obj.get("schaeffler_sharpEdgeVertexPairs") or []
_mark_seams_from_index_pairs(obj, seam_pairs)
_mark_sharp_from_index_pairs(obj, sharp_pairs)
...
```
**Acceptance gate:** Blender log shows `[USD_IMPORT] 25 parts, 5044 seam edges restored from primvars` — no KD-tree needed.
---
## Phase 4 — Blender Render from USD (Priority 5)
### Task 4.1 — `blender_render.py` USD path
**File:** `render-worker/scripts/blender_render.py`
Add `--usd_path` argument. When provided:
1. Call `import_usd.py` instead of `export_gltf.py` GLB import
2. Read `schaeffler:partKey` and `schaeffler:canonicalMaterialName` per mesh object after import
3. Apply materials by `partKey → material library name` lookup instead of object-name heuristics
**Migration:** Keep `--glb_path` working in parallel; switch production task to prefer `--usd_path` when `usd_master` asset exists.
### Task 4.2 — Backend render service update
**File:** `backend/app/services/render_blender.py`
When building `blender_cmd`, check if `usd_master` MediaAsset exists for the CadFile:
- If yes: pass `--usd_path <usd_master_local_path>`
- If no: fall back to `--glb_path <gltf_production_path>` (unchanged)
---
## Phase 5 — Frontend Migration (Priority 4)
### Task 5.1 — Scene manifest fetch on product load
**File:** `frontend/src/api/sceneManifest.ts`
```typescript
export interface PartEntry {
part_key: string;
source_name: string;
prim_path: string;
effective_material: string | null;
assignment_provenance: 'manual' | 'auto' | 'default';
is_unassigned: boolean;
}
export interface SceneManifest {
cad_file_id: string;
parts: PartEntry[];
unmatched_source_rows: string[];
unassigned_parts: string[];
}
export async function fetchSceneManifest(cadFileId: string): Promise<SceneManifest> {
const res = await api.get<SceneManifest>(`/cad/${cadFileId}/scene-manifest`);
return res.data;
}
```
### Task 5.2 — ThreeDViewer partKey integration
**File:** `frontend/src/components/cad/ThreeDViewer.tsx`
- On GLB load: iterate `scene.children`, read `mesh.userData.partKey` (set by preview GLB derivation)
- On click: identify `partKey` from `userData`, not from `mesh.name`
- Pass `partKey` to `MaterialPanel` for display and persistence
### Task 5.3 — MaterialPanel reconciliation section
**File:** `frontend/src/components/cad/MaterialPanel.tsx`
Add reconciliation section showing:
- "N unmatched Excel rows" (source names with no partKey match)
- "M unassigned parts" (partKeys with no resolved material)
Clicking an unassigned part in the viewer auto-focuses it in the MaterialPanel.
---
## Open Questions (must be decided before Phase 1 coding starts)
| # | Question | Options | Default recommendation |
|---|---|---|---|
| 1 | USD authoring library | `pxr` (full OpenUSD, heavy) / `usda` text templating (no deps) / `usd-core` pip | Start with `pxr` — pip-installable, same OCC kernel available |
| 2 | Seam/sharp payload encoding | Custom primvars on mesh prim / separate JSON sidecar / GLB extras (current) | Index-space primvars — cleaner, survives transforms |
| 3 | Preview GLB derivation | USD → GLB export pass / co-author from same tessellation pass | Co-author during migration (avoid round-trip loss) |
| 4 | Single-file USD or override layers | Flat single file / canonical + overlay layers (flattened for delivery) | RFC recommends Option B (overlay layers, flatten for delivery) |
| 5 | `pxr` install in render-worker | Add to `render-worker/Dockerfile` / use system package | `pip install usd-core` — no NVIDIA/Pixar GPU tools needed |
---
## Non-Regression Checklist
Before merging any Priority 25 work:
- [ ] Click a part in ThreeDViewer → selection resolves to stable `partKey`
- [ ] Pin selection → isolate, hide, ghost all work as before
- [ ] Unassigned parts are visually highlighted
- [ ] Assign a Blender asset-library material name via browser → persisted by `partKey`
- [ ] Reload page → same part still assigned
- [ ] Subsequent Blender render uses the same assignment
- [ ] CAD file with mismatched Excel names → system produces canonical scene, preview asset, unmatched row count
- [ ] `geometryGltfUrl` / `productionGltfUrl` distinction no longer required by frontend (removed from API contract)