Hartmut/HartOMat
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: update ROADMAP.md + USD plan after Phase B completion

Browse Source 
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Hartmut Nörenberg
2026-03-11 21:51:38 +01:00
parent d938c4db1b
commit 4f0fe2c8c7
2 changed files with 79 additions and 37 deletions
Download Patch File Download Diff File Expand all files Collapse all files
ROADMAP.md
+69 -36
View File
@@ -10,7 +10,7 @@
| Area | Detail |
|---|---|
| Phase A | Flamenco removed, blender-renderer → render-worker Celery container, threejs-renderer removed, MinIO added |
| Phase A runtime | Flamenco removed from the active pipeline, `render-worker` replaced the old Blender service path, MinIO added; legacy `blender-renderer/` and `threejs-renderer/` directories still remain in the repo |
| Phase B | Domain-driven project structure, Tenant model + RLS migrations 035/036, Tenant management UI |
| Phase C | WorkflowDefinition model, standard workflows seeded, React Flow Workflow Editor |
| Phase D | OCC mesh attributes extraction, Blender integration |
@@ -20,6 +20,26 @@
| Media cache-bust | `?v={file_size_bytes}` in download URLs + `Cache-Control: no-cache` headers |
| GPU activation order | Fix: `_activate_gpu()` called before AND after `open_mainfile` to survive engine reset |
| Material system | Aliases-first lookup, `get_material_library_path()` via AssetLibrary |
| Pipeline task split | `backend/app/tasks/step_tasks.py` is now a 23-line compatibility shim; active task implementations live in `backend/app/domains/pipeline/tasks/` |
| Render job tracking | `RenderJobDocument`, `PipelineLogger`, and cancel-via-real-`celery_task_id` are already wired into the render pipeline |
| Tenant isolation baseline | `TenantContextMiddleware`, JWT `tenant_id`, and the `global_admin` / `tenant_admin` role hierarchy are in place for HTTP requests |
| Hash groundwork | `compute_step_hash()` exists and `CadFile.step_file_hash` is already persisted during thumbnail processing |
---
## 🔎 Status Snapshot
Verified against the repository on `2026-03-11`.
| Priority | Status | Re-evaluated state |
|---|---|---|
| 1. Pipeline Cleanup Foundation | In progress | `step_tasks.py` decomposition is done; dead-code cleanup and `blender_render.py` decomposition are still open |
| 2. USD Foundation Without Viewer Regression | Not started in code | Decisions are documented, but there is no `export_step_to_usd.py`, `usd_master`, `scene-manifest`, or `partKey` implementation yet |
| 3. Tessellation and Topology Quality | Not started in code | No `gmsh` install/wiring, no `tessellation_engine` setting, no Admin dropdown yet |
| 7. Render Job Tracking and Structured Logging | Done | `RenderJobDocument`, migration `048`, `PipelineLogger`, and revoke-by-real-task-id are present |
| 8. Tenant Isolation Completion | In progress | HTTP-side RLS context is wired; Celery task-side `set_tenant_context()` propagation still needs to be added |
| 9. Hash-Based Scene Conversion Caching | Partial foundation | Existing `step_file_hash` and STL-cache utilities should be extended, not rebuilt from scratch |
| 10. UI/UX Polish | Partial | Admin help tooltips, mobile nav, and some empty states exist; notification batching and remaining polish items are still open |
---
@@ -41,9 +61,11 @@ That removes the old assumption that USD work must wait for a Three.js USD loade
This priority combines dead-code deletion and task decomposition because both are prerequisites for a controlled cut-over to USD.
**Status:** In progress. M2 is complete; M1 and M3 remain open.
**Milestones:**
- M1: Dead code deleted — Pillow block, STL settings, orphaned directories
- M2: `step_tasks.py` decomposed into `backend/app/tasks/pipeline/` submodules
- M2: `step_tasks.py` decomposed into `backend/app/domains/pipeline/tasks/` submodules
- M3: `blender_render.py` decomposed into `render-worker/scripts/_blender_*.py` submodules
**File targets:**
@@ -55,13 +77,13 @@ This priority combines dead-code deletion and task decomposition because both ar
| DELETE | `threejs-renderer/` directory |
| DELETE | `flamenco/` directory |
| DELETE | `renderproblems_tmp/` |
| DELETE lines 705–1050 | `backend/app/tasks/step_tasks.py` (`render_order_line_task` — duplicates `rendering/tasks`) |
| DONE | `backend/app/tasks/step_tasks.py` — reduced to compatibility shim only |
| REMOVE settings | `admin.py`: `VALID_STL_QUALITIES`, `stl_quality`, `generate-missing-stls` endpoint |
| REMOVE endpoint | `cad.py`: `POST /cad/{id}/generate-stl/{quality}` |
| CREATE | `backend/app/tasks/pipeline/extract.py` — metadata extraction (OCC parsing, < 2s) |
| CREATE | `backend/app/tasks/pipeline/thumbnail.py` — Blender thumbnail render |
| CREATE | `backend/app/tasks/pipeline/stills.py` — still render task |
| CREATE | `backend/app/tasks/pipeline/turntable.py` — turntable render task |
| DONE | `backend/app/domains/pipeline/tasks/extract_metadata.py` — metadata extraction |
| DONE | `backend/app/domains/pipeline/tasks/render_thumbnail.py` — thumbnail render task |
| DONE (combined) | `backend/app/domains/pipeline/tasks/render_order_line.py` — still/dispatch pipeline entry |
| OPEN | turntable-specific pipeline task split still needs to be carved out explicitly if kept as a separate concern |
| THIN (< 80 lines) | `backend/app/tasks/step_tasks.py` — dispatch only |
| CREATE | `render-worker/scripts/_blender_gpu.py` |
| CREATE | `render-worker/scripts/_blender_import.py` |
@@ -81,6 +103,8 @@ This priority combines dead-code deletion and task decomposition because both ar
**Goal:** Introduce canonical part identity and the three-layer material assignment model while keeping the current GLB-based browser UX working end-to-end.
**Status:** Not started in code. Architecture decisions are documented, but repo work has not begun.
**Milestones:**
- M1: `export_step_to_usd.py` produces valid USD with part hierarchy and `schaeffler:partKey` on every prim
- M2: `usd_master` MediaAsset type exists in DB and is stored after each export
@@ -106,8 +130,7 @@ This priority combines dead-code deletion and task decomposition because both ar
| CREATE | `frontend/src/api/sceneManifest.ts` — `SceneManifest` interface, `fetchSceneManifest()` |
**Open questions to decide before M1:**
- USD authoring library: `pxr` (OpenUSD full Python SDK) vs. plain USDA text templating vs. `usd-core` pip package
- seam/sharp payload encoding: custom primvars (`primvars:schaeffler:seamEdgeVertexPairs`) or a separate JSON sidecar
- None blocking at the architecture level. The roadmap decisions for `usd-core` and index-space seam/sharp primvars are already captured in `docs/plans/0001-step-to-usd-implementation.md`.
**Acceptance gates:**
- `python3 export_step_to_usd.py --step_path 81113-l_cut.stp` → valid `.usd` file, 25 part prims, each has `schaeffler:partKey` attribute
@@ -124,6 +147,8 @@ This priority combines dead-code deletion and task decomposition because both ar
**Goal:** Eliminate fan triangles on cylindrical surfaces (rings, bearings) and produce clean seams for UV unwrap.
**Status:** Not started in code. This is still a pure planning workstream at the moment.
**Milestones:**
- M1: GMSH 4.15+ installed in render-worker container
- M2: `export_step_to_gltf.py --tessellation_engine gmsh` produces fan-free GLB
@@ -232,6 +257,8 @@ This priority combines dead-code deletion and task decomposition because both ar
**Goal:** Fix broken render job cancellation (synthetic `render-{line_id}` ID never matches real Celery task ID) and establish structured per-step logging.
**Status:** Done, aside from any follow-up polish.
**Milestones:**
- M1: `RenderJobDocument` schema + migration; tasks write real `self.request.id` to DB
- M2: Cancel endpoint reads `celery_task_id` from job doc and calls `revoke()` — actually stops task
@@ -243,7 +270,7 @@ This priority combines dead-code deletion and task decomposition because both ar
|---|---|
| CREATE | `backend/app/domains/rendering/job_document.py` — `RenderJobDocument` Pydantic model, `update_step()`, `set_state()` |
| CREATE | `backend/app/core/pipeline_logger.py` — `PipelineLogger(step_start/done/error)` writing to logging + Redis SSE |
| CREATE migration | `backend/alembic/versions/062_render_job_document.py` — add `render_job_doc JSONB` to `order_lines` |
| DONE | `backend/alembic/versions/048_render_job_document.py` — adds `render_job_doc JSONB` to `order_lines` |
| MODIFY | `backend/app/domains/pipeline/tasks/render_order_line.py` — write `celery_task_id` + step events to job doc |
| MODIFY | `backend/app/domains/pipeline/tasks/render_thumbnail.py` — same |
| MODIFY | `backend/app/api/routers/orders.py` — cancel reads `render_job_doc.celery_task_id`, calls `celery.control.revoke()` |
@@ -255,7 +282,9 @@ This priority combines dead-code deletion and task decomposition because both ar
### Priority 8 — Tenant Isolation Completion
**Goal:** Make PostgreSQL RLS enforcement real. Currently `build_tenant_db_dep()` yields `db` without calling `SET LOCAL app.current_tenant_id`, making all tenant isolation a silent no-op.
**Goal:** Finish tenant isolation hardening, especially for non-HTTP execution paths.
**Status:** In progress. HTTP-side RLS enforcement is now real; task-side propagation is the remaining gap.
**Milestones:**
- M1: `TenantContextMiddleware` registered; all HTTP requests set RLS context from JWT
@@ -270,8 +299,8 @@ This priority combines dead-code deletion and task decomposition because both ar
| MODIFY | `backend/app/main.py` — `app.add_middleware(TenantContextMiddleware)` |
| MODIFY | `backend/app/utils/auth.py` — `create_access_token()` embeds `tenant_id` in JWT claims |
| MODIFY | `backend/app/tasks/pipeline/thumbnail.py`, `extract.py`, `stills.py`, `turntable.py` — `set_tenant_context()` at start |
| CREATE migration | `backend/alembic/versions/063_role_hierarchy.py` — rename `admin` → `global_admin`, add `tenant_admin` |
| MODIFY | All routers using `require_admin()` → `require_global_admin()` |
| DONE | `backend/alembic/versions/049_role_hierarchy.py` — adds `global_admin` and `tenant_admin` |
| PARTIAL | Routers are mixed between new `require_global_admin()` usage and backward-compatible `require_admin()` aliases |
**Acceptance gates:**
- Login as tenant A user → `GET /api/products` returns 0 results when tenant A has no products, even if tenant B has 50
@@ -281,10 +310,12 @@ This priority combines dead-code deletion and task decomposition because both ar
### Priority 9 — Hash-Based Scene Conversion Caching
**Goal:** Skip re-tessellation when the STEP file has not changed. Cache canonical scene + preview derivatives by `SHA256(step_file)`.
**Goal:** Extend the existing STEP-hash plumbing so canonical scene + preview derivatives can skip unnecessary re-tessellation.
**Status:** Partial foundation already exists.
**Milestones:**
- M1: `cad_files.step_hash` column in DB; hash computed and stored on each export
- M1: Existing `cad_files.step_file_hash` is reused or renamed for canonical-scene caching
- M2: Export task checks hash before processing — returns cached asset UUID on hit
- M3: Hash invalidated correctly when admin forces reprocess or deflection settings change
@@ -292,10 +323,10 @@ This priority combines dead-code deletion and task decomposition because both ar
| Action | Path |
|---|---|
| ADD column | `backend/app/domains/products/models.py` — `CadFile.step_hash: str \| None` |
| CREATE migration | `backend/alembic/versions/064_step_hash.py` — `ADD COLUMN step_hash VARCHAR(64)` |
| MODIFY | `backend/app/domains/pipeline/tasks/export_glb.py` (or future USD task) — hash check before subprocess call |
| ADD util | `backend/app/services/step_processor.py` — `compute_step_hash(file_path) -> str` |
| DONE | `backend/app/domains/products/models.py` — `CadFile.step_file_hash: str \| None` already exists |
| DONE | `backend/app/domains/products/cache_service.py` — `compute_step_hash(file_path)` already exists |
| OPEN | `backend/app/domains/pipeline/tasks/export_glb.py` (or future USD task) — hash check before subprocess call |
| OPEN | optional migration to rename `step_file_hash` → `step_hash` only if naming consistency is worth the churn |
**Acceptance gates:**
- Upload same STEP file twice → second task completes in < 2s (cache hit logged: `[CACHE] hash match, skipping tessellation`)
@@ -306,11 +337,13 @@ This priority combines dead-code deletion and task decomposition because both ar
**Goal:** Address independent UI items from `visual-audit-report.md` that require no backend changes.
**Status:** Partial. Some milestones are already shipped and should be removed from the active queue.
**Milestones:**
- M1: Tooltip/help text on every Admin settings input
- M2: Empty state messages in MediaBrowser, ProductLibrary, Orders
- M1: Tooltip/help text on every Admin settings input — mostly done
- M2: Empty state messages in MediaBrowser, ProductLibrary, Orders — partially done
- M3: Notification batching — group per-render noise into job summaries
- M4: Mobile navigation — hamburger menu at < 768px
- M4: Mobile navigation — hamburger menu at < 768px — done
- M5: Kanban rejection flow — drag-to-reject with reason field
**File targets:**
@@ -345,9 +378,8 @@ Priority 1 (Pipeline Cleanup Foundation)
Priority 3 (Tessellation and Topology Quality)
└── Priority 5 (Canonical USD Export and Render Migration)
Priority 7 (Render Job Tracking and Structured Logging) — can run in parallel
Priority 8 (Tenant Isolation Completion) — can run in parallel
Priority 10 (UI/UX Polish) — independent
Priority 8 remaining work (Celery tenant context) — can run in parallel
Priority 10 remaining polish — independent
```
---
@@ -355,19 +387,20 @@ Priority 10 (UI/UX Polish) — independent
## What To Do Next
**Recommended execution path:**
1. Do Priority 1 first: clean up and split the current pipeline.
2. Start Priority 2 immediately after: add `partKey`, assignment-layer semantics, and scene manifest without changing the browser UX.
3. Run Priority 3 in parallel or immediately after, depending on whether current tessellation quality blocks scene-authoring confidence.
4. Use the implementation plan in `docs/plans/0001-step-to-usd-implementation.md` as the execution checklist for the USD workstream.
1. Finish the remaining Priority 1 work first: remove STL-era dead code and split `blender_render.py`.
2. Start Priority 2 immediately after that cleanup baseline is stable: add `partKey`, assignment layers, and scene manifest without changing browser UX.
3. Run Priority 3 in parallel only if cylinder tessellation is actively blocking confidence in seam/sharp payload work; otherwise keep it behind Priority 2.
4. Treat Priority 8 as a short parallel hardening task: add Celery-side tenant context propagation.
5. Use `docs/plans/0001-step-to-usd-implementation.md` as the execution checklist for the USD workstream.
**Parallel sprint option (2 agents):**
- Agent 1: Priority 1 (pipeline cleanup foundation)
- Agent 2: Priority 3 (tessellation and topology quality)
- Agent 1: Priority 1 remainder (dead-code cleanup + `blender_render.py` split)
- Agent 2: Priority 8 remainder or Priority 3, depending on whether tessellation quality is currently blocking work
**Parallel sprint option (3 agents):**
- Agent 1: Priority 1
- Agent 2: Priority 3
- Agent 3: Priority 7 or Priority 8
- Agent 1: Priority 1 remainder
- Agent 2: Priority 2 groundwork (`usd_master`, `part_key_service`, `scene-manifest`)
- Agent 3: Priority 8 remainder or targeted Priority 10 polish
**Do not defer anymore:**
- canonical `partKey`
@@ -383,8 +416,8 @@ These are now considered implementation prerequisites for the long-term refactor
Old planning files are kept for reference but superseded by this document:
- `PLAN.md` — original Phase A–F plan (Phases A–E complete, Phase F = Priority 9 here)
- `PLAN_REFACTOR.md` — 1,173-line architectural plan (Phases 1–8 mapped to Priorities 2–8 above)
- `plan.md` — active GMSH implementation plan (Priority 1)
- `docs/rfcs/0001-step-to-usd-workflow.md` — USD RFC (Priority 10)
- `plan.md` — GMSH tessellation implementation plan (Priority 3)
- `docs/rfcs/0001-step-to-usd-workflow.md` — USD RFC (Priorities 2, 4, and 5)
- `docs/plans/0001-step-to-usd-implementation.md` — actionable USD implementation plan
- `review-report.md` — latest code review results
- `visual-audit-report.md` — UX audit results