diff --git a/ROADMAP.md b/ROADMAP.md index 3561d79..f8a526c 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -10,7 +10,7 @@ | Area | Detail | |---|---| -| Phase A | Flamenco removed, blender-renderer → render-worker Celery container, threejs-renderer removed, MinIO added | +| Phase A runtime | Flamenco removed from the active pipeline, `render-worker` replaced the old Blender service path, MinIO added; legacy `blender-renderer/` and `threejs-renderer/` directories still remain in the repo | | Phase B | Domain-driven project structure, Tenant model + RLS migrations 035/036, Tenant management UI | | Phase C | WorkflowDefinition model, standard workflows seeded, React Flow Workflow Editor | | Phase D | OCC mesh attributes extraction, Blender integration | @@ -20,6 +20,26 @@ | Media cache-bust | `?v={file_size_bytes}` in download URLs + `Cache-Control: no-cache` headers | | GPU activation order | Fix: `_activate_gpu()` called before AND after `open_mainfile` to survive engine reset | | Material system | Aliases-first lookup, `get_material_library_path()` via AssetLibrary | +| Pipeline task split | `backend/app/tasks/step_tasks.py` is now a 23-line compatibility shim; active task implementations live in `backend/app/domains/pipeline/tasks/` | +| Render job tracking | `RenderJobDocument`, `PipelineLogger`, and cancel-via-real-`celery_task_id` are already wired into the render pipeline | +| Tenant isolation baseline | `TenantContextMiddleware`, JWT `tenant_id`, and the `global_admin` / `tenant_admin` role hierarchy are in place for HTTP requests | +| Hash groundwork | `compute_step_hash()` exists and `CadFile.step_file_hash` is already persisted during thumbnail processing | + +--- + +## 🔎 Status Snapshot + +Verified against the repository on `2026-03-11`. + +| Priority | Status | Re-evaluated state | +|---|---|---| +| 1. Pipeline Cleanup Foundation | In progress | `step_tasks.py` decomposition is done; dead-code cleanup and `blender_render.py` decomposition are still open | +| 2. USD Foundation Without Viewer Regression | Not started in code | Decisions are documented, but there is no `export_step_to_usd.py`, `usd_master`, `scene-manifest`, or `partKey` implementation yet | +| 3. Tessellation and Topology Quality | Not started in code | No `gmsh` install/wiring, no `tessellation_engine` setting, no Admin dropdown yet | +| 7. Render Job Tracking and Structured Logging | Done | `RenderJobDocument`, migration `048`, `PipelineLogger`, and revoke-by-real-task-id are present | +| 8. Tenant Isolation Completion | In progress | HTTP-side RLS context is wired; Celery task-side `set_tenant_context()` propagation still needs to be added | +| 9. Hash-Based Scene Conversion Caching | Partial foundation | Existing `step_file_hash` and STL-cache utilities should be extended, not rebuilt from scratch | +| 10. UI/UX Polish | Partial | Admin help tooltips, mobile nav, and some empty states exist; notification batching and remaining polish items are still open | --- @@ -41,9 +61,11 @@ That removes the old assumption that USD work must wait for a Three.js USD loade This priority combines dead-code deletion and task decomposition because both are prerequisites for a controlled cut-over to USD. +**Status:** In progress. M2 is complete; M1 and M3 remain open. + **Milestones:** - M1: Dead code deleted — Pillow block, STL settings, orphaned directories -- M2: `step_tasks.py` decomposed into `backend/app/tasks/pipeline/` submodules +- M2: `step_tasks.py` decomposed into `backend/app/domains/pipeline/tasks/` submodules - M3: `blender_render.py` decomposed into `render-worker/scripts/_blender_*.py` submodules **File targets:** @@ -55,13 +77,13 @@ This priority combines dead-code deletion and task decomposition because both ar | DELETE | `threejs-renderer/` directory | | DELETE | `flamenco/` directory | | DELETE | `renderproblems_tmp/` | -| DELETE lines 705–1050 | `backend/app/tasks/step_tasks.py` (`render_order_line_task` — duplicates `rendering/tasks`) | +| DONE | `backend/app/tasks/step_tasks.py` — reduced to compatibility shim only | | REMOVE settings | `admin.py`: `VALID_STL_QUALITIES`, `stl_quality`, `generate-missing-stls` endpoint | | REMOVE endpoint | `cad.py`: `POST /cad/{id}/generate-stl/{quality}` | -| CREATE | `backend/app/tasks/pipeline/extract.py` — metadata extraction (OCC parsing, < 2s) | -| CREATE | `backend/app/tasks/pipeline/thumbnail.py` — Blender thumbnail render | -| CREATE | `backend/app/tasks/pipeline/stills.py` — still render task | -| CREATE | `backend/app/tasks/pipeline/turntable.py` — turntable render task | +| DONE | `backend/app/domains/pipeline/tasks/extract_metadata.py` — metadata extraction | +| DONE | `backend/app/domains/pipeline/tasks/render_thumbnail.py` — thumbnail render task | +| DONE (combined) | `backend/app/domains/pipeline/tasks/render_order_line.py` — still/dispatch pipeline entry | +| OPEN | turntable-specific pipeline task split still needs to be carved out explicitly if kept as a separate concern | | THIN (< 80 lines) | `backend/app/tasks/step_tasks.py` — dispatch only | | CREATE | `render-worker/scripts/_blender_gpu.py` | | CREATE | `render-worker/scripts/_blender_import.py` | @@ -81,6 +103,8 @@ This priority combines dead-code deletion and task decomposition because both ar **Goal:** Introduce canonical part identity and the three-layer material assignment model while keeping the current GLB-based browser UX working end-to-end. +**Status:** Not started in code. Architecture decisions are documented, but repo work has not begun. + **Milestones:** - M1: `export_step_to_usd.py` produces valid USD with part hierarchy and `schaeffler:partKey` on every prim - M2: `usd_master` MediaAsset type exists in DB and is stored after each export @@ -106,8 +130,7 @@ This priority combines dead-code deletion and task decomposition because both ar | CREATE | `frontend/src/api/sceneManifest.ts` — `SceneManifest` interface, `fetchSceneManifest()` | **Open questions to decide before M1:** -- USD authoring library: `pxr` (OpenUSD full Python SDK) vs. plain USDA text templating vs. `usd-core` pip package -- seam/sharp payload encoding: custom primvars (`primvars:schaeffler:seamEdgeVertexPairs`) or a separate JSON sidecar +- None blocking at the architecture level. The roadmap decisions for `usd-core` and index-space seam/sharp primvars are already captured in `docs/plans/0001-step-to-usd-implementation.md`. **Acceptance gates:** - `python3 export_step_to_usd.py --step_path 81113-l_cut.stp` → valid `.usd` file, 25 part prims, each has `schaeffler:partKey` attribute @@ -124,6 +147,8 @@ This priority combines dead-code deletion and task decomposition because both ar **Goal:** Eliminate fan triangles on cylindrical surfaces (rings, bearings) and produce clean seams for UV unwrap. +**Status:** Not started in code. This is still a pure planning workstream at the moment. + **Milestones:** - M1: GMSH 4.15+ installed in render-worker container - M2: `export_step_to_gltf.py --tessellation_engine gmsh` produces fan-free GLB @@ -232,6 +257,8 @@ This priority combines dead-code deletion and task decomposition because both ar **Goal:** Fix broken render job cancellation (synthetic `render-{line_id}` ID never matches real Celery task ID) and establish structured per-step logging. +**Status:** Done, aside from any follow-up polish. + **Milestones:** - M1: `RenderJobDocument` schema + migration; tasks write real `self.request.id` to DB - M2: Cancel endpoint reads `celery_task_id` from job doc and calls `revoke()` — actually stops task @@ -243,7 +270,7 @@ This priority combines dead-code deletion and task decomposition because both ar |---|---| | CREATE | `backend/app/domains/rendering/job_document.py` — `RenderJobDocument` Pydantic model, `update_step()`, `set_state()` | | CREATE | `backend/app/core/pipeline_logger.py` — `PipelineLogger(step_start/done/error)` writing to logging + Redis SSE | -| CREATE migration | `backend/alembic/versions/062_render_job_document.py` — add `render_job_doc JSONB` to `order_lines` | +| DONE | `backend/alembic/versions/048_render_job_document.py` — adds `render_job_doc JSONB` to `order_lines` | | MODIFY | `backend/app/domains/pipeline/tasks/render_order_line.py` — write `celery_task_id` + step events to job doc | | MODIFY | `backend/app/domains/pipeline/tasks/render_thumbnail.py` — same | | MODIFY | `backend/app/api/routers/orders.py` — cancel reads `render_job_doc.celery_task_id`, calls `celery.control.revoke()` | @@ -255,7 +282,9 @@ This priority combines dead-code deletion and task decomposition because both ar ### Priority 8 — Tenant Isolation Completion -**Goal:** Make PostgreSQL RLS enforcement real. Currently `build_tenant_db_dep()` yields `db` without calling `SET LOCAL app.current_tenant_id`, making all tenant isolation a silent no-op. +**Goal:** Finish tenant isolation hardening, especially for non-HTTP execution paths. + +**Status:** In progress. HTTP-side RLS enforcement is now real; task-side propagation is the remaining gap. **Milestones:** - M1: `TenantContextMiddleware` registered; all HTTP requests set RLS context from JWT @@ -270,8 +299,8 @@ This priority combines dead-code deletion and task decomposition because both ar | MODIFY | `backend/app/main.py` — `app.add_middleware(TenantContextMiddleware)` | | MODIFY | `backend/app/utils/auth.py` — `create_access_token()` embeds `tenant_id` in JWT claims | | MODIFY | `backend/app/tasks/pipeline/thumbnail.py`, `extract.py`, `stills.py`, `turntable.py` — `set_tenant_context()` at start | -| CREATE migration | `backend/alembic/versions/063_role_hierarchy.py` — rename `admin` → `global_admin`, add `tenant_admin` | -| MODIFY | All routers using `require_admin()` → `require_global_admin()` | +| DONE | `backend/alembic/versions/049_role_hierarchy.py` — adds `global_admin` and `tenant_admin` | +| PARTIAL | Routers are mixed between new `require_global_admin()` usage and backward-compatible `require_admin()` aliases | **Acceptance gates:** - Login as tenant A user → `GET /api/products` returns 0 results when tenant A has no products, even if tenant B has 50 @@ -281,10 +310,12 @@ This priority combines dead-code deletion and task decomposition because both ar ### Priority 9 — Hash-Based Scene Conversion Caching -**Goal:** Skip re-tessellation when the STEP file has not changed. Cache canonical scene + preview derivatives by `SHA256(step_file)`. +**Goal:** Extend the existing STEP-hash plumbing so canonical scene + preview derivatives can skip unnecessary re-tessellation. + +**Status:** Partial foundation already exists. **Milestones:** -- M1: `cad_files.step_hash` column in DB; hash computed and stored on each export +- M1: Existing `cad_files.step_file_hash` is reused or renamed for canonical-scene caching - M2: Export task checks hash before processing — returns cached asset UUID on hit - M3: Hash invalidated correctly when admin forces reprocess or deflection settings change @@ -292,10 +323,10 @@ This priority combines dead-code deletion and task decomposition because both ar | Action | Path | |---|---| -| ADD column | `backend/app/domains/products/models.py` — `CadFile.step_hash: str \| None` | -| CREATE migration | `backend/alembic/versions/064_step_hash.py` — `ADD COLUMN step_hash VARCHAR(64)` | -| MODIFY | `backend/app/domains/pipeline/tasks/export_glb.py` (or future USD task) — hash check before subprocess call | -| ADD util | `backend/app/services/step_processor.py` — `compute_step_hash(file_path) -> str` | +| DONE | `backend/app/domains/products/models.py` — `CadFile.step_file_hash: str \| None` already exists | +| DONE | `backend/app/domains/products/cache_service.py` — `compute_step_hash(file_path)` already exists | +| OPEN | `backend/app/domains/pipeline/tasks/export_glb.py` (or future USD task) — hash check before subprocess call | +| OPEN | optional migration to rename `step_file_hash` → `step_hash` only if naming consistency is worth the churn | **Acceptance gates:** - Upload same STEP file twice → second task completes in < 2s (cache hit logged: `[CACHE] hash match, skipping tessellation`) @@ -306,11 +337,13 @@ This priority combines dead-code deletion and task decomposition because both ar **Goal:** Address independent UI items from `visual-audit-report.md` that require no backend changes. +**Status:** Partial. Some milestones are already shipped and should be removed from the active queue. + **Milestones:** -- M1: Tooltip/help text on every Admin settings input -- M2: Empty state messages in MediaBrowser, ProductLibrary, Orders +- M1: Tooltip/help text on every Admin settings input — mostly done +- M2: Empty state messages in MediaBrowser, ProductLibrary, Orders — partially done - M3: Notification batching — group per-render noise into job summaries -- M4: Mobile navigation — hamburger menu at < 768px +- M4: Mobile navigation — hamburger menu at < 768px — done - M5: Kanban rejection flow — drag-to-reject with reason field **File targets:** @@ -345,9 +378,8 @@ Priority 1 (Pipeline Cleanup Foundation) Priority 3 (Tessellation and Topology Quality) └── Priority 5 (Canonical USD Export and Render Migration) -Priority 7 (Render Job Tracking and Structured Logging) — can run in parallel -Priority 8 (Tenant Isolation Completion) — can run in parallel -Priority 10 (UI/UX Polish) — independent +Priority 8 remaining work (Celery tenant context) — can run in parallel +Priority 10 remaining polish — independent ``` --- @@ -355,19 +387,20 @@ Priority 10 (UI/UX Polish) — independent ## What To Do Next **Recommended execution path:** -1. Do Priority 1 first: clean up and split the current pipeline. -2. Start Priority 2 immediately after: add `partKey`, assignment-layer semantics, and scene manifest without changing the browser UX. -3. Run Priority 3 in parallel or immediately after, depending on whether current tessellation quality blocks scene-authoring confidence. -4. Use the implementation plan in `docs/plans/0001-step-to-usd-implementation.md` as the execution checklist for the USD workstream. +1. Finish the remaining Priority 1 work first: remove STL-era dead code and split `blender_render.py`. +2. Start Priority 2 immediately after that cleanup baseline is stable: add `partKey`, assignment layers, and scene manifest without changing browser UX. +3. Run Priority 3 in parallel only if cylinder tessellation is actively blocking confidence in seam/sharp payload work; otherwise keep it behind Priority 2. +4. Treat Priority 8 as a short parallel hardening task: add Celery-side tenant context propagation. +5. Use `docs/plans/0001-step-to-usd-implementation.md` as the execution checklist for the USD workstream. **Parallel sprint option (2 agents):** -- Agent 1: Priority 1 (pipeline cleanup foundation) -- Agent 2: Priority 3 (tessellation and topology quality) +- Agent 1: Priority 1 remainder (dead-code cleanup + `blender_render.py` split) +- Agent 2: Priority 8 remainder or Priority 3, depending on whether tessellation quality is currently blocking work **Parallel sprint option (3 agents):** -- Agent 1: Priority 1 -- Agent 2: Priority 3 -- Agent 3: Priority 7 or Priority 8 +- Agent 1: Priority 1 remainder +- Agent 2: Priority 2 groundwork (`usd_master`, `part_key_service`, `scene-manifest`) +- Agent 3: Priority 8 remainder or targeted Priority 10 polish **Do not defer anymore:** - canonical `partKey` @@ -383,8 +416,8 @@ These are now considered implementation prerequisites for the long-term refactor Old planning files are kept for reference but superseded by this document: - `PLAN.md` — original Phase A–F plan (Phases A–E complete, Phase F = Priority 9 here) - `PLAN_REFACTOR.md` — 1,173-line architectural plan (Phases 1–8 mapped to Priorities 2–8 above) -- `plan.md` — active GMSH implementation plan (Priority 1) -- `docs/rfcs/0001-step-to-usd-workflow.md` — USD RFC (Priority 10) +- `plan.md` — GMSH tessellation implementation plan (Priority 3) +- `docs/rfcs/0001-step-to-usd-workflow.md` — USD RFC (Priorities 2, 4, and 5) - `docs/plans/0001-step-to-usd-implementation.md` — actionable USD implementation plan - `review-report.md` — latest code review results - `visual-audit-report.md` — UX audit results diff --git a/docs/plans/0001-step-to-usd-implementation.md b/docs/plans/0001-step-to-usd-implementation.md index fa354a6..46c9f1d 100644 --- a/docs/plans/0001-step-to-usd-implementation.md +++ b/docs/plans/0001-step-to-usd-implementation.md @@ -8,12 +8,21 @@ ## Prerequisites -- [ ] Priority 1 complete (step_tasks.py decomposed, blender_render.py decomposed) +- [x] `step_tasks.py` decomposition is already done; `backend/app/tasks/step_tasks.py` is now a compatibility shim over `backend/app/domains/pipeline/tasks/*` +- [ ] `blender_render.py` decomposition is still pending; current file remains monolithic +- [ ] Legacy STL-era cleanup is still pending (`stl_quality`, STL endpoints, orphaned directories) - [x] Decision: USD authoring library → **`usd-core` (pip)** — provides `pxr` module, no GPU tools needed, pip-installable in render-worker - [x] Decision: seam/sharp payload encoding → **index-space primvars** (`primvars:schaeffler:seamEdgeVertexPairs`, `primvars:schaeffler:sharpEdgeVertexPairs`) — survives transforms, no KD-tree needed - [x] Decision: preview GLB derivation → **co-author from same tessellation pass** during migration (avoid round-trip loss from USD→GLB export) - [x] Decision: single-file vs override layers → **Option B: canonical geometry layer + material override layer, flattened via `UsdUtils.FlattenLayerStack()` for delivery** — preserves hierarchy AND allows instancing later (`FlattenLayerStack` keeps `instanceable` prims; `UsdStage.Flatten` would expand them). Note: Phase 1 uses no instancing (matching current GLB pipeline), but the delivery path is already instancing-safe. +## Current Baseline + +- No USD exporter exists yet in `render-worker/scripts/` +- No `usd_master` asset type or scene-manifest endpoint exists yet in backend/frontend code +- No `partKey` payload exists yet in the GLB export/viewer contract +- No `gmsh` tessellation wiring exists yet; Phase 1 should assume current OCC tessellation unless Priority 3 is pulled forward + --- ## Phase 1 — Dual-Write USD Beside GLB (Priority 2, M1–M2)