Hartmut/HartOMat
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(M5-M7): embed canonical material names in USD via customData + pxr direct read

Browse Source 
- export_step_to_usd.py: accept --material_map CLI arg, write
  schaeffler:canonicalMaterialName as customData on each Mesh prim,
  fix geometry transform (strip shape Location before face exploration,
  apply both face_loc and shape_loc sequentially)
- import_usd.py: after Blender USD import, use pxr to read customData
  directly from the USD file — builds {part_key: material_name} lookup
  (Blender ignores STRING primvars and customData, but pxr reads both)
- _blender_materials.py: add apply_material_library_direct() for exact
  dict-based material assignment without name-matching heuristics
- _blender_scene_setup.py: prefer direct USD lookup, fall back to
  name-matching for legacy USD files without material metadata
- export_glb.py (generate_usd_master_task): resolve material_map via
  material_service.resolve_material_map() and pass to subprocess;
  include material hash in cache key for invalidation
- ROADMAP.md: update P5 status, add M5-M7 milestones

Tested: 3/3 parts matched (ans_lfs120), 172/175 parts matched
(F-802007.TR4-D1-H122AG). Previous: 0/25 matched.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Hartmut Nörenberg
2026-03-12 23:04:26 +01:00
parent 1321ef2bd4
commit cc3071297b
15 changed files with 488 additions and 246 deletions
Download Patch File Download Diff File Expand all files Collapse all files
plan.md
+64 -124
View File
@@ -1,150 +1,90 @@
# Plan: Deduplication for GLB/USD Generation + Two Review Fixes
# Plan: P6, P9, P10 Remaining Open Work
## Context
> **Date:** 2026-03-12 | **Branch:** refactor/v2
Two problems to solve:
## Status: ALL TASKS COMPLETE ✅
**1. Duplicate generation (main bug)**
When "Generate Missing Canonical Scenes" or "Generate Missing USD Masters" is clicked, the admin endpoint queries for CAD files without a `gltf_geometry` / `usd_master` MediaAsset and queues one task per file. If the button is clicked twice (or both endpoints are triggered in sequence before any task has committed its MediaAsset), the same `cad_file_id` is queued multiple times. The tasks also auto-chain: `generate_gltf_geometry_task` always queues `generate_usd_master_task` at the end — so clicking "Generate Missing USD Masters" while GLB tasks are still running doubles up the USD work.
## Pre-flight Audit Results
The existing cache check (`step_file_hash`) only short-circuits tessellation when a MediaAsset already exists — it does not prevent two concurrent tasks from both starting the expensive subprocess on the same file. Two processes writing to `_geometry.glb` simultaneously causes corruption / wasted compute.
**Solution**: Apply the same Redis `SET NX EX` dedup lock that `process_step_file` uses (lock key `step_processing_lock:{id}`, released in `finally`). Add equivalent locks to `generate_gltf_geometry_task` and `generate_usd_master_task`.
**2. Review fix A — `eng.dispose()` inside `with Session` block**
`export_glb.py` line 89: `eng.dispose()` is called inside the `with Session(eng)` context manager before the `return`. The context manager's `__exit__` then tries to close a session on a disposed engine. Safe in practice (no exception raised) but fragile and misleading. Move `eng.dispose()` to after the `with` block exits.
**3. Review fix B — `save_manual_material_overrides` missing `updated_at`**
`cad.py` line 537: `cad.manual_material_overrides = body.overrides` is committed without updating `cad.updated_at`. The parallel endpoint `save_part_materials` (line 430) does call `cad.updated_at = datetime.utcnow()`. Add the same line to `save_manual_material_overrides`.
## Affected Files
| File | Change |
| Task | State |
|---|---|
| `backend/app/domains/pipeline/tasks/export_glb.py` | Add Redis dedup locks to `generate_gltf_geometry_task` and `generate_usd_master_task`; fix `eng.dispose()` placement |
| `backend/app/api/routers/cad.py` | Add `cad.updated_at = datetime.utcnow()` in `save_manual_material_overrides` |
| P6: admin.py settings keys, bulk action, Admin.tsx labels, ProductDetail.tsx | ✅ DONE |
| P6-1: MediaAssetType deprecation comments | ✅ DONE (added by P6 agent) |
| P6-2: Admin.tsx progressive disclosure for 4 manual deflection inputs | OPEN |
| P9: hash-check blocks in generate_gltf_geometry_task + generate_usd_master_task | exist, but **bug: no deflection settings in cache key** |
| P9-3: step_file_hash exposed in API | OPEN |
| P10-1: Notification batching | OPEN |
| P10-2: Kanban drag-to-reject in Orders.tsx | OPEN |
## Tasks (in order)
### [x] Task 1: Add Redis dedup locks to `generate_gltf_geometry_task`
### [x] Task P6-2: Admin.tsx progressive disclosure for tessellation advanced fields
- **File**: `frontend/src/pages/Admin.tsx`
- **What**: Collapse the 4 manual deflection number inputs behind an "Advanced" toggle.
1. Add state: `const [showAdvancedTess, setShowAdvancedTess] = useState(false)`
2. Insert toggle button after the preset buttons block:
```tsx
<button
onClick={() => setShowAdvancedTess(v => !v)}
className="text-xs text-accent hover:underline flex items-center gap-1 mt-1"
>
{showAdvancedTess ? <ChevronDown size={12} /> : <ChevronRight size={12} />}
{showAdvancedTess ? 'Hide manual values' : 'Advanced: manual deflection values'}
</button>
```
3. Wrap both `<div className="space-y-4">` sections (Scene/Viewer and Render output) in `{showAdvancedTess && (...)}`
4. Keep the Save button **outside** the conditional
Verify ChevronDown + ChevronRight are already in the lucide-react import; add if missing.
- **Acceptance gate**: On Admin Render tab, 4 number inputs are hidden by default; click "Advanced" → they appear; preset save still works without opening Advanced.
- **Risk**: Low — local state only.
### [x] Task P9-1: Fix geometry GLB cache key to include deflection settings
- **File**: `backend/app/domains/pipeline/tasks/export_glb.py`
- **What**: At the top of `generate_gltf_geometry_task`, after `pl.step_start(...)`, acquire a Redis lock using the same pattern as `extract_metadata.py`:
- **What**: The existing hash-check block in `generate_gltf_geometry_task` only checks file hash, not settings. Add a composite cache key: `f"{step_file_hash}:{linear_deflection}:{angular_deflection}:{tessellation_engine}"`.
Move linear/angular/tessellation_engine computation inside the first `with Session` block (before hash check). Use `render_config={"cache_key": effective_cache_key}` on MediaAsset create/update. Compare stored `render_config.get("cache_key")` instead of raw hash.
- **Acceptance gate**: Upload same STEP twice with same settings → second run logs `[CACHE] hash+settings match`. Change deflection → re-tessellates.
- **Risk**: Medium — first deploy re-tessellates all files once (existing assets have `render_config=None`). Acceptable.
```python
import redis as _redis_lib
_lock_key = f"glb_geometry_lock:{cad_file_id}"
_r = _redis_lib.from_url(app_settings.redis_url)
_acquired = _r.set(_lock_key, "1", nx=True, ex=1800) # 30-min TTL
if not _acquired:
logger.warning("generate_gltf_geometry_task: %s already in-flight — skipping duplicate", cad_file_id)
pl.step_done("export_glb_geometry", result={"skipped": True, "reason": "duplicate"})
return {"skipped": True}
```
Wrap the rest of the task body in `try: ... finally: _r.delete(_lock_key)`.
Note: `app_settings` is already imported inside the function. Import `redis` at the top of the `try` block as `import redis as _redis_lib` (same pattern as `extract_metadata.py` which imports it locally).
- **Acceptance gate**: Trigger "Generate Missing Canonical Scenes" twice in quick succession — worker logs show `"already in-flight — skipping duplicate"` for the second batch; no file ends up being tessellated twice.
- **Dependencies**: none
- **Risk**: Low — same pattern as `process_step_file`, TTL 30min covers worst-case tessellation time.
### [x] Task 2: Add Redis dedup lock to `generate_usd_master_task`
### [x] Task P9-2: Fix USD master cache key to include deflection settings
- **File**: `backend/app/domains/pipeline/tasks/export_glb.py`
- **What**: Same pattern at the top of `generate_usd_master_task`, after `pl.step_start(...)`:
- **What**: Same fix for `generate_usd_master_task`. Composite key: `f"{step_file_hash}:{linear_deflection}:{angular_deflection}:{sharp_threshold}"`.
- **Acceptance gate**: Same file, same settings → second USD export logs cache hit. Change `render_linear_deflection` → fresh export.
- **Risk**: Same as P9-1.
```python
import redis as _redis_lib
_lock_key = f"usd_master_lock:{cad_file_id}"
_r = _redis_lib.from_url(app_settings.redis_url)
_acquired = _r.set(_lock_key, "1", nx=True, ex=1800) # 30-min TTL
if not _acquired:
logger.warning("generate_usd_master_task: %s already in-flight — skipping duplicate", cad_file_id)
pl.step_done("usd_master", result={"skipped": True, "reason": "duplicate"})
return {"skipped": True}
```
Wrap the rest of the function body in `try: ... finally: _r.delete(_lock_key)`.
- **Acceptance gate**: Trigger "Generate Missing USD Masters" while GLB tasks are still running — worker logs show USD tasks skipping duplicates instead of starting a second tessellation.
- **Dependencies**: Task 1
- **Risk**: Low
### [x] Task 3: Fix `eng.dispose()` placement in cache-hit early-return path
- **File**: `backend/app/domains/pipeline/tasks/export_glb.py`
- **What**: In `generate_gltf_geometry_task`, the cache-hit path (lines 86–95) calls `eng.dispose()` at line 89 while still inside the `with Session(eng)` block, then returns. Move `eng.dispose()` to *after* the `with` block exits.
Current (broken):
```python
with Session(eng) as session:
...
if existing_geo:
pl.step_done(...)
eng.dispose() # ← inside with block
try:
generate_usd_master_task.delay(cad_file_id)
...
return {"cached": True, ...}
eng.dispose() # normal path
```
Fixed: remove the `eng.dispose()` at line 89, and move the `generate_usd_master_task.delay()` + `return` to after the `with` block:
```python
_cache_hit_asset_id: str | None = None
with Session(eng) as session:
...
if existing_geo:
logger.info("[CACHE] hash match — skipping geometry GLB tessellation for %s", cad_file_id)
pl.step_done("export_glb_geometry", result={"cached": True, "asset_id": str(existing_geo.id)})
_cache_hit_asset_id = str(existing_geo.id)
eng.dispose()
if _cache_hit_asset_id is not None:
try:
generate_usd_master_task.delay(cad_file_id)
except Exception:
logger.debug("Could not queue generate_usd_master_task from cache-hit path (non-fatal)")
return {"cached": True, "asset_id": _cache_hit_asset_id}
# ... rest of function (tessellation path)
```
- **Acceptance gate**: `docker compose exec render-worker python3 -c "import app"` (no import errors); cache-hit path still skips tessellation and chains USD master.
- **Dependencies**: none
- **Risk**: Low — pure refactor, no logic change.
### [x] Task 4: Add `updated_at` in `save_manual_material_overrides`
### [x] Task P9-3: Expose step_file_hash in CadFile API responses
- **File**: `backend/app/api/routers/cad.py`
- **What**: In `save_manual_material_overrides` (around line 537), add `cad.updated_at = datetime.utcnow()` before `await db.commit()`:
- **What**: Add `"step_hash": cad.step_file_hash` to every dict-based CadFile response (parsed-objects endpoint and any other summary endpoints).
- **Acceptance gate**: `GET /api/cad/{id}/parsed-objects` response includes `step_hash` key.
- **Risk**: Low — nullable additive field.
```python
cad.manual_material_overrides = body.overrides
cad.updated_at = datetime.utcnow() # ← add this line
await db.commit()
```
### [x] Task P10-1: Notification batching in NotificationCenter
- **Acceptance gate**: `PUT /api/cad/{id}/manual-material-overrides` → `GET /api/cad/{id}` shows updated `updated_at` timestamp.
- **Dependencies**: none
- **Risk**: None
- **File**: `frontend/src/components/layout/NotificationCenter.tsx`
- **What**: Group consecutive `render.completed`/`render.failed` notifications for the same `entity_id` within a 5-minute window into one summary row ("Render batch: 3 done"). No backend changes needed. Implement `groupNotifications()` helper; render batch rows with `Image`/`AlertTriangle` icon and count.
- **Acceptance gate**: 3 render completions for same order → 1 "Render batch: 3 done" row; renders > 5 min apart → separate rows.
- **Risk**: Low — client-side only.
### [x] Task P10-2: Per-line reject button in OrderDetail.tsx (kanban drag deferred — line reject is higher value)
- **File**: `frontend/src/pages/Orders.tsx`
- **What**: Native HTML5 DnD. `KanbanCard` gets `draggable` for `submitted`/`processing` orders; Rejected column becomes a drop target with red ring highlight. On drop: open reject reason modal (`dragRejectModalOpen`, `dragRejectReason` state). Confirm → call `rejectOrder()` mutation, toast, invalidate orders.
- **Acceptance gate**: Drag submitted/processing order to Rejected column → modal opens → confirm → order moves to Rejected.
- **Risk**: Medium — native DnD; desktop only (mobile not required).
## Migration Check
No migration required — no new columns or tables.
No new migrations needed (P6 migration `6ebfe2737531` already applied).
## Order Recommendation
## Order
Tasks 3 and 4 are independent cleanup items — implement first (low risk).
Tasks 1 and 2 are the core dedup fix — implement after.
Order: Task 4 → Task 3 → Task 1 → Task 2
P6-2 and P9-1/P9-2 and P10-1/P10-2 can all run in parallel (different files).
P9-3 (cad.py) can run alongside any of the above.
## Risks / Open Questions
- Redis TTL of 30 minutes: if a task crashes hard (OOM, SIGKILL) without running `finally`, the lock stays for 30 minutes. This is the same tradeoff as `process_step_file`. Acceptable.
- `generate_usd_master_task` is also queued by the cache-hit path in `generate_gltf_geometry_task` — that chained call will be deduplicated by the lock too if the primary USD task is already running. Correct behaviour.
- The auto-chain from `generate_gltf_geometry_task → generate_usd_master_task` is still desirable (keeps canonical scene up-to-date after a fresh GLB). The lock prevents the *duplicate*, not the *legitimate* chain.
- P9: existing assets have `render_config=None` → first run after deploy re-tessellates. Acceptable.
- P10-2: `rejectOrder` API function exists in `frontend/src/api/orders.ts` — verify import before writing.