Hartmut/HartOMat
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: cinematic highlight render — 20s procedural camera animation

Browse Source 
New render type: 4-segment cinematic camera animation (480 frames @ 24fps)
for professional product highlight videos.

Camera sequence:
1. Establishing (5s): slow 45° orbit + push-in, 50mm lens
2. Detail sweep (5s): low-angle close arc, 85mm telephoto, shallow DOF
3. Crane up (5s): rising 30°→60°, 35mm wide reveal, pull-back
4. Hero close (5s): push-in to beauty angle, 65mm, smooth ease-out

Technical:
- cinematic_render.py: procedural camera from bounding sphere, cubic easing,
  per-frame keyframes (location, rotation, focal length, DOF)
- render_cinematic_to_file(): service function (same pattern as turntable)
- Pipeline routing: render_settings.cinematic flag → cinematic path
- Depth of field enabled (f-stop scales with product size)
- use_persistent_data for BVH caching between frames
- Same material/template/USD pipeline as turntable

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Hartmut Nörenberg
2026-03-15 21:25:56 +01:00
parent c82f2a894d
commit f22b963be9
4 changed files with 1210 additions and 105 deletions
Download Patch File Download Diff File Expand all files Collapse all files
plan.md
+69 -104
View File
@@ -1,122 +1,87 @@
# Plan: Rich Product Metadata Extraction from STEP Files
# Plan: Cinematic Highlight Render
## Context
The AI chat agent was asked "What is the biggest product from my order?" and couldn't answer because dimensional data wasn't available in tool results. While `cad_files.mesh_attributes` already stores bounding box dimensions, much more metadata is extractable from STEP files via OCC that would make the AI agent and the product library significantly more useful.
Add a new render type: a 20-second cinematic product highlight video with procedural multi-shot camera animation. Unlike turntable (constant orbit), this has varying camera distances, focal lengths, elevations, and bezier-eased transitions between 4 segments.
**Currently extracted**: part names, bounding box (xyz), sharp edges, smooth angle
**Available but not extracted**: per-part volume, surface area, assembly hierarchy, instance counts, embedded colors, triangle counts, geometric complexity
## Tasks
**Goal**: Expand the STEP metadata extraction to compute richer product characteristics and store them in a structured `cad_metadata` JSONB field, accessible to the AI agent, product search, and frontend.
### [ ] Task 1: Cinematic Blender script
## Affected Files
- **File**: `render-worker/scripts/cinematic_render.py` (new)
- **What**: Blender Python script that:
1. Imports product geometry (GLB or USD, same as turntable_render.py)
2. Applies materials (same pipeline as turntable)
3. Computes bounding sphere from imported meshes
4. Creates a procedural 4-segment camera animation (480 frames @ 24fps = 20s):
- Segment 1 (0-120): Establishing — slow 45° orbit + push-in, 50mm lens
- Segment 2 (121-240): Detail sweep — low-angle close arc, 85mm lens, shallow DOF
- Segment 3 (241-360): Crane up — rising from 30° to 60° elevation, 35mm wide
- Segment 4 (361-480): Hero close — slow push-in, 65mm, ease-out to still
5. Each segment: camera position from spherical coords (azimuth, elevation, distance), bezier interpolation
6. Depth of field enabled (f-stop scales with product size)
7. Renders all frames as PNG to temp directory
8. FFmpeg assembles frames → MP4 (H.264, yuv420p, 24fps)
| File | Change |
|------|--------|
| `backend/app/services/step_processor.py` | Expand `extract_step_metadata()` with volume, surface area, hierarchy, complexity |
| `backend/app/domains/products/models.py` | Add `cad_metadata` JSONB column to Product |
| `backend/alembic/versions/XXX_add_cad_metadata.py` | Migration |
| `backend/app/domains/pipeline/tasks/extract_metadata.py` | Populate `cad_metadata` after STEP processing |
| `backend/app/domains/products/schemas.py` | Expose `cad_metadata` in ProductOut |
| `backend/app/services/chat_service.py` | Include metadata in search_products and system prompt |
| `frontend/src/pages/ProductDetail.tsx` | Display rich metadata (volume, part count, complexity) |
## Tasks (in order)
### [ ] Task 1: Expand STEP metadata extraction
- **File**: `backend/app/services/step_processor.py`
- **What**: Expand `extract_step_metadata()` to compute additional properties after the existing bbox/edge extraction. Add a new function `extract_rich_metadata(doc, shape_tool)` that returns:
```python
{
"part_count": 42, # Number of leaf parts
"assembly_depth": 3, # Max nesting depth
"total_volume_cm3": 1250.4, # Sum of all part volumes (cm³)
"total_surface_area_cm2": 3400.2, # Sum of all surface areas (cm²)
"total_triangle_count": 45000, # After tessellation
"total_vertex_count": 23000, # After tessellation
"largest_part": { # Part with largest volume
"name": "OuterRing",
"volume_cm3": 450.2,
},
"smallest_dimension_mm": 0.5, # Smallest bbox dimension across all parts
"instance_count": 36, # Total instances (parts may repeat)
"unique_part_count": 12, # Distinct shapes
"complexity_score": "high", # low/medium/high based on triangle count
}
CLI args (same pattern as turntable_render.py):
```
Use OCC:
- `GProp_GProps` + `BRepGProp.VolumeProperties()` for volume
- `BRepGProp.SurfaceProperties()` for surface area
- `Poly_Triangulation` for triangle/vertex counts (after tessellation)
- Assembly tree walk (already done in `_collect_part_key_map`) for hierarchy depth + instance count
- **Acceptance gate**: `extract_rich_metadata()` returns all fields for a test STEP file
blender --background --python cinematic_render.py -- \
<glb_path> <output_mp4> <width> <height> <engine> <samples> \
<smooth_angle> <cycles_device> <transparent_bg> <template_path> \
<target_collection> <material_library_path> <material_map_json> \
<part_names_ordered_json> <lighting_only> <shadow_catcher> \
<rotation_x> <rotation_y> <rotation_z> \
[--usd-path ...] [--focal-length ...] [--material-override ...]
```
- **Acceptance gate**: Script renders a 20s MP4 from a STEP file
- **Dependencies**: None
### [ ] Task 2: Add cad_metadata column to Product model
### [ ] Task 2: Render service function
- **File**: `backend/app/domains/products/models.py`
- **What**: Add `cad_metadata: Mapped[dict | None] = mapped_column(JSONB, nullable=True, default=None)` to the Product model. This stores the rich metadata at the product level (not cad_file) because products are the user-facing entity.
- **Migration**: `alembic revision --autogenerate -m "add cad_metadata to products"`
- **Also**: Add to ProductOut schema in `backend/app/domains/products/schemas.py`
- **Acceptance gate**: Column exists, schema includes it
- **Dependencies**: None
- **File**: `backend/app/services/render_blender.py`
- **What**: Add `render_cinematic_to_file()` function with same signature as `render_turntable_to_file()` but:
- Calls `cinematic_render.py` instead of `turntable_render.py`
- Fixed 24fps, 480 frames (20s)
- Output format: mp4
- Passes all the same material/template/position args
- **Acceptance gate**: Function callable, builds correct subprocess command
- **Dependencies**: Task 1
### [ ] Task 3: Populate cad_metadata during STEP processing
### [ ] Task 3: Pipeline integration
- **File**: `backend/app/domains/pipeline/tasks/extract_metadata.py`
- **What**: After `process_step_file` extracts objects and queues thumbnail, call `extract_rich_metadata()` and store the result on the Product's `cad_metadata` field. Also store it on `cad_files.mesh_attributes` (merge with existing data).
- **Also**: Add a "reextract metadata" admin action that re-runs this for all existing products
- **Acceptance gate**: After STEP processing, product.cad_metadata is populated with volume, part_count, etc.
- **Dependencies**: Tasks 1, 2
### [ ] Task 4: Expose metadata in AI agent tools
- **File**: `backend/app/services/chat_service.py`
- **What**:
1. Update `_tool_search_products()` to include `cad_metadata` fields (part_count, total_volume_cm3, complexity_score) in results
2. Update `query_database` tool description to mention `products.cad_metadata` JSONB field
3. Update system prompt to mention available metadata
- **Acceptance gate**: AI agent can answer "What is the biggest product?" using volume data
- **Dependencies**: Task 3
### [ ] Task 5: Display rich metadata on ProductDetail page
- **File**: `frontend/src/pages/ProductDetail.tsx`
- **What**: Add a "CAD Metadata" section on the product detail page showing:
- Part count + unique parts + instances
- Total volume (cm³) + surface area (cm²)
- Largest part name + volume
- Complexity score badge (low/medium/high)
- Triangle/vertex count
- Assembly depth
- **Acceptance gate**: Metadata displayed on product page; empty gracefully when not available
- **File**: `backend/app/domains/pipeline/tasks/render_order_line.py`
- **What**: In the render task, detect when output type has a cinematic flag. Add a check:
- If `output_type.render_settings.get("cinematic")` is True → call `render_cinematic_to_file()` instead of `render_turntable_to_file()`
- OR: if output_type name contains "Cinematic" → route to cinematic
- **Acceptance gate**: Order line with cinematic output type renders via the new script
- **Dependencies**: Task 2
### [ ] Task 6: Batch re-extract metadata for existing products
### [ ] Task 4: Output type + test
- **File**: `backend/app/api/routers/admin.py`
- **What**: Add a "Re-extract Rich Metadata" button in System Tools that queues a Celery task to re-process all completed STEP files and populate `cad_metadata` for all products.
- **Acceptance gate**: Button triggers batch job; existing products get metadata populated
- **Dependencies**: Tasks 1, 3
- **What**: Create the "Cinematic Highlight" output type via API:
```json
{
"name": "Cinematic Highlight",
"renderer": "blender",
"output_format": "mp4",
"render_backend": "celery",
"is_animation": true,
"transparent_bg": false,
"cycles_device": "gpu",
"render_settings": {
"width": 1920,
"height": 1080,
"engine": "cycles",
"samples": 128,
"cinematic": true,
"fps": 24,
"frame_count": 480
}
}
```
Link to BlenderStudio template. Test with a real product.
- **Acceptance gate**: A cinematic MP4 renders successfully for a TRB product
- **Dependencies**: Task 3
## Migration Check
**Yes** — one new JSONB column on `products` table.
## Order Recommendation
1. Task 1 (extraction logic) + Task 2 (model + migration) — parallel
2. Task 3 (wire up in pipeline)
3. Task 4 (AI agent) + Task 5 (frontend) — parallel
4. Task 6 (batch re-extract)
## Risks / Open Questions
1. **Volume calculation accuracy**: OCC `BRepGProp` computes exact B-rep volume, not mesh-based. This is accurate but can be slow for very complex shapes. Cap at 5s per file.
2. **Performance**: Rich metadata extraction adds ~100-500ms per STEP file. This is acceptable since STEP processing already takes 1-5s.
3. **Existing products**: ~45 products with STEP files need backfill. Task 6 handles this.
4. **Triangle count varies**: Depends on tessellation settings (deflection angles). Store the count at the current tessellation quality for reference, with a note that it's approximate.
**No** — no DB changes needed.