Hartmut/HartOMat
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
40 Commits 1 Branch 0 Tags
f15b035b8864c57908dc7f4e447eb62c895dd3ba
Commit Graph

30 Commits

Author SHA1 Message Date
Hartmut f15b035b88 feat(L1): modular widget dashboard — 15 configurable widgets 
Replaces monolithic AdminDashboard/ClientDashboard with a per-user
configurable widget grid. 15 widget types: ProductionStats, QueueStatus,
RecentRenders, CostOverview, WorkerStatus, KPISummary, OrderThroughput,
Revenue, ItemStatus, ProcessingTimes, RenderTimeByOutputType,
OutputTypeUsage, TopProducts, OrdersByUser, RenderBackendStats.

DashboardTimeframeContext provides shared timeframe state. Dashboard
config persisted in DB via GET/PUT /api/dashboard/config.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-03-06 23:11:13 +01:00
Hartmut a70cb55d01 feat(N): workflow pipeline, 3D viewer, worker management, QC tests 
- workflow_builder.py: fix broken stubs, add render_order_line_still_task
  (resolves step_path from DB instead of passing order_line_id as step_path)
- domains/rendering/tasks.py: add render_order_line_still_task,
  export_gltf_for_order_line_task, export_blend_for_order_line_task,
  generate_gltf_geometry_task (trimesh STL→GLB, no Blender needed)
- tasks/step_tasks.py: add generate_gltf_geometry_task for CadFile GLB export
- cad router: POST /{id}/generate-gltf-geometry endpoint (admin/PM)
- worker router: GET /celery-workers + POST /scale (docker compose subprocess)
- Dockerfile: pip install -e "[dev]" to enable pytest
- docker-compose.yml: docker socket + compose file mount on backend
- ThreeDViewer.tsx: mode toggle (geometry/production), wireframe, env presets,
  download buttons (GLB + .blend)
- CadPreview.tsx: load gltf_geometry/gltf_production/blend_production assets
  from MediaAsset table and pass URLs to ThreeDViewer
- ProductDetail.tsx: "View 3D" button → /cad/:id, "Generate GLB" button
- media router/service: cad_file_id filter on GET /api/media
- WorkerManagement.tsx: new page with worker status, queue depth, scale controls
- App.tsx + Layout.tsx: /workers route + sidebar link (admin/PM)
- tests: test_rendering_service.py, test_orders_service.py (backend)
- tests: WorkerActivity.test.tsx, WorkerManagement.test.tsx (frontend)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-03-06 22:56:53 +01:00
Hartmut f1e02ded78 feat(F1): wire MinIO STL cache into render_still + render_turntable_to_file 
Previously the cache_service was only used in the generate_stl_cache Celery task.
All render paths (render_still, render_turntable_to_file, render_turntable_task)
only checked for a local file and converted from scratch if missing.

Changes:
- render_blender.py: add _stl_from_cache_or_convert() helper that checks MinIO
  cache before falling back to local STEP→STL conversion. Wire into render_still()
  and render_turntable_to_file() (both STL conversion blocks).
- domains/rendering/tasks.py: wire MinIO cache check into render_turntable_task()
  inline before convert_step_to_stl(). All errors are non-fatal (falls back to
  fresh conversion).

Now a STEP file converted on one worker is available to all workers via MinIO,
avoiding redundant cadquery conversions on re-renders.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-03-06 21:54:03 +01:00
Hartmut bfc0050580 feat(L+M): configurable dashboard widget system + test framework 
Phase L: Dashboard widget system
- Migration 046: dashboard_configs table (user/tenant/role fallback cascade)
- DashboardConfig model + dashboard_service with get/upsert per-user and tenant-default
- API router: GET/PUT /api/dashboard/config, GET/PUT /api/dashboard/tenant-default
- Frontend: 5 widget components (ProductionStats, QueueStatus, RecentRenders, CostOverview, WorkerStatus)
- DashboardGrid with API-backed config, DashboardCustomizeModal (checkbox selection, save/cancel)
- Dashboard.tsx: widget grid + analytics section (privileged users)
- Admin.tsx: restructured with new section order and Maintenance 2-col grid

Phase M: Test framework
- Backend: pytest-asyncio + pytest-cov + factory-boy in pyproject.toml
- conftest.py: excel_dir fixtures + async DB fixtures + mock storage/celery stubs
- Domain tests: billing_service, cache_service, notifications_service, imports_sanity
- Integration: test_api_health.py smoke test (requires running backend)
- Frontend: vitest + @testing-library/react + msw added to package.json
- vite.config.ts: test block (jsdom + globals + setupFiles)
- tsconfig.json: exclude src/__tests__ from main tsc (test runner handles its own types)
- MSW handlers for /api/auth/me, Billing.test.tsx, formatters.test.ts

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-03-06 21:50:07 +01:00
Hartmut 71f8df67f3 fix(cache): pass Path object to storage.upload in store_stl_cache 
storage.upload() expects Path, not str — str has no .name attribute,
causing 'str object has no attribute name' warnings on every STL cache write.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-03-06 21:35:10 +01:00
Hartmut ced64055f2 refactor(admin): remove Pillow as selectable renderer, restructure admin page 
- Backend: VALID_RENDERERS = {"blender"} only; remove pillow from renderer-status response
- Frontend: remove renderer picker (pillow/blender buttons) — Blender is the only renderer
- Blender options always visible, grouped into Render Quality / Performance / Output sections
- Maintenance buttons in 2-column grid with descriptions
- Page reorder: Pricing Summary → Users → Blender Settings → Render Templates →
  Asset Libraries → Output Types → Pricing Tiers → SMTP → Templates
- Pillow code kept internally as fallback (not exposed in UI)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-03-06 21:20:28 +01:00
Hartmut fef12743a7 fix(K): asset library catalog refresh — fix task binding and model imports 
- fix(tasks): @shared_task → @celery_app.task so task uses configured redis://redis broker
  (@shared_task without explicit app bound to default app → localhost:6379 → Connection refused)
- fix(tasks): import app.models before DB access so SQLAlchemy can resolve Material.creator → User
  relationship (was: InvalidRequestError: 'User' failed to locate a name)

Verified: catalog refresh now finds and stores 35 materials from uploaded .blend file.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-03-06 21:09:14 +01:00
Hartmut 3ac3ca1d70 fix(K): correct CATALOG_SCRIPT path + register AssetLibrary in models/__init__ 
- fix(tasks): use RENDER_SCRIPTS_DIR env var for catalog_assets.py path
  (was computing wrong path via __file__ parents → /render-worker/scripts/ which doesn't exist in container)
- fix(models): add AssetLibrary to app/models/__init__.py so alembic autogenerate discovers it
- fix(api): remove unused FileResponse import from asset_libraries.py

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-03-06 21:02:41 +01:00
Hartmut a18d4c23ec feat(K): Blender Asset Library + production exports (GLB + .blend) 
- feat(migration): 045_asset_libraries — new asset_libraries table (blend_file_path, catalog JSONB)
- feat(model): AssetLibrary SQLAlchemy model in domains/materials/models.py
- feat(api): POST/GET/PATCH/DELETE /api/asset-libraries + /upload-blend + /refresh-catalog endpoints
- feat(celery): refresh_asset_library_catalog task on thumbnail_rendering queue — runs Blender headless
- feat(blender): catalog_assets.py — extracts asset-marked materials + node_groups from .blend
- feat(blender): asset_library.py — apply_asset_library_materials + apply_asset_library_node_groups helpers
- feat(blender): export_gltf.py — STEP→STL→GLB production export with optional asset library
- feat(blender): export_blend.py — STEP→STL→.blend production export with pack_all()
- feat(frontend): api/assetLibraries.ts — full CRUD API client
- feat(frontend): AssetLibraryPanel in Admin.tsx — upload, refresh, expand catalog view
- docs: Blender asset_data marking requirement learning in LEARNINGS.md

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-03-06 20:56:26 +01:00
Hartmut 7a1329958d feat(J): WebSocket live-events + replace polling + fix ffmpeg turntable timeout 
- fix(render): ffmpeg overlay=0:0 -> overlay=0:0:shortest=1 to prevent hang on finite PNG sequences
- feat(ws): add core/websocket.py ConnectionManager + Redis Pub/Sub subscriber loop
- feat(ws): add /api/ws WebSocket endpoint with JWT query-param auth in main.py
- feat(ws): emit render_complete/failed + cad_processing_complete events from step_tasks.py
- feat(ws): emit order_status_change events from orders router
- feat(ws): add beat_tasks.py broadcast_queue_status task (every 10s via Redis __broadcast__)
- feat(frontend): add useWebSocket hook with auto-reconnect (exponential backoff, 25s ping)
- feat(frontend): add WebSocketContext + WebSocketProvider wrapping App
- refactor(frontend): remove polling from WorkerActivity (was 5s/3s) + OrderDetail (was 5s)
- refactor(frontend): reduce polling in Layout (8s->60s) + NotificationCenter (15s->60s)
- docs: add ffmpeg shortest=1 + WebSocket JWT auth learnings to LEARNINGS.md

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-03-06 20:49:34 +01:00
Hartmut ceb0143cb6 fix: animation output types now render turntable mp4 via Celery 
- render_order_line_task: detect is_animation flag, branch to turntable
  pipeline instead of still-image pipeline
- render_blender: add render_turntable_to_file() — STL conversion,
  blender turntable_render.py for frames, ffmpeg compose to mp4
- render-worker/Dockerfile: add ffmpeg package
- Fix: Turntable Video_White was producing a .jpg still instead of .mp4

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-03-06 20:10:33 +01:00
Hartmut d138bc4bc4 feat(activity): merge CAD + render jobs into unified timeline 
- Single chronological list sorted by updated_at (newest first)
- Type badges distinguish CAD processing (FileCode2) from Render jobs (Image)
- Render job rows now link directly to the order (/orders/:id)
- Remove separate "Render Jobs" and "CAD File Processing" sections
- Stat cards simplified to 4: CAD active/failed + Render active/failed
- Backend: add order_id to RenderJobEntry response
- Frontend: add order_id to RenderJobEntry interface, remove flamenco_job_id

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-03-06 20:01:03 +01:00
Hartmut ab3f9c734a fix: render pipeline + multi-tenancy bugs (B-Fix-1 through B-Fix-9) 
- Remove worker-thumbnail (no Blender, was competing on thumbnail_rendering)
- Move render_order_line_task to thumbnail_rendering queue (render-worker)
- Restore template_service.py real implementation (fix circular import shim)
- Thread tenant_id through STEP upload, Excel import, product create
- Make system tables (output_types, materials, etc.) tenant_id nullable
- Fix tenants frontend 307-redirect: use trailing slash /tenants/
- Remove Flamenco + Three.js from Admin UI (unsupported)
- Set all output_types render_backend to celery (was flamenco)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-03-06 19:34:20 +01:00
Hartmut 381f44bc8b feat: render health endpoint + test script + pipeline fixes 
- GET /api/worker/health/render: checks render-worker (thumbnail_rendering
  queue), Blender availability via active_queues inspect, queue depth,
  last render recency — returns ok/degraded/down status
- scripts/test_render_pipeline.py: integration test for full pipeline
  (--health, --sample, --full modes)
- PLAN.md: appended Render Pipeline Fixes section with all B-Fixes
- LEARNINGS.md: documented 5 new learnings (queue mismatch, circular
  import, 307 redirect, worker capability detection)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-03-06 19:34:12 +01:00
Hartmut 979b0082ec fix(api): remove 307 redirect on GET /api/workflows 
Change list/create route paths from '/' to '' on workflow_router so
GET /api/workflows and POST /api/workflows respond directly without
307 redirect. The redirect was causing axios to drop the Authorization
header, making the WorkflowEditor always show an empty list.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-03-06 18:26:27 +01:00
Hartmut c0ea60d984 fix: resolve open risks — invoice race condition, SMTP config, workflow seeds 
- billing/service.py: pg_advisory_xact_lock on invoice_number_seq per year
  → prevents duplicate INV-YYYY-NNNN under concurrent requests
- admin.py: SMTP settings in system_settings (smtp_host/port/user/password/
  from_address/enabled) with GET+PUT support; seed-workflows endpoint creates
  4 standard workflow definitions (still-cycles, still-eevee, turntable,
  multi-angle) idempotently
- notifications/service.py: send_email_notification_stub now sends real
  SMTP email via smtplib when smtp_enabled=true in system_settings
- Admin.tsx: SMTP settings panel (host/port/user/password/from + enable
  toggle, save button); Seed Standard Workflows maintenance button
- Upload.tsx: fix TS error — title→aria-label on Lucide icons
- Admin.tsx Settings type: add render_backend/flamenco_* fields (TS fix)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-03-06 18:15:45 +01:00
Hartmut f19a6ccde8 feat(F-G-H-I): STL cache, invoices, import validation, notification settings 
Phase F — STL Hash Cache:
- Migration 041: step_file_hash column on cad_files
- cache_service.py: SHA256 hash + MinIO-backed STL cache (check/store)
- render_step_thumbnail: compute+persist hash before render
- generate_stl_cache: check MinIO cache before cadquery conversion, store after

Phase G — Invoices:
- Migration 042: invoices + invoice_lines tables with RLS
- Invoice/InvoiceLine models + schemas
- billing service: generate_invoice_number (INV-YYYY-NNNN), create/list/get/delete/PDF
- WeasyPrint PDF generation; backend Dockerfile + pyproject.toml deps
- invoice_router with 6 endpoints; registered in main.py
- frontend: Billing.tsx page + api/billing.ts; route + nav link

Phase H — Import Sanity Check:
- Migration 043: import_validations table
- ImportValidation model + schemas
- run_sanity_check: material fuzzy-match (cutoff=0.8), STEP availability, duplicate detection
- validate_excel_import Celery task (queue: step_processing)
- uploads.py: create ImportValidation on /excel, fire task, expose GET /validations/{id}
- frontend: Upload.tsx polling ValidationDialog with Ampel status indicators

Phase I — Notification Settings:
- Migration 044: notification_configs table (user×event×channel toggles)
- NotificationConfig model + seeds (in_app=true, email=false)
- get/upsert/reset config endpoints on /notifications/config
- frontend: NotificationSettings.tsx page + api/notifications.ts extensions

Infrastructure:
- docker-compose.yml: add worker-thumbnail service (concurrency=1, Q=thumbnail_rendering)
- Fix Dockerfile: libgdk-pixbuf-2.0-0 (correct Debian bookworm package name)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-03-06 18:05:01 +01:00
Hartmut 7706c514c8 fix(deploy): fix render-worker build context + migration 040 idempotency 
- docker-compose.yml: change render-worker build context from ./render-worker
  to . (project root) so pyproject.toml is accessible; update dockerfile path
- render-worker/Dockerfile: update COPY paths for new build context;
  install Python 3.11 via deadsnakes PPA (Ubuntu 22.04 ships 3.10 which
  fails the >=3.11 requirement in pyproject.toml)
- 040_media_assets.py: rewrite upgrade() with raw idempotent SQL (CREATE TYPE
  inside DO $$ EXCEPTION WHEN duplicate_object $$; CREATE TABLE IF NOT EXISTS;
  CREATE INDEX IF NOT EXISTS) to handle pre-existing enum from partial runs

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-03-06 17:32:42 +01:00
Hartmut c74e118b98 feat(E): add MediaAsset catalog — model, CRUD API, MediaBrowser UI 
Migration 040: media_assets table with RLS (tenant_isolation + admin_bypass).
domains/media/: MediaAsset model, schemas, service, router with ZIP-download.
publish_asset Celery task in rendering/tasks.py.
core/storage.py: download_bytes() method for MinIO + LocalStorage.
frontend: MediaBrowser.tsx (grid/list, multi-select, zip-download, pagination) + api/media.ts.
Route /media (AdminRoute) + sidebar link with Image icon for admin+pm.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-03-06 17:11:17 +01:00
Hartmut 716451ff76 feat(D): OCC mesh attribute extraction + Blender smooth shading integration 
Migration 039: cad_files.mesh_attributes JSONB column.
domains/products/tasks.py: extract_mesh_attributes Celery task using pythonOCC.
still_render.py + turntable_render.py: _apply_mesh_attributes() sets auto-smooth
based on curved_ratio and topology threshold from OCC analysis.
render_blender.py: passes --mesh-attributes JSON arg to Blender subprocess.
render_still_task: loads mesh_attributes from DB and passes to renderer.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-03-06 17:07:55 +01:00
Hartmut 7e47e4aca7 feat(C1+C2): workflow system — WorkflowDefinition + Celery Canvas builder 
Migrations 037 (workflow tables + 3 seed definitions) + 038 (output_types.workflow_definition_id).
WorkflowDefinition/Run/NodeResult SQLAlchemy models in domains/rendering/models.py.
workflow_builder.py: dispatch_workflow() with Celery Canvas for still/turntable/multi_angle.
workflow_router.py: CRUD endpoints at /api/workflows (admin/PM guards).
dispatch_service.py: dispatch_render_with_workflow() prefers workflow path when
  OutputType.workflow_definition_id is set, falls back to legacy dispatch otherwise.
main.py: registers workflows_router.
models/__init__.py: re-exports WorkflowDefinition, WorkflowRun, WorkflowNodeResult.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-03-06 17:07:21 +01:00
Hartmut 251dd703ed feat(B2): add tenant model + migrations 035/036 + RLS policies 
Migration 035: tenants table with 'Schaeffler' default seed.
Migration 036: tenant_id FK on all tables, RLS policies, backfill.
New domains/tenants/ with CRUD router (admin only).
All domain models extended with tenant_id FK.
core/database.py: get_db_for_tenant with RLS context setter.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-03-06 16:30:41 +01:00
Hartmut b87df4a3e5 refactor(B1): migrate to domain-driven project structure 
Move all models/schemas/services/routers into app/domains/.
Keep backward-compat shims in old locations for imports.
Preserves domains/rendering/tasks.py from Phase A.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-03-06 16:24:11 +01:00
Hartmut 5764118b8e feat(A5): add app_config table + typed config service 
- Migration 034: creates app_config table with JSONB columns (render, storage,
  worker, notifications, billing); migrates existing system_settings values
- backend/app/core/config_service.py:
  - Typed Pydantic models: RenderConfig, WorkerConfig, StorageConfig, etc.
  - AppConfig aggregate model
  - get_app_config(db) async + get_app_config_sync() for Celery tasks
  - update_render_config() / update_worker_config() for partial updates
- system_settings table preserved for backward compat during Phase B migration

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-03-06 15:51:56 +01:00
Hartmut c728358fb6 feat(A4): add MinIO service + storage abstraction (core/storage.py) 
- Add MinIO service to docker-compose.yml (port 9000 API, 9001 console)
- Add minio-data volume for persistent object storage
- Create backend/app/core/storage.py: MinIOStorage + LocalStorage abstraction
  - MinIOStorage: boto3-based, auto-creates bucket, upload/download/exists/delete/presign
  - LocalStorage: fallback for dev (UPLOAD_DIR filesystem, backward compat)
  - get_storage() singleton: auto-selects based on MINIO_URL env var
- Add MINIO_URL/USER/PASSWORD/BUCKET env vars to all service definitions
- backend/pyproject.toml: docker>=6.1.0 → boto3>=1.34.0
- Add docker-compose.worker.yml: external render-worker for remote machines
- Fix .gitignore: 'core' rule was too broad, now only matches root /core dump
- Update .env.example: MinIO connection vars documented

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-03-06 15:51:06 +01:00
Hartmut c8ecc29d40 refactor(A3): remove threejs-renderer service, add migration 033 
- Migration 033: reset threejs thumbnail_renderer setting → blender,
  remove obsolete threejs_render_size system_setting
- step_processor.py threejs code paths already fall through to Pillow
  (committed in A2); Three.js browser viewer (ThreeDViewer.tsx) remains
- threejs-renderer/ directory kept for reference but no longer built

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-03-06 15:49:16 +01:00
Hartmut 9d1a820295 refactor(A2): replace blender-renderer HTTP service with render-worker Celery container 
- Create render-worker/ with Dockerfile (Ubuntu + cadquery + Blender via host mount)
- Add render-worker/check_version.py: verifies Blender >= 5.0.1 at startup, Exit 1 on failure
- Add render-worker/scripts/: blender_render.py, still_render.py, turntable_render.py
- Create backend/app/services/render_blender.py: direct subprocess rendering
  - convert_step_to_stl() and export_per_part_stls() using cadquery
  - render_still(): STEP → STL → PNG via Blender subprocess
  - is_blender_available(): detects BLENDER_BIN env for render-worker context
- Create backend/app/domains/rendering/tasks.py: render_still_task + render_turntable_task
- Update step_processor.py: use subprocess path when BLENDER_BIN env is set (render-worker)
- Update step_tasks.py: generate_stl_cache uses direct cadquery instead of HTTP
- Remove blender-renderer and threejs-renderer from docker-compose.yml
- Replace worker-thumbnail with render-worker (Ubuntu + cadquery + Blender mount)
- Remove Docker SDK from backend Dockerfile (was only for flamenco scaling)
- Update .env.example: BLENDER_VERSION=5.0.1 documented
- Update celery_app.py: include domains.rendering.tasks in autodiscover

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-03-06 15:48:46 +01:00
Hartmut 1d6864fb64 refactor(A1): remove Flamenco, simplify render pipeline to Celery-only 
- Remove flamenco-manager and flamenco-worker from docker-compose.yml
- Delete flamenco_client.py, flamenco_tasks.py, docker_scaler.py
- Simplify render_dispatcher.py to Celery-only (removes ~300 lines)
- Remove Flamenco beat schedule from celery_app.py
- Clean admin.py: remove flamenco settings, endpoints, threejs validation
- Clean orders.py cancel-render: Celery revoke only
- Clean worker.py: remove flamenco_job_id from activity response
- Migration 032: cancel lingering flamenco jobs, remove flamenco settings
- PLAN.md: mark all decisions confirmed, status IN UMSETZUNG

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-03-06 15:38:37 +01:00
Hartmut 552922eb8a chore: remove build artifacts from tracking, extend .gitignore 2026-03-05 22:13:21 +01:00
Hartmut bce762a783 feat: initial commit 2026-03-05 22:12:38 +01:00