refactor: rename thumbnail_rendering queue to asset_pipeline

The queue handles far more than thumbnails: OCC tessellation, USD master
generation, GLB production, order line renders, and workflow renders.
asset_pipeline better reflects its role as the render-worker's primary queue.

Updated all references in: task decorators, celery_app.py, beat_tasks.py,
docker-compose.yml worker command, worker.py MONITORED_QUEUES, admin.py,
CLAUDE.md, LEARNINGS.md, Dockerfile, helpTexts.ts, test files,
and all .claude/commands/*.md skill files.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
2026-03-12 22:28:38 +01:00
parent e7b70a35ea
commit 1321ef2bd4
39 changed files with 540 additions and 122 deletions
+6 -6
View File
@@ -15,7 +15,7 @@ Schaeffler Automat is a working Blender-based media production pipeline with:
- 7 Docker services with GPU render-worker
- PostgreSQL with tenant_id columns + Row Level Security (RLS) enabled but inconsistently
applied at the application layer
- Celery task queues with two workers (step_processing + thumbnail_rendering)
- Celery task queues with two workers (step_processing + asset_pipeline)
- WebSocket real-time events via Redis Pub/Sub
- React/Vite frontend with workflow editor (ReactFlow), media browser, notifications
@@ -584,7 +584,7 @@ internal teams), RLS + tenant_id partitioning is sufficient.
- Each tenant gets own PostgreSQL schema (not separate DB) with schema-based routing
- Shared MinIO with per-tenant bucket policies
- Separate Redis database (0-15) per tenant (max 16 tenants)
- Celery routing: per-tenant queue prefix `{tenant_slug}.thumbnail_rendering`
- Celery routing: per-tenant queue prefix `{tenant_slug}.asset_pipeline`
### 4.4 Per-Tenant Feature Flags
@@ -751,7 +751,7 @@ export const HELP_TEXTS: Record<string, HelpText> = {
},
"action.regenerate_thumbnails": {
title: "Regenerate All Thumbnails",
body: "Re-renders thumbnails for all STEP files using current settings. This queues all files on the thumbnail_rendering worker. Expected time: N × 30s. Only needed after changing renderer settings.",
body: "Re-renders thumbnails for all STEP files using current settings. This queues all files on the asset_pipeline worker. Expected time: N × 30s. Only needed after changing renderer settings.",
warning: "This will queue a large number of tasks. Only run during off-peak hours.",
},
// ... all settings
@@ -889,7 +889,7 @@ rejection UI. `rejected_at` column exists but there is no rejection reason field
### 8.1 Current Concurrency Controls
- `worker` (step_processing): `CELERY_WORKER_CONCURRENCY` env var, default 8
- `render-worker` (thumbnail_rendering): hardcoded 1 (Blender serial access)
- `render-worker` (asset_pipeline): hardcoded 1 (Blender serial access)
- Both require Docker service restart to change concurrency
### 8.2 Dynamic Worker Scaling
@@ -901,7 +901,7 @@ Use Celery's built-in `autoscale` option:
render-worker:
command: celery -A app.tasks.celery_app worker
--loglevel=info
-Q thumbnail_rendering
-Q asset_pipeline
--autoscale=1,1 # min=1, max=1 (single Blender concurrency)
--concurrency=1
```
@@ -984,7 +984,7 @@ After Phase 2 decomposition, update `celery_app.conf.update(task_routes={...})`:
```python
task_routes = {
"app.domains.pipeline.tasks.*": {"queue": "step_processing"},
"app.domains.rendering.tasks.*": {"queue": "thumbnail_rendering"},
"app.domains.rendering.tasks.*": {"queue": "asset_pipeline"},
"app.domains.media.tasks.*": {"queue": "step_processing"},
"app.tasks.ai_tasks.*": {"queue": "ai_validation"},
"app.tasks.beat_tasks.*": {"queue": "step_processing"},