HartOMat/docs/workflows/NODE_BASED_PRODUCTION_ARCHITECTURE.md

Node-Based Production Architecture

Purpose

Define the target model for a reusable, node-based production system where workflow steps are backend-owned modules, the editor is schema-driven, and legacy rendering stays operational during migration.

Problem Statement

The current workflow system already has meaningful extraction work:

• bridge/runtime services exist for setup, template resolution, material mapping, bbox resolution, publish, and notify
• graph execution can already orchestrate still, turntable, and blend export flows
• the editor already consumes backend node definitions

What is still missing is a clean production model.

Today, three different concerns are still partially collapsed into each other:

1. OutputType as user-facing commercial/render choice
2. workflow graph as orchestration definition
3. legacy/internal render settings as implicit execution contract

That makes it hard to:

• reuse a process step like CAD import as a true module
• expose all node settings coherently in the editor
• bind output types to workflows without fragile implicit assumptions
• preserve legacy behavior while enabling graph-native production

Target Model

1. Production Module

A production module is the canonical backend capability unit.

Examples:

• cad.resolve_step_path
• cad.extract_objects
• cad.export_glb
• cad.compute_bbox
• materials.resolve_map
• materials.auto_populate
• render.resolve_template
• render.blender_still
• render.blender_turntable
• output.publish_asset
• output.notify

Rules:

• modules are backend-owned
• modules define typed input contract, output contract, defaults, and execution semantics
• modules are reusable from legacy code, graph runtime, shadow mode, and tests
• modules must not depend on editor-only UI metadata

2. Workflow Node

A workflow node is an orchestration wrapper around a production module.

It adds:

• node id
• graph connectivity
• per-instance parameter overrides
• editor UI metadata
• retry/failure policy

It must not redefine business behavior that belongs to the production module itself.

Implication:

• the node registry should evolve from "palette metadata + field definitions" to "module-backed node definitions"
• step should remain the stable runtime key, but internally map to a reusable module contract

3. Workflow Family

Workflows must be separated into runtime families, not inferred ad hoc:

• cad_file
• order_line

Family drives:

• valid entry context
• allowed node palette
• validation rules
• available output contracts
• preflight expectations

Mixed-family graphs may still exist temporarily for migration visibility, but must not be the target authoring model.

4. Output Type as Invocation Profile

OutputType should no longer be treated as a loose bag of renderer flags.

It should be the product-facing invocation profile for a workflow:

• commercial name and visibility
• compatible categories
• pricing tier binding
• workflow family
• linked workflow definition
• invocation-level parameter overrides
• output artifact contract

Examples of invocation-level overrides:

• resolution
• samples
• engine
• transparency
• animation timing
• material override

Examples of artifact contract:

• still image
• turntable video
• production .blend
• preview thumbnail
• future exported package types

This keeps the responsibility split clean:

• workflow definition answers: "what steps run and in what order?"
• output type answers: "what productized variant of that workflow do we sell and with which defaults/constraints?"

Required Refactor Direction

A. Formalize Node Contracts

Extend the node registry so each definition exposes:

• family
• module_key
• input_contract
• output_contract
• param_schema
• artifact_roles_produced
• artifact_roles_consumed
• legacy_source

Current definitions already cover labels, categories, defaults, and fields. They do not yet fully express machine-usable production contracts.

B. Promote Runtime Services to Module Layer

The extracted bridge/runtime services are the right foundation. They now need a clear module boundary so both legacy and graph runtimes call the same backend operation layer.

Desired shape:

• legacy pipeline calls module layer directly
• graph runtime calls module layer directly or via async task adapters
• Celery task mapping becomes transport/adaptation, not the primary execution model

C. Split Graph Authoring by Family

The editor should author against family-scoped graphs:

• CAD Intake graph
• Order Rendering graph

That includes:

• family-specific starter templates
• family-specific node palette groups
• validation that rejects wrong-family entry nodes early
• cleaner organization than a single mixed library

D. Reframe Output Type Creation

Output type creation is currently too close to legacy render settings and too far from workflow invocation.

Create/edit flow should become:

1. choose family
2. choose or create workflow
3. choose artifact kind
4. set invocation overrides
5. bind pricing/category/material constraints

The current workflow_definition_id field is directionally correct, but too weak on its own because there is no explicit invocation contract or family validation around it yet.

Compatibility Rules

Legacy Safety

• legacy dispatch remains the fallback path
• existing output types without workflow linkage remain valid
• graph rollout must be opt-in per output type/workflow

Migration Safety

• old output types may continue to store render settings in their current shape
• a compatibility adapter should map legacy render settings into invocation overrides
• workflow definitions must remain canonical JSON DAGs

Recommended Implementation Sequence

Phase A: Stabilize Broken Contracts

• align frontend/backend OutputType defaults and allowed values
• add backend validation for output-type family/workflow compatibility
• make output type creation/editing reflect current real backend constraints

Phase B: Contract-First Registry

• add family, contracts, and module metadata to node definitions
• expose them over /api/workflows/node-definitions
• move editor grouping/validation to registry-owned family metadata

Phase C: Invocation Profiles

• extend OutputType into a workflow invocation profile
• add explicit artifact kind and workflow family
• separate invocation overrides from raw render settings

Phase D: Module Unification

• route legacy and graph execution through the same module layer
• keep Celery as transport where async work is needed
• reduce duplicate logic in tasks and runtime adapters

Phase E: Full Parity Authoring

• ship family-specific starter workflows
• expose all module settings in editor
• support end-to-end preflight, dispatch, run inspection, and parity verification

Immediate Code Implications

• workflow_node_registry.py is the correct extension point for module contracts
• workflow_schema.py will need stronger family- and contract-aware validation
• OutputType needs a clearer model than raw renderer/backend defaults plus optional workflow id
• editor UX should follow model cleanup, not lead it

Decision

We should simplify and refactor before doing more isolated workflow-editor UX work.

The next implementation blocks should prioritize:

1. fixing the output-type/workflow contract
2. formalizing node/module contracts
3. only then expanding editor affordances on top of the cleaned model