[snehabankar-bit]

What

Task annotations, with durations like PT30M or P1DT2H:

• tangleml.com/orchestration/max_pending_duration: from launch until the container starts.
• tangleml.com/orchestration/max_run_duration: from start until the container ends.
• tangleml.com/orchestration/max_pipeline_run_duration (root task): from run creation until the run ends.

A task over its limit is stopped like a cancel and ends as FAILED, with the reason in orchestration_error_message. Downstream tasks are skipped. No new status and no ComponentSpec change.

How

After each status refresh, a node over its limit is flagged desired_state: TIMED_OUT with a reason. The existing cancel code stops the container on the next sweep. A queued task in a run over its limit is failed without launching.

• A user cancel wins and ends CANCELLED.
• A shared container stops only when every node using it is over its limit. Until then, downstream tasks keep waiting.
• A container that already started and waits again is not timed out on pending.
• An invalid value, including a number such as 3600, is ignored with one warning per value.

Timing is approximate

A task stops at its limit, plus up to one status check, plus the launcher's stop grace period. In local tests a PT30S limit stopped tasks at 32-49s: status checks were 1-14s apart depending on load, and Docker waits 10s before killing.

Known limit

On the Kubernetes Job launcher the run clock starts when the Job is accepted, not when the container starts, so max_run_duration includes waiting time there.

Testing

Unit tests in tests/test_orchestrator_sql.py. The suite passes except test_request_id_concurrency, which also fails on master.

End to end on a local Tangle (start_local.py) built from this branch:

• Docker launcher, via Colima.
• Kubernetes Job launcher (Local_Kubernetes_UsingHostPathStorage_KubernetesJobLauncher) on a kind cluster, for the pending case.

#	Case	Result
1	No annotations (Docker, kind)	SUCCEEDED
2	Generous limits (PT10M) on task and root	SUCCEEDED, nothing flagged
3	max_run_duration: PT30S on sleep 300 (Docker, kind)	FAILED with reason, downstream SKIPPED
4	max_pending_duration: PT45S, missing image tag (kind)	pod in ImagePullBackOff, FAILED at ~46s, downstream SKIPPED
5	Root PT1M, a → b → c at 40s each	a SUCCEEDED, b stopped and FAILED, c SKIPPED
6	Root PT10S, orchestrator down for 20s after submit	a FAILED at launch with no container, b SKIPPED
7	Cancel a task that has a limit	CANCELLED, no timeout reason
8	Invalid values soon, 30m, PT0S	runs normally, one warning per value
9	Orchestrator restarted mid-run	limit still enforced
10	Two runs share a container, only one has a run limit	container keeps running for the other run; the limited run's next task FAILED at launch
11	Cached result reused under a PT1S limit	reused, not timed out

Found and fixed along the way: the Docker launcher returns a naive start time, which made the run check raise TypeError (SYSTEM_ERROR). It is now treated as UTC, with a unit test.

Related

Overlaps #292 (one global pending timeout, ending as SYSTEM_ERROR). This one is per task and adds a run limit. Happy to rebase on #292 or the other way round.

🤖 Generated with Claude Code

[Ark-kun]

This changes the DB schema and the DBs would need to be properly migrated. However, we would prefer not to add manual migrations right now as we're in process of adding alembic-based migartions.

Best to reuse the Failed status for now.

[Ark-kun]

Interesting idea.
But let's first implement such experimental features as annotations (see Tangle annotations supported by the Kubernetes Launcher). The ComponentSpec schema has stability and compatibility guarantees. New features that are only supported by some engines are usually added via annotations where possible.

[snehabankar-bit]

Thanks, reworked: the limits are now task annotations (no ExecutionOptionsSpec change) and a timed-out task ends as FAILED, so there is no new status or DB migration.

[snehabankar-bit]

can we add run_timed_out to should_terminate