[fal.ai/livepeer-staging] Media input loop fails with TransferEncodingError 400 — 'Not enough data to satisfy transfer length header'

[livepeer-tessa]

Summary

The media input loop and control channel subscription both crash with a TransferEncodingError: 400 when the trickle connection to the orchestrator fails mid-stream. This causes the worker to attempt to cleanly disconnect but hit ConnectionRefusedError trying to reach the orchestrator host.

Errors

2026-04-02 00:08:00,317 - scope.cloud.livepeer_app - ERROR - Media input loop failed: Response payload is not completed: <TransferEncodingError: 400, message='Not enough data to satisfy transfer length header.'>

2026-04-02 00:08:01,605 - scope.cloud.livepeer_app - ERROR - Control channel subscription error: Trickle JSONL subscription error: ClientPayloadError: Response payload is not completed: <TransferEncodingError: 400, message='Not enough data to satisfy transfer length header.'>

Then cascading connection errors during cleanup:

aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host 34.169.235.70:8935 ssl:default [Connect call failed ('34.169.235.70', 8935)]
ConnectionRefusedError: [Errno 111] Connect call failed ('34.169.235.70', 8935)

Log Context

• Timestamp: 2026-04-02 at 00:07:59 – 00:08:02 UTC
• fal_job_id: 89db37ff-*
• Orchestrator host: 34.169.235.70:8935 (direct IP, not staging-1.daydream.monster)
• Also seen earlier: 2026-04-01 23:03:55 – 23:03:57 UTC

Relationship to Existing Issues

• Issue [fal.ai/livepeer-staging] Media input loop fails with EOFError on stream disconnect — no graceful teardown #771 covers a similar loop failure but from EOFError on clean disconnect
• This is a distinct failure mode: the orchestrator truncates the transfer mid-stream (HTTP 400 + incomplete transfer encoding), then goes fully unreachable

Impact

Worker terminates the session uncleanly. The trickle DELETE calls also fail with ClientConnectorError, suggesting the orchestrator went down or was restarted while the session was active.

Possible Fix

In livepeer_app.py (and trickle_subscriber.py):

• Treat TransferEncodingError as a network-level disconnect, not an application error — catch it alongside EOFError in the input loop error handling, and do a graceful shutdown instead of propagating as an error
• Don't attempt trickle DELETE/POST cleanup if the host is already unreachable (pre-check or suppress ClientConnectorError during teardown)

[livepeer-tessa]

Now also on production orchestrator — 2026-04-15 21:06–21:48 UTC

Same TransferEncodingError pattern now occurring on the prod orchestrator (orch-prod-1.daydream.live:8935), not just staging:

scope.cloud.livepeer_app - ERROR - Media input loop failed: Response payload is not completed: <TransferEncodingError: 400, message='Not enough data to satisfy transfer length header.'>
scope.cloud.livepeer_app - ERROR - Control channel subscription error: Trickle JSONL subscription error: ClientPayloadError: Response payload is not completed: <TransferEncodingError: 400, message='Not enough data to satisfy transfer length header.'>
livepeer_gateway.trickle_subscriber - ERROR - Trickle sub failed GET https://orch-prod-1.daydream.live:8935/ai/trickle/... status=502 msg=
livepeer_gateway.trickle_publisher - ERROR - Trickle POST failed url=https://orch-prod-1.daydream.live:8935/ai/trickle/... status=502 body=''

Affected apps/jobs:

• scope-livepeer--prod job ac940416 at ~21:48 UTC
• scope-livepeer-emran job 03454ac7 at ~21:06 UTC

Both ended with 502 from orch-prod-1, followed by TransferEncodingError. Also saw Trickle publisher channel does not exist and Stream not found (404) from orch-prod-1 in a separate RTX5090 job at ~03:09 UTC.