Powered by

Ingestion Attempt State

Ingestion Attempt State is VySol's in-memory backend contract for representing one active source-ingestion attempt. It gives backend callers a truthful state model for whether ingestion has not started, is running, is stopping, is paused with resumable staged work, or has completed successfully, and which ingestion phase the current attempt has reached.

This page is for developers, power users, and AI coding agents that need to understand ingestion attempt state before changing ingestion orchestration, cancellation handling, source staging integration, future route handlers, one-button UI behavior, or logging.

Why It Exists

VySol's future ingestion UI is expected to use one primary control for start, pause or stop, resume, and complete feedback. That control needs backend state that does not pretend cancellation is instant and does not confuse a never-started flow with a successful finish.

The attempt state model keeps that lifecycle separate from parsing, chunking, source commits, and UI rendering. It also carries attempt IDs so late results from an older attempt or a cancelled current attempt can be rejected instead of mutating lifecycle state or cleaning live Deep Pause work.

The phase model exists because some decisions need more detail than RUNNING. Draft leave safety, for example, must know whether the attempt is still before durable text commit or has already reached the point where source text and chunks are safely stored.

Ownership Boundary

Ingestion Attempt State owns:

  • Representing the single active backend ingestion attempt in process memory.
  • Distinguishing IDLE, RUNNING, STOPPING, PAUSED, and COMPLETE attempt statuses.
  • Distinguishing attempt phases: not_started, preflight, parsing, splitting, atomic_text_commit, and text_committed.
  • Creating a new attempt ID when a fresh attempt starts from IDLE or COMPLETE.
  • Coordinating fresh attempt workspace creation before entering RUNNING.
  • Letting backend callers update the phase for the current running attempt by attempt ID.
  • Coordinating attempt cancellation flag setup, pause requests, resume clearing, and terminal cleanup.
  • Cancelling active or pausable attempts during app shutdown without treating shutdown as a successful completion.
  • Preserving the same attempt ID while a paused attempt resumes.
  • Verifying that a paused attempt still has its temporary workspace before resume.
  • Coordinating temporary workspace cleanup after successful completion and terminal cancellation to IDLE.
  • Abandoning the active temporary workspace record during app-close cancellation so startup cleanup owns filesystem removal.
  • Rejecting stale completion or cancellation results from old attempt IDs.
  • Rejecting late completion results from the cancelled current attempt while it is STOPPING or PAUSED.
  • Representing whether staged work remains when cooperative cancellation finishes.
  • Logging cancellation completion at INFO.
  • Logging valid state transitions at INFO.
  • Logging invalid transitions and stale attempt results at WARNING.
  • Logging unexpected state-model failures at ERROR.

Ingestion Attempt State does not own:

  • Rendering the one-button UI or choosing button labels.
  • Creating or managing temporary workspace directories beyond coordinating with Temporary Ingestion Workspace.
  • Running parsers, source access checks, source hashing, duplicate checks, chunk generation, provider calls, or source commits beyond tracking the phase reported by orchestration and rejecting lifecycle results that no longer match the current state.
  • Persisting attempt state to app.sqlite, world.sqlite, files, manifests, or saved-world data.
  • Creating durable failed source, failed world, or failed attempt statuses.
  • Managing external job queues, worker processes, or background task scheduling.

Normal Flow

Before ingestion starts, the registry reports IDLE with no attempt ID, no staged work remaining, and phase not_started. When a caller starts a fresh attempt, a new attempt ID is generated, Temporary Ingestion Workspace creates scratch storage for that ID, the state moves to RUNNING, staged work is considered present, and phase starts at preflight.

While the attempt is running, future ingestion orchestration can move the same attempt through preflight, parsing, splitting, atomic_text_commit, and text_committed. Phase updates are accepted only for the current running attempt ID. Stale attempt IDs and non-running states are rejected without changing the current state.

If the user asks to stop while ingestion is running, the cancellation flag is set and the state moves to STOPPING. This tells future UI code that cancellation has been requested but cooperative cancellation has not finished yet. A repeated stop request while already STOPPING is treated as an idempotent no-op and returns the current state.

When cancellation finishes, the caller reports whether staged work remains. This is the only point where PAUSED can be exposed. If work remains, the state moves to PAUSED and keeps the same attempt ID and temporary workspace so the same attempt can resume later without discarding already prepared temporary work. If no work remains, the state moves back to IDLE and the temporary workspace is removed.

Resuming a paused attempt first verifies that the same attempt ID still has its temporary workspace, clears the previous cancellation request, then moves PAUSED back to RUNNING with that same attempt ID and preserved phase. Completing a running attempt with the current attempt ID moves the state to COMPLETE, resets phase to not_started, clears the cancellation flag, removes the temporary workspace, and allows the UI to distinguish successful completion from a never-started idle state.

If a completion result arrives for the same attempt after cancellation has moved it to STOPPING or PAUSED, the result is rejected. The attempt state, cancellation flag, and temporary workspace remain unchanged so a cancelled attempt cannot accidentally complete or delete resumable Deep Pause data.

When the app is closing, the shutdown boundary cancels any current RUNNING, STOPPING, or PAUSED attempt. It leaves the cancellation flag requested, preserves the current phase, records the lifecycle as stopping, and abandons the temporary workspace record instead of deleting the workspace during shutdown. App-close cancellation is not a successful commit and is not a resumable pause; it exists so late completion and commit guards reject the attempt while backend startup remains responsible for removing abandoned temporary workspace folders.

Inputs

Ingestion Attempt State receives caller intent to start, stop, resume, finish cancellation, complete an attempt, or update the phase of the current running attempt. Cancellation, completion, and phase update calls must include the attempt ID they belong to. Cancellation completion also receives a boolean staged-work signal so it can choose between PAUSED and IDLE. Stop requests also coordinate with Ingestion Attempt Cancellation so downstream work can see that Pause was requested.

On FastAPI shutdown, the app-close cancellation boundary can ask this system to cancel the current active or pausable attempt without preserving it for restart resume.

It does not receive source file paths, source text, parser output, chunks, hashes, provider responses, database connections, or UI request objects.

Outputs

The system returns immutable in-memory attempt state objects containing:

  • The current attempt status.
  • The current attempt phase.
  • The current attempt ID, when an attempt exists.
  • Whether staged work remains available for resume.
  • The current temporary workspace when looked up by the current attempt ID.

It does not produce database rows, files, committed source metadata, chunks, provider calls, HTTP responses, or visible UI directly.

Saved State And Resume Behavior

Attempt state is in memory only. It is intended to represent the currently running app process, not crash recovery or durable saved progress.

PAUSED means cancellation finished while staged work still exists and the same attempt can resume with its existing temporary workspace and preserved phase. That workspace may contain intermediate parsed or split data needed for a true Deep Pause continuation. IDLE means no current attempt is resumable and phase is not_started. COMPLETE means an attempt finished successfully, its temporary workspace has been cleaned, phase is reset to not_started, and the completed state can be shown separately from IDLE until a new attempt starts.

App-close cancellation discards the same-process resume boundary. A paused attempt that exists only because the app is still open must not be persisted across restart, and shutdown must not convert paused work into a committed or resumable attempt.

Retry, Pause, And Abort Behavior

This model supports cooperative cancellation state, but it does not interrupt running work itself. The future ingestion runner must request a stop, check the cancellation flag between work-launch boundaries, finish its own cancellation work, and then report whether staged work remains.

Fresh starts from IDLE or COMPLETE create new attempt IDs, new temporary workspaces, fresh cancellation flags, and phase preflight. Resume from PAUSED keeps the existing attempt ID, phase, and workspace while clearing the previous cancellation request. Terminal cancellation and successful completion remove the temporary workspace, reset phase to not_started, and clear the cancellation flag. Late completion or cancellation results for older attempt IDs are rejected and leave the current state unchanged. Late completion for the cancelled current attempt is also rejected while STOPPING or PAUSED.

App-close cancellation is stricter than same-process Pause. It handles RUNNING, STOPPING, and PAUSED as discard-on-close states, keeps the cancellation signal requested, and does not expose a restart resume path.

Failure Behavior

Invalid transitions are logged at WARNING and rejected without changing state. Examples include stopping while idle, resuming while not paused, starting while already running, updating phase while not running, or completing an attempt while it is not running. Repeated stop requests while already STOPPING are logged at WARNING and return the current state.

If the cancellation flag cannot be set for a valid stop request, the failure is logged at ERROR, the attempt remains RUNNING, and the caller receives a cancellation error.

Stale attempt results are logged at WARNING and rejected without changing state. Late completion results from a cancelled current attempt are also logged at WARNING, rejected without changing state, and must not clean the paused workspace. Workspace creation failures and unexpected state-model failures are logged at ERROR and raised so callers do not continue with an unknown lifecycle state.

App-close cancellation is logged at INFO. It must not commit staged work, must not clear the cancellation flag in a way that permits a late commit, and must leave temporary workspace folder removal to the next startup cleanup boundary.

Logs must not include raw source paths, full temporary workspace paths, source text, local machine details, provider output, or user data.

System Interactions

Ingestion Attempt State currently interacts with:

  • Future ingestion orchestration, which will use it before and after cooperative cancellation, resume, and completion.
  • Ingestion Attempt Cancellation, which stores the attempt-scoped cancellation signal used by future orchestration and commit guards.
  • Temporary Ingestion Workspace, which provides current-attempt scratch storage for Start and Resume attempts.
  • Temporary Source Staging State, whose remaining staged work determines whether cancellation ends in PAUSED or IDLE.
  • App-close cancellation, which discards draft or staged source state and uses attempt state to reject late completion.
  • Draft World Leave Safety, which reads the current phase to decide whether leaving a draft is safe after durable text commit.
  • Future one-button UI state, which can map the backend lifecycle to start, stopping, resume, and completion feedback.
  • The central logger, which records transition summaries and rejected transitions without source data.

It must stay separate from parser internals, source preflight checks, committed source storage, chunk storage, embeddings, graph extraction, retrieval, and UI rendering.

Current Edge Cases

Internal edge cases:

  • The initial state is IDLE with no attempt ID.
  • The initial phase is not_started.
  • Starting from IDLE creates a new attempt ID and enters preflight.
  • Starting from COMPLETE creates a new attempt ID and enters preflight.
  • Starting creates a temporary workspace before the state becomes RUNNING.
  • Starting from RUNNING, STOPPING, or PAUSED is rejected.
  • Phase updates are accepted only for the current RUNNING attempt.
  • Stale phase updates are rejected without changing state.
  • Non-running phase updates are rejected without changing state.
  • Stopping is allowed only from RUNNING.
  • Repeated stopping while already STOPPING returns the current state and logs a warning.
  • Stop requests set the current attempt's cancellation flag before the state moves to STOPPING.
  • Stop requests preserve the current phase.
  • Cancellation completion is accepted only from STOPPING and only for the current attempt ID.
  • Cancellation with staged work moves to PAUSED, logs cancellation completion, and preserves the existing attempt ID, phase, and temporary workspace.
  • Cancellation without staged work moves to IDLE, resets phase to not_started, clears the cancellation flag, and cleans the temporary workspace.
  • Resume is allowed only from PAUSED with staged work remaining and the existing temporary workspace available, then clears the cancellation flag and preserves phase.
  • Completion is accepted only from RUNNING and only for the current attempt ID, then resets phase to not_started, clears the cancellation flag, and cleans the temporary workspace.
  • COMPLETE and IDLE remain distinct statuses.
  • Stale attempt IDs cannot complete or cancel a newer attempt.
  • Late completion from the current attempt while STOPPING or PAUSED is rejected without changing state or cleaning the workspace.
  • App-close cancellation accepts RUNNING, STOPPING, and PAUSED attempts as discard-on-close states.
  • App-close cancellation keeps the cancellation signal requested so late completion and commit work are rejected.
  • App-close cancellation preserves phase for late-guard visibility while marking the attempt as stopping.
  • App-close cancellation abandons workspace registry ownership instead of deleting the temporary workspace folder during shutdown.

Cross-system edge cases:

  • Future ingestion orchestration must pass the current attempt ID into completion and cancellation-finished calls.
  • Future ingestion orchestration must use the current attempt ID when asking for a temporary workspace.
  • Future cancellation code must report whether staged work remains instead of guessing from status alone.
  • Future orchestration must stop launching new work after the cancellation flag is requested.
  • Future orchestration must report cancellation completion before the backend exposes PAUSED.
  • Future orchestration must treat completion results that arrive after cancellation as rejected late results.
  • Future resume code must not create a new workspace for the paused attempt.
  • Future cleanup code must preserve PAUSED workspaces during the same process and clean only terminal attempt workspaces.
  • Future UI code must treat STOPPING as an in-progress cancellation state, not as already paused.
  • Future UI code must treat PAUSED as resumable work and IDLE as ready for a fresh start.
  • Future source commit work must not infer that attempt state has committed, parsed, copied, hashed, or chunked any source.
  • Future source commit work must set text_committed only after source text and chunks are durably committed to SQLite.
  • Draft leave safety must not treat RUNNING alone as safe; it must read the phase.
  • Future source commit work must not treat temporary workspace files as committed source files.
  • FastAPI shutdown must run app-close cancellation before closing the global database connection so draft and staging cleanup can still use app-owned registries.
  • App-close cancellation must not preserve PAUSED state for restart resume.
  • Attempt state must not create or migrate app-global or per-world database tables.

Invariants

  • Attempt state must remain in memory for this system.
  • Only one active attempt may exist in the registry.
  • IDLE must not have an attempt ID or staged work remaining.
  • IDLE and COMPLETE must use phase not_started.
  • PAUSED must mean staged work remains.
  • COMPLETE must stay distinct from IDLE.
  • Fresh starts must create new attempt IDs.
  • Fresh starts must enter phase preflight.
  • Phase updates must be scoped to the current running attempt ID.
  • Fresh starts must initialize a not-requested cancellation flag.
  • Resume must keep the paused attempt ID.
  • Resume must preserve the paused attempt phase.
  • Resume must clear the previous cancellation request.
  • Fresh starts must create new temporary workspaces.
  • Resume must keep the paused attempt's existing temporary workspace.
  • PAUSED must only be exposed after cancellation completion is reported.
  • Completion and terminal cancellation must clean the attempt workspace without changing the meaning of COMPLETE, IDLE, or PAUSED.
  • Completion and terminal cancellation must reset phase to not_started.
  • Terminal completion and terminal cancellation must clear the attempt cancellation flag.
  • Repeated stop requests while STOPPING must not change state or create a new attempt.
  • App close must never be treated as successful completion.
  • App close must not preserve paused staged work across restart.
  • Stale attempt results must never mutate current state.
  • Stale phase updates must never mutate current state.
  • Late completion results from a cancelled current attempt must never mutate current state or clean Deep Pause workspace data.
  • Invalid transitions must leave current state unchanged.
  • The system must not parse, hash, copy, commit, persist, or assign book numbers for sources.
  • No database migration is required for ingestion attempt state.

Implementation Landmarks

  • app/ingestion/attempt_state.py owns the in-memory attempt state and phase model, registry, transitions, stale-result rejection, terminal workspace cleanup coordination, and logging.
  • app/ingestion/app_close_cancellation.py coordinates app-close discard behavior across attempt state, active staged-batch metadata, draft worlds, source staging, and temporary workspace ownership.
  • app/ingestion/attempt_cancellation.py owns the in-memory cancellation flag used when Pause is requested.
  • app/ingestion/attempt_workspace.py owns temporary workspace creation, lookup, current-attempt scoping, and cleanup.
  • app/ingestion/__init__.py exports the public attempt-state and workspace API for backend callers.
  • tests/test_ingestion_attempt_state.py covers lifecycle transitions, phase updates, workspace creation and reuse, stale attempt rejection, invalid transition logging, unexpected failure logging, and database non-persistence.

What AI/Coders Must Check Before Changing This System

Before editing Ingestion Attempt State, check:

  • Whether the change belongs in attempt state or in ingestion orchestration, source staging, parser routing, commit logic, routes, or UI behavior.
  • Whether IDLE, PAUSED, and COMPLETE remain semantically distinct.
  • Whether phase updates remain scoped to the current running attempt.
  • Whether text_committed still means source text and chunks have been durably committed.
  • Whether stale attempt IDs still cannot mutate current state.
  • Whether cancellation completion still chooses PAUSED only when staged work remains.
  • Whether PAUSED is exposed only after cancellation completion, never directly from Pause request.
  • Whether valid Pause requests still set the cancellation flag before entering STOPPING.
  • Whether repeated Pause requests while STOPPING remain idempotent.
  • Whether late completion after cancellation is still rejected without deleting paused workspace data.
  • Whether app-close cancellation still discards RUNNING, STOPPING, and PAUSED attempts without creating restart-resumable work.
  • Whether cancellation to IDLE and successful completion still clean the temporary workspace.
  • Whether terminal cleanup and Resume clear cancellation flag state at the right time.
  • Whether Start and Resume still preserve the Temporary Ingestion Workspace contract.
  • Whether logs avoid source paths, full temporary workspace paths, source text, provider output, local machine details, and user data.
  • Whether attempt state remains independent of database schema, source commits, parser work, chunk storage, embeddings, graph extraction, and durable failed statuses.