Stream the ABC query CSV download through the event loopDownloading a large query result built the whole CSV in the HTTP output
queue before the first byte went out, which cost about 2 GiB for a large
result, and did so synchronously inside the single-process event loop, so
the app server could not answer heartbeats and the client gave up on the
session once the file arrived.
Give Posda::QueryRowStore a forward-only cursor with its own file handle,...
Page ABC query results from disk instead of holding every row in memoryThe Activity Based Curation UI kept the full result of every foreground
query as a Perl array of arrays for as long as the result stayed open,
even though only 30 rows are shown at a time. At over a hundred bytes per
cell, large results pushed the app server past its 8Gi container limit.
Add Posda::QueryRowStore, which appends rows to a temp file as the query
subprocess streams them in (one JS...
Allow date_shift 0 from patient_mapping table for apply dispositionsBecause a Postgres interval of 0 returns as "00:00:00", and Posda
expects the shift to be of the form "X days", a special exception is
needed for the 0 shift.
Edit and import private-disposition files one at a timeApplyPrivateDispositions.py edited every file in the activity before
importing any of them, so the temp directory grew to a full edited copy
of the activity. Each worker now edits, imports, and deletes its own
temp file, bounding temp usage to MAX_WORKERS files.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VAxJdwEt6KFWHr99NoyDZN
Let POSDA_WORKER_QUEUE name the Redis list and label Jobs by queueThe dispatcher will roll out beside the existing worker-low and
worker-high Deployments, reading a Redis list of its own, and in Kueue
mode the priority split has no meaning. POSDA_WORKER_QUEUE now overrides
the work_queue_<priority> name outright, for every mode; unset, nothing
changes. It is read straight from the environment rather than through
Config.get so the legacy loop does not gain a w...
Describe the worker node's three modes in ARCHITECTURE.mdCo-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014DWGayuQXqtajFQLUK8ZPH
Add the Kueue dispatcher and the single-item Job entrypointdispatch.py pops work ids from Redis and creates one Kueue-labelled Job
each. Redis stays the queue of record: nothing is popped while more than
POSDA_DISPATCH_MAX_OPEN_JOBS Jobs are unfinished, and a Job that cannot
be created marks the row errored instead of dropping the id. Every
POSDA_DISPATCH_RECONCILE_SECONDS it lists its Jobs and marks the row of
any Failed Job that never reported (OOMKi...
Add a minimal in-cluster Kubernetes client and the Job manifest builderThe dispatcher needs four API calls (get its own pod, create a Job, list
Jobs, list pods), which does not justify the official client and its
dependency tree; k8s.py does them with requests and the mounted
ServiceAccount token, re-read per call because bound tokens rotate.
jobspec.py builds a Job from the dispatcher's own pod spec rather than
from a template: the dispatcher Deployment carries ...
Give WorkerNode.py subcommands and pass arguments through the entrypointsWorkerNode.py becomes a shim over posda.worker.cli. With no arguments it
runs the legacy Redis loop exactly as before; "dispatch" and "run
<work_id>" select the two Kueue modes added in the following commits.
worker_start.sh and docker-entrypoint forward their arguments so
"docker-entrypoint worker dispatch" works once the image is rebuilt.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic....
Move the worker node's logic into a posda.worker packageWorkerNode.py was a 500-line script with module globals, which made it
impossible to import from tests and awkward to extend. The logic now
lives in posda/worker: settings (environment), api (PosdaApi client),
execute (one work item), workqueue (Redis) and legacy (the BRPOP loop).
The legacy loop is a line-for-line port. The only intended behaviour
changes are two token leaks: set_status_runni...
Split RunQueryInBackground output into spreadsheet-sized filesLarge query results exceeded the 1,048,576-row limit of Excel and
LibreOffice Calc. Output is now split into files of at most 75% of that
limit (786,432 lines including the header), named "Query Results NN of
M" so they sort in order.
The summary block (query name, run by, timing, args) moves from the top
of the CSV into the email, so every file starts with the column header.
Rows are streame...
Add RunQueryWithRowCount for streaming selects with an up-front countDBD::Pg buffers a whole select result client-side during execute and
returns the row count, but RunQuery discarded that count for selects, so
callers that needed the total before writing output had to buffer every
row again in Perl. RunQueryWithRowCount delivers the count first, then
streams rows to a callback.
The prepare/execute/reconnect-once logic from _RunQueryBlocking is
factored into _E...
Fix StructFixer creating empty Structure Set ROI itemsThe loop that sets Referenced Frame of Reference UID on each ROI
iterated the CLOSED_PLANAR contour list and used each contour's ROI
Contour Sequence (3006,0039) index as a Structure Set ROI Sequence
(3006,0020) index. When a file has more ROI Contour items than ROIs,
InsertElementBySig created new items holding only (3006,0024), with no
ROI Number, Name or Generation Algorithm.
Iterate the St...
Pull request #442: Deduplicate unique_pixel_data before adding its unique indexMerge in PT/oneposda from dedupe-unique-pixel-data to master
* commit 'f15e90a6080d95a43d4f12189b27216d59ee5f24':
Deduplicate unique_pixel_data before adding its unique index
Pass defer_size to dcmread so large elements stay on diskThe ingest worker read every element into memory before skip_large was
ever consulted, which OOMed on large files. Posda's parser seeks past
anything over 256 bytes; pydicom only matches that when dcmread is given
defer_size, so parse() now passes skip_large through (and nothing when
the limit is 0, as in the pixel worker and the load_large_values fix).
pydicom's get_item quietly reads a defer...
Deduplicate unique_pixel_data before adding its unique indexThe ingest lookup-unique migration skipped deduplicating unique_pixel_data
because the table was empty in the dev database, but production holds
duplicates over (digest, size), so the index creation would fail there.
Remap image and pixel_location references to the lowest id per
(digest, size) and delete the rest, as for slope_intercept and
window_level. Rows with a NULL size are left alone, m...
Pull request #436: Ingest rewriteMerge in PT/oneposda from ingest-rewrite to master
* commit 'c06b1d11537e891149ac13e7265337cad655f34b':
Add atomic per-file imports, make the lookup upserts unconditional
Decode zero-length implicit-VR elements instead of leaking pydicom types
Move the ingest lookup migration into database/migrations
add psycopg2-binary as a global dep
Run the ingest workers as independent deployment...
Add DICOM tag and type to study/series consistency report headersAnalyzeStudySeriesConsistencyByActivity.pl produced StudyInconsistencies
and SeriesInconsistencies reports whose column headers were only the
internal table column names (e.g. "patient_id", "laterality"). Headers
now also carry the DICOM tag and attribute type:
study_instance_uid,"patient_id (0010,0020) Type 2",...
Changes:
- Add a %TagInfo lookup mapping every column returned by the
Seri...
Fix StructFixer rejecting near-axial images as non-axialCheckImagGeo compared ImageOrientationPatient against 1,0,0,0,1,0 with
==, rejecting images with a third direction cosine of -0.00002359.
Compare against a 1e-4 tolerance instead; acceptance is a strict
superset of before. The sprintf("%.8f") moves out of the comparison
and into the error message, which prints the same text.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Add "Launch Any Operation" picker to Activity Operations paletteMany spreadsheet operations have no dedicated button in the workflow
definition. Add a final "Launch Any Operation" category whose panel
renders a text input backed by a datalist of every loaded operation,
plus a Launch button. Typing filters the list; a name can also be
entered by hand.
The new InvokeNamedOperation handler trims the name, verifies it via
GetOperationDescription (so operations...
Remove extraneous print after calling background.finish()BackgroundProcess redirects STDOUT to a file (the "email" report), and
when finish() is called, it closes that file. So printing after finish
causes a crash, and should never be necessary anyway.
Add CreateTimepointFromImportEventId operationNew Python background script that creates an activity timepoint from a
single import_event_id given as a parameter (unlike the Perl
CreateActivityTimepointFromImportEventId.pl, which reads a list from
STDIN). Selects DICOM files by default; an optional trailing
include_all_files argument (truthy to enable) includes every file in
the import event. Aborts before creating the timepoint if no files...
Add atomic per-file imports, make the lookup upserts unconditionalPOSDA_INGEST_FIX_ATOMIC_IMPORT wraps each file's import in one
transaction. Under the inherited autocommit behaviour, a handler that
dies partway leaves the rows it already wrote, and because
is_dicom_file was committed first the file drops out of the replenish
query and sits partial forever; re-processing it by hand duplicates
rows. With the flag on, a failure rolls the whole file back, then a...
Decode zero-length implicit-VR elements instead of leaking pydicom typespydicom stores a zero-length element in an implicit-VR file as a
RawDataElement whose value is None, because the VR is unknown at read
time, and Dataset.get_item routes value-None elements through the
converting __getitem__. The shim's already-converted branch then
returned the converted value verbatim -- PersonName("") for an empty
PatientName -- which psycopg2 cannot bind: every implicit-VR f...
Move the ingest lookup migration into database/migrationsIt landed in the legacy Posda/sql/migrations directory, but new
migrations belong in database/migrations/posda_files where ./migrate
applies them. Renamed to the date-prefixed convention and dropped the
explicit begin/commit, since ./migrate wraps each file in
--single-transaction. Updated the two references in fixes.py and
Ingest-Fixes.md.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com...
Pull request #434: Background editor status fixesMerge in PT/oneposda from background-editor-status-fixes to master
* commit 'be956c6ef7e75683003de429a50aab6d95ca152c':
Make stalled activity_task_status rows visible
Gate NewSubprocessEditor per-file debug output behind POSDA_DEBUG
Stop WorkerNode losing subprocess stderr
Fix LineReader EOF fall-through and two BackgroundEditorTp bugs
Close out activity_task_status from an END block...
Throttle activity status updates in CreateActivityTimepointFromFileIdsSetActivityStatus was called after every single-row insert; now it only
fires every 1000 files (and on the final file) to cut DB churn.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J7koX7JPvgJbNTwprRessU
Run the ingest workers as independent deploymentsThe pipeline was a daemon that forked five file workers and five pixel
workers, which does not survive being scheduled: the tiers could not be
scaled, restarted or placed independently.
The blocker was shutdown. Both halves signalled it through a Redis 'quit'
key that every process polled, so a daemon receiving SIGTERM stopped every
worker, and on startup the daemon deleted the key again -- ra...
Give every blocking-pop loop a socket_timeout above its block timeredis-py 8 changed socket_timeout from None to 5s. Every poller here uses
brpop(..., 5), so on an empty queue the server holds the connection for the
full block time and the client's read deadline expires at the same moment:
"TimeoutError: Timeout reading from socket", and only ever when there is no
work to do. redis-py 8 retries with backoff before raising, so each idle poll
also stalled the p...
Document the POSDA_INGEST_FIX_* flagsThe ten flags were only described in fixes.py's docstrings and in commit
messages, which is no use to anyone deciding whether to turn one on.
docs/programming/Ingest-Fixes.md now covers them properly: how to set one,
why they default to off, and what each actually changes.
They are grouped by blast radius, which is the thing that matters when
scheduling them:
- three fill columns that are u...
Port RtBeams and retire the RT Perl bridgeThe ingestion pipeline is now entirely Python. RtBeams was the last
handler on the Perl side: one loop over the beam sequence, ten sub-loops
for the things hanging off a beam, and a control point loop with four
sub-loops of its own, naming a little over 130 elements between them. The
parameter tables were extracted mechanically from the Perl rather than
retyped.
Verified against the unmodified...
Port five of the six RT Plan handlers, and fix RtPatientSetupRtPlan, RtPrescription, RtToleranceTables, RtPatientSetup and
RtFractionScheme now run natively. RtBeams is the only handler left on the
Perl side.
Verified against the unmodified Perl on a plan with two referenced doses,
a related plan, two dose references, a tolerance table with two limiting
devices, a patient setup with a setup image and a shielding device, and a
fraction group with a dose ...
Port RTDose and RTDvh, and the private-tag dictionary they needRT Dose files no longer touch Perl. RTDose consumes the image_id that
ImagePixel publishes and hands rt_dose_id to RTDvh, both through hist, so
what used to cross a process boundary is now an ordinary dict entry.
RTDvh forced two pieces of infrastructure:
- Search(), the forgiving sibling of Substitutions. Same walk, but a
literal index naming a non-dataset, or a sequence whose value is not...
Port the RT Structure Set handlers to PythonStructureSet, RoiContour and RtRoiObservations now run natively; an RT
Structure Set no longer touches Perl at all. They keep sharing state
through hist -- structure_set_id, ss_for_by_item (which memoises one
ss_for row per frame-of-reference item) and roi (which maps ROI number to
the row, so the later two can resolve ReferencedROINumber and complain
about dangling references).
Verified again...
Port Posda::Dataset's sequence navigationAdds the two pieces of navigation machinery the RT handlers are built on,
as the first step of removing Perl from the ingestion pipeline entirely.
ExtractElementBySig's suffixes: "[n]" indexes the value itself, "[n]rest"
and "[n]>rest" descend into sequence item n and carry on. Posda checks the
bare-index form before it checks for SQ, so a trailing [n] serves both
multi-valued elements and seq...
Add opt-in corrections for the six verified ingest defectsThe port reproduces these by default, because that equivalence is what
makes the cutover verifiable. Each correction is therefore a flag that is
off unless POSDA_INGEST_FIX_<NAME> says otherwise.
Sequencing matters: enabling any of these breaks the equivalence with the
Perl, so they should be turned on only after the Python workers have
replaced the Perl ones. Enabling one while both implement...
Add differential verification harness for the ingest portRuns both implementations over the same corpus into two throwaway copies
of the posda_files schema and diffs every populated table row by row.
The Perl side runs the unmodified FastFileProcessWorker.pl and
FastPixelInfoWorker.pl -- nothing is transcribed for the comparison. The
scratch database is substituted underneath them by pointing
POSDA_DATABASE_CONFIG at a generated databases.json whose...
Bridge the RT module handlers to the legacy Perl implementationThe RT handlers are ~2,000 lines that lean entirely on Posda::Dataset's
sequence navigation: all 353 of the "(3006,0010)[<0>](3006,00c0)" style
signatures in Modules.pm are in the RT region. Rather than port that
machinery before anything could ship, Python runs the leading non-RT
handlers and hands the trailing RT block to a new Perl helper.
Three facts checked against the Perl make the split...
Port the fast-file-process daemon and both worker loopsAdds the three entry points: the queue replenisher, the file worker
(file/dicom_file/file_meta plus SOP dispatch) and the pixel worker, along
with the parse layer that stands in for Posda::NewTry and Posda::Try.
The Redis protocol is kept byte-identical -- payloads are bare
[file_id, path] JSON arrays on the same 'files' and 'pixel_location'
lists, with the same 'quit' key -- so Perl and Pytho...
Port DicomIod dispatch, file_sop_common and ctp_fileAdds the SOP-class dispatch layer: the 81-class handler table, sop_common
and CTP private-tag extraction, the creation date/time reformatting, the
"sop class has a space" repair (which also redirects handler lookup), and
the flush of accumulated complaints into dicom_process_errors.
Both large tables are generated from the Perl rather than transcribed,
with the generators kept in the scratchpa...
Port the core non-RT Posda::DB::Modules handlersAdds the ten handlers that do real work outside the RT modules -- Patient,
Study, Series, Equipment, ImagePixel, ImagePlane, FrameOfReference,
SlopeIntercept, WindowLevel and CTImage -- plus a thin database layer. The
other twenty-odd non-RT handlers are print-only stubs in the Perl and are
registered as named no-ops.
Handlers keep the Perl's contract, including the `hist` dict threaded
betwee...
Add Posda parser value semantics shim for the Python ingest portFirst step of rewriting posda-fast-file-processor in Python/pydicom. This
is the foundation layer only: reading values out of a dataset the way
Posda::Parser does, and the GetAttrs coercion table.
The port is deliberately bug-for-bug so that it can be verified by exact
row diff against the Perl; the defects it reproduces are recorded for
separate fixes later.
Two semantics dominate and are wh...
Make stalled activity_task_status rows visibleThe table had no primary key, no unique constraint and no index at all, so
every heartbeat was a sequential scan and nothing prevented duplicate rows.
Add an index on (activity_id, subprocess_invocation_id) plus a partial index
over open rows.
Deliberately not a primary key yet: existing deployments may already hold
duplicates, which would make ADD PRIMARY KEY fail mid-deploy. The migration
ca...
Gate NewSubprocessEditor per-file debug output behind POSDA_DEBUGEight of these run concurrently under BackgroundEditorTp.pl and all inherit the
worker's stderr capture file. Dumping the whole edit struct for every DICOM
file made that file large enough to slow the upload appreciably and widen the
window in which a straggler could still be writing to it. Genuine errors and
warnings stay unconditional.
Uses the same env flag as the existing checks in Dispatc...
Stop WorkerNode losing subprocess stderrThree problems, all on the path that captures a job's output.
An upload failure left stdout_file_id/stderr_file_id unbound, so the reference
below raised UnboundLocalError, which the blanket handler in work_loop caught
and logged in place of the real error. Bind them before the try and log the
traceback.
wait() returns when the direct child exits, but background scripts leave
children running...
Quasar Jarosz
Michael Rutherford
Quasar Jarosz