Add outbox worker processing policy

docs/event_outbox_spec.md

@@ -384,7 +384,7 @@ row exists.
 
 ## Worker/Rake Task
 
-Current implemented worker-facing command:
+Current implemented Redmine-side worker-facing command:
 
 ```sh
 bundle exec rake redmine_event_outbox:dump RAILS_ENV=production
@@ -392,7 +392,7 @@ bundle exec rake redmine_event_outbox:dump RAILS_ENV=production
 
 This prints pending rows as JSON and does not mark them processed.
 
-Next worker command:
+Historical planned Redmine-side publish command:
 
 ```sh
 bundle exec rake redmine_event_outbox:publish RAILS_ENV=production
@@ -411,10 +411,11 @@ bundle exec rake redmine_event_outbox:publish
 bundle exec rake redmine_event_outbox:publish RETRY=1
 ```
 
-Initial publisher target can be stdout or a local JSONL file. That lets us test
-the Redmine side before choosing Redis Streams or RabbitMQ.
+The current external publisher is `redmine_outbox_worker.py`. Its initial
+publisher target is JSONL at `/tmp/redmine-outbox/derived_documents.jsonl`,
+which lets us test the Redmine side before choosing Redis Streams or RabbitMQ.
 
-The next implementation should keep the worker small and conservative:
+The worker stays small and conservative:
 
 - select pending rows in id order
 - lock or claim a bounded batch
@@ -424,6 +425,19 @@ The next implementation should keep the worker small and conservative:
 - leave failed rows available for retry
 - make duplicate delivery acceptable to consumers
 
+Current external worker commands:
+
+```sh
+./redmine_outbox_worker.py --status
+./redmine_outbox_worker.py --dry-run --batch-size 10
+./redmine_outbox_worker.py --batch-size 20
+./redmine_outbox_worker.py --purge-processed-days 30
+./redmine_outbox_worker.py --purge-processed-days 30 --apply-purge
+```
+
+Processed rows are retained by default. Purge is explicit, age-gated, and
+intended for test-instance cleanup, not normal delivery.
+
 Later publisher targets:
 
 - Redis Streams
@@ -550,7 +564,8 @@ decision code fails.
 
 ## Open Questions
 
-- How long should processed outbox rows be retained?
+- Do production processed rows need a formal retention window, or should they
+  be retained until external index/message-bus replay policy is settled?
 - Do we need a Redmine admin page to inspect outbox health, or is a rake task/log enough?
 - What is the current production mail ingestion path?
 - Which vector-capable external search index should be used first?

docs/outbox_worker_policy.md

@@ -0,0 +1,63 @@
+# Outbox Worker Policy
+
+`redmine_outbox_worker.py` is the first external publisher for
+`event_outbox_events`. It currently publishes enriched documents to JSONL, but
+the processing policy is intended to carry forward to a later message bus.
+
+## Default Output
+
+The default output is:
+
+```sh
+/tmp/redmine-outbox/derived_documents.jsonl
+```
+
+Use `--output /path/to/file.jsonl` to override it. The worker creates parent
+directories as needed.
+
+## Processing Semantics
+
+- Read eligible pending rows globally in id order.
+- Claim a bounded batch with `locked_at` and `locked_by`.
+- Enrich each event with read-only joins.
+- Append derived JSON documents to the output.
+- Mark a row `processed_at` only after the output write succeeds.
+- On failure, increment `attempts`, write `last_error`, and release the lock.
+
+This is at-least-once delivery. Downstream consumers must tolerate duplicate
+documents keyed by `event_id` or `doc_id`.
+
+## Operations
+
+Inspect queue state:
+
+```sh
+./redmine_outbox_worker.py --status
+```
+
+Preview a batch without claiming or marking rows processed:
+
+```sh
+./redmine_outbox_worker.py --dry-run --batch-size 10
+```
+
+Process a bounded batch:
+
+```sh
+./redmine_outbox_worker.py --batch-size 20
+```
+
+Preview processed-row cleanup:
+
+```sh
+./redmine_outbox_worker.py --purge-processed-days 30
+```
+
+Apply cleanup explicitly:
+
+```sh
+./redmine_outbox_worker.py --purge-processed-days 30 --apply-purge
+```
+
+Processed rows are kept by default. Purge is an administrative cleanup tool for
+test instances and should not be part of normal publishing.

docs/redmineup_local_fork_changelog.md

@@ -30,7 +30,37 @@ environment. Before risky edits, archive the current plugin directories in
   - Helpdesk outbox/worker validation now has a repeatable live script.
 - Next meaningful milestone:
   - Run `validate_helpdesk_outbox_worker.py` after outbox or worker changes,
-    then choose the external index target and durable processing policy.
+    then choose the external index target.
+
+## 2026-04-25 - Outbox Worker Processing Policy
+
+- Touched areas:
+  - External outbox worker
+  - Worker operations documentation
+- Purpose:
+  - Make the worker usable for bounded JSONL publishing before a message bus is
+    chosen.
+  - Add queue visibility and explicit processed-row cleanup for test-instance
+    maintenance.
+- Behavior changed:
+  - Default worker output is `/tmp/redmine-outbox/derived_documents.jsonl`.
+  - `--status` reports pending, ready, locked, failed, and processed counts.
+  - `--purge-processed-days N` previews processed-row cleanup; `--apply-purge`
+    is required to delete rows.
+  - Normal processing still marks rows processed only after successful output.
+- LAN test result:
+  - `./redmine_outbox_worker.py --status` reported 100 total rows, 100 ready,
+    0 locked, 0 failed, and 0 processed before the processing run.
+  - `./redmine_outbox_worker.py --dry-run --batch-size 5` previewed the first
+    five rows without marking them processed.
+  - `./redmine_outbox_worker.py --batch-size 5` processed five rows and wrote
+    six JSONL documents to `/tmp/redmine-outbox/derived_documents.jsonl`.
+  - Follow-up status reported 95 pending, 95 ready, 0 locked, 0 failed, and
+    5 processed.
+  - `./redmine_outbox_worker.py --purge-processed-days 30` previewed zero rows.
+  - Follow-up `./validate_helpdesk_outbox_worker.py` passed with controlled
+    Helpdesk issue `#39875`; final status reported 118 total rows, 113 ready,
+    0 locked, 0 failed, and 5 processed.
 
 ## 2026-04-25 - Helpdesk Outbox Worker Validation
 

docs/test_instance_post_import.md

@@ -149,7 +149,35 @@ Helpdesk behavior, checks the matching `event_outbox_events` rows, and dry-runs
 worker enrichment without claiming or marking rows processed. Details are in
 `docs/helpdesk_outbox_worker_validation.md`.
 
-## 9. Re-Run The Read-Only Validator
+## 9. Check Or Process The Outbox Worker
+
+Inspect outbox queue state:
+
+```sh
+./redmine_outbox_worker.py --status
+```
+
+Preview worker output without marking rows processed:
+
+```sh
+./redmine_outbox_worker.py --dry-run --batch-size 10
+```
+
+Process a small bounded batch only after the dry-run output looks correct:
+
+```sh
+./redmine_outbox_worker.py --batch-size 5
+```
+
+The default JSONL output path is
+`/tmp/redmine-outbox/derived_documents.jsonl`. Use `--output` to override it.
+Processed rows are retained by default. To preview test-instance cleanup:
+
+```sh
+./redmine_outbox_worker.py --purge-processed-days 30
+```
+
+## 10. Re-Run The Read-Only Validator
 
 Finish by running: