Add outbox worker processing policy

This commit is contained in:
Jason Thistlethwaite
2026-04-25 00:53:49 +00:00
parent c9f4c69525
commit f109fdcb91
8 changed files with 249 additions and 33 deletions
+21 -6
View File
@@ -384,7 +384,7 @@ row exists.
## Worker/Rake Task
Current implemented worker-facing command:
Current implemented Redmine-side worker-facing command:
```sh
bundle exec rake redmine_event_outbox:dump RAILS_ENV=production
@@ -392,7 +392,7 @@ bundle exec rake redmine_event_outbox:dump RAILS_ENV=production
This prints pending rows as JSON and does not mark them processed.
Next worker command:
Historical planned Redmine-side publish command:
```sh
bundle exec rake redmine_event_outbox:publish RAILS_ENV=production
@@ -411,10 +411,11 @@ bundle exec rake redmine_event_outbox:publish
bundle exec rake redmine_event_outbox:publish RETRY=1
```
Initial publisher target can be stdout or a local JSONL file. That lets us test
the Redmine side before choosing Redis Streams or RabbitMQ.
The current external publisher is `redmine_outbox_worker.py`. Its initial
publisher target is JSONL at `/tmp/redmine-outbox/derived_documents.jsonl`,
which lets us test the Redmine side before choosing Redis Streams or RabbitMQ.
The next implementation should keep the worker small and conservative:
The worker stays small and conservative:
- select pending rows in id order
- lock or claim a bounded batch
@@ -424,6 +425,19 @@ The next implementation should keep the worker small and conservative:
- leave failed rows available for retry
- make duplicate delivery acceptable to consumers
Current external worker commands:
```sh
./redmine_outbox_worker.py --status
./redmine_outbox_worker.py --dry-run --batch-size 10
./redmine_outbox_worker.py --batch-size 20
./redmine_outbox_worker.py --purge-processed-days 30
./redmine_outbox_worker.py --purge-processed-days 30 --apply-purge
```
Processed rows are retained by default. Purge is explicit, age-gated, and
intended for test-instance cleanup, not normal delivery.
Later publisher targets:
- Redis Streams
@@ -550,7 +564,8 @@ decision code fails.
## Open Questions
- How long should processed outbox rows be retained?
- Do production processed rows need a formal retention window, or should they
be retained until external index/message-bus replay policy is settled?
- Do we need a Redmine admin page to inspect outbox health, or is a rake task/log enough?
- What is the current production mail ingestion path?
- Which vector-capable external search index should be used first?