Add outbox worker processing policy
This commit is contained in:
@@ -0,0 +1,63 @@
|
||||
# Outbox Worker Policy
|
||||
|
||||
`redmine_outbox_worker.py` is the first external publisher for
|
||||
`event_outbox_events`. It currently publishes enriched documents to JSONL, but
|
||||
the processing policy is intended to carry forward to a later message bus.
|
||||
|
||||
## Default Output
|
||||
|
||||
The default output is:
|
||||
|
||||
```sh
|
||||
/tmp/redmine-outbox/derived_documents.jsonl
|
||||
```
|
||||
|
||||
Use `--output /path/to/file.jsonl` to override it. The worker creates parent
|
||||
directories as needed.
|
||||
|
||||
## Processing Semantics
|
||||
|
||||
- Read eligible pending rows globally in id order.
|
||||
- Claim a bounded batch with `locked_at` and `locked_by`.
|
||||
- Enrich each event with read-only joins.
|
||||
- Append derived JSON documents to the output.
|
||||
- Mark a row `processed_at` only after the output write succeeds.
|
||||
- On failure, increment `attempts`, write `last_error`, and release the lock.
|
||||
|
||||
This is at-least-once delivery. Downstream consumers must tolerate duplicate
|
||||
documents keyed by `event_id` or `doc_id`.
|
||||
|
||||
## Operations
|
||||
|
||||
Inspect queue state:
|
||||
|
||||
```sh
|
||||
./redmine_outbox_worker.py --status
|
||||
```
|
||||
|
||||
Preview a batch without claiming or marking rows processed:
|
||||
|
||||
```sh
|
||||
./redmine_outbox_worker.py --dry-run --batch-size 10
|
||||
```
|
||||
|
||||
Process a bounded batch:
|
||||
|
||||
```sh
|
||||
./redmine_outbox_worker.py --batch-size 20
|
||||
```
|
||||
|
||||
Preview processed-row cleanup:
|
||||
|
||||
```sh
|
||||
./redmine_outbox_worker.py --purge-processed-days 30
|
||||
```
|
||||
|
||||
Apply cleanup explicitly:
|
||||
|
||||
```sh
|
||||
./redmine_outbox_worker.py --purge-processed-days 30 --apply-purge
|
||||
```
|
||||
|
||||
Processed rows are kept by default. Purge is an administrative cleanup tool for
|
||||
test instances and should not be part of normal publishing.
|
||||
Reference in New Issue
Block a user