redmine/docs/event_outbox_spec.md

Redmine Event Outbox Spec

Purpose

Add a low-risk event boundary around the legacy Redmine install so external tools can react to Redmine changes without polling heavily and without making Redmine dependent on a message bus, search service, or experimental automation code.

The first version should capture issue and journal activity into a local database outbox table. A separate worker can later publish those rows to Redis, RabbitMQ, a webhook, or a search/indexing service.

Goals

• Record issue creation and update events.
• Record issue journal/comment events.
• Record contact creation and update events from redmine_contacts.
• Keep Redmine issue saves working even if external messaging/search systems are down.
• Make event processing replayable from a durable local table.
• Keep the plugin small and compatible with Redmine 3.4.4-era plugin patterns.
• Support future indexing for queries like "recent events for customer A".

Non-Goals For V1

• No direct Redis/RabbitMQ publish inside Redmine request callbacks.
• No semantic/vector search inside Redmine.
• No replacement of Redmine email handling in the first version.
• No guarantee that external consumers receive each event exactly once.
• No broad UI in Redmine beyond optional admin/status views later.
• No binary image indexing inside Redmine.

Safety Rule

Redmine must not fail an issue create/update because a broker, worker, search index, DNS lookup, network call, or external service failed.

V1 should therefore only write to Redmine's own database from inside Redmine. All network publishing happens outside Redmine in a worker.

There is one important tradeoff:

• If outbox insert failures are rescued, ticket saves are maximally protected but rare event loss is possible.
• If outbox rows are written transactionally and failures are not rescued, event durability is stronger but ticket saves can fail because the outbox write failed.

For this project, prefer protecting ticket saves. The outbox insert should be simple and local, and failures should be logged with enough detail to diagnose.

Proposed Plugin

Plugin name:

redmine_event_outbox

Directory:

redmine-copy/plugins/redmine_event_outbox/

Initial structure:

init.rb
db/migrate/001_create_event_outbox_events.rb
app/models/event_outbox_event.rb
lib/redmine_event_outbox.rb
lib/redmine_event_outbox/hooks/issues_hook.rb
lib/redmine_event_outbox/patches/contact_patch.rb
lib/redmine_event_outbox/patches/journal_patch.rb
lib/tasks/redmine_event_outbox.rake

Event Table

Table name:

event_outbox_events

Columns:

id                integer primary key
event_type        string, required
source_type       string, required
source_id         integer, required
project_id        integer, nullable
issue_id          integer, nullable
journal_id        integer, nullable
user_id           integer, nullable
occurred_at       datetime, required
payload           text/json, required
processed_at      datetime, nullable
attempts          integer, default 0
last_error        text, nullable
locked_at         datetime, nullable
locked_by         string, nullable
created_at        datetime
updated_at        datetime

Production Redmine currently uses MySQL. Store payload as text for Redmine 3.4.4 compatibility and serialize JSON in application code. Do not rely on native MySQL JSON behavior in V1.

Useful indexes:

index_event_outbox_events_on_processed_at_id
index_event_outbox_events_on_event_type
index_event_outbox_events_on_issue_id
index_event_outbox_events_on_project_id
index_event_outbox_events_on_occurred_at

V1 Event Types

issue.created

Created after a new issue is saved.

Trigger candidate found in Redmine 3.4.4:

controller_issues_new_after_save

Payload:

{
  "event_type": "issue.created",
  "issue_id": 123,
  "project_id": 4,
  "tracker_id": 1,
  "status_id": 1,
  "priority_id": 2,
  "author_id": 5,
  "author_name": "Jane User",
  "assigned_to_id": null,
  "assigned_to_name": null,
  "subject": "Example",
  "created_on": "2026-04-21T12:00:00Z",
  "updated_on": "2026-04-21T12:00:00Z"
}

issue.updated

Created after an existing issue update succeeds.

Trigger candidate found in Redmine 3.4.4:

controller_issues_edit_after_save

Payload should include issue identity plus the current journal id when present:

{
  "event_type": "issue.updated",
  "issue_id": 123,
  "journal_id": 456,
  "project_id": 4,
  "status_id": 2,
  "assigned_to_id": 7,
  "assigned_to_name": "Support User",
  "actor_id": 5,
  "actor_name": "Jane User",
  "subject": "Example",
  "updated_on": "2026-04-21T12:15:00Z"
}

journal.created

Created when a journal row for an issue is committed.

Trigger candidate found in Redmine 3.4.4:

Journal.after_commit :on => :create

The plugin can patch Journal with a separate after_commit callback.

Payload:

{
  "event_type": "journal.created",
  "journal_id": 456,
  "issue_id": 123,
  "project_id": 4,
  "user_id": 5,
  "user_name": "Jane User",
  "subject": "Example",
  "private_notes": false,
  "has_notes": true,
  "changed_fields": ["status_id", "assigned_to_id"],
  "created_on": "2026-04-21T12:15:00Z"
}

V1 should not put full private note text into the payload. The indexing worker can fetch detail with appropriate credentials if needed.

contact.created

Created after a contact is committed.

Trigger candidate:

Contact.after_commit :on => :create

The plugin can patch Contact from redmine_contacts when that plugin is installed.

Payload:

{
  "event_type": "contact.created",
  "contact_id": 321,
  "project_ids": [4],
  "is_company": false,
  "name": "Customer Name",
  "company": "Customer Company",
  "author_id": 5,
  "author_name": "Jane User",
  "assigned_to_id": 7,
  "assigned_to_name": "Support User",
  "created_on": "2026-04-21T12:20:00Z",
  "updated_on": "2026-04-21T12:20:00Z"
}

contact.updated

Created after a contact update is committed.

Trigger candidate:

Contact.after_commit :on => :update

Payload:

{
  "event_type": "contact.updated",
  "contact_id": 321,
  "project_ids": [4],
  "is_company": false,
  "name": "Customer Name",
  "company": "Customer Company",
  "actor_id": 5,
  "actor_name": "Jane User",
  "assigned_to_id": 7,
  "assigned_to_name": "Support User",
  "updated_on": "2026-04-21T12:25:00Z"
}

Contact payloads should include enough human context for downstream consumers to decide whether to fetch the full contact. Avoid embedding all phone/email/address data in V1 payloads unless a consumer proves it needs those fields inline.

helpdesk_ticket.created

Created after a HelpdeskTicket row is committed when redmine_contacts_helpdesk is installed.

Payload:

{
  "event_type": "helpdesk_ticket.created",
  "helpdesk_ticket_id": 10,
  "issue_id": 123,
  "project_id": 4,
  "contact_id": 321,
  "message_id": "<message@example>",
  "is_incoming": true,
  "source": 0,
  "from_address": "customer@example.com",
  "to_address": "support@example.com",
  "cc_address": null,
  "subject": "Example",
  "ticket_date": "2026-04-21T12:00:00Z"
}

helpdesk_ticket.updated

Created after an existing HelpdeskTicket row is updated. This is useful when the contact, source, or message metadata is corrected after ticket creation.

journal_message.created

Created after a JournalMessage row is committed. This is the authoritative per-email metadata layer for helpdesk conversation search.

Payload:

{
  "event_type": "journal_message.created",
  "journal_message_id": 20,
  "journal_id": 456,
  "issue_id": 123,
  "project_id": 4,
  "contact_id": 321,
  "message_id": "<reply@example>",
  "is_incoming": false,
  "source": 0,
  "from_address": "support@example.com",
  "to_address": "customer@example.com",
  "cc_address": null,
  "has_bcc_address": false,
  "private_notes": false,
  "has_notes": true,
  "message_date": "2026-04-21T12:15:00Z"
}

The event records whether BCC metadata exists but does not store the BCC address itself.

journal_message.updated

Created after an existing JournalMessage row is updated. This lets downstream indexes repair message-level documents when message metadata is corrected.

Payload Context Policy

Payloads should be lightweight but useful. Include:

• ids needed for follow-up fetches
• event type and timestamp
• issue/contact subject or display name
• user id and user name when known
• project id(s)
• changed field names when available

Avoid:

• full private notes
• large descriptions/background fields
• attachments or binary content
• full email bodies
• BCC addresses
• large custom field dumps

This gives consumers enough information to decide whether they care about an event while keeping the outbox table compact and lower-risk.

Delivery Semantics

V1 should provide at-least-once processing from the outbox to downstream systems.

Consumers must be idempotent. Use the outbox row id as the event id:

{
  "event_id": 98765,
  "event_type": "issue.updated"
}

Duplicates are acceptable. Silent message loss is not acceptable once an outbox row exists.

Worker/Rake Task

Current implemented Redmine-side worker-facing command:

bundle exec rake redmine_event_outbox:dump RAILS_ENV=production

This prints pending rows as JSON and does not mark them processed.

Historical planned Redmine-side publish command:

bundle exec rake redmine_event_outbox:publish RAILS_ENV=production

V1 modes:

# Print pending rows as JSON without marking processed.
bundle exec rake redmine_event_outbox:dump

# Process pending rows and mark success.
bundle exec rake redmine_event_outbox:publish

# Retry failed/unprocessed rows.
bundle exec rake redmine_event_outbox:publish RETRY=1

The current external publisher is redmine_outbox_worker.py. Its initial publisher target is JSONL at /tmp/redmine-outbox/derived_documents.jsonl, which lets us test the Redmine side before choosing Redis Streams or RabbitMQ.

The worker stays small and conservative:

• select pending rows in id order
• lock or claim a bounded batch
• publish each row
• mark processed_at only after publish succeeds
• increment attempts and write last_error on failure
• leave failed rows available for retry
• make duplicate delivery acceptable to consumers

Current external worker commands:

./redmine_outbox_worker.py --status
./redmine_outbox_worker.py --dry-run --batch-size 10
./redmine_outbox_worker.py --batch-size 20
./redmine_outbox_worker.py --purge-processed-days 30
./redmine_outbox_worker.py --purge-processed-days 30 --apply-purge

Processed rows are retained by default. Purge is explicit, age-gated, and intended for test-instance cleanup, not normal delivery.

Later publisher targets:

• Redis Streams
• RabbitMQ
• webhook HTTP POST
• local search/indexing service

Search/Indexing Direction

The semantic/fuzzy search system should be external to Redmine.

The first external search index should have strong vector-search support because future work will use embeddings heavily, including image embeddings. Redmine should only emit events and identifiers; the external indexer should fetch, transform, chunk, embed, and store searchable material.

Likely derived entities:

• contacts
• issues
• journals
• helpdesk email messages
• status/assignee changes
• timestamps and project links
• future image/document embeddings

Likely query examples:

recent events for customer A
open issues involving customer A
recent emails from customer A
status changes for customer A this week

The outbox does not need to contain all searchable text. It only needs enough identity and timestamps for a worker to fetch and update the external index.

Candidate vector-capable search/index options to evaluate later:

• PostgreSQL with pgvector
• Qdrant
• Weaviate
• OpenSearch/Elasticsearch with vector fields
• LanceDB

Since production Redmine uses MySQL, the search/index database should be treated as a separate derived system rather than as a feature of the Redmine database.

Email Handling Direction

Email handling should be handled after the basic issue/journal outbox is stable.

Observed areas to inspect further:

• Redmine core MailHandler
• redmine_contacts rake tasks under lib/tasks/contacts_email.rake
• redmine_contacts_helpdesk
• helpdesk_mailer endpoint and related mail handling code

Future event types may include:

email.received
email.ignored
email.created_issue
email.updated_issue
helpdesk_ticket.created
helpdesk_ticket.updated

Custom mail routing should have a safe fallback to current behavior if external decision code fails.

Implementation Phases

Phase 1: Outbox Skeleton

• Status: implemented and tested on the LAN Redmine copy.
• Created plugin.
• Added migration and model.
• Added helper method for safe event creation.
• Added issue create/update hooks.
• Added basic rake task to dump pending rows.
• Verified issue create/update writes outbox rows in the Meetings project.

Phase 2: Journal Events

• Status: implemented and tested on the LAN Redmine copy.
• Patched Journal with after_commit.
• Records journal.created.
• Includes changed field names from journal details.
• Avoids full private note content in payload.

Phase 3: Contact Events

• Status: implemented and tested on the LAN Redmine copy.
• Patched Contact from redmine_contacts if available.
• Records contact.created.
• Records contact.updated.
• Includes contact display name, company, project ids, and relevant user context.
• Keeps payloads small; full contact detail can be fetched by external workers.

Phase 4: Worker

• Add locking fields.
• Process pending rows in batches.
• Mark processed_at only after successful publish.
• Track attempts and last error.
• Support dry-run/dump mode.

Phase 5: External Index Prototype

• Build a small external worker outside Redmine.
• Read outbox rows.
• Fetch affected issue/contact/journal details through API or DB.
• Store in a vector-capable search database.
• Add a CLI for recent customer timelines.

Phase 6: Message Bus

• Choose Redis Streams or RabbitMQ based on actual consumer needs.
• Keep Redmine unchanged; only the worker publisher changes.

Open Questions

• Do production processed rows need a formal retention window, or should they be retained until external index/message-bus replay policy is settled?
• Do we need a Redmine admin page to inspect outbox health, or is a rake task/log enough?
• What is the current production mail ingestion path?
• Which vector-capable external search index should be used first?
• Should contact payloads include normalized primary email/phone, or is that too much inline data?

Current Checkpoint

Known-good archive:

dist/redmine_event_outbox-0.0.1-known-good-20260421T143957Z.tar.gz

Manifest:

dist/redmine_event_outbox-0.0.1-known-good-20260421T143957Z.MANIFEST.md

Tested on LAN Redmine copy:

• Migrated successfully.
• Passenger restart via touch tmp/restart.txt.
• Verified issue.created.
• Verified issue.updated.
• Verified journal.created.
• Verified contact.created.
• Verified contact.updated.
• Verified redmine_event_outbox:dump.

Test artifacts left on LAN copy for traceability:

• Issue #39858 in project Meetings.
• Contact #4337 in project Email Test.
• Six outbox rows in event_outbox_events.