Initial Redmine tooling and local plugin forks

This commit is contained in:
Jason Thistlethwaite
2026-04-24 22:01:18 +00:00
commit 9f682af0eb
683 changed files with 56878 additions and 0 deletions
+588
View File
@@ -0,0 +1,588 @@
# Redmine Event Outbox Spec
## Purpose
Add a low-risk event boundary around the legacy Redmine install so external tools
can react to Redmine changes without polling heavily and without making Redmine
dependent on a message bus, search service, or experimental automation code.
The first version should capture issue and journal activity into a local database
outbox table. A separate worker can later publish those rows to Redis, RabbitMQ,
a webhook, or a search/indexing service.
## Goals
- Record issue creation and update events.
- Record issue journal/comment events.
- Record contact creation and update events from `redmine_contacts`.
- Keep Redmine issue saves working even if external messaging/search systems are
down.
- Make event processing replayable from a durable local table.
- Keep the plugin small and compatible with Redmine 3.4.4-era plugin patterns.
- Support future indexing for queries like "recent events for customer A".
## Non-Goals For V1
- No direct Redis/RabbitMQ publish inside Redmine request callbacks.
- No semantic/vector search inside Redmine.
- No replacement of Redmine email handling in the first version.
- No guarantee that external consumers receive each event exactly once.
- No broad UI in Redmine beyond optional admin/status views later.
- No binary image indexing inside Redmine.
## Safety Rule
Redmine must not fail an issue create/update because a broker, worker, search
index, DNS lookup, network call, or external service failed.
V1 should therefore only write to Redmine's own database from inside Redmine. All
network publishing happens outside Redmine in a worker.
There is one important tradeoff:
- If outbox insert failures are rescued, ticket saves are maximally protected but
rare event loss is possible.
- If outbox rows are written transactionally and failures are not rescued, event
durability is stronger but ticket saves can fail because the outbox write
failed.
For this project, prefer protecting ticket saves. The outbox insert should be
simple and local, and failures should be logged with enough detail to diagnose.
## Proposed Plugin
Plugin name:
```text
redmine_event_outbox
```
Directory:
```text
redmine-copy/plugins/redmine_event_outbox/
```
Initial structure:
```text
init.rb
db/migrate/001_create_event_outbox_events.rb
app/models/event_outbox_event.rb
lib/redmine_event_outbox.rb
lib/redmine_event_outbox/hooks/issues_hook.rb
lib/redmine_event_outbox/patches/contact_patch.rb
lib/redmine_event_outbox/patches/journal_patch.rb
lib/tasks/redmine_event_outbox.rake
```
## Event Table
Table name:
```text
event_outbox_events
```
Columns:
```text
id integer primary key
event_type string, required
source_type string, required
source_id integer, required
project_id integer, nullable
issue_id integer, nullable
journal_id integer, nullable
user_id integer, nullable
occurred_at datetime, required
payload text/json, required
processed_at datetime, nullable
attempts integer, default 0
last_error text, nullable
locked_at datetime, nullable
locked_by string, nullable
created_at datetime
updated_at datetime
```
Production Redmine currently uses MySQL. Store `payload` as `text` for Redmine
3.4.4 compatibility and serialize JSON in application code. Do not rely on
native MySQL JSON behavior in V1.
Useful indexes:
```text
index_event_outbox_events_on_processed_at_id
index_event_outbox_events_on_event_type
index_event_outbox_events_on_issue_id
index_event_outbox_events_on_project_id
index_event_outbox_events_on_occurred_at
```
## V1 Event Types
### issue.created
Created after a new issue is saved.
Trigger candidate found in Redmine 3.4.4:
```ruby
controller_issues_new_after_save
```
Payload:
```json
{
"event_type": "issue.created",
"issue_id": 123,
"project_id": 4,
"tracker_id": 1,
"status_id": 1,
"priority_id": 2,
"author_id": 5,
"author_name": "Jane User",
"assigned_to_id": null,
"assigned_to_name": null,
"subject": "Example",
"created_on": "2026-04-21T12:00:00Z",
"updated_on": "2026-04-21T12:00:00Z"
}
```
### issue.updated
Created after an existing issue update succeeds.
Trigger candidate found in Redmine 3.4.4:
```ruby
controller_issues_edit_after_save
```
Payload should include issue identity plus the current journal id when present:
```json
{
"event_type": "issue.updated",
"issue_id": 123,
"journal_id": 456,
"project_id": 4,
"status_id": 2,
"assigned_to_id": 7,
"assigned_to_name": "Support User",
"actor_id": 5,
"actor_name": "Jane User",
"subject": "Example",
"updated_on": "2026-04-21T12:15:00Z"
}
```
### journal.created
Created when a journal row for an issue is committed.
Trigger candidate found in Redmine 3.4.4:
```ruby
Journal.after_commit :on => :create
```
The plugin can patch `Journal` with a separate `after_commit` callback.
Payload:
```json
{
"event_type": "journal.created",
"journal_id": 456,
"issue_id": 123,
"project_id": 4,
"user_id": 5,
"user_name": "Jane User",
"subject": "Example",
"private_notes": false,
"has_notes": true,
"changed_fields": ["status_id", "assigned_to_id"],
"created_on": "2026-04-21T12:15:00Z"
}
```
V1 should not put full private note text into the payload. The indexing worker can
fetch detail with appropriate credentials if needed.
### contact.created
Created after a contact is committed.
Trigger candidate:
```ruby
Contact.after_commit :on => :create
```
The plugin can patch `Contact` from `redmine_contacts` when that plugin is
installed.
Payload:
```json
{
"event_type": "contact.created",
"contact_id": 321,
"project_ids": [4],
"is_company": false,
"name": "Customer Name",
"company": "Customer Company",
"author_id": 5,
"author_name": "Jane User",
"assigned_to_id": 7,
"assigned_to_name": "Support User",
"created_on": "2026-04-21T12:20:00Z",
"updated_on": "2026-04-21T12:20:00Z"
}
```
### contact.updated
Created after a contact update is committed.
Trigger candidate:
```ruby
Contact.after_commit :on => :update
```
Payload:
```json
{
"event_type": "contact.updated",
"contact_id": 321,
"project_ids": [4],
"is_company": false,
"name": "Customer Name",
"company": "Customer Company",
"actor_id": 5,
"actor_name": "Jane User",
"assigned_to_id": 7,
"assigned_to_name": "Support User",
"updated_on": "2026-04-21T12:25:00Z"
}
```
Contact payloads should include enough human context for downstream consumers to
decide whether to fetch the full contact. Avoid embedding all phone/email/address
data in V1 payloads unless a consumer proves it needs those fields inline.
### helpdesk_ticket.created
Created after a `HelpdeskTicket` row is committed when
`redmine_contacts_helpdesk` is installed.
Payload:
```json
{
"event_type": "helpdesk_ticket.created",
"helpdesk_ticket_id": 10,
"issue_id": 123,
"project_id": 4,
"contact_id": 321,
"message_id": "<message@example>",
"is_incoming": true,
"source": 0,
"from_address": "customer@example.com",
"to_address": "support@example.com",
"cc_address": null,
"subject": "Example",
"ticket_date": "2026-04-21T12:00:00Z"
}
```
### helpdesk_ticket.updated
Created after an existing `HelpdeskTicket` row is updated. This is useful when
the contact, source, or message metadata is corrected after ticket creation.
### journal_message.created
Created after a `JournalMessage` row is committed. This is the authoritative
per-email metadata layer for helpdesk conversation search.
Payload:
```json
{
"event_type": "journal_message.created",
"journal_message_id": 20,
"journal_id": 456,
"issue_id": 123,
"project_id": 4,
"contact_id": 321,
"message_id": "<reply@example>",
"is_incoming": false,
"source": 0,
"from_address": "support@example.com",
"to_address": "customer@example.com",
"cc_address": null,
"has_bcc_address": false,
"private_notes": false,
"has_notes": true,
"message_date": "2026-04-21T12:15:00Z"
}
```
The event records whether BCC metadata exists but does not store the BCC address
itself.
### journal_message.updated
Created after an existing `JournalMessage` row is updated. This lets downstream
indexes repair message-level documents when message metadata is corrected.
## Payload Context Policy
Payloads should be lightweight but useful. Include:
- ids needed for follow-up fetches
- event type and timestamp
- issue/contact subject or display name
- user id and user name when known
- project id(s)
- changed field names when available
Avoid:
- full private notes
- large descriptions/background fields
- attachments or binary content
- full email bodies
- BCC addresses
- large custom field dumps
This gives consumers enough information to decide whether they care about an
event while keeping the outbox table compact and lower-risk.
## Delivery Semantics
V1 should provide at-least-once processing from the outbox to downstream systems.
Consumers must be idempotent. Use the outbox row id as the event id:
```json
{
"event_id": 98765,
"event_type": "issue.updated"
}
```
Duplicates are acceptable. Silent message loss is not acceptable once an outbox
row exists.
## Worker/Rake Task
Current implemented worker-facing command:
```sh
bundle exec rake redmine_event_outbox:dump RAILS_ENV=production
```
This prints pending rows as JSON and does not mark them processed.
Next worker command:
```sh
bundle exec rake redmine_event_outbox:publish RAILS_ENV=production
```
V1 modes:
```sh
# Print pending rows as JSON without marking processed.
bundle exec rake redmine_event_outbox:dump
# Process pending rows and mark success.
bundle exec rake redmine_event_outbox:publish
# Retry failed/unprocessed rows.
bundle exec rake redmine_event_outbox:publish RETRY=1
```
Initial publisher target can be stdout or a local JSONL file. That lets us test
the Redmine side before choosing Redis Streams or RabbitMQ.
The next implementation should keep the worker small and conservative:
- select pending rows in id order
- lock or claim a bounded batch
- publish each row
- mark `processed_at` only after publish succeeds
- increment `attempts` and write `last_error` on failure
- leave failed rows available for retry
- make duplicate delivery acceptable to consumers
Later publisher targets:
- Redis Streams
- RabbitMQ
- webhook HTTP POST
- local search/indexing service
## Search/Indexing Direction
The semantic/fuzzy search system should be external to Redmine.
The first external search index should have strong vector-search support because
future work will use embeddings heavily, including image embeddings. Redmine
should only emit events and identifiers; the external indexer should fetch,
transform, chunk, embed, and store searchable material.
Likely derived entities:
- contacts
- issues
- journals
- helpdesk email messages
- status/assignee changes
- timestamps and project links
- future image/document embeddings
Likely query examples:
```text
recent events for customer A
open issues involving customer A
recent emails from customer A
status changes for customer A this week
```
The outbox does not need to contain all searchable text. It only needs enough
identity and timestamps for a worker to fetch and update the external index.
Candidate vector-capable search/index options to evaluate later:
- PostgreSQL with `pgvector`
- Qdrant
- Weaviate
- OpenSearch/Elasticsearch with vector fields
- LanceDB
Since production Redmine uses MySQL, the search/index database should be treated
as a separate derived system rather than as a feature of the Redmine database.
## Email Handling Direction
Email handling should be handled after the basic issue/journal outbox is stable.
Observed areas to inspect further:
- Redmine core `MailHandler`
- `redmine_contacts` rake tasks under `lib/tasks/contacts_email.rake`
- `redmine_contacts_helpdesk`
- `helpdesk_mailer` endpoint and related mail handling code
Future event types may include:
```text
email.received
email.ignored
email.created_issue
email.updated_issue
helpdesk_ticket.created
helpdesk_ticket.updated
```
Custom mail routing should have a safe fallback to current behavior if external
decision code fails.
## Implementation Phases
### Phase 1: Outbox Skeleton
- Status: implemented and tested on the LAN Redmine copy.
- Created plugin.
- Added migration and model.
- Added helper method for safe event creation.
- Added issue create/update hooks.
- Added basic rake task to dump pending rows.
- Verified issue create/update writes outbox rows in the `Meetings` project.
### Phase 2: Journal Events
- Status: implemented and tested on the LAN Redmine copy.
- Patched `Journal` with `after_commit`.
- Records `journal.created`.
- Includes changed field names from journal details.
- Avoids full private note content in payload.
### Phase 3: Contact Events
- Status: implemented and tested on the LAN Redmine copy.
- Patched `Contact` from `redmine_contacts` if available.
- Records `contact.created`.
- Records `contact.updated`.
- Includes contact display name, company, project ids, and relevant user context.
- Keeps payloads small; full contact detail can be fetched by external workers.
### Phase 4: Worker
- Add locking fields.
- Process pending rows in batches.
- Mark `processed_at` only after successful publish.
- Track attempts and last error.
- Support dry-run/dump mode.
### Phase 5: External Index Prototype
- Build a small external worker outside Redmine.
- Read outbox rows.
- Fetch affected issue/contact/journal details through API or DB.
- Store in a vector-capable search database.
- Add a CLI for recent customer timelines.
### Phase 6: Message Bus
- Choose Redis Streams or RabbitMQ based on actual consumer needs.
- Keep Redmine unchanged; only the worker publisher changes.
## Open Questions
- How long should processed outbox rows be retained?
- Do we need a Redmine admin page to inspect outbox health, or is a rake task/log enough?
- What is the current production mail ingestion path?
- Which vector-capable external search index should be used first?
- Should contact payloads include normalized primary email/phone, or is that too much inline data?
## Current Checkpoint
Known-good archive:
```text
dist/redmine_event_outbox-0.0.1-known-good-20260421T143957Z.tar.gz
```
Manifest:
```text
dist/redmine_event_outbox-0.0.1-known-good-20260421T143957Z.MANIFEST.md
```
Tested on LAN Redmine copy:
- Migrated successfully.
- Passenger restart via `touch tmp/restart.txt`.
- Verified `issue.created`.
- Verified `issue.updated`.
- Verified `journal.created`.
- Verified `contact.created`.
- Verified `contact.updated`.
- Verified `redmine_event_outbox:dump`.
Test artifacts left on LAN copy for traceability:
- Issue `#39858` in project `Meetings`.
- Contact `#4337` in project `Email Test`.
- Six outbox rows in `event_outbox_events`.
+94
View File
@@ -0,0 +1,94 @@
# Pre-Existing Issues Log
This log tracks bugs, warnings, and confusing behaviors noticed while working on
the Redmine 3.4.4 local fork. It is not a task tracker; it is a place to keep
context so future plugin edits do not have to rediscover old problems.
## 2026-04-21 - Duplicate Contact/Helpdesk Avatar IDs
- Area: `redmine_contacts`, `redmine_contacts_helpdesk`
- Status: observed/analyzing
- Symptom: pages that display multiple contact/user avatars may show or behave
as if every row has the same thumbnail/avatar.
- Relevant code:
- `redmine_contacts/lib/redmine_contacts/helpers/contacts_helper.rb`
- helpdesk/contact views that call `link_to ..., :id => "avatar"` repeatedly
- Current assessment:
- Several views/helpers render repeated `id="avatar"` attributes on lists of
contacts, issues, notes, deals, and helpdesk journals.
- Repeated DOM ids are invalid HTML and can cause JavaScript, tooltip, popup,
or CSS selectors using `#avatar` to bind to the wrong element or reuse the
first matching element.
- This is a better fit for the long-standing "same thumbnail for every user"
symptom than the Rails cache digest warning below.
- Next diagnostic/fix idea:
- Replace repeated `:id => "avatar"` with `:class => "avatar"` where no unique
id is required.
- Where an id is required, generate stable unique ids such as
`contact-avatar-#{contact.id}` or `journal-avatar-#{journal.id}`.
- Test on a page that currently displays multiple contacts/users with distinct
avatars.
## 2026-04-21 - `attachments/contacts_thumbnail` Cache Digest Warning
- Area: `redmine_contacts`
- Status: observed/analyzing
- Log message:
```text
Couldn't find template for digesting: attachments/contacts_thumbnail
```
- Relevant code:
- `redmine_contacts/lib/redmine_contacts/patches/attachments_controller_patch.rb`
- route: `attachments/contacts_thumbnail/:id(/:size)`
- Current assessment:
- `AttachmentsController#contacts_thumbnail` streams a generated thumbnail via
`send_file` or returns 404; it normally does not render a view template.
- Rails 4 cache digesting still probes for a conventional action template and
logs the missing-template warning.
- This is likely log noise and probably not the cause of the duplicate-avatar
symptom.
- Possible fix:
- Add a blank placeholder template at
`redmine_contacts/app/views/attachments/contacts_thumbnail.html.erb` with a
local fork comment explaining that the action streams files.
- Do this only after confirming it does not mask the duplicate-avatar bug.
## 2026-04-21 - Helpdesk Search Manual URL Confusion
- Area: local `redmine_contacts_helpdesk` search API change
- Status: mitigated in current working copy and LAN deployment
- Symptom:
- Visiting `/helpdesk_search/issues` or `/helpdesk_search/issues/1` produced
`ActionController::RoutingError` stack traces in `production.log`.
- Current assessment:
- The originally implemented API route was
`/helpdesk_search/issues/:issue_id/ticket`.
- Manual browser tests naturally tried the shorter paths.
- Mitigation:
- Added usage routes for `/helpdesk_search`, `/helpdesk_search/issues`, and
`/helpdesk_search/contacts`.
- Added `/helpdesk_search/issues/:issue_id` as an alias for the ticket lookup.
## 2026-04-21 - `acts_as_list` Redmine 4 Deprecation Warning
- Area: Redmine core/plugin compatibility
- Status: observed; low urgency while Redmine 3.4.4 remains the baseline
- Log message:
```text
DEPRECATION WARNING: The acts_as_list plugin will be removed from Redmine 4 core, use the acts_as_list gem or similar implementation instead. (called from acts_as_list at /usr/share/redmine/lib/plugins/acts_as_list/lib/active_record/acts/list.rb:34)
```
- Current assessment:
- This is an upgrade-compatibility warning from the Redmine/Rails stack, not a
current runtime failure.
- It means some installed plugin or model uses `acts_as_list` from Redmine
core. Redmine 4 removes that bundled implementation, so any future Redmine 4+
migration would need an explicit `acts_as_list` gem or a local replacement.
- Since the near-term baseline is Redmine 3.4.4 and upgrading is not currently
a goal, this should not block helpdesk search work.
- Next diagnostic/fix idea:
- If upgrade work resumes, search installed plugins and app models for
`acts_as_list`, then decide whether to add the gem or patch each caller.
+85
View File
@@ -0,0 +1,85 @@
# RedmineUP Local Fork Changelog
The installed RedmineUP `redmine_contacts` and `redmine_contacts_helpdesk`
plugins are treated as locally maintained legacy code for this Redmine 3.4.4
environment. Before risky edits, archive the current plugin directories in
`dist/` and record the purpose, touched behavior, and LAN test result here.
## Current Checkpoint
- Baseline:
- Redmine `3.4.4`
- `redmine_contacts` `4.1.2 PRO`
- `redmine_contacts_helpdesk` `3.0.9 PRO`
- Strategic direction:
- Treat helpdesk/customer data as first-class.
- Prefer local-fork plugin edits when they unlock safer search/indexing.
- Keep Redmine request paths independent from external worker/index failures.
- Implemented locally:
- `redmine_event_outbox` plugin with issue/journal/contact events.
- Optional helpdesk outbox hooks for `HelpdeskTicket` and `JournalMessage`.
- Read-only `helpdesk_search/*` JSON endpoints in the local helpdesk fork.
- Standalone contact CLI and read-only helpdesk export/search CLI.
- LAN deployment status:
- Helpdesk search routes were deployed and route-loaded successfully on the
LAN Redmine copy.
- Short alias/usage routes were added to avoid noisy routing errors during
manual browser testing.
- Full end-to-end helpdesk outbox validation is still pending.
- Next meaningful milestone:
- Build the external worker/indexer that consumes `event_outbox_events`,
enriches via read-only MySQL joins, and emits deterministic ticket/message
documents for external indexing.
## 2026-04-24 - POP3 Get Mail Compatibility Fix
- Touched plugin:
- `redmine_contacts`
- `redmine_contacts_helpdesk`
- Purpose:
- Fix Helpdesk POP3 retrieval on the LAN test host when Ruby 2.5 raises
`FrozenError: can't modify frozen String` inside `Net::POP3`.
- Allow Helpdesk outbound mail to use Mailpit's unauthenticated SMTP listener.
- Behavior changed:
- Changed POP3 message retrieval from `msg.pop` to `msg.pop(String.new)` so
Ruby's POP3 code appends chunks into an explicit mutable destination string.
- This does not change message handling semantics; it only avoids relying on
Ruby's default empty string argument being mutable.
- Changed Helpdesk SMTP delivery option construction to omit
`authentication`, `user_name`, and `password` when the project SMTP
authentication setting is blank.
- LAN test result:
- Deployed to `/usr/share/redmine/plugins/redmine_contacts`.
- `HelpdeskMailer.check_project(Project.find("fud-helpdesk").id)` completed
successfully and processed 1 message.
- Deployed to `/usr/share/redmine/plugins/redmine_contacts_helpdesk`.
- Mailpit rejected `AUTH PLAIN` with `502 5.5.1 Command not implemented`.
After blanking SMTP auth settings and omitting auth options, a Helpdesk test
mail for issue `#39863` was delivered to Mailpit.
## 2026-04-21 - Helpdesk Search Foundation
- Archives created before plugin edits:
- `dist/redmine_contacts-4.1.2-local-before-helpdesk-search-20260421T215548Z.tar.gz`
- `dist/redmine_contacts_helpdesk-3.0.9-local-before-helpdesk-search-20260421T215548Z.tar.gz`
- Touched plugins:
- `redmine_contacts_helpdesk`
- `redmine_event_outbox`
- Purpose:
- Make helpdesk ticket and message identity available to external search and
indexing workers.
- Avoid relying on Redmine issue author when helpdesk-created tickets use
`Anonymous`.
- Behavior changed:
- Added read-only `helpdesk_search/*` JSON endpoints guarded by the existing
`view_helpdesk_tickets` permission.
- Added optional outbox hooks for `HelpdeskTicket` and `JournalMessage`.
- Payload/content policy:
- Include ids, source, direction, message id, and non-body address metadata.
- Do not copy email bodies, private note text, attachments, or BCC addresses
into event rows or the read API.
- LAN test result:
- Pending. Validate on the LAN Redmine copy by creating/updating a controlled
helpdesk ticket and journal message, checking `event_outbox_events`, and
calling the new `helpdesk_search/*` endpoints with a user/API key that has
`view_helpdesk_tickets`.