Legacy Redmine Search And Integration Project

This repository exists to make a heavily customized, business-critical Redmine 3.4.4 installation easier to work with without forcing a risky near-term platform upgrade.

The original production-like copy used for testing is on the LAN at:

http://192.168.50.170/

This project started because a large amount of customer/vendor communication is stored in Redmine through the old RedmineUP CRM and Helpdesk plugins, but the default UI and documented APIs are not good enough for operational search, contact lookup, message history review, and future automation.

The goal is not "modernize Redmine" in one jump. The goal is to create safe, pragmatic tooling around the existing system so the business can search and use its real communication history.

Why This Project Exists

The Redmine install has been tightly integrated into day-to-day business operations. It stores:

• CRM contacts
• helpdesk tickets
• helpdesk email conversations
• issue status and assignment history
• internal journal notes

The old RedmineUP plugin stack is effectively local legacy code now:

• redmine_contacts 4.1.2 PRO
• redmine_contacts_helpdesk 3.0.9 PRO

Tracked local plugin source lives under:

• plugins

The full redmine-copy/ tree is an ignored working/reference copy of the legacy install. Make local plugin changes in plugins/ first, then deploy or copy them into the test Redmine instance or redmine-copy/ as needed.

The Redmine API/MCP wrapper project now lives in:

• redMCP

That subproject contains the PHP wrapper that composes normal Redmine issue API responses with local Helpdesk metadata. Its dependencies are managed by Composer; redMCP/vendor/, local env files, and composer.phar are ignored.

There is no realistic short-term plan to:

• migrate to a newer RedmineUP package
• upgrade cleanly to Redmine 6
• replace the whole system before extracting value from the data already inside it

So this repository treats those plugins as maintainable local forks where needed.

Core Problem We Are Solving

The main operational need is better search over real customer/vendor communications.

The crucial discovery was that core Redmine issue data is not sufficient on its own. In the helpdesk workflow, the issue author can be Anonymous, while the actual customer identity and email metadata live in:

• helpdesk_tickets
• journal_messages
• contacts

That means any serious search/indexing system must treat helpdesk data as first-class and must not rely only on Redmine issue API fields.

Architecture Direction

The working architecture is:

1. Redmine records low-risk local outbox events.
2. An external worker reads outbox rows.
3. The worker enriches those rows with read-only MySQL joins against helpdesk/contact tables.
4. The worker builds ticket-level and message-level documents.
5. Those documents are indexed externally, eventually with vector search support.

The important safety rule is:

External search/indexing failures must not break normal Redmine saves.

That is why the event boundary lives in a local DB outbox table rather than in request-time webhooks, brokers, or direct embedding calls.

What Has Been Finished

1. Contact API Exploration And CLI

The old RedmineUP contacts plugin already exposes useful JSON routes such as:

/projects/customer-service/contacts.json

That led to the first standalone helper:

• redmine_contacts.py

It currently supports:

• fetching contacts to a local cache
• fuzzy-ish contact search over cached JSON
• light contact updates through the Redmine API
• helpdesk metadata lookup through the local read-only helpdesk API routes

2. Event Outbox Plugin

A small plugin was created at:

• redmine-copy/plugins/redmine_event_outbox

It records local database events into event_outbox_events.

Known-good archive:

• dist/redmine_event_outbox-0.0.1-known-good-20260421T143957Z.tar.gz
• manifest

Tested event types on the LAN copy:

• issue.created
• issue.updated
• journal.created
• contact.created
• contact.updated
• helpdesk_ticket.created
• helpdesk_ticket.updated
• journal_message.created

Planned/implemented locally but not yet observed in the controlled LAN validation workflow:

• journal_message.updated

The Helpdesk user workflow itself is now live-smoke-tested, including inbound Mailpit import, issueWithHelpdesk() metadata, default non-email updates, and explicit customer-visible Helpdesk replies. The repeatable outbox validation uses that same controlled workflow to verify Helpdesk ticket/message outbox rows and worker-derived documents without marking rows processed.

3. Local Helpdesk Plugin Fork Changes

We made targeted changes to the local fork of redmine_contacts_helpdesk:

• added a read-only JSON controller:
  • helpdesk_search_controller.rb
• added routes for:
  • ticket by issue
  • issues by contact
  • messages by issue
  • contact timeline
• added shorter alias/usage routes to avoid noisy routing errors in logs
• guarded those endpoints with the existing view_helpdesk_tickets permission

These changes were deployed to the LAN Redmine copy and route-loaded successfully.

4. Local Fork Hygiene

Before touching the RedmineUP plugin forks, rollback archives were created:

• redmine_contacts-4.1.2-local-before-helpdesk-search-20260421T215548Z.tar.gz
• redmine_contacts_helpdesk-3.0.9-local-before-helpdesk-search-20260421T215548Z.tar.gz

Manifests:

• contacts manifest
• helpdesk manifest

Change tracking docs:

• docs/redmineup_local_fork_changelog.md
• redmine-copy/plugins/redmine_contacts_helpdesk/LOCAL_CHANGELOG.md

5. Read-Only Helpdesk Export/Search CLI

We also built:

• redmine_helpdesk_search.py

Purpose:

• prove that helpdesk/customer/message data can be pulled correctly using read-only SQL joins
• produce ticket-level and message-level documents locally
• provide rough CLI search/timeline tooling for debugging and validation

Current state:

• works over SSH to the LAN Redmine host
• reads MySQL credentials from remote config/database.yml
• fetches helpdesk ticket and journal message documents into local JSONL cache
• can search/timeline/issues-by-contact over the local cache

Important note:

This script is a diagnostic/export tool, not the final search architecture. We intentionally stopped short of treating CLI speed optimization as the main goal.

6. External Outbox Worker Prototype

The first external worker prototype is:

• redmine_outbox_worker.py

It runs outside Redmine and consumes event_outbox_events over SSH/MySQL. The initial output target is deterministic local JSONL rather than a live search service:

• default output: /tmp/redmine-outbox/derived_documents.jsonl

Current behavior:

• dry-runs pending rows without marking them processed
• reports queue counts with --status
• claims bounded batches with locked_at and locked_by
• enriches helpdesk ticket/message/contact-related events with read-only joins
• writes derived event/ticket/message/contact documents as JSONL
• marks rows processed only after a successful local write
• increments attempts and writes last_error when processing fails
• previews or applies processed-row cleanup with --purge-processed-days

The worker processing policy is documented in:

• docs/outbox_worker_policy.md

7. Test Helpdesk Mail Reset

After importing a production database into the LAN test instance, reset all active projects to use the local Mailpit test mailbox for Helpdesk settings with:

• reset_helpdesk_mail_settings.py

The complete post-import workflow is documented in:

• docs/test_instance_post_import.md

Use the read-only validator to check the test instance without changing it:

./validate_test_instance.py

Run the Helpdesk/redMCP live smoke test after the post-import checks pass:

./helpdesk_smoke_test.py

That test is documented in:

• docs/helpdesk_smoke_test.md

Run the Helpdesk outbox worker validation when changing outbox hooks, worker enrichment, or Helpdesk/redMCP behavior:

./validate_helpdesk_outbox_worker.py

That test is documented in:

• docs/helpdesk_outbox_worker_validation.md

Preview the affected projects and settings:

./reset_helpdesk_mail_settings.py --dry-run

Apply the reset:

./reset_helpdesk_mail_settings.py

Defaults match the current Mailpit test setup:

• incoming POP3: 192.168.1.105:1110
• outgoing SMTP: 192.168.1.105:1025
• incoming username/password: test / testpass
• outgoing SMTP authentication: none
• answer-from pattern: helpdesk-{identifier}@example.test

If Mailpit moves, pass the host that Redmine can reach:

./reset_helpdesk_mail_settings.py --mailpit-host 192.168.50.170

8. redMCP Helpdesk Semantics

The redMCP wrapper now makes Helpdesk behavior explicit:

• redMCP/bin/redmcp-server.php runs as a stdio MCP server.
• redMCP/bin/redmcp-http-server.php runs as a bearer-token-protected Streamable HTTP MCP server for network client testing, with PID/status/stop helpers and optional debug JSONL logging.
• redMCP/bin/generate-bearer-token.php generates local MCP bearer tokens.
• projects() and project() expose Redmine's built-in /projects.json project list/detail APIs.
• users(), user(), and projectMemberships() expose Redmine's built-in user and project membership APIs.
• issues() and filterIssues() expose Redmine's built-in /issues.json issue filters.
• search() and searchIssues() expose Redmine's built-in /search.json text search.
• MCP list tools accept friendly limit, page, offset, sort, status, and date options while still allowing raw Redmine filters/params overrides.
• issueWithHelpdesk() composes normal issue data with Helpdesk ticket/message metadata.
• updateIssue() is safe by default and does not send customer email.
• updateIssue(..., ['send_helpdesk_email' => true]) and sendHelpdeskIssueResponse() deliberately use the Helpdesk email path.

The live smoke test verifies both default non-email updates and explicit customer-visible replies through Mailpit.

LAN Deployment Progress

The LAN Redmine copy at 192.168.50.170 was inspected and updated via SSH.

Confirmed environment facts:

• SSH user used: reddev
• remote Redmine path: /usr/share/redmine
• remote Ruby: 2.5.1
• remote Bundler: 1.17.3
• remote DB config: MySQL via config/database.yml

Remote plugin rollback archives were created on the LAN host under:

/home/reddev/redmine-plugin-backups/

The deployed helpdesk search routes were verified with:

• remote rake routes
• HTTP requests returning normal login redirects when unauthenticated

What Is Not Finished Yet

1. External Search Index

The worker can now publish bounded JSONL batches and mark successful rows processed. We have not yet built the actual external index.

Planned direction:

• Qdrant first
• OpenAI embeddings first
• ticket-level docs for "which issue mentioned this?"
• message-level docs for "how did we handle a similar case?"

This should wait until the worker validation pass has documented the derived document shape that will feed the index.

2. Pre-Existing UI/Plugin Bugs

We discovered old plugin issues while working:

• duplicate avatar DOM ids likely causing repeated/wrong thumbnails
• attachments/contacts_thumbnail digest warning
• acts_as_list Redmine 4 compatibility warning

These are logged in:

• docs/pre_existing_issues.md

They are important context but are not the primary search deliverable.

Current Repo Files That Matter Most

Project docs:

• docs/event_outbox_spec.md
• docs/redmineup_local_fork_changelog.md
• docs/helpdesk_smoke_test.md
• docs/test_instance_post_import.md
• docs/pre_existing_issues.md

Tooling:

• redmine_contacts.py
• redmine_helpdesk_search.py

Local plugin work:

• redmine-copy/plugins/redmine_event_outbox
• redmine-copy/plugins/redmine_contacts_helpdesk/app/controllers/helpdesk_search_controller.rb

Current Recommended Next Steps

If continuing this project, the next best work is:

1. Run the post-import validator and Helpdesk smoke test to establish a clean baseline.
2. Generate controlled Helpdesk activity in fud-helpdesk.
3. Inspect event_outbox_events for the corresponding issue, journal, Helpdesk ticket, and journal message rows.
4. Check worker queue state with redmine_outbox_worker.py --status.
5. Process a bounded JSONL batch after dry-run output is correct.
6. Choose the first external index target after JSONL publishing is stable.

Practical Commands

Fetch contacts:

./redmine_contacts.py fetch

Search contacts:

./redmine_contacts.py search "customer name or phone"

Preview a contact update:

./redmine_contacts.py update 123 --set first_name=Corrected

Fetch helpdesk metadata through the Redmine read API:

./redmine_contacts.py helpdesk-ticket 39858
./redmine_contacts.py helpdesk-issues 4337 --limit 50
./redmine_contacts.py helpdesk-messages 39858 --limit 100
./redmine_contacts.py helpdesk-timeline 4337 --limit 100

Fetch read-only helpdesk documents directly over SSH/MySQL:

./redmine_helpdesk_search.py fetch

Preview pending outbox events and their derived documents without marking them processed:

./redmine_outbox_worker.py --dry-run --batch-size 10

Check queue status:

./redmine_outbox_worker.py --status

Process a bounded outbox batch into JSONL and mark successful rows:

./redmine_outbox_worker.py --batch-size 20

Preview processed-row cleanup:

./redmine_outbox_worker.py --purge-processed-days 30

Search the cached helpdesk documents:

./redmine_helpdesk_search.py search "inventory not updated" --type message --limit 10

Show issues by contact:

./redmine_helpdesk_search.py issues-by-contact 1299 --limit 20

Summary

This project began as a way to search and use legacy Redmine data without breaking the system or forcing a full upgrade. The important progress so far is not cosmetic UI work. It is the architectural clarification that helpdesk data is the authoritative customer communication layer, plus the first safe event and export tooling around it.

The repo is now at the point where bounded external worker publishing is proven against the LAN test instance. The next meaningful leap is choosing and building the first external search/index target, likely Qdrant with OpenAI embeddings.