Add Helpdesk outbox worker validation

AGENTS.md

@@ -11,9 +11,10 @@ environment. The canonical plugin sources live in `plugins/`:
 
 Top-level Python helpers such as `redmine_outbox_worker.py` and
 `reset_helpdesk_mail_settings.py` support LAN test-instance operations.
-`validate_test_instance.py` performs read-only post-import checks. Project
-notes and design records live in `docs/`. `redMCP/` contains the PHP
-Redmine API/MCP wrapper. `dist/*.MANIFEST.md` files are tracked; rollback
+`validate_test_instance.py` performs read-only post-import checks, and
+`validate_helpdesk_outbox_worker.py` runs the controlled Helpdesk/outbox live
+validator. Project notes and design records live in `docs/`. `redMCP/`
+contains the PHP Redmine API/MCP wrapper. `dist/*.MANIFEST.md` files are tracked; rollback
 tarballs are intentionally ignored. `redmine-copy/` is an ignored
 working/reference copy, not the source of truth.
 
@@ -25,7 +26,7 @@ Use targeted syntax checks before committing:
 ruby -c plugins/redmine_contacts/lib/redmine_contacts/utils/check_mail.rb
 ruby -c plugins/redmine_contacts_helpdesk/app/models/helpdesk_mailer.rb
 ruby -c plugins/redmine_event_outbox/lib/redmine_event_outbox.rb
-python3 -m py_compile redmine_outbox_worker.py reset_helpdesk_mail_settings.py validate_test_instance.py helpdesk_smoke_test.py
+python3 -m py_compile redmine_outbox_worker.py reset_helpdesk_mail_settings.py validate_test_instance.py helpdesk_smoke_test.py validate_helpdesk_outbox_worker.py
 cd redMCP && php -l app/RedmineClient.php && composer validate
 ```
 
@@ -35,6 +36,7 @@ Dry-run operational helpers before applying changes:
 ./reset_helpdesk_mail_settings.py --dry-run
 ./validate_test_instance.py
 ./helpdesk_smoke_test.py
+./validate_helpdesk_outbox_worker.py
 ./redmine_outbox_worker.py --dry-run --batch-size 10
 ```
 
@@ -53,7 +55,8 @@ against the LAN Redmine copy using controlled projects such as `fud-helpdesk` or
 `fud-nohelpdesk`. Record notable manual validation in `docs/` when behavior or
 deployment assumptions change. The post-import workflow lives in
 `docs/test_instance_post_import.md`; the Helpdesk/redMCP live smoke test is
-documented in `docs/helpdesk_smoke_test.md`.
+documented in `docs/helpdesk_smoke_test.md`; the Helpdesk outbox worker
+validator is documented in `docs/helpdesk_outbox_worker_validation.md`.
 
 ## Commit & Pull Request Guidelines
 

README.md

@@ -138,19 +138,20 @@ Tested event types on the LAN copy:
 - `journal.created`
 - `contact.created`
 - `contact.updated`
-
-Planned/implemented locally but not fully LAN-validated as a complete workflow:
-
 - `helpdesk_ticket.created`
 - `helpdesk_ticket.updated`
 - `journal_message.created`
+
+Planned/implemented locally but not yet observed in the controlled LAN
+validation workflow:
+
 - `journal_message.updated`
 
 The Helpdesk user workflow itself is now live-smoke-tested, including inbound
 Mailpit import, `issueWithHelpdesk()` metadata, default non-email updates, and
-explicit customer-visible Helpdesk replies. The remaining gap is validating that
-those same controlled actions produce the expected outbox rows and derived
-worker documents.
+explicit customer-visible Helpdesk replies. The repeatable outbox validation
+uses that same controlled workflow to verify Helpdesk ticket/message outbox rows
+and worker-derived documents without marking rows processed.
 
 ### 3. Local Helpdesk Plugin Fork Changes
 
@@ -259,6 +260,17 @@ That test is documented in:
 
 - [docs/helpdesk_smoke_test.md](/home/iadnah/redmine/docs/helpdesk_smoke_test.md:1)
 
+Run the Helpdesk outbox worker validation when changing outbox hooks, worker
+enrichment, or Helpdesk/redMCP behavior:
+
+```sh
+./validate_helpdesk_outbox_worker.py
+```
+
+That test is documented in:
+
+- [docs/helpdesk_outbox_worker_validation.md](/home/iadnah/redmine/docs/helpdesk_outbox_worker_validation.md:1)
+
 Preview the affected projects and settings:
 
 ```sh
@@ -323,18 +335,14 @@ The deployed helpdesk search routes were verified with:
 
 ## What Is Not Finished Yet
 
-### 1. Worker Validation
+### 1. Worker Processing Policy
 
-This is the next implementation milestone.
+The Helpdesk outbox worker now has a repeatable live validator. The next worker
+decision is operational policy:
 
-Still needed:
-
-- run and document end-to-end validation of `redmine_outbox_worker.py` against
-  controlled Helpdesk activity on the LAN copy
-- prove that imported Helpdesk issues, Helpdesk replies, and normal issue
-  updates create the expected event rows
-- verify the derived JSONL document shape is useful and does not leak unsafe
-  content
+- choose when to mark rows processed on the test instance
+- decide where durable JSONL/index output should live
+- define retention or replay expectations for `event_outbox_events`
 
 ### 2. External Search Index
 

docs/helpdesk_outbox_worker_validation.md

@@ -0,0 +1,60 @@
+# Helpdesk Outbox Worker Validation
+
+Use `validate_helpdesk_outbox_worker.py` to verify that controlled Helpdesk
+activity creates the expected `event_outbox_events` rows and that
+`redmine_outbox_worker.py` can enrich those rows without processing them.
+
+## Run
+
+```sh
+./validate_helpdesk_outbox_worker.py
+```
+
+The script first runs the full Helpdesk/redMCP smoke path:
+
+- sends an inbound email to Mailpit for `fud-helpdesk`
+- triggers Helpdesk mail import on the LAN Redmine host
+- verifies `issueWithHelpdesk()` metadata
+- verifies normal `updateIssue()` does not send customer mail
+- verifies explicit Helpdesk reply delivery through Mailpit
+- closes the controlled Helpdesk issue unless `--keep-open` is passed
+
+It then queries `event_outbox_events` for the created issue and dry-runs
+`redmine_outbox_worker.enrich_event()` against those exact rows.
+
+## Expected Checks
+
+The validator fails if any of these are missing:
+
+- `helpdesk_ticket.created`
+- `helpdesk_ticket.updated`
+- `journal_message.created`
+- `journal.created`
+- `issue.updated`
+- derived worker document types: `event`, `ticket`, and `message`
+
+It also confirms the new rows have not been locked or marked processed, and
+that worker documents do not expose raw `bcc_address` data.
+
+## Configuration
+
+Defaults match the LAN test setup:
+
+- Redmine: `http://192.168.50.170`
+- SSH: `reddev@192.168.50.170`, key `/tmp/reddev`
+- remote Redmine path: `/usr/share/redmine`
+- Mailpit: `192.168.1.105`, SMTP `1025`, HTTP `8025`
+- API key source: `REDMINE_API_KEY`, `REDMNINE_API_KEY`, or `redMCP/.env`
+
+Useful options:
+
+```sh
+./validate_helpdesk_outbox_worker.py --skip-preflight
+./validate_helpdesk_outbox_worker.py --keep-open
+```
+
+## Expected Output
+
+The final summary should include the created issue id, Helpdesk ticket ids,
+journal message ids, event row ids/types, and derived document type counts.
+The worker mode should report dry-run enrichment only.

docs/helpdesk_smoke_test.md

@@ -48,3 +48,7 @@ Smoke test passed.
 
 Closed smoke-test tickets are intentionally retained in `fud-helpdesk` as audit
 evidence. Their subjects start with `[redMCP-smoke ...]`.
+
+For outbox and worker coverage, run `./validate_helpdesk_outbox_worker.py`.
+That script reuses this same live smoke path, then checks the matching
+`event_outbox_events` rows and dry-runs worker enrichment.

docs/redmineup_local_fork_changelog.md

@@ -27,11 +27,38 @@ environment. Before risky edits, archive the current plugin directories in
     manual browser testing.
   - Helpdesk mail import, Helpdesk metadata lookup, default non-email updates,
     and explicit outbound Helpdesk replies are live-smoke-tested through redMCP.
-  - Helpdesk outbox/worker validation is still pending.
+  - Helpdesk outbox/worker validation now has a repeatable live script.
 - Next meaningful milestone:
-  - Validate `redmine_outbox_worker.py` end to end against controlled Helpdesk
-    activity, document the derived JSONL shape, then choose the external index
-    target.
+  - Run `validate_helpdesk_outbox_worker.py` after outbox or worker changes,
+    then choose the external index target and durable processing policy.
+
+## 2026-04-25 - Helpdesk Outbox Worker Validation
+
+- Touched areas:
+  - Helpdesk test tooling
+  - External outbox worker validation docs
+- Purpose:
+  - Make Helpdesk outbox coverage repeatable after database refreshes and
+    worker changes.
+  - Validate worker enrichment against a real Mailpit-imported Helpdesk ticket
+    without claiming rows or marking them processed.
+- Behavior checked:
+  - Controlled Helpdesk import produces `helpdesk_ticket.created`,
+    `helpdesk_ticket.updated`, `journal_message.created`, `journal.created`,
+    and `issue.updated` rows.
+  - Worker dry-run enrichment emits `event`, `ticket`, and `message` documents.
+  - Derived message documents expose `has_bcc_address` but do not expose raw
+    `bcc_address`.
+- LAN test result:
+  - `./validate_helpdesk_outbox_worker.py` passed against `fud-helpdesk` and
+    `fud-nohelpdesk`.
+  - Latest documented passing run created and closed controlled Helpdesk issue
+    `#39873`.
+  - Observed outbox rows: 1 `helpdesk_ticket.created`, 3
+    `helpdesk_ticket.updated`, 1 `journal_message.created`, 3
+    `journal.created`, and 2 `issue.updated`.
+  - Worker dry-run enrichment produced 10 `event`, 6 `ticket`, and 2
+    `message` documents without claiming or marking rows processed.
 
 ## 2026-04-24 - Helpdesk/redMCP Smoke Validation
 

docs/test_instance_post_import.md

@@ -136,7 +136,20 @@ non-Helpdesk CRUD, confirms default updates do not send Helpdesk email, verifies
 explicit outbound Mailpit delivery, and closes the created test ticket. Details
 are in `docs/helpdesk_smoke_test.md`.
 
-## 8. Re-Run The Read-Only Validator
+## 8. Validate Helpdesk Outbox Worker Enrichment
+
+When Helpdesk/outbox behavior matters, run the repeatable live validator:
+
+```sh
+./validate_helpdesk_outbox_worker.py
+```
+
+This creates one controlled Helpdesk ticket through Mailpit, verifies redMCP
+Helpdesk behavior, checks the matching `event_outbox_events` rows, and dry-runs
+worker enrichment without claiming or marking rows processed. Details are in
+`docs/helpdesk_outbox_worker_validation.md`.
+
+## 9. Re-Run The Read-Only Validator
 
 Finish by running:
 

helpdesk_smoke_test.py

@@ -54,6 +54,16 @@ class SmokeConfig:
     keep_open: bool
 
 
+@dataclass(frozen=True)
+class SmokeResult:
+    token: str
+    subject: str
+    issue_id: int
+    from_address: str
+    reply_token: str
+    control_issue_id: int
+
+
 def main() -> int:
     parser = argparse.ArgumentParser(description="Run a live Helpdesk/redMCP smoke test against the LAN Redmine copy.")
     parser.add_argument("--redmine-url", default=os.getenv("REDMINE_URL", DEFAULT_REDMINE_URL))
@@ -101,7 +111,7 @@ def main() -> int:
         return 1
 
 
-def run_smoke(config: SmokeConfig) -> None:
+def run_smoke(config: SmokeConfig) -> SmokeResult:
     token = time.strftime("%Y%m%d%H%M%S")
     subject = f"[redMCP-smoke {token}] Helpdesk inbound validation"
     from_address = f"redmcp-smoke-{token}@example.test"
@@ -158,6 +168,14 @@ def run_smoke(config: SmokeConfig) -> None:
     print("\nSmoke test passed.")
     print(f"  issue_id: {issue_id}")
     print(f"  subject: {subject}")
+    return SmokeResult(
+        token=token,
+        subject=subject,
+        issue_id=issue_id,
+        from_address=from_address,
+        reply_token=reply_token,
+        control_issue_id=control_id,
+    )
 
 
 def load_env_file(path: Path) -> dict[str, str]:

validate_helpdesk_outbox_worker.py

@@ -0,0 +1,228 @@
+#!/usr/bin/env python3
+"""Live Helpdesk outbox/worker validation for the LAN Redmine copy.
+
+This creates one controlled Helpdesk ticket through Mailpit, then inspects the
+outbox rows for that issue and dry-runs worker enrichment without claiming or
+marking rows processed.
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import sys
+from collections import Counter
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Any
+
+from helpdesk_smoke_test import (
+    DEFAULT_CONTROL_PROJECT,
+    DEFAULT_MAILPIT_HOST,
+    DEFAULT_MAILPIT_HTTP_PORT,
+    DEFAULT_MAILPIT_SMTP_PORT,
+    DEFAULT_PROJECT,
+    DEFAULT_REDMINE_URL,
+    DEFAULT_REMOTE_REDMINE,
+    DEFAULT_SSH_HOST,
+    DEFAULT_SSH_KEY,
+    SmokeConfig,
+    SmokeError,
+    load_env_file,
+    run_smoke,
+)
+from redmine_outbox_worker import OutboxWorkerError, RemoteRedmine, enrich_event, sql_int
+
+
+EXPECTED_EVENT_TYPES = {
+    "helpdesk_ticket.created",
+    "helpdesk_ticket.updated",
+    "journal_message.created",
+    "journal.created",
+    "issue.updated",
+}
+
+
+class ValidationError(RuntimeError):
+    pass
+
+
+@dataclass(frozen=True)
+class ValidationConfig:
+    smoke: SmokeConfig
+    remote: RemoteRedmine
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser(
+        description="Create a controlled Helpdesk issue and validate outbox worker enrichment for it."
+    )
+    parser.add_argument("--redmine-url", default=os.getenv("REDMINE_URL", DEFAULT_REDMINE_URL))
+    parser.add_argument("--api-key", default=os.getenv("REDMINE_API_KEY") or os.getenv("REDMNINE_API_KEY"))
+    parser.add_argument("--env-file", type=Path, default=Path("redMCP/.env"))
+    parser.add_argument("--mailpit-host", default=DEFAULT_MAILPIT_HOST)
+    parser.add_argument("--mailpit-http-port", type=int, default=DEFAULT_MAILPIT_HTTP_PORT)
+    parser.add_argument("--mailpit-smtp-port", type=int, default=DEFAULT_MAILPIT_SMTP_PORT)
+    parser.add_argument("--project", default=DEFAULT_PROJECT)
+    parser.add_argument("--control-project", default=DEFAULT_CONTROL_PROJECT)
+    parser.add_argument("--ssh-host", default=os.getenv("REDMINE_SSH_HOST", DEFAULT_SSH_HOST))
+    parser.add_argument("--ssh-key", type=Path, default=Path(os.getenv("REDMINE_SSH_KEY", str(DEFAULT_SSH_KEY))))
+    parser.add_argument("--remote-redmine", default=os.getenv("REDMINE_REMOTE_PATH", DEFAULT_REMOTE_REDMINE))
+    parser.add_argument("--skip-preflight", action="store_true", help="Skip validate_test_instance.py.")
+    parser.add_argument("--keep-open", action="store_true", help="Do not close the created Helpdesk issue.")
+    args = parser.parse_args()
+
+    try:
+        config = build_config(args)
+        run_validation(config)
+        return 0
+    except (SmokeError, OutboxWorkerError, ValidationError) as exc:
+        print(f"error: {exc}", file=sys.stderr)
+        return 1
+
+
+def build_config(args: argparse.Namespace) -> ValidationConfig:
+    env = load_env_file(args.env_file)
+    redmine_url = args.redmine_url or env.get("REDMINE_URL") or DEFAULT_REDMINE_URL
+    api_key = args.api_key or env.get("REDMINE_API_KEY") or env.get("REDMNINE_API_KEY")
+    if not api_key:
+        raise ValidationError("missing Redmine API key. Set REDMINE_API_KEY or redMCP/.env.")
+
+    smoke = SmokeConfig(
+        redmine_url=redmine_url.rstrip("/"),
+        api_key=api_key,
+        mailpit_host=args.mailpit_host,
+        mailpit_http_port=args.mailpit_http_port,
+        mailpit_smtp_port=args.mailpit_smtp_port,
+        project=args.project,
+        control_project=args.control_project,
+        ssh_host=args.ssh_host,
+        ssh_key=args.ssh_key,
+        remote_redmine=args.remote_redmine,
+        skip_preflight=args.skip_preflight,
+        keep_open=args.keep_open,
+    )
+    remote = RemoteRedmine(args.ssh_host, args.ssh_key, args.remote_redmine)
+    return ValidationConfig(smoke=smoke, remote=remote)
+
+
+def run_validation(config: ValidationConfig) -> None:
+    smoke_result = run_smoke(config.smoke)
+    events = fetch_issue_events(config.remote, smoke_result.issue_id)
+    assert_expected_events(events)
+    assert_events_unprocessed(events)
+
+    documents: list[dict[str, Any]] = []
+    for event in events:
+        documents.extend(enrich_event(config.remote, event))
+
+    assert_worker_documents(events, documents)
+    assert_no_bcc_address_leak(documents)
+    print_summary(smoke_result.issue_id, events, documents)
+
+
+def fetch_issue_events(remote: RemoteRedmine, issue_id: int) -> list[dict[str, Any]]:
+    events = remote.mysql_json_lines(
+        f"""
+SELECT HEX(CAST(JSON_OBJECT(
+  'id', id,
+  'event_type', event_type,
+  'source_type', source_type,
+  'source_id', source_id,
+  'project_id', project_id,
+  'issue_id', issue_id,
+  'journal_id', journal_id,
+  'user_id', user_id,
+  'occurred_at', DATE_FORMAT(occurred_at, '%Y-%m-%dT%H:%i:%sZ'),
+  'attempts', attempts,
+  'processed_at', IF(processed_at IS NULL, NULL, DATE_FORMAT(processed_at, '%Y-%m-%dT%H:%i:%sZ')),
+  'locked_at', IF(locked_at IS NULL, NULL, DATE_FORMAT(locked_at, '%Y-%m-%dT%H:%i:%sZ')),
+  'payload', payload
+) AS CHAR)) AS document
+FROM event_outbox_events
+WHERE issue_id = {sql_int(issue_id)}
+ORDER BY id;
+"""
+    )
+    if not events:
+        raise ValidationError(f"no event_outbox_events rows found for issue #{issue_id}")
+    return events
+
+
+def assert_expected_events(events: list[dict[str, Any]]) -> None:
+    found = {str(event.get("event_type") or "") for event in events}
+    missing = sorted(EXPECTED_EVENT_TYPES - found)
+    if missing:
+        raise ValidationError(f"missing expected Helpdesk outbox event types: {', '.join(missing)}")
+
+
+def assert_events_unprocessed(events: list[dict[str, Any]]) -> None:
+    mutated = [
+        event
+        for event in events
+        if event.get("processed_at") is not None or event.get("locked_at") is not None
+    ]
+    if mutated:
+        ids = ", ".join(str(event.get("id")) for event in mutated)
+        raise ValidationError(f"new smoke-test outbox rows were already processed or locked: {ids}")
+
+
+def assert_worker_documents(events: list[dict[str, Any]], documents: list[dict[str, Any]]) -> None:
+    doc_types = {str(document.get("doc_type") or "") for document in documents}
+    required_doc_types = {"event", "ticket", "message"}
+    missing_doc_types = sorted(required_doc_types - doc_types)
+    if missing_doc_types:
+        raise ValidationError(f"worker dry-run did not derive document types: {', '.join(missing_doc_types)}")
+
+    event_doc_ids = {document.get("event_id") for document in documents if document.get("doc_type") == "event"}
+    missing_events = [str(event.get("id")) for event in events if event.get("id") not in event_doc_ids]
+    if missing_events:
+        raise ValidationError(f"worker dry-run did not emit event documents for rows: {', '.join(missing_events)}")
+
+
+def assert_no_bcc_address_leak(documents: list[dict[str, Any]]) -> None:
+    for document in documents:
+        if contains_key(document, "bcc_address"):
+            raise ValidationError(f"worker document leaked raw bcc_address: {document.get('doc_id')}")
+
+
+def contains_key(value: Any, needle: str) -> bool:
+    if isinstance(value, dict):
+        return any(key == needle or contains_key(child, needle) for key, child in value.items())
+    if isinstance(value, list):
+        return any(contains_key(child, needle) for child in value)
+    return False
+
+
+def print_summary(issue_id: int, events: list[dict[str, Any]], documents: list[dict[str, Any]]) -> None:
+    event_types = Counter(str(event.get("event_type") or "") for event in events)
+    doc_types = Counter(str(document.get("doc_type") or "") for document in documents)
+    ticket_ids = sorted(
+        {
+            int(document["helpdesk_ticket_id"])
+            for document in documents
+            if document.get("doc_type") == "ticket" and document.get("helpdesk_ticket_id")
+        }
+    )
+    message_ids = sorted(
+        {
+            int(document["journal_message_id"])
+            for document in documents
+            if document.get("doc_type") == "message" and document.get("journal_message_id")
+        }
+    )
+    event_rows = [f"{event.get('id')}:{event.get('event_type')}" for event in events]
+
+    print("\nHelpdesk outbox worker validation passed.")
+    print(f"  issue_id: {issue_id}")
+    print(f"  helpdesk_ticket_ids: {json.dumps(ticket_ids)}")
+    print(f"  journal_message_ids: {json.dumps(message_ids)}")
+    print(f"  event_rows: {', '.join(event_rows)}")
+    print(f"  event_type_counts: {json.dumps(dict(sorted(event_types.items())), sort_keys=True)}")
+    print(f"  derived_doc_type_counts: {json.dumps(dict(sorted(doc_types.items())), sort_keys=True)}")
+    print("  worker_mode: dry-run enrichment only; rows were not claimed or marked processed")
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())