DigitalData.MessagingService/AGENTS.md

MessagingService - Agent Notes and Future Enhancements

Purpose

This document contains important notes, decisions, and future enhancement plans for the MessagingService application. This is intended for AI agents and developers who will continue development.

---

Important Notes

1. Database Schema - DO NOT MODIFY

CRITICAL: The database schema must NEVER be modified. All Entity Framework entities must map to existing legacy tables using [Table] and [Column] attributes.

Naming Convention:

• Database: SNAKE_CASE with prefixes (TBEMLP_, TBDD_)
• C# Entities: PascalCase without prefixes
• Use [Table("TBDD_FOO")] and [Column("COLUMN_NAME")] attributes

Example:

[Table("TBDD_EMAIL_ACCOUNT")]
public class EmailAccount
{
    [Column("EMAIL_ACCOUNT_ID")]
    public int Id { get; set; }
    
    [Column("ACCOUNT_NAME")]
    public string AccountName { get; set; }
}

2. Message ID Hash Algorithm

The MessageIdGenerator in Domain.Services must use exactly the same algorithm as the legacy system to ensure duplicate detection works correctly.

Algorithm: SHA256 hash of {originalMessageId}|{sender}|{date:yyyyMMddHHmmss}|{subject}

3. DateTime Usage - ALWAYS Use Local Time

CRITICAL: Always use DateTime.Now instead of DateTime.UtcNow throughout the entire application.

Reason: The legacy system uses local server time, and the database stores all timestamps as local time. Using UTC would break compatibility and cause incorrect time comparisons.

Examples:

// ✅ CORRECT
profile.CreatedDate = DateTime.Now;
var lastPoll = DateTime.Now.AddMinutes(-profile.PollIntervalMinutes);

// ❌ WRONG - DO NOT USE
profile.CreatedDate = DateTime.UtcNow;  // NEVER USE UTC
var lastPoll = DateTime.UtcNow.AddMinutes(-profile.PollIntervalMinutes);  // NEVER USE UTC

Important: This applies to:

• All entity audit fields (CreatedDate, ModifiedDate, LastPollDate, etc.)
• All date comparisons in business logic
• All timestamps in logs and error messages
• All date parameters in queries

4. Git Operations - NEVER Without Explicit Permission

CRITICAL: NEVER execute git commit or git push commands automatically. ALWAYS wait for explicit user instruction.

Rules:

• Only commit when user explicitly says "commit" or "commit this"
• Only push when user explicitly says "push" or "push to remote"
• Stage files with git add ONLY when about to commit per user request

5. MediatR Command/Query File Organization

IMPORTANT: Commands/Queries and their Handlers must be in the SAME file.

Example:

// ✅ CORRECT - CreateEmailProfileCommand.cs contains BOTH
public record CreateEmailProfileCommand : IRequest<int> { ... }

public class CreateEmailProfileCommandHandler : IRequestHandler<CreateEmailProfileCommand, int> { ... }

// ❌ WRONG - Separate files
// CreateEmailProfileCommand.cs (command only)
// CreateEmailProfileCommandHandler.cs (handler only)

File Naming:

• Commands: {Verb}{Entity}Command.cs (e.g., CreateEmailProfileCommand.cs)
• Queries: {Verb}{Entity}Query.cs (e.g., GetEmailProfilesQuery.cs)

Folder Structure (NO Features/ prefix):

Application/
├── EmailProfiles/
│   ├── Commands/CreateEmailProfileCommand.cs
│   ├── Queries/GetEmailProfilesQuery.cs
│   └── Validators/CreateEmailProfileCommandValidator.cs
├── EmailAccounts/
│   ├── Commands/CreateEmailAccountCommand.cs
│   └── Queries/GetEmailAccountsQuery.cs
└── Common/
    ├── Dtos/EmailProfileDto.cs (single DTOs at root)
    ├── Dtos/EmailHistories/ (multiple DTOs in subfolder)
    └── Interfaces/IEmailService.cs

6. Repository Pattern - NO UnitOfWork, Generic CRUD with AutoMapper

CRITICAL: DO NOT use IUnitOfWork pattern. Use generic repository pattern with AutoMapper-based CRUD operations.

Key Principles:

• ✅ Each operation auto-saves changes - NO explicit SaveChangesAsync needed
• ✅ Use UpdateSingleAsync / DeleteSingleAsync for single-record safety
• ✅ Use UpdateAsync / DeleteAsync only when intentionally modifying multiple records
• ✅ AutoMapper handles all DTO → Entity mappings

Pattern:

// IRepository<T> generic interface
public interface IRepository<TEntity> where TEntity : class
{
    // Query operations
    Task<TEntity?> GetByIdAsync(int id, CancellationToken cancellationToken = default);
    Task<IEnumerable<TEntity>> FindAsync(Expression<Func<TEntity, bool>> predicate, ...);
    
    // Create - auto-saves
    Task<TEntity> CreateAsync<TDto>(TDto dto, CancellationToken cancellationToken = default);
    
    // Update - auto-saves
    Task<int> UpdateAsync<TDto>(Expression<...> predicate, TDto dto, ...); // Multiple records
    Task UpdateSingleAsync<TDto>(Expression<...> predicate, TDto dto, ...); // SAFE: Single record only
    
    // Delete - auto-saves
    Task<int> DeleteAsync(Expression<...> predicate, ...); // Multiple records
    Task DeleteSingleAsync(Expression<...> predicate, ...); // SAFE: Single record only
}

Command Handler Examples:

// ✅ CORRECT - CreateAsync auto-saves
public class CreateEmailProfileCommandHandler(IRepository<EmailProfile> repository) 
    : IRequestHandler<CreateEmailProfileCommand, int>
{
    public async Task<int> Handle(CreateEmailProfileCommand request, CancellationToken cancellationToken)
    {
        var profile = await repository.CreateAsync(request, cancellationToken);
        return profile.Id; // NO SaveChangesAsync needed!
    }
}

// ✅ CORRECT - UpdateSingleAsync for safety (throws if 0 or 2+ records match)
public class UpdateEmailProfileCommandHandler(IRepository<EmailProfile> repository) 
    : IRequestHandler<UpdateEmailProfileCommand, int>
{
    public async Task<int> Handle(UpdateEmailProfileCommand request, CancellationToken cancellationToken)
    {
        await repository.UpdateSingleAsync(p => p.Id == request.Id, request, cancellationToken);
        return request.Id; // NO SaveChangesAsync needed!
    }
}

// ✅ CORRECT - DeleteSingleAsync for safety (throws if 0 or 2+ records match)
public class DeleteEmailProfileCommandHandler(IRepository<EmailProfile> repository) 
    : IRequestHandler<DeleteEmailProfileCommand, int>
{
    public async Task<int> Handle(DeleteEmailProfileCommand request, CancellationToken cancellationToken)
    {
        await repository.DeleteSingleAsync(p => p.Id == request.Id, cancellationToken);
        return request.Id; // NO SaveChangesAsync needed!
    }
}

// ❌ WRONG - Manual entity creation (use AutoMapper instead)
var profile = new EmailProfile 
{ 
    ProfileName = request.ProfileName, 
    EmailAccountId = request.EmailAccountId,
    // ... 15 more properties
};

// ❌ WRONG - Using IUnitOfWork (removed)
public CreateEmailProfileCommandHandler(IUnitOfWork unitOfWork) { ... }

// ❌ WRONG - Calling SaveChangesAsync (removed)
await repository.SaveChangesAsync(cancellationToken);

Safety Rules:

1. UpdateSingleAsync - Use for ID-based updates. Throws InvalidOperationException if:

   • Zero records match (entity not found)
   • Multiple records match (predicate too broad)

2. DeleteSingleAsync - Use for ID-based deletes. Throws InvalidOperationException if:

   • Zero records match (entity not found)
   • Multiple records match (predicate too broad)

3. UpdateAsync / DeleteAsync - Use ONLY when intentionally modifying multiple records:

   // ✅ CORRECT - Intentional bulk operation
   await repository.UpdateAsync(
       p => p.EmailAccountId == accountId, 
       new { IsActive = false }, 
       cancellationToken);
   
   // ✅ Returns count of updated/deleted records
   var count = await repository.DeleteAsync(p => p.IsActive == false, cancellationToken);
   

DTO Mapping Responsibility:

• Each DTO creator must define their own AutoMapper profile
• Example: CreateEmailProfileCommand → EmailProfile mapping must be defined in EmailProfileMappingProfile.cs
• Repository implementation uses IMapper.Map<TEntity>(dto) internally

---

Future Enhancements

7. RabbitMQ Command Bus Integration (IMPLEMENTED)

Purpose: Asynchronous command processing via RabbitMQ message broker for POST/PUT/DELETE operations.

Architecture:

• GET Queries: Synchronous (immediate response via MediatR)
• POST/PUT/DELETE Commands: Can be asynchronous (published to RabbitMQ, processed by background worker)

RabbitMQ Server:

• Management UI: http://172.24.12.56:15672
• AMQP Port: 5672 (default)
• Exchange: emailprofiler.commands (Direct)
• Queue: emailprofiler.command.queue
• Routing Key: command

Implementation Components:

1. ICommandPublisher (Application/Common/Interfaces/ICommandPublisher.cs):

   • Interface for publishing commands to message broker
   • Generic method: PublishAsync<TCommand>(TCommand command, CancellationToken)

2. RabbitMqCommandPublisher (Infrastructure/Messaging/RabbitMqCommandPublisher.cs):

   • Implements ICommandPublisher
   • Serializes command to JSON with metadata envelope (CommandType, Payload, CorrelationId, PublishedAt)
   • Publishes to RabbitMQ exchange with persistent delivery mode

3. RabbitMqCommandConsumer (Infrastructure/Messaging/RabbitMqCommandConsumer.cs):

   • BackgroundService that consumes commands from RabbitMQ
   • Deserializes command envelope
   • Resolves command type from assembly
   • Executes command via MediatR in scoped service
   • Acknowledges message on success, requeues on error

4. RabbitMqConfiguration (Infrastructure/Messaging/RabbitMqConfiguration.cs):

   • Configuration model for RabbitMQ connection
   • Binds to appsettings.json section: RabbitMq

Configuration (appsettings.json):

{
  "RabbitMq": {
    "HostName": "172.24.12.56",
    "Port": 5672,
    "UserName": "guest",
    "Password": "guest",
    "VirtualHost": "/",
    "ExchangeName": "emailprofiler.commands",
    "QueueName": "emailprofiler.command.queue",
    "RoutingKey": "command",
    "AutomaticRecoveryEnabled": true,
    "NetworkRecoveryIntervalSeconds": 10
  }
}

Dependency Injection (Infrastructure/DependencyInjection.cs):

services.Configure<RabbitMqConfiguration>(configuration.GetSection(RabbitMqConfiguration.SectionName));
services.AddSingleton<ICommandPublisher, RabbitMqCommandPublisher>();
services.AddHostedService<RabbitMqCommandConsumer>();

Usage in API Controllers (Future):

// Option 1: Synchronous (immediate execution via MediatR)
var result = await _mediator.Send(new CreateEmailProfileCommand(...), cancellationToken);
return Ok(result);

// Option 2: Asynchronous (publish to RabbitMQ for background processing)
await _commandPublisher.PublishAsync(new CreateEmailProfileCommand(...), cancellationToken);
return Accepted(); // HTTP 202 - command queued for processing

Benefits:

• Decouples API from long-running command processing
• Improves API responsiveness (fire-and-forget)
• Enables horizontal scaling (multiple consumers)
• Automatic retries on failure (requeue mechanism)
• Message persistence (survives application restarts)

---

HIGH PRIORITY: Email Queue for Outgoing Messages (Future)

Implementation Steps:

1. Add NuGet Package:

   dotnet add package RabbitMQ.Client
   

2. Create RabbitMqEmailQueue.cs:

   // src/DigitalData.MessagingService.Infrastructure/Queue/RabbitMqEmailQueue.cs
   public class RabbitMqEmailQueue : IEmailQueue
   {
       private readonly IConnection _connection;
       private readonly IModel _channel;
       private const string QueueName = "email-outbox";
   
       public RabbitMqEmailQueue(IOptions<RabbitMqConfiguration> config)
       {
           var factory = new ConnectionFactory
           {
               HostName = config.Value.HostName,
               Port = config.Value.Port,
               UserName = config.Value.UserName,
               Password = config.Value.Password
           };
   
           _connection = factory.CreateConnection();
           _channel = _connection.CreateModel();
   
           _channel.QueueDeclare(
               queue: QueueName,
               durable: true,
               exclusive: false,
               autoDelete: false,
               arguments: null);
       }
   
       public async Task EnqueueAsync(SendingEmail email, CancellationToken cancellationToken)
       {
           var json = JsonSerializer.Serialize(email);
           var body = Encoding.UTF8.GetBytes(json);
   
           var properties = _channel.CreateBasicProperties();
           properties.Persistent = true;
   
           _channel.BasicPublish(
               exchange: "",
               routingKey: QueueName,
               basicProperties: properties,
               body: body);
   
           await Task.CompletedTask;
       }
   
       public async Task<SendingEmail?> DequeueAsync(CancellationToken cancellationToken)
       {
           var result = _channel.BasicGet(QueueName, autoAck: false);
   
           if (result == null)
               return null;
   
           var json = Encoding.UTF8.GetString(result.Body.ToArray());
           var email = JsonSerializer.Deserialize<SendingEmail>(json);
   
           _channel.BasicAck(result.DeliveryTag, false);
   
           return await Task.FromResult(email);
       }
   }
   

3. Configuration (appsettings.json):

   {
     "RabbitMq": {
       "HostName": "localhost",
       "Port": 5672,
       "UserName": "guest",
       "Password": "guest"
     }
   }
   

4. Dependency Injection (Program.cs):

   // Replace InMemoryEmailQueue with RabbitMqEmailQueue
   // builder.Services.AddSingleton<IEmailQueue, InMemoryEmailQueue>();
   builder.Services.AddSingleton<IEmailQueue, RabbitMqEmailQueue>();
   

Benefits:

• Message persistence (survives application restarts)
• Scalability (multiple worker instances can consume from queue)
• Reliability (automatic retries, dead letter queues)
• Monitoring (RabbitMQ management UI)

Migration Path:

1. Deploy RabbitMQ server (Docker recommended)
2. Test RabbitMqEmailQueue in staging environment
3. Switch DI registration from InMemoryEmailQueue to RabbitMqEmailQueue
4. Monitor queue depth and worker performance

---

Pending Implementation Tasks

Phase 2: Application Layer (COMPLETE)

Status: ✅ Complete - All Commands, Queries, Handlers, Validators, AutoMapper Profiles, and Interfaces implemented

Completed:

• ✅ MediatR Commands (CreateEmailProfileCommand, ProcessEmailCommand, etc.)
• ✅ MediatR Queries (GetEmailProfilesQuery, GetEmailHistoryQuery, etc.)
• ✅ Command/Query Handlers
• ✅ FluentValidation Validators
• ✅ AutoMapper Profiles
• ✅ Application Interfaces (IEmailService, IPdfProcessingService, IDmsService, etc.)

Example Command:

// src/DigitalData.MessagingService.Application/EmailProfiles/Commands/CreateEmailProfileCommand.cs
public record CreateEmailProfileCommand(string ProfileName, int EmailAccountId) : IRequest<int>;

public class CreateEmailProfileCommandHandler : IRequestHandler<CreateEmailProfileCommand, int>
{
    private readonly IRepository<EmailProfile> _repository;
    
    public async Task<int> Handle(CreateEmailProfileCommand request, CancellationToken cancellationToken)
    {
        var profile = await _repository.CreateAsync(request, cancellationToken);
        return profile.Id;
    }
}

8. Email Library - Limilabs Mail.dll

IMPORTANT: This project uses Limilabs Mail.dll (https://www.limilabs.com/) for email operations, NOT MailKit/MimeKit.

Why Limilabs?:

• Commercial-grade IMAP/POP3/SMTP library
• Better OAuth2 support (Microsoft 365, Gmail)
• More reliable with Exchange servers
• Superior attachment handling
• Built-in retry mechanisms

NuGet Package:

dotnet add package Limilabs.Mail

Key Classes:

• Imap - IMAP client for receiving emails
• Smtp - SMTP client for sending emails
• Mail.Message - Email message representation
• OAuth2 - OAuth2 authentication helper

Implementation Example:

// Limilabs IMAP with OAuth2
using Limilabs.Client.IMAP;
using Limilabs.Mail;

public class LimilabsEmailService : IEmailService
{
    public async Task<IEnumerable<EmailMessage>> ReceiveEmailsAsync(EmailAccountDto account)
    {
        using var imap = new Imap();
        
        if (account.UseOAuth2)
        {
            await imap.ConnectSSLAsync(account.ImapServer, account.ImapPort);
            await imap.LoginOAUTH2Async(account.Username, account.OAuth2AccessToken);
        }
        else
        {
            await imap.ConnectSSLAsync(account.ImapServer, account.ImapPort);
            await imap.LoginAsync(account.Username, account.EncryptedPassword);
        }
        
        imap.SelectInbox();
        var uids = imap.Search(Flag.Unseen);
        
        var messages = new List<EmailMessage>();
        foreach (var uid in uids)
        {
            var eml = imap.GetMessageByUID(uid);
            var mail = new MailBuilder().CreateFromEml(eml);
            messages.Add(ConvertToEmailMessage(mail));
        }
        
        imap.Close();
        return messages;
    }
}

DO NOT USE:

• ❌ MailKit
• ❌ MimeKit
• ❌ System.Net.Mail (obsolete)

Phase 3: Infrastructure Layer (COMPLETE)

Status: ✅ Complete - DbContext, Repository, Services, RabbitMQ, and DI implemented

Completed:

• ✅ MessagingServiceDbContext with DbSet for all entities (attribute-only config, no overrides)
• ✅ Generic Repository implementing IRepository with AutoMapper-based CRUD
• ✅ LimilabsEmailService (IMAP/SMTP with OAuth2 using Limilabs Mail.dll - TODO: Add Limilabs.Mail NuGet)
• ✅ GdPicturePdfProcessingService (using GdPicture.NET 14 - TODO: Add GdPicture NuGet and license)
• ✅ WindreamDmsService (COM Interop - TODO: Add windream COM Interop references)
• ✅ DataProtectionEncryptionService (Data Protection API)
• ✅ InMemoryEmailQueue (TODO: Upgrade to RabbitMqEmailQueue later)
• ✅ RabbitMqCommandPublisher and RabbitMqCommandConsumer
• ✅ DependencyInjection.cs with all service registrations

Implementation Notes:

• All services have real implementations with commented TODO blocks for external dependencies
• LimilabsEmailService uses Microsoft.Identity.Client for OAuth2 token acquisition
• GdPicturePdfProcessingService uses GdPicture.NET 14.x API (GetAttachmentCount, ExtractEmbeddedFile)
• WindreamDmsService uses COM Interop (WMSession, WMConnect, WMObjects) based on legacy patterns
• NO EF Core migrations (legacy DB must not be modified)

DbContext Example:

public class MessagingServiceDbContext : DbContext
{
    public DbSet<EmailAccount> EmailAccounts { get; set; }
    public DbSet<EmailProfile> EmailProfiles { get; set; }
    // ... other DbSets
    
    protected override void OnModelCreating(ModelBuilder modelBuilder)
    {
        modelBuilder.ApplyConfigurationsFromAssembly(Assembly.GetExecutingAssembly());
        
        // Important: Check for triggers
        modelBuilder.Entity<EmailHistory>().ToTable(tb => tb.HasTrigger("TR_TBEMLP_HISTORY_AUDIT"));
    }
}

Phase 4: API Layer

Status: Minimal structure exists

TODO:

• Create Controllers (EmailProfilesController, EmailAccountsController, EmailHistoryController)
• Create Background Workers (EmailPollingWorker, EmailSenderWorker)
• Configure Serilog
• Configure Scalar (OpenAPI documentation)
• Add Exception Handling Middleware
• Configure DI for all layers
• Support both IIS and Windows Service hosting

Worker Configuration (appsettings.json):

{
  "Workers": {
    "EmailPolling": {
      "Enabled": true,
      "IntervalSeconds": 60
    },
    "EmailSender": {
      "Enabled": true,
      "IntervalSeconds": 5
    }
  },
  "Hosting": {
    "Mode": "IIS"  // or "WindowsService"
  }
}

Phase 5: Testing

Status: Not started

TODO:

• Unit tests for Domain entities
• Unit tests for Application handlers (using FakeItEasy)
• Integration tests for Repositories (using Testcontainers)
• API tests (using WebApplicationFactory)
• Generate fake test data (using Bogus)

Test Example:

public class MessageIdGeneratorTests
{
    [Fact]
    public void Generate_ShouldProduceSameHashAsLegacy()
    {
        // Arrange
        var generator = new MessageIdGenerator();
        var original = "msg-123";
        var sender = "test@example.com";
        var date = new DateTime(2026, 1, 1, 12, 0, 0);
        var subject = "Test Subject";
        
        // Act
        var messageId = generator.Generate(original, sender, date, subject);
        
        // Assert
        messageId.Hash.Should().NotBeNullOrEmpty();
        // TODO: Verify against known legacy hash
    }
}

---

Architecture Decisions

Clean Architecture Layers

1. Domain: Core business logic, no dependencies
2. Application: Use cases, depends on Domain
3. Infrastructure: External concerns, depends on Domain + Application
4. API: Entry point, depends on all

CQRS Pattern with MediatR

• Commands: Modify state (Create, Update, Delete)
• Queries: Read data (Get, List)
• Separate models for read and write operations

Repository Pattern

• Interface in Application layer
• Implementation in Infrastructure layer
• One repository per Aggregate Root

---

Known Issues and Limitations

1. PdfSharp Embedded File Extraction

PdfSharp has limited support for embedded file extraction from PDFs. If advanced PDF processing is needed, consider:

• iText7 (AGPL or commercial license)
• Aspose.PDF (commercial license)
• Custom PDF parsing using PDF specification

2. windream COM Interop

The windream DMS integration uses COM Interop which is Windows-only. The application cannot be fully cross-platform unless windream provides a REST API alternative.

3. OAuth2 Token Refresh

Current implementation acquires new tokens on each request. Consider implementing token caching:

• Use Microsoft.Identity.Web for automatic token management
• Cache tokens in memory or distributed cache (Redis)

---

Development Guidelines

1. Code Style

• All code and comments: English
• README.md and user documentation: German
• Follow C# naming conventions (PascalCase, camelCase)
• Use nullable reference types (#nullable enable)

2. Logging

Use Serilog with structured logging:

_logger.LogInformation("Processing email {MessageId} from profile {ProfileId}", messageId, profileId);

3. Configuration

• Development: appsettings.Development.json + User Secrets
• Production: appsettings.json + Environment Variables + Azure Key Vault

4. Error Handling

• Domain: Throw DomainException for business rule violations
• Application: Use FluentValidation for input validation
• API: Use exception handling middleware to return proper HTTP status codes

---

Deployment Scenarios

IIS Hosting (Default)

{
  "Hosting": {
    "Mode": "IIS"
  }
}

Windows Service Hosting

{
  "Hosting": {
    "Mode": "WindowsService"
  }
}

In Program.cs:

var builder = WebApplication.CreateBuilder(args);

if (builder.Configuration["Hosting:Mode"] == "WindowsService")
{
    builder.Host.UseWindowsService();
}

Install as Windows Service:

sc create MessagingService binPath="C:\Path\To\DigitalData.MessagingService.API.exe"

---

Contact and Support

For questions about this implementation, consult:

• Legacy system analysis: legacy/PROJECT_ANALYSIS.md
• Migration plan: MIGRATION_PLAN.md (if created)
• This document: agents.md

---

Last Updated: 2026-07-07 Version: 1.0 Status: Phase 1 Complete (Domain Layer), Phase 2-8 Pending