DigitalData.MessagingService/agents.md

EmailProfiler - Agent Notes and Future Enhancements

Purpose

This document contains important notes, decisions, and future enhancement plans for the EmailProfiler application. This is intended for AI agents and developers who will continue development.

---

Important Notes

1. Database Schema - DO NOT MODIFY

CRITICAL: The database schema must NEVER be modified. All Entity Framework entities must map to existing legacy tables using [Table] and [Column] attributes.

Naming Convention:

• Database: SNAKE_CASE with prefixes (TBEMLP_, TBDD_)
• C# Entities: PascalCase without prefixes
• Use [Table("TBDD_FOO")] and [Column("COLUMN_NAME")] attributes

Example:

[Table("TBDD_EMAIL_ACCOUNT")]
public class EmailAccount
{
    [Column("EMAIL_ACCOUNT_ID")]
    public int Id { get; set; }
    
    [Column("ACCOUNT_NAME")]
    public string AccountName { get; set; }
}

2. Message ID Hash Algorithm

The MessageIdGenerator in Domain.Services must use exactly the same algorithm as the legacy system to ensure duplicate detection works correctly.

Algorithm: SHA256 hash of {originalMessageId}|{sender}|{date:yyyyMMddHHmmss}|{subject}

3. DateTime Usage - ALWAYS Use Local Time

CRITICAL: Always use DateTime.Now instead of DateTime.UtcNow throughout the entire application.

Reason: The legacy system uses local server time, and the database stores all timestamps as local time. Using UTC would break compatibility and cause incorrect time comparisons.

Examples:

// ✅ CORRECT
profile.CreatedDate = DateTime.Now;
var lastPoll = DateTime.Now.AddMinutes(-profile.PollIntervalMinutes);

// ❌ WRONG - DO NOT USE
profile.CreatedDate = DateTime.UtcNow;  // NEVER USE UTC
var lastPoll = DateTime.UtcNow.AddMinutes(-profile.PollIntervalMinutes);  // NEVER USE UTC

Important: This applies to:

• All entity audit fields (CreatedDate, ModifiedDate, LastPollDate, etc.)
• All date comparisons in business logic
• All timestamps in logs and error messages
• All date parameters in queries

4. No Commits Without Permission

NEVER commit changes to git automatically. Always wait for explicit user instruction to commit.

---

Future Enhancements

HIGH PRIORITY: RabbitMQ Queue Implementation

Current State:

• Email queue is implemented using in-memory Channel<T> in InMemoryEmailQueue.cs
• Location: src/DigitalData.EmailProfiler.Infrastructure/Queue/InMemoryEmailQueue.cs

Future Enhancement: Replace the in-memory queue with RabbitMQ for production resilience and scalability.

Implementation Steps:

1. Add NuGet Package:

   dotnet add package RabbitMQ.Client
   

2. Create RabbitMqEmailQueue.cs:

   // src/DigitalData.EmailProfiler.Infrastructure/Queue/RabbitMqEmailQueue.cs
   public class RabbitMqEmailQueue : IEmailQueue
   {
       private readonly IConnection _connection;
       private readonly IModel _channel;
       private const string QueueName = "email-outbox";
   
       public RabbitMqEmailQueue(IOptions<RabbitMqConfiguration> config)
       {
           var factory = new ConnectionFactory
           {
               HostName = config.Value.HostName,
               Port = config.Value.Port,
               UserName = config.Value.UserName,
               Password = config.Value.Password
           };
   
           _connection = factory.CreateConnection();
           _channel = _connection.CreateModel();
   
           _channel.QueueDeclare(
               queue: QueueName,
               durable: true,
               exclusive: false,
               autoDelete: false,
               arguments: null);
       }
   
       public async Task EnqueueAsync(OutgoingEmail email, CancellationToken cancellationToken)
       {
           var json = JsonSerializer.Serialize(email);
           var body = Encoding.UTF8.GetBytes(json);
   
           var properties = _channel.CreateBasicProperties();
           properties.Persistent = true;
   
           _channel.BasicPublish(
               exchange: "",
               routingKey: QueueName,
               basicProperties: properties,
               body: body);
   
           await Task.CompletedTask;
       }
   
       public async Task<OutgoingEmail?> DequeueAsync(CancellationToken cancellationToken)
       {
           var result = _channel.BasicGet(QueueName, autoAck: false);
   
           if (result == null)
               return null;
   
           var json = Encoding.UTF8.GetString(result.Body.ToArray());
           var email = JsonSerializer.Deserialize<OutgoingEmail>(json);
   
           _channel.BasicAck(result.DeliveryTag, false);
   
           return await Task.FromResult(email);
       }
   }
   

3. Configuration (appsettings.json):

   {
     "RabbitMq": {
       "HostName": "localhost",
       "Port": 5672,
       "UserName": "guest",
       "Password": "guest"
     }
   }
   

4. Dependency Injection (Program.cs):

   // Replace InMemoryEmailQueue with RabbitMqEmailQueue
   // builder.Services.AddSingleton<IEmailQueue, InMemoryEmailQueue>();
   builder.Services.AddSingleton<IEmailQueue, RabbitMqEmailQueue>();
   

Benefits:

• Message persistence (survives application restarts)
• Scalability (multiple worker instances can consume from queue)
• Reliability (automatic retries, dead letter queues)
• Monitoring (RabbitMQ management UI)

Migration Path:

1. Deploy RabbitMQ server (Docker recommended)
2. Test RabbitMqEmailQueue in staging environment
3. Switch DI registration from InMemoryEmailQueue to RabbitMqEmailQueue
4. Monitor queue depth and worker performance

---

Pending Implementation Tasks

Phase 2: Application Layer (IN PROGRESS)

Status: Partially complete - DTOs created, Commands/Queries needed

TODO:

• Create MediatR Commands (CreateEmailProfileCommand, ProcessEmailCommand, etc.)
• Create MediatR Queries (GetEmailProfilesQuery, GetEmailHistoryQuery, etc.)
• Create Command/Query Handlers
• Create FluentValidation Validators
• Create AutoMapper Profiles
• Create Application Interfaces (IEmailService, IPdfProcessingService, IDmsService, etc.)

Example Command:

// src/DigitalData.EmailProfiler.Application/EmailProfiles/Commands/CreateEmailProfileCommand.cs
public record CreateEmailProfileCommand(string ProfileName, int EmailAccountId) : IRequest<int>;

public class CreateEmailProfileCommandHandler : IRequestHandler<CreateEmailProfileCommand, int>
{
    private readonly IEmailProfileRepository _repository;
    
    public async Task<int> Handle(CreateEmailProfileCommand request, CancellationToken cancellationToken)
    {
        var profile = new EmailProfile
        {
            ProfileName = request.ProfileName,
            EmailAccountId = request.EmailAccountId,
            IsActive = true
        };
        
        await _repository.AddAsync(profile, cancellationToken);
        return profile.Id;
    }
}

Phase 3: Infrastructure Layer

Status: Not started

TODO:

• Create EmailProfilerDbContext with DbSet for all entities
• Create Entity Configurations (Fluent API) for all entities
• Create Repositories implementing Application interfaces
• Create MailKitEmailService (IMAP/SMTP with OAuth2)
• Create PdfSharpProcessingService
• Create WindreamDmsService (COM Interop)
• Create EncryptionService (Data Protection API)
• Create initial EF Core migration

DbContext Example:

public class EmailProfilerDbContext : DbContext
{
    public DbSet<EmailAccount> EmailAccounts { get; set; }
    public DbSet<EmailProfile> EmailProfiles { get; set; }
    // ... other DbSets
    
    protected override void OnModelCreating(ModelBuilder modelBuilder)
    {
        modelBuilder.ApplyConfigurationsFromAssembly(Assembly.GetExecutingAssembly());
        
        // Important: Check for triggers
        modelBuilder.Entity<EmailHistory>().ToTable(tb => tb.HasTrigger("TR_TBEMLP_HISTORY_AUDIT"));
    }
}

Phase 4: API Layer

Status: Minimal structure exists

TODO:

• Create Controllers (ProfilesController, EmailAccountsController, HistoryController)
• Create Background Workers (EmailPollingWorker, EmailSenderWorker)
• Configure Serilog
• Configure Scalar (OpenAPI documentation)
• Add Exception Handling Middleware
• Configure DI for all layers
• Support both IIS and Windows Service hosting

Worker Configuration (appsettings.json):

{
  "Workers": {
    "EmailPolling": {
      "Enabled": true,
      "IntervalSeconds": 60
    },
    "EmailSender": {
      "Enabled": true,
      "IntervalSeconds": 5
    }
  },
  "Hosting": {
    "Mode": "IIS"  // or "WindowsService"
  }
}

Phase 5: Testing

Status: Not started

TODO:

• Unit tests for Domain entities
• Unit tests for Application handlers (using FakeItEasy)
• Integration tests for Repositories (using Testcontainers)
• API tests (using WebApplicationFactory)
• Generate fake test data (using Bogus)

Test Example:

public class MessageIdGeneratorTests
{
    [Fact]
    public void Generate_ShouldProduceSameHashAsLegacy()
    {
        // Arrange
        var generator = new MessageIdGenerator();
        var original = "msg-123";
        var sender = "test@example.com";
        var date = new DateTime(2026, 1, 1, 12, 0, 0);
        var subject = "Test Subject";
        
        // Act
        var messageId = generator.Generate(original, sender, date, subject);
        
        // Assert
        messageId.Hash.Should().NotBeNullOrEmpty();
        // TODO: Verify against known legacy hash
    }
}

---

Architecture Decisions

Clean Architecture Layers

1. Domain: Core business logic, no dependencies
2. Application: Use cases, depends on Domain
3. Infrastructure: External concerns, depends on Domain + Application
4. API: Entry point, depends on all

CQRS Pattern with MediatR

• Commands: Modify state (Create, Update, Delete)
• Queries: Read data (Get, List)
• Separate models for read and write operations

Repository Pattern

• Interface in Application layer
• Implementation in Infrastructure layer
• One repository per Aggregate Root

---

Known Issues and Limitations

1. PdfSharp Embedded File Extraction

PdfSharp has limited support for embedded file extraction from PDFs. If advanced PDF processing is needed, consider:

• iText7 (AGPL or commercial license)
• Aspose.PDF (commercial license)
• Custom PDF parsing using PDF specification

2. windream COM Interop

The windream DMS integration uses COM Interop which is Windows-only. The application cannot be fully cross-platform unless windream provides a REST API alternative.

3. OAuth2 Token Refresh

Current implementation acquires new tokens on each request. Consider implementing token caching:

• Use Microsoft.Identity.Web for automatic token management
• Cache tokens in memory or distributed cache (Redis)

---

Development Guidelines

1. Code Style

• All code and comments: English
• README.md and user documentation: German
• Follow C# naming conventions (PascalCase, camelCase)
• Use nullable reference types (#nullable enable)

2. Logging

Use Serilog with structured logging:

_logger.LogInformation("Processing email {MessageId} from profile {ProfileId}", messageId, profileId);

3. Configuration

• Development: appsettings.Development.json + User Secrets
• Production: appsettings.json + Environment Variables + Azure Key Vault

4. Error Handling

• Domain: Throw DomainException for business rule violations
• Application: Use FluentValidation for input validation
• API: Use exception handling middleware to return proper HTTP status codes

---

Deployment Scenarios

IIS Hosting (Default)

{
  "Hosting": {
    "Mode": "IIS"
  }
}

Windows Service Hosting

{
  "Hosting": {
    "Mode": "WindowsService"
  }
}

In Program.cs:

var builder = WebApplication.CreateBuilder(args);

if (builder.Configuration["Hosting:Mode"] == "WindowsService")
{
    builder.Host.UseWindowsService();
}

Install as Windows Service:

sc create EmailProfiler binPath="C:\Path\To\DigitalData.EmailProfiler.API.exe"

---

Contact and Support

For questions about this implementation, consult:

• Legacy system analysis: legacy/PROJECT_ANALYSIS.md
• Migration plan: MIGRATION_PLAN.md (if created)
• This document: agents.md

---

Last Updated: 2026-07-07 Version: 1.0 Status: Phase 1 Complete (Domain Layer), Phase 2-8 Pending