# EmailProfiler - Agent Notes and Future Enhancements ## Purpose This document contains important notes, decisions, and future enhancement plans for the EmailProfiler application. This is intended for AI agents and developers who will continue development. --- ## Important Notes ### 1. Database Schema - DO NOT MODIFY **CRITICAL**: The database schema must NEVER be modified. All Entity Framework entities must map to existing legacy tables using `[Table]` and `[Column]` attributes. **Naming Convention**: - Database: `SNAKE_CASE` with prefixes (TBEMLP_, TBDD_) - C# Entities: `PascalCase` without prefixes - Use `[Table("TBDD_FOO")]` and `[Column("COLUMN_NAME")]` attributes **Example**: ```csharp [Table("TBDD_EMAIL_ACCOUNT")] public class EmailAccount { [Column("EMAIL_ACCOUNT_ID")] public int Id { get; set; } [Column("ACCOUNT_NAME")] public string AccountName { get; set; } } ``` ### 2. Message ID Hash Algorithm The `MessageIdGenerator` in `Domain.Services` must use **exactly the same algorithm** as the legacy system to ensure duplicate detection works correctly. **Algorithm**: SHA256 hash of `{originalMessageId}|{sender}|{date:yyyyMMddHHmmss}|{subject}` ### 3. DateTime Usage - ALWAYS Use Local Time **CRITICAL**: Always use `DateTime.Now` instead of `DateTime.UtcNow` throughout the entire application. **Reason**: The legacy system uses local server time, and the database stores all timestamps as local time. Using UTC would break compatibility and cause incorrect time comparisons. **Examples**: ```csharp // ✅ CORRECT profile.CreatedDate = DateTime.Now; var lastPoll = DateTime.Now.AddMinutes(-profile.PollIntervalMinutes); // ❌ WRONG - DO NOT USE profile.CreatedDate = DateTime.UtcNow; // NEVER USE UTC var lastPoll = DateTime.UtcNow.AddMinutes(-profile.PollIntervalMinutes); // NEVER USE UTC ``` **Important**: This applies to: - All entity audit fields (CreatedDate, ModifiedDate, LastPollDate, etc.) - All date comparisons in business logic - All timestamps in logs and error messages - All date parameters in queries ### 4. Git Operations - NEVER Without Explicit Permission **CRITICAL**: NEVER execute `git commit` or `git push` commands automatically. ALWAYS wait for explicit user instruction. **Rules**: - Only commit when user explicitly says "commit" or "commit this" - Only push when user explicitly says "push" or "push to remote" - Stage files with `git add` ONLY when about to commit per user request ### 5. MediatR Command/Query File Organization **IMPORTANT**: Commands/Queries and their Handlers must be in the SAME file. **Example**: ```csharp // ✅ CORRECT - CreateEmailProfileCommand.cs contains BOTH public record CreateEmailProfileCommand : IRequest { ... } public class CreateEmailProfileCommandHandler : IRequestHandler { ... } // ❌ WRONG - Separate files // CreateEmailProfileCommand.cs (command only) // CreateEmailProfileCommandHandler.cs (handler only) ``` **File Naming**: - Commands: `{Verb}{Entity}Command.cs` (e.g., `CreateEmailProfileCommand.cs`) - Queries: `{Verb}{Entity}Query.cs` (e.g., `GetEmailProfilesQuery.cs`) **Folder Structure** (NO Features/ prefix): ``` Application/ ├── EmailProfiles/ │ ├── Commands/CreateEmailProfileCommand.cs │ ├── Queries/GetEmailProfilesQuery.cs │ └── Validators/CreateEmailProfileCommandValidator.cs ├── EmailAccounts/ │ ├── Commands/CreateEmailAccountCommand.cs │ └── Queries/GetEmailAccountsQuery.cs └── Common/ ├── Dtos/EmailProfileDto.cs (single DTOs at root) ├── Dtos/EmailHistories/ (multiple DTOs in subfolder) └── Interfaces/IEmailService.cs ``` ### 6. Repository Pattern - NO UnitOfWork, Generic CRUD with AutoMapper **CRITICAL**: DO NOT use IUnitOfWork pattern. Use generic repository pattern with AutoMapper-based CRUD operations. **Key Principles**: - ✅ Each operation auto-saves changes - NO explicit SaveChangesAsync needed - ✅ Use `UpdateSingleAsync` / `DeleteSingleAsync` for single-record safety - ✅ Use `UpdateAsync` / `DeleteAsync` only when intentionally modifying multiple records - ✅ AutoMapper handles all DTO → Entity mappings **Pattern**: ```csharp // IRepository generic interface public interface IRepository where TEntity : class { // Query operations Task GetByIdAsync(int id, CancellationToken cancellationToken = default); Task> FindAsync(Expression> predicate, ...); // Create - auto-saves Task CreateAsync(TDto dto, CancellationToken cancellationToken = default); // Update - auto-saves Task UpdateAsync(Expression<...> predicate, TDto dto, ...); // Multiple records Task UpdateSingleAsync(Expression<...> predicate, TDto dto, ...); // SAFE: Single record only // Delete - auto-saves Task DeleteAsync(Expression<...> predicate, ...); // Multiple records Task DeleteSingleAsync(Expression<...> predicate, ...); // SAFE: Single record only } ``` **Command Handler Examples**: ```csharp // ✅ CORRECT - CreateAsync auto-saves public class CreateEmailProfileCommandHandler(IRepository repository) : IRequestHandler { public async Task Handle(CreateEmailProfileCommand request, CancellationToken cancellationToken) { var profile = await repository.CreateAsync(request, cancellationToken); return profile.Id; // NO SaveChangesAsync needed! } } // ✅ CORRECT - UpdateSingleAsync for safety (throws if 0 or 2+ records match) public class UpdateEmailProfileCommandHandler(IRepository repository) : IRequestHandler { public async Task Handle(UpdateEmailProfileCommand request, CancellationToken cancellationToken) { await repository.UpdateSingleAsync(p => p.Id == request.Id, request, cancellationToken); return request.Id; // NO SaveChangesAsync needed! } } // ✅ CORRECT - DeleteSingleAsync for safety (throws if 0 or 2+ records match) public class DeleteEmailProfileCommandHandler(IRepository repository) : IRequestHandler { public async Task Handle(DeleteEmailProfileCommand request, CancellationToken cancellationToken) { await repository.DeleteSingleAsync(p => p.Id == request.Id, cancellationToken); return request.Id; // NO SaveChangesAsync needed! } } // ❌ WRONG - Manual entity creation (use AutoMapper instead) var profile = new EmailProfile { ProfileName = request.ProfileName, EmailAccountId = request.EmailAccountId, // ... 15 more properties }; // ❌ WRONG - Using IUnitOfWork (removed) public CreateEmailProfileCommandHandler(IUnitOfWork unitOfWork) { ... } // ❌ WRONG - Calling SaveChangesAsync (removed) await repository.SaveChangesAsync(cancellationToken); ``` **Safety Rules**: 1. **UpdateSingleAsync** - Use for ID-based updates. Throws `InvalidOperationException` if: - Zero records match (entity not found) - Multiple records match (predicate too broad) 2. **DeleteSingleAsync** - Use for ID-based deletes. Throws `InvalidOperationException` if: - Zero records match (entity not found) - Multiple records match (predicate too broad) 3. **UpdateAsync / DeleteAsync** - Use ONLY when intentionally modifying multiple records: ```csharp // ✅ CORRECT - Intentional bulk operation await repository.UpdateAsync( p => p.EmailAccountId == accountId, new { IsActive = false }, cancellationToken); // ✅ Returns count of updated/deleted records var count = await repository.DeleteAsync(p => p.IsActive == false, cancellationToken); ``` **DTO Mapping Responsibility**: - Each DTO creator must define their own AutoMapper profile - Example: `CreateEmailProfileCommand` → `EmailProfile` mapping must be defined in `EmailProfileMappingProfile.cs` - Repository implementation uses `IMapper.Map(dto)` internally --- ## Future Enhancements ### 7. RabbitMQ Command Bus Integration (IMPLEMENTED) **Purpose**: Asynchronous command processing via RabbitMQ message broker for POST/PUT/DELETE operations. **Architecture**: - **GET Queries**: Synchronous (immediate response via MediatR) - **POST/PUT/DELETE Commands**: Can be asynchronous (published to RabbitMQ, processed by background worker) **RabbitMQ Server**: - Management UI: `http://172.24.12.56:15672` - AMQP Port: `5672` (default) - Exchange: `emailprofiler.commands` (Direct) - Queue: `emailprofiler.command.queue` - Routing Key: `command` **Implementation Components**: 1. **ICommandPublisher** (`Application/Common/Interfaces/ICommandPublisher.cs`): - Interface for publishing commands to message broker - Generic method: `PublishAsync(TCommand command, CancellationToken)` 2. **RabbitMqCommandPublisher** (`Infrastructure/Messaging/RabbitMqCommandPublisher.cs`): - Implements `ICommandPublisher` - Serializes command to JSON with metadata envelope (CommandType, Payload, CorrelationId, PublishedAt) - Publishes to RabbitMQ exchange with persistent delivery mode 3. **RabbitMqCommandConsumer** (`Infrastructure/Messaging/RabbitMqCommandConsumer.cs`): - BackgroundService that consumes commands from RabbitMQ - Deserializes command envelope - Resolves command type from assembly - Executes command via MediatR in scoped service - Acknowledges message on success, requeues on error 4. **RabbitMqConfiguration** (`Infrastructure/Messaging/RabbitMqConfiguration.cs`): - Configuration model for RabbitMQ connection - Binds to `appsettings.json` section: `RabbitMq` **Configuration** (`appsettings.json`): ```json { "RabbitMq": { "HostName": "172.24.12.56", "Port": 5672, "UserName": "guest", "Password": "guest", "VirtualHost": "/", "ExchangeName": "emailprofiler.commands", "QueueName": "emailprofiler.command.queue", "RoutingKey": "command", "AutomaticRecoveryEnabled": true, "NetworkRecoveryIntervalSeconds": 10 } } ``` **Dependency Injection** (`Infrastructure/DependencyInjection.cs`): ```csharp services.Configure(configuration.GetSection(RabbitMqConfiguration.SectionName)); services.AddSingleton(); services.AddHostedService(); ``` **Usage in API Controllers** (Future): ```csharp // Option 1: Synchronous (immediate execution via MediatR) var result = await _mediator.Send(new CreateEmailProfileCommand(...), cancellationToken); return Ok(result); // Option 2: Asynchronous (publish to RabbitMQ for background processing) await _commandPublisher.PublishAsync(new CreateEmailProfileCommand(...), cancellationToken); return Accepted(); // HTTP 202 - command queued for processing ``` **Benefits**: - Decouples API from long-running command processing - Improves API responsiveness (fire-and-forget) - Enables horizontal scaling (multiple consumers) - Automatic retries on failure (requeue mechanism) - Message persistence (survives application restarts) --- ### HIGH PRIORITY: Email Queue for Outgoing Messages (Future) **Implementation Steps**: 1. **Add NuGet Package**: ```bash dotnet add package RabbitMQ.Client ``` 2. **Create RabbitMqEmailQueue.cs**: ```csharp // src/DigitalData.EmailProfiler.Infrastructure/Queue/RabbitMqEmailQueue.cs public class RabbitMqEmailQueue : IEmailQueue { private readonly IConnection _connection; private readonly IModel _channel; private const string QueueName = "email-outbox"; public RabbitMqEmailQueue(IOptions config) { var factory = new ConnectionFactory { HostName = config.Value.HostName, Port = config.Value.Port, UserName = config.Value.UserName, Password = config.Value.Password }; _connection = factory.CreateConnection(); _channel = _connection.CreateModel(); _channel.QueueDeclare( queue: QueueName, durable: true, exclusive: false, autoDelete: false, arguments: null); } public async Task EnqueueAsync(OutgoingEmail email, CancellationToken cancellationToken) { var json = JsonSerializer.Serialize(email); var body = Encoding.UTF8.GetBytes(json); var properties = _channel.CreateBasicProperties(); properties.Persistent = true; _channel.BasicPublish( exchange: "", routingKey: QueueName, basicProperties: properties, body: body); await Task.CompletedTask; } public async Task DequeueAsync(CancellationToken cancellationToken) { var result = _channel.BasicGet(QueueName, autoAck: false); if (result == null) return null; var json = Encoding.UTF8.GetString(result.Body.ToArray()); var email = JsonSerializer.Deserialize(json); _channel.BasicAck(result.DeliveryTag, false); return await Task.FromResult(email); } } ``` 3. **Configuration** (appsettings.json): ```json { "RabbitMq": { "HostName": "localhost", "Port": 5672, "UserName": "guest", "Password": "guest" } } ``` 4. **Dependency Injection** (Program.cs): ```csharp // Replace InMemoryEmailQueue with RabbitMqEmailQueue // builder.Services.AddSingleton(); builder.Services.AddSingleton(); ``` **Benefits**: - Message persistence (survives application restarts) - Scalability (multiple worker instances can consume from queue) - Reliability (automatic retries, dead letter queues) - Monitoring (RabbitMQ management UI) **Migration Path**: 1. Deploy RabbitMQ server (Docker recommended) 2. Test RabbitMqEmailQueue in staging environment 3. Switch DI registration from InMemoryEmailQueue to RabbitMqEmailQueue 4. Monitor queue depth and worker performance --- ## Pending Implementation Tasks ### Phase 2: Application Layer (COMPLETE) **Status**: ✅ Complete - All Commands, Queries, Handlers, Validators, AutoMapper Profiles, and Interfaces implemented **Completed**: - ✅ MediatR Commands (CreateEmailProfileCommand, ProcessEmailCommand, etc.) - ✅ MediatR Queries (GetEmailProfilesQuery, GetEmailHistoryQuery, etc.) - ✅ Command/Query Handlers - ✅ FluentValidation Validators - ✅ AutoMapper Profiles - ✅ Application Interfaces (IEmailService, IPdfProcessingService, IDmsService, etc.) **Example Command**: ```csharp // src/DigitalData.EmailProfiler.Application/EmailProfiles/Commands/CreateEmailProfileCommand.cs public record CreateEmailProfileCommand(string ProfileName, int EmailAccountId) : IRequest; public class CreateEmailProfileCommandHandler : IRequestHandler { private readonly IRepository _repository; public async Task Handle(CreateEmailProfileCommand request, CancellationToken cancellationToken) { var profile = await _repository.CreateAsync(request, cancellationToken); return profile.Id; } } ``` ### 8. Email Library - Limilabs Mail.dll **IMPORTANT**: This project uses **Limilabs Mail.dll** (https://www.limilabs.com/) for email operations, NOT MailKit/MimeKit. **Why Limilabs?**: - Commercial-grade IMAP/POP3/SMTP library - Better OAuth2 support (Microsoft 365, Gmail) - More reliable with Exchange servers - Superior attachment handling - Built-in retry mechanisms **NuGet Package**: ```bash dotnet add package Limilabs.Mail ``` **Key Classes**: - `Imap` - IMAP client for receiving emails - `Smtp` - SMTP client for sending emails - `Mail.Message` - Email message representation - `OAuth2` - OAuth2 authentication helper **Implementation Example**: ```csharp // Limilabs IMAP with OAuth2 using Limilabs.Client.IMAP; using Limilabs.Mail; public class LimilabsEmailService : IEmailService { public async Task> ReceiveEmailsAsync(EmailAccountDto account) { using var imap = new Imap(); if (account.UseOAuth2) { await imap.ConnectSSLAsync(account.ImapServer, account.ImapPort); await imap.LoginOAUTH2Async(account.Username, account.OAuth2AccessToken); } else { await imap.ConnectSSLAsync(account.ImapServer, account.ImapPort); await imap.LoginAsync(account.Username, account.EncryptedPassword); } imap.SelectInbox(); var uids = imap.Search(Flag.Unseen); var messages = new List(); foreach (var uid in uids) { var eml = imap.GetMessageByUID(uid); var mail = new MailBuilder().CreateFromEml(eml); messages.Add(ConvertToEmailMessage(mail)); } imap.Close(); return messages; } } ``` **DO NOT USE**: - ❌ MailKit - ❌ MimeKit - ❌ System.Net.Mail (obsolete) ### Phase 3: Infrastructure Layer (COMPLETE) **Status**: ✅ Complete - DbContext, Repository, Services, RabbitMQ, and DI implemented **Completed**: - ✅ EmailProfilerDbContext with DbSet for all entities (attribute-only config, no overrides) - ✅ Generic Repository implementing IRepository with AutoMapper-based CRUD - ✅ LimilabsEmailService (IMAP/SMTP with OAuth2 using Limilabs Mail.dll - TODO: Add Limilabs.Mail NuGet) - ✅ GdPicturePdfProcessingService (using GdPicture.NET 14 - TODO: Add GdPicture NuGet and license) - ✅ WindreamDmsService (COM Interop - TODO: Add windream COM Interop references) - ✅ DataProtectionEncryptionService (Data Protection API) - ✅ InMemoryEmailQueue (TODO: Upgrade to RabbitMqEmailQueue later) - ✅ RabbitMqCommandPublisher and RabbitMqCommandConsumer - ✅ DependencyInjection.cs with all service registrations **Implementation Notes**: - All services have real implementations with commented TODO blocks for external dependencies - LimilabsEmailService uses Microsoft.Identity.Client for OAuth2 token acquisition - GdPicturePdfProcessingService uses GdPicture.NET 14.x API (GetAttachmentCount, ExtractEmbeddedFile) - WindreamDmsService uses COM Interop (WMSession, WMConnect, WMObjects) based on legacy patterns - NO EF Core migrations (legacy DB must not be modified) **DbContext Example**: ```csharp public class EmailProfilerDbContext : DbContext { public DbSet EmailAccounts { get; set; } public DbSet EmailProfiles { get; set; } // ... other DbSets protected override void OnModelCreating(ModelBuilder modelBuilder) { modelBuilder.ApplyConfigurationsFromAssembly(Assembly.GetExecutingAssembly()); // Important: Check for triggers modelBuilder.Entity().ToTable(tb => tb.HasTrigger("TR_TBEMLP_HISTORY_AUDIT")); } } ``` ### Phase 4: API Layer **Status**: Minimal structure exists **TODO**: - [ ] Create Controllers (EmailProfilesController, EmailAccountsController, EmailHistoryController) - [ ] Create Background Workers (EmailPollingWorker, EmailSenderWorker) - [ ] Configure Serilog - [ ] Configure Scalar (OpenAPI documentation) - [ ] Add Exception Handling Middleware - [ ] Configure DI for all layers - [ ] Support both IIS and Windows Service hosting **Worker Configuration** (appsettings.json): ```json { "Workers": { "EmailPolling": { "Enabled": true, "IntervalSeconds": 60 }, "EmailSender": { "Enabled": true, "IntervalSeconds": 5 } }, "Hosting": { "Mode": "IIS" // or "WindowsService" } } ``` ### Phase 5: Testing **Status**: Not started **TODO**: - [ ] Unit tests for Domain entities - [ ] Unit tests for Application handlers (using FakeItEasy) - [ ] Integration tests for Repositories (using Testcontainers) - [ ] API tests (using WebApplicationFactory) - [ ] Generate fake test data (using Bogus) **Test Example**: ```csharp public class MessageIdGeneratorTests { [Fact] public void Generate_ShouldProduceSameHashAsLegacy() { // Arrange var generator = new MessageIdGenerator(); var original = "msg-123"; var sender = "test@example.com"; var date = new DateTime(2026, 1, 1, 12, 0, 0); var subject = "Test Subject"; // Act var messageId = generator.Generate(original, sender, date, subject); // Assert messageId.Hash.Should().NotBeNullOrEmpty(); // TODO: Verify against known legacy hash } } ``` --- ## Architecture Decisions ### Clean Architecture Layers 1. **Domain**: Core business logic, no dependencies 2. **Application**: Use cases, depends on Domain 3. **Infrastructure**: External concerns, depends on Domain + Application 4. **API**: Entry point, depends on all ### CQRS Pattern with MediatR - **Commands**: Modify state (Create, Update, Delete) - **Queries**: Read data (Get, List) - Separate models for read and write operations ### Repository Pattern - Interface in Application layer - Implementation in Infrastructure layer - One repository per Aggregate Root --- ## Known Issues and Limitations ### 1. PdfSharp Embedded File Extraction PdfSharp has limited support for embedded file extraction from PDFs. If advanced PDF processing is needed, consider: - **iText7** (AGPL or commercial license) - **Aspose.PDF** (commercial license) - Custom PDF parsing using PDF specification ### 2. windream COM Interop The windream DMS integration uses COM Interop which is Windows-only. The application cannot be fully cross-platform unless windream provides a REST API alternative. ### 3. OAuth2 Token Refresh Current implementation acquires new tokens on each request. Consider implementing token caching: - Use `Microsoft.Identity.Web` for automatic token management - Cache tokens in memory or distributed cache (Redis) --- ## Development Guidelines ### 1. Code Style - All code and comments: **English** - README.md and user documentation: **German** - Follow C# naming conventions (PascalCase, camelCase) - Use nullable reference types (`#nullable enable`) ### 2. Logging Use Serilog with structured logging: ```csharp _logger.LogInformation("Processing email {MessageId} from profile {ProfileId}", messageId, profileId); ``` ### 3. Configuration - Development: `appsettings.Development.json` + User Secrets - Production: `appsettings.json` + Environment Variables + Azure Key Vault ### 4. Error Handling - Domain: Throw `DomainException` for business rule violations - Application: Use `FluentValidation` for input validation - API: Use exception handling middleware to return proper HTTP status codes --- ## Deployment Scenarios ### IIS Hosting (Default) ```json { "Hosting": { "Mode": "IIS" } } ``` ### Windows Service Hosting ```json { "Hosting": { "Mode": "WindowsService" } } ``` In `Program.cs`: ```csharp var builder = WebApplication.CreateBuilder(args); if (builder.Configuration["Hosting:Mode"] == "WindowsService") { builder.Host.UseWindowsService(); } ``` Install as Windows Service: ```bash sc create EmailProfiler binPath="C:\Path\To\DigitalData.EmailProfiler.API.exe" ``` --- ## Contact and Support For questions about this implementation, consult: - Legacy system analysis: `legacy/PROJECT_ANALYSIS.md` - Migration plan: `MIGRATION_PLAN.md` (if created) - This document: `agents.md` --- **Last Updated**: 2026-07-07 **Version**: 1.0 **Status**: Phase 1 Complete (Domain Layer), Phase 2-8 Pending