diff --git a/agents.md b/agents.md new file mode 100644 index 0000000..bd8cbb7 --- /dev/null +++ b/agents.md @@ -0,0 +1,402 @@ +# EmailProfiler - Agent Notes and Future Enhancements + +## Purpose +This document contains important notes, decisions, and future enhancement plans for the EmailProfiler application. This is intended for AI agents and developers who will continue development. + +--- + +## Important Notes + +### 1. Database Schema - DO NOT MODIFY +**CRITICAL**: The database schema must NEVER be modified. All Entity Framework entities must map to existing legacy tables using `[Table]` and `[Column]` attributes. + +**Naming Convention**: +- Database: `SNAKE_CASE` with prefixes (TBEMLP_, TBDD_) +- C# Entities: `PascalCase` without prefixes +- Use `[Table("TBDD_FOO")]` and `[Column("COLUMN_NAME")]` attributes + +**Example**: +```csharp +[Table("TBDD_EMAIL_ACCOUNT")] +public class EmailAccount +{ + [Column("EMAIL_ACCOUNT_ID")] + public int Id { get; set; } + + [Column("ACCOUNT_NAME")] + public string AccountName { get; set; } +} +``` + +### 2. Message ID Hash Algorithm +The `MessageIdGenerator` in `Domain.Services` must use **exactly the same algorithm** as the legacy system to ensure duplicate detection works correctly. + +**Algorithm**: SHA256 hash of `{originalMessageId}|{sender}|{date:yyyyMMddHHmmss}|{subject}` + +### 3. No Commits Without Permission +**NEVER** commit changes to git automatically. Always wait for explicit user instruction to commit. + +--- + +## Future Enhancements + +### HIGH PRIORITY: RabbitMQ Queue Implementation + +**Current State**: +- Email queue is implemented using in-memory `Channel` in `InMemoryEmailQueue.cs` +- Location: `src/DigitalData.EmailProfiler.Infrastructure/Queue/InMemoryEmailQueue.cs` + +**Future Enhancement**: +Replace the in-memory queue with **RabbitMQ** for production resilience and scalability. + +**Implementation Steps**: + +1. **Add NuGet Package**: + ```bash + dotnet add package RabbitMQ.Client + ``` + +2. **Create RabbitMqEmailQueue.cs**: + ```csharp + // src/DigitalData.EmailProfiler.Infrastructure/Queue/RabbitMqEmailQueue.cs + public class RabbitMqEmailQueue : IEmailQueue + { + private readonly IConnection _connection; + private readonly IModel _channel; + private const string QueueName = "email-outbox"; + + public RabbitMqEmailQueue(IOptions config) + { + var factory = new ConnectionFactory + { + HostName = config.Value.HostName, + Port = config.Value.Port, + UserName = config.Value.UserName, + Password = config.Value.Password + }; + + _connection = factory.CreateConnection(); + _channel = _connection.CreateModel(); + + _channel.QueueDeclare( + queue: QueueName, + durable: true, + exclusive: false, + autoDelete: false, + arguments: null); + } + + public async Task EnqueueAsync(OutgoingEmail email, CancellationToken cancellationToken) + { + var json = JsonSerializer.Serialize(email); + var body = Encoding.UTF8.GetBytes(json); + + var properties = _channel.CreateBasicProperties(); + properties.Persistent = true; + + _channel.BasicPublish( + exchange: "", + routingKey: QueueName, + basicProperties: properties, + body: body); + + await Task.CompletedTask; + } + + public async Task DequeueAsync(CancellationToken cancellationToken) + { + var result = _channel.BasicGet(QueueName, autoAck: false); + + if (result == null) + return null; + + var json = Encoding.UTF8.GetString(result.Body.ToArray()); + var email = JsonSerializer.Deserialize(json); + + _channel.BasicAck(result.DeliveryTag, false); + + return await Task.FromResult(email); + } + } + ``` + +3. **Configuration** (appsettings.json): + ```json + { + "RabbitMq": { + "HostName": "localhost", + "Port": 5672, + "UserName": "guest", + "Password": "guest" + } + } + ``` + +4. **Dependency Injection** (Program.cs): + ```csharp + // Replace InMemoryEmailQueue with RabbitMqEmailQueue + // builder.Services.AddSingleton(); + builder.Services.AddSingleton(); + ``` + +**Benefits**: +- Message persistence (survives application restarts) +- Scalability (multiple worker instances can consume from queue) +- Reliability (automatic retries, dead letter queues) +- Monitoring (RabbitMQ management UI) + +**Migration Path**: +1. Deploy RabbitMQ server (Docker recommended) +2. Test RabbitMqEmailQueue in staging environment +3. Switch DI registration from InMemoryEmailQueue to RabbitMqEmailQueue +4. Monitor queue depth and worker performance + +--- + +## Pending Implementation Tasks + +### Phase 2: Application Layer (IN PROGRESS) +**Status**: Partially complete - DTOs created, Commands/Queries needed + +**TODO**: +- [ ] Create MediatR Commands (CreateEmailProfileCommand, ProcessEmailCommand, etc.) +- [ ] Create MediatR Queries (GetEmailProfilesQuery, GetEmailHistoryQuery, etc.) +- [ ] Create Command/Query Handlers +- [ ] Create FluentValidation Validators +- [ ] Create AutoMapper Profiles +- [ ] Create Application Interfaces (IEmailService, IPdfProcessingService, IDmsService, etc.) + +**Example Command**: +```csharp +// src/DigitalData.EmailProfiler.Application/EmailProfiles/Commands/CreateEmailProfileCommand.cs +public record CreateEmailProfileCommand(string ProfileName, int EmailAccountId) : IRequest; + +public class CreateEmailProfileCommandHandler : IRequestHandler +{ + private readonly IEmailProfileRepository _repository; + + public async Task Handle(CreateEmailProfileCommand request, CancellationToken cancellationToken) + { + var profile = new EmailProfile + { + ProfileName = request.ProfileName, + EmailAccountId = request.EmailAccountId, + IsActive = true + }; + + await _repository.AddAsync(profile, cancellationToken); + return profile.Id; + } +} +``` + +### Phase 3: Infrastructure Layer +**Status**: Not started + +**TODO**: +- [ ] Create EmailProfilerDbContext with DbSet for all entities +- [ ] Create Entity Configurations (Fluent API) for all entities +- [ ] Create Repositories implementing Application interfaces +- [ ] Create MailKitEmailService (IMAP/SMTP with OAuth2) +- [ ] Create PdfSharpProcessingService +- [ ] Create WindreamDmsService (COM Interop) +- [ ] Create EncryptionService (Data Protection API) +- [ ] Create initial EF Core migration + +**DbContext Example**: +```csharp +public class EmailProfilerDbContext : DbContext +{ + public DbSet EmailAccounts { get; set; } + public DbSet EmailProfiles { get; set; } + // ... other DbSets + + protected override void OnModelCreating(ModelBuilder modelBuilder) + { + modelBuilder.ApplyConfigurationsFromAssembly(Assembly.GetExecutingAssembly()); + + // Important: Check for triggers + modelBuilder.Entity().ToTable(tb => tb.HasTrigger("TR_TBEMLP_HISTORY_AUDIT")); + } +} +``` + +### Phase 4: API Layer +**Status**: Minimal structure exists + +**TODO**: +- [ ] Create Controllers (ProfilesController, EmailAccountsController, HistoryController) +- [ ] Create Background Workers (EmailPollingWorker, EmailSenderWorker) +- [ ] Configure Serilog +- [ ] Configure Scalar (OpenAPI documentation) +- [ ] Add Exception Handling Middleware +- [ ] Configure DI for all layers +- [ ] Support both IIS and Windows Service hosting + +**Worker Configuration** (appsettings.json): +```json +{ + "Workers": { + "EmailPolling": { + "Enabled": true, + "IntervalSeconds": 60 + }, + "EmailSender": { + "Enabled": true, + "IntervalSeconds": 5 + } + }, + "Hosting": { + "Mode": "IIS" // or "WindowsService" + } +} +``` + +### Phase 5: Testing +**Status**: Not started + +**TODO**: +- [ ] Unit tests for Domain entities +- [ ] Unit tests for Application handlers (using FakeItEasy) +- [ ] Integration tests for Repositories (using Testcontainers) +- [ ] API tests (using WebApplicationFactory) +- [ ] Generate fake test data (using Bogus) + +**Test Example**: +```csharp +public class MessageIdGeneratorTests +{ + [Fact] + public void Generate_ShouldProduceSameHashAsLegacy() + { + // Arrange + var generator = new MessageIdGenerator(); + var original = "msg-123"; + var sender = "test@example.com"; + var date = new DateTime(2026, 1, 1, 12, 0, 0); + var subject = "Test Subject"; + + // Act + var messageId = generator.Generate(original, sender, date, subject); + + // Assert + messageId.Hash.Should().NotBeNullOrEmpty(); + // TODO: Verify against known legacy hash + } +} +``` + +--- + +## Architecture Decisions + +### Clean Architecture Layers +1. **Domain**: Core business logic, no dependencies +2. **Application**: Use cases, depends on Domain +3. **Infrastructure**: External concerns, depends on Domain + Application +4. **API**: Entry point, depends on all + +### CQRS Pattern with MediatR +- **Commands**: Modify state (Create, Update, Delete) +- **Queries**: Read data (Get, List) +- Separate models for read and write operations + +### Repository Pattern +- Interface in Application layer +- Implementation in Infrastructure layer +- One repository per Aggregate Root + +--- + +## Known Issues and Limitations + +### 1. PdfSharp Embedded File Extraction +PdfSharp has limited support for embedded file extraction from PDFs. If advanced PDF processing is needed, consider: +- **iText7** (AGPL or commercial license) +- **Aspose.PDF** (commercial license) +- Custom PDF parsing using PDF specification + +### 2. windream COM Interop +The windream DMS integration uses COM Interop which is Windows-only. The application cannot be fully cross-platform unless windream provides a REST API alternative. + +### 3. OAuth2 Token Refresh +Current implementation acquires new tokens on each request. Consider implementing token caching: +- Use `Microsoft.Identity.Web` for automatic token management +- Cache tokens in memory or distributed cache (Redis) + +--- + +## Development Guidelines + +### 1. Code Style +- All code and comments: **English** +- README.md and user documentation: **German** +- Follow C# naming conventions (PascalCase, camelCase) +- Use nullable reference types (`#nullable enable`) + +### 2. Logging +Use Serilog with structured logging: +```csharp +_logger.LogInformation("Processing email {MessageId} from profile {ProfileId}", messageId, profileId); +``` + +### 3. Configuration +- Development: `appsettings.Development.json` + User Secrets +- Production: `appsettings.json` + Environment Variables + Azure Key Vault + +### 4. Error Handling +- Domain: Throw `DomainException` for business rule violations +- Application: Use `FluentValidation` for input validation +- API: Use exception handling middleware to return proper HTTP status codes + +--- + +## Deployment Scenarios + +### IIS Hosting (Default) +```json +{ + "Hosting": { + "Mode": "IIS" + } +} +``` + +### Windows Service Hosting +```json +{ + "Hosting": { + "Mode": "WindowsService" + } +} +``` + +In `Program.cs`: +```csharp +var builder = WebApplication.CreateBuilder(args); + +if (builder.Configuration["Hosting:Mode"] == "WindowsService") +{ + builder.Host.UseWindowsService(); +} +``` + +Install as Windows Service: +```bash +sc create EmailProfiler binPath="C:\Path\To\DigitalData.EmailProfiler.API.exe" +``` + +--- + +## Contact and Support + +For questions about this implementation, consult: +- Legacy system analysis: `legacy/PROJECT_ANALYSIS.md` +- Migration plan: `MIGRATION_PLAN.md` (if created) +- This document: `agents.md` + +--- + +**Last Updated**: 2026-07-07 +**Version**: 1.0 +**Status**: Phase 1 Complete (Domain Layer), Phase 2-8 Pending