DigitalData.MessagingService/AGENTS.md

# EmailProfiler - Agent Notes and Future Enhancements

## Purpose
This document contains important notes, decisions, and future enhancement plans for the EmailProfiler application. This is intended for AI agents and developers who will continue development.

---

## Important Notes

### 1. Database Schema - DO NOT MODIFY
**CRITICAL**: The database schema must NEVER be modified. All Entity Framework entities must map to existing legacy tables using `[Table]` and `[Column]` attributes.

**Naming Convention**:
- Database: `SNAKE_CASE` with prefixes (TBEMLP_, TBDD_)
- C# Entities: `PascalCase` without prefixes
- Use `[Table("TBDD_FOO")]` and `[Column("COLUMN_NAME")]` attributes

**Example**:
```csharp
[Table("TBDD_EMAIL_ACCOUNT")]
public class EmailAccount
{
    [Column("EMAIL_ACCOUNT_ID")]
    public int Id { get; set; }
    
    [Column("ACCOUNT_NAME")]
    public string AccountName { get; set; }
}
```

### 2. Message ID Hash Algorithm
The `MessageIdGenerator` in `Domain.Services` must use **exactly the same algorithm** as the legacy system to ensure duplicate detection works correctly.

**Algorithm**: SHA256 hash of `{originalMessageId}|{sender}|{date:yyyyMMddHHmmss}|{subject}`

### 3. DateTime Usage - ALWAYS Use Local Time
**CRITICAL**: Always use `DateTime.Now` instead of `DateTime.UtcNow` throughout the entire application.

**Reason**: The legacy system uses local server time, and the database stores all timestamps as local time. Using UTC would break compatibility and cause incorrect time comparisons.

**Examples**:
```csharp
// ✅ CORRECT
profile.CreatedDate = DateTime.Now;
var lastPoll = DateTime.Now.AddMinutes(-profile.PollIntervalMinutes);

// ❌ WRONG - DO NOT USE
profile.CreatedDate = DateTime.UtcNow;  // NEVER USE UTC
var lastPoll = DateTime.UtcNow.AddMinutes(-profile.PollIntervalMinutes);  // NEVER USE UTC
```

**Important**: This applies to:
- All entity audit fields (CreatedDate, ModifiedDate, LastPollDate, etc.)
- All date comparisons in business logic
- All timestamps in logs and error messages
- All date parameters in queries

### 4. Git Operations - NEVER Without Explicit Permission
**CRITICAL**: NEVER execute `git commit` or `git push` commands automatically. ALWAYS wait for explicit user instruction.

**Rules**:
- Only commit when user explicitly says "commit" or "commit this"
- Only push when user explicitly says "push" or "push to remote"
- Stage files with `git add` ONLY when about to commit per user request

### 5. MediatR Command/Query File Organization
**IMPORTANT**: Commands/Queries and their Handlers must be in the SAME file.

**Example**:
```csharp
// ✅ CORRECT - CreateEmailProfileCommand.cs contains BOTH
public record CreateEmailProfileCommand : IRequest<int> { ... }

public class CreateEmailProfileCommandHandler : IRequestHandler<CreateEmailProfileCommand, int> { ... }

// ❌ WRONG - Separate files
// CreateEmailProfileCommand.cs (command only)
// CreateEmailProfileCommandHandler.cs (handler only)
```

**File Naming**:
- Commands: `{Verb}{Entity}Command.cs` (e.g., `CreateEmailProfileCommand.cs`)
- Queries: `{Verb}{Entity}Query.cs` (e.g., `GetEmailProfilesQuery.cs`)

---

## Future Enhancements

### HIGH PRIORITY: RabbitMQ Queue Implementation

**Current State**: 
- Email queue is implemented using in-memory `Channel<T>` in `InMemoryEmailQueue.cs`
- Location: `src/DigitalData.EmailProfiler.Infrastructure/Queue/InMemoryEmailQueue.cs`

**Future Enhancement**:
Replace the in-memory queue with **RabbitMQ** for production resilience and scalability.

**Implementation Steps**:

1. **Add NuGet Package**:
   ```bash
   dotnet add package RabbitMQ.Client
   ```

2. **Create RabbitMqEmailQueue.cs**:
   ```csharp
   // src/DigitalData.EmailProfiler.Infrastructure/Queue/RabbitMqEmailQueue.cs
   public class RabbitMqEmailQueue : IEmailQueue
   {
       private readonly IConnection _connection;
       private readonly IModel _channel;
       private const string QueueName = "email-outbox";
       
       public RabbitMqEmailQueue(IOptions<RabbitMqConfiguration> config)
       {
           var factory = new ConnectionFactory
           {
               HostName = config.Value.HostName,
               Port = config.Value.Port,
               UserName = config.Value.UserName,
               Password = config.Value.Password
           };
           
           _connection = factory.CreateConnection();
           _channel = _connection.CreateModel();
           
           _channel.QueueDeclare(
               queue: QueueName,
               durable: true,
               exclusive: false,
               autoDelete: false,
               arguments: null);
       }
       
       public async Task EnqueueAsync(OutgoingEmail email, CancellationToken cancellationToken)
       {
           var json = JsonSerializer.Serialize(email);
           var body = Encoding.UTF8.GetBytes(json);
           
           var properties = _channel.CreateBasicProperties();
           properties.Persistent = true;
           
           _channel.BasicPublish(
               exchange: "",
               routingKey: QueueName,
               basicProperties: properties,
               body: body);
           
           await Task.CompletedTask;
       }
       
       public async Task<OutgoingEmail?> DequeueAsync(CancellationToken cancellationToken)
       {
           var result = _channel.BasicGet(QueueName, autoAck: false);
           
           if (result == null)
               return null;
           
           var json = Encoding.UTF8.GetString(result.Body.ToArray());
           var email = JsonSerializer.Deserialize<OutgoingEmail>(json);
           
           _channel.BasicAck(result.DeliveryTag, false);
           
           return await Task.FromResult(email);
       }
   }
   ```

3. **Configuration** (appsettings.json):
   ```json
   {
     "RabbitMq": {
       "HostName": "localhost",
       "Port": 5672,
       "UserName": "guest",
       "Password": "guest"
     }
   }
   ```

4. **Dependency Injection** (Program.cs):
   ```csharp
   // Replace InMemoryEmailQueue with RabbitMqEmailQueue
   // builder.Services.AddSingleton<IEmailQueue, InMemoryEmailQueue>();
   builder.Services.AddSingleton<IEmailQueue, RabbitMqEmailQueue>();
   ```

**Benefits**:
- Message persistence (survives application restarts)
- Scalability (multiple worker instances can consume from queue)
- Reliability (automatic retries, dead letter queues)
- Monitoring (RabbitMQ management UI)

**Migration Path**:
1. Deploy RabbitMQ server (Docker recommended)
2. Test RabbitMqEmailQueue in staging environment
3. Switch DI registration from InMemoryEmailQueue to RabbitMqEmailQueue
4. Monitor queue depth and worker performance

---

## Pending Implementation Tasks

### Phase 2: Application Layer (IN PROGRESS)
**Status**: Partially complete - DTOs created, Commands/Queries needed

**TODO**:
- [ ] Create MediatR Commands (CreateEmailProfileCommand, ProcessEmailCommand, etc.)
- [ ] Create MediatR Queries (GetEmailProfilesQuery, GetEmailHistoryQuery, etc.)
- [ ] Create Command/Query Handlers
- [ ] Create FluentValidation Validators
- [ ] Create AutoMapper Profiles
- [ ] Create Application Interfaces (IEmailService, IPdfProcessingService, IDmsService, etc.)

**Example Command**:
```csharp
// src/DigitalData.EmailProfiler.Application/EmailProfiles/Commands/CreateEmailProfileCommand.cs
public record CreateEmailProfileCommand(string ProfileName, int EmailAccountId) : IRequest<int>;

public class CreateEmailProfileCommandHandler : IRequestHandler<CreateEmailProfileCommand, int>
{
    private readonly IEmailProfileRepository _repository;
    
    public async Task<int> Handle(CreateEmailProfileCommand request, CancellationToken cancellationToken)
    {
        var profile = new EmailProfile
        {
            ProfileName = request.ProfileName,
            EmailAccountId = request.EmailAccountId,
            IsActive = true
        };
        
        await _repository.AddAsync(profile, cancellationToken);
        return profile.Id;
    }
}
```

### Phase 3: Infrastructure Layer
**Status**: Not started

**TODO**:
- [ ] Create EmailProfilerDbContext with DbSet<T> for all entities
- [ ] Create Entity Configurations (Fluent API) for all entities
- [ ] Create Repositories implementing Application interfaces
- [ ] Create MailKitEmailService (IMAP/SMTP with OAuth2)
- [ ] Create PdfSharpProcessingService
- [ ] Create WindreamDmsService (COM Interop)
- [ ] Create EncryptionService (Data Protection API)
- [ ] Create initial EF Core migration

**DbContext Example**:
```csharp
public class EmailProfilerDbContext : DbContext
{
    public DbSet<EmailAccount> EmailAccounts { get; set; }
    public DbSet<EmailProfile> EmailProfiles { get; set; }
    // ... other DbSets
    
    protected override void OnModelCreating(ModelBuilder modelBuilder)
    {
        modelBuilder.ApplyConfigurationsFromAssembly(Assembly.GetExecutingAssembly());
        
        // Important: Check for triggers
        modelBuilder.Entity<EmailHistory>().ToTable(tb => tb.HasTrigger("TR_TBEMLP_HISTORY_AUDIT"));
    }
}
```

### Phase 4: API Layer
**Status**: Minimal structure exists

**TODO**:
- [ ] Create Controllers (ProfilesController, EmailAccountsController, HistoryController)
- [ ] Create Background Workers (EmailPollingWorker, EmailSenderWorker)
- [ ] Configure Serilog
- [ ] Configure Scalar (OpenAPI documentation)
- [ ] Add Exception Handling Middleware
- [ ] Configure DI for all layers
- [ ] Support both IIS and Windows Service hosting

**Worker Configuration** (appsettings.json):
```json
{
  "Workers": {
    "EmailPolling": {
      "Enabled": true,
      "IntervalSeconds": 60
    },
    "EmailSender": {
      "Enabled": true,
      "IntervalSeconds": 5
    }
  },
  "Hosting": {
    "Mode": "IIS"  // or "WindowsService"
  }
}
```

### Phase 5: Testing
**Status**: Not started

**TODO**:
- [ ] Unit tests for Domain entities
- [ ] Unit tests for Application handlers (using FakeItEasy)
- [ ] Integration tests for Repositories (using Testcontainers)
- [ ] API tests (using WebApplicationFactory)
- [ ] Generate fake test data (using Bogus)

**Test Example**:
```csharp
public class MessageIdGeneratorTests
{
    [Fact]
    public void Generate_ShouldProduceSameHashAsLegacy()
    {
        // Arrange
        var generator = new MessageIdGenerator();
        var original = "msg-123";
        var sender = "test@example.com";
        var date = new DateTime(2026, 1, 1, 12, 0, 0);
        var subject = "Test Subject";
        
        // Act
        var messageId = generator.Generate(original, sender, date, subject);
        
        // Assert
        messageId.Hash.Should().NotBeNullOrEmpty();
        // TODO: Verify against known legacy hash
    }
}
```

---

## Architecture Decisions

### Clean Architecture Layers
1. **Domain**: Core business logic, no dependencies
2. **Application**: Use cases, depends on Domain
3. **Infrastructure**: External concerns, depends on Domain + Application
4. **API**: Entry point, depends on all

### CQRS Pattern with MediatR
- **Commands**: Modify state (Create, Update, Delete)
- **Queries**: Read data (Get, List)
- Separate models for read and write operations

### Repository Pattern
- Interface in Application layer
- Implementation in Infrastructure layer
- One repository per Aggregate Root

---

## Known Issues and Limitations

### 1. PdfSharp Embedded File Extraction
PdfSharp has limited support for embedded file extraction from PDFs. If advanced PDF processing is needed, consider:
- **iText7** (AGPL or commercial license)
- **Aspose.PDF** (commercial license)
- Custom PDF parsing using PDF specification

### 2. windream COM Interop
The windream DMS integration uses COM Interop which is Windows-only. The application cannot be fully cross-platform unless windream provides a REST API alternative.

### 3. OAuth2 Token Refresh
Current implementation acquires new tokens on each request. Consider implementing token caching:
- Use `Microsoft.Identity.Web` for automatic token management
- Cache tokens in memory or distributed cache (Redis)

---

## Development Guidelines

### 1. Code Style
- All code and comments: **English**
- README.md and user documentation: **German**
- Follow C# naming conventions (PascalCase, camelCase)
- Use nullable reference types (`#nullable enable`)

### 2. Logging
Use Serilog with structured logging:
```csharp
_logger.LogInformation("Processing email {MessageId} from profile {ProfileId}", messageId, profileId);
```

### 3. Configuration
- Development: `appsettings.Development.json` + User Secrets
- Production: `appsettings.json` + Environment Variables + Azure Key Vault

### 4. Error Handling
- Domain: Throw `DomainException` for business rule violations
- Application: Use `FluentValidation` for input validation
- API: Use exception handling middleware to return proper HTTP status codes

---

## Deployment Scenarios

### IIS Hosting (Default)
```json
{
  "Hosting": {
    "Mode": "IIS"
  }
}
```

### Windows Service Hosting
```json
{
  "Hosting": {
    "Mode": "WindowsService"
  }
}
```

In `Program.cs`:
```csharp
var builder = WebApplication.CreateBuilder(args);

if (builder.Configuration["Hosting:Mode"] == "WindowsService")
{
    builder.Host.UseWindowsService();
}
```

Install as Windows Service:
```bash
sc create EmailProfiler binPath="C:\Path\To\DigitalData.EmailProfiler.API.exe"
```

---

## Contact and Support

For questions about this implementation, consult:
- Legacy system analysis: `legacy/PROJECT_ANALYSIS.md`
- Migration plan: `MIGRATION_PLAN.md` (if created)
- This document: `agents.md`

---

**Last Updated**: 2026-07-07
**Version**: 1.0
**Status**: Phase 1 Complete (Domain Layer), Phase 2-8 Pending