# EmailProfiler - Agent Notes and Future Enhancements

## Purpose
This document contains important notes, decisions, and future enhancement plans for the EmailProfiler application. This is intended for AI agents and developers who will continue development.

---

## Important Notes

### 1. Database Schema - DO NOT MODIFY
**CRITICAL**: The database schema must NEVER be modified. All Entity Framework entities must map to existing legacy tables using `[Table]` and `[Column]` attributes.

**Naming Convention**:
- Database: `SNAKE_CASE` with prefixes (TBEMLP_, TBDD_)
- C# Entities: `PascalCase` without prefixes
- Use `[Table("TBDD_FOO")]` and `[Column("COLUMN_NAME")]` attributes

**Example**:
```csharp
[Table("TBDD_EMAIL_ACCOUNT")]
public class EmailAccount
{
    [Column("EMAIL_ACCOUNT_ID")]
    public int Id { get; set; }
    
    [Column("ACCOUNT_NAME")]
    public string AccountName { get; set; }
}
```

### 2. Message ID Hash Algorithm
The `MessageIdGenerator` in `Domain.Services` must use **exactly the same algorithm** as the legacy system to ensure duplicate detection works correctly.

**Algorithm**: SHA256 hash of `{originalMessageId}|{sender}|{date:yyyyMMddHHmmss}|{subject}`

### 3. DateTime Usage - ALWAYS Use Local Time
**CRITICAL**: Always use `DateTime.Now` instead of `DateTime.UtcNow` throughout the entire application.

**Reason**: The legacy system uses local server time, and the database stores all timestamps as local time. Using UTC would break compatibility and cause incorrect time comparisons.

**Examples**:
```csharp
// ✅ CORRECT
profile.CreatedDate = DateTime.Now;
var lastPoll = DateTime.Now.AddMinutes(-profile.PollIntervalMinutes);

// ❌ WRONG - DO NOT USE
profile.CreatedDate = DateTime.UtcNow;  // NEVER USE UTC
var lastPoll = DateTime.UtcNow.AddMinutes(-profile.PollIntervalMinutes);  // NEVER USE UTC
```

**Important**: This applies to:
- All entity audit fields (CreatedDate, ModifiedDate, LastPollDate, etc.)
- All date comparisons in business logic
- All timestamps in logs and error messages
- All date parameters in queries

### 4. Git Operations - NEVER Without Explicit Permission
**CRITICAL**: NEVER execute `git commit` or `git push` commands automatically. ALWAYS wait for explicit user instruction.

**Rules**:
- Only commit when user explicitly says "commit" or "commit this"
- Only push when user explicitly says "push" or "push to remote"
- Stage files with `git add` ONLY when about to commit per user request

### 5. MediatR Command/Query File Organization
**IMPORTANT**: Commands/Queries and their Handlers must be in the SAME file.

**Example**:
```csharp
// ✅ CORRECT - CreateEmailProfileCommand.cs contains BOTH
public record CreateEmailProfileCommand : IRequest<int> { ... }

public class CreateEmailProfileCommandHandler : IRequestHandler<CreateEmailProfileCommand, int> { ... }

// ❌ WRONG - Separate files
// CreateEmailProfileCommand.cs (command only)
// CreateEmailProfileCommandHandler.cs (handler only)
```

**File Naming**:
- Commands: `{Verb}{Entity}Command.cs` (e.g., `CreateEmailProfileCommand.cs`)
- Queries: `{Verb}{Entity}Query.cs` (e.g., `GetEmailProfilesQuery.cs`)

### 6. Repository Pattern - NO UnitOfWork, Generic CRUD with AutoMapper
**CRITICAL**: DO NOT use IUnitOfWork pattern. Use generic repository pattern with AutoMapper-based CRUD operations.

**Key Principles**:
- ✅ Each operation auto-saves changes - NO explicit SaveChangesAsync needed
- ✅ Use `UpdateSingleAsync` / `DeleteSingleAsync` for single-record safety
- ✅ Use `UpdateAsync` / `DeleteAsync` only when intentionally modifying multiple records
- ✅ AutoMapper handles all DTO → Entity mappings

**Pattern**:
```csharp
// IRepository<T> generic interface
public interface IRepository<TEntity> where TEntity : class
{
    // Query operations
    Task<TEntity?> GetByIdAsync(int id, CancellationToken cancellationToken = default);
    Task<IEnumerable<TEntity>> FindAsync(Expression<Func<TEntity, bool>> predicate, ...);
    
    // Create - auto-saves
    Task<TEntity> CreateAsync<TDto>(TDto dto, CancellationToken cancellationToken = default);
    
    // Update - auto-saves
    Task<int> UpdateAsync<TDto>(Expression<...> predicate, TDto dto, ...); // Multiple records
    Task UpdateSingleAsync<TDto>(Expression<...> predicate, TDto dto, ...); // SAFE: Single record only
    
    // Delete - auto-saves
    Task<int> DeleteAsync(Expression<...> predicate, ...); // Multiple records
    Task DeleteSingleAsync(Expression<...> predicate, ...); // SAFE: Single record only
}
```

**Command Handler Examples**:
```csharp
// ✅ CORRECT - CreateAsync auto-saves
public class CreateEmailProfileCommandHandler(IRepository<EmailProfile> repository) 
    : IRequestHandler<CreateEmailProfileCommand, int>
{
    public async Task<int> Handle(CreateEmailProfileCommand request, CancellationToken cancellationToken)
    {
        var profile = await repository.CreateAsync(request, cancellationToken);
        return profile.Id; // NO SaveChangesAsync needed!
    }
}

// ✅ CORRECT - UpdateSingleAsync for safety (throws if 0 or 2+ records match)
public class UpdateEmailProfileCommandHandler(IRepository<EmailProfile> repository) 
    : IRequestHandler<UpdateEmailProfileCommand, int>
{
    public async Task<int> Handle(UpdateEmailProfileCommand request, CancellationToken cancellationToken)
    {
        await repository.UpdateSingleAsync(p => p.Id == request.Id, request, cancellationToken);
        return request.Id; // NO SaveChangesAsync needed!
    }
}

// ✅ CORRECT - DeleteSingleAsync for safety (throws if 0 or 2+ records match)
public class DeleteEmailProfileCommandHandler(IRepository<EmailProfile> repository) 
    : IRequestHandler<DeleteEmailProfileCommand, int>
{
    public async Task<int> Handle(DeleteEmailProfileCommand request, CancellationToken cancellationToken)
    {
        await repository.DeleteSingleAsync(p => p.Id == request.Id, cancellationToken);
        return request.Id; // NO SaveChangesAsync needed!
    }
}

// ❌ WRONG - Manual entity creation (use AutoMapper instead)
var profile = new EmailProfile 
{ 
    ProfileName = request.ProfileName, 
    EmailAccountId = request.EmailAccountId,
    // ... 15 more properties
};

// ❌ WRONG - Using IUnitOfWork (removed)
public CreateEmailProfileCommandHandler(IUnitOfWork unitOfWork) { ... }

// ❌ WRONG - Calling SaveChangesAsync (removed)
await repository.SaveChangesAsync(cancellationToken);
```

**Safety Rules**:
1. **UpdateSingleAsync** - Use for ID-based updates. Throws `InvalidOperationException` if:
   - Zero records match (entity not found)
   - Multiple records match (predicate too broad)

2. **DeleteSingleAsync** - Use for ID-based deletes. Throws `InvalidOperationException` if:
   - Zero records match (entity not found)
   - Multiple records match (predicate too broad)

3. **UpdateAsync / DeleteAsync** - Use ONLY when intentionally modifying multiple records:
   ```csharp
   // ✅ CORRECT - Intentional bulk operation
   await repository.UpdateAsync(
       p => p.EmailAccountId == accountId, 
       new { IsActive = false }, 
       cancellationToken);
   
   // ✅ Returns count of updated/deleted records
   var count = await repository.DeleteAsync(p => p.IsActive == false, cancellationToken);
   ```

**DTO Mapping Responsibility**:
- Each DTO creator must define their own AutoMapper profile
- Example: `CreateEmailProfileCommand` → `EmailProfile` mapping must be defined in `EmailProfileMappingProfile.cs`
- Repository implementation uses `IMapper.Map<TEntity>(dto)` internally

---

## Future Enhancements

### 7. RabbitMQ Command Bus Integration (IMPLEMENTED)

**Purpose**: Asynchronous command processing via RabbitMQ message broker for POST/PUT/DELETE operations.

**Architecture**:
- **GET Queries**: Synchronous (immediate response via MediatR)
- **POST/PUT/DELETE Commands**: Can be asynchronous (published to RabbitMQ, processed by background worker)

**RabbitMQ Server**:
- Management UI: `http://172.24.12.56:15672`
- AMQP Port: `5672` (default)
- Exchange: `emailprofiler.commands` (Direct)
- Queue: `emailprofiler.command.queue`
- Routing Key: `command`

**Implementation Components**:

1. **ICommandPublisher** (`Application/Common/Interfaces/ICommandPublisher.cs`):
   - Interface for publishing commands to message broker
   - Generic method: `PublishAsync<TCommand>(TCommand command, CancellationToken)`

2. **RabbitMqCommandPublisher** (`Infrastructure/Messaging/RabbitMqCommandPublisher.cs`):
   - Implements `ICommandPublisher`
   - Serializes command to JSON with metadata envelope (CommandType, Payload, CorrelationId, PublishedAt)
   - Publishes to RabbitMQ exchange with persistent delivery mode

3. **RabbitMqCommandConsumer** (`Infrastructure/Messaging/RabbitMqCommandConsumer.cs`):
   - BackgroundService that consumes commands from RabbitMQ
   - Deserializes command envelope
   - Resolves command type from assembly
   - Executes command via MediatR in scoped service
   - Acknowledges message on success, requeues on error

4. **RabbitMqConfiguration** (`Infrastructure/Messaging/RabbitMqConfiguration.cs`):
   - Configuration model for RabbitMQ connection
   - Binds to `appsettings.json` section: `RabbitMq`

**Configuration** (`appsettings.json`):
```json
{
  "RabbitMq": {
    "HostName": "172.24.12.56",
    "Port": 5672,
    "UserName": "guest",
    "Password": "guest",
    "VirtualHost": "/",
    "ExchangeName": "emailprofiler.commands",
    "QueueName": "emailprofiler.command.queue",
    "RoutingKey": "command",
    "AutomaticRecoveryEnabled": true,
    "NetworkRecoveryIntervalSeconds": 10
  }
}
```

**Dependency Injection** (`Infrastructure/DependencyInjection.cs`):
```csharp
services.Configure<RabbitMqConfiguration>(configuration.GetSection(RabbitMqConfiguration.SectionName));
services.AddSingleton<ICommandPublisher, RabbitMqCommandPublisher>();
services.AddHostedService<RabbitMqCommandConsumer>();
```

**Usage in API Controllers** (Future):
```csharp
// Option 1: Synchronous (immediate execution via MediatR)
var result = await _mediator.Send(new CreateEmailProfileCommand(...), cancellationToken);
return Ok(result);

// Option 2: Asynchronous (publish to RabbitMQ for background processing)
await _commandPublisher.PublishAsync(new CreateEmailProfileCommand(...), cancellationToken);
return Accepted(); // HTTP 202 - command queued for processing
```

**Benefits**:
- Decouples API from long-running command processing
- Improves API responsiveness (fire-and-forget)
- Enables horizontal scaling (multiple consumers)
- Automatic retries on failure (requeue mechanism)
- Message persistence (survives application restarts)

---

### HIGH PRIORITY: Email Queue for Outgoing Messages (Future)

**Implementation Steps**:

1. **Add NuGet Package**:
   ```bash
   dotnet add package RabbitMQ.Client
   ```

2. **Create RabbitMqEmailQueue.cs**:
   ```csharp
   // src/DigitalData.EmailProfiler.Infrastructure/Queue/RabbitMqEmailQueue.cs
   public class RabbitMqEmailQueue : IEmailQueue
   {
       private readonly IConnection _connection;
       private readonly IModel _channel;
       private const string QueueName = "email-outbox";
       
       public RabbitMqEmailQueue(IOptions<RabbitMqConfiguration> config)
       {
           var factory = new ConnectionFactory
           {
               HostName = config.Value.HostName,
               Port = config.Value.Port,
               UserName = config.Value.UserName,
               Password = config.Value.Password
           };
           
           _connection = factory.CreateConnection();
           _channel = _connection.CreateModel();
           
           _channel.QueueDeclare(
               queue: QueueName,
               durable: true,
               exclusive: false,
               autoDelete: false,
               arguments: null);
       }
       
       public async Task EnqueueAsync(OutgoingEmail email, CancellationToken cancellationToken)
       {
           var json = JsonSerializer.Serialize(email);
           var body = Encoding.UTF8.GetBytes(json);
           
           var properties = _channel.CreateBasicProperties();
           properties.Persistent = true;
           
           _channel.BasicPublish(
               exchange: "",
               routingKey: QueueName,
               basicProperties: properties,
               body: body);
           
           await Task.CompletedTask;
       }
       
       public async Task<OutgoingEmail?> DequeueAsync(CancellationToken cancellationToken)
       {
           var result = _channel.BasicGet(QueueName, autoAck: false);
           
           if (result == null)
               return null;
           
           var json = Encoding.UTF8.GetString(result.Body.ToArray());
           var email = JsonSerializer.Deserialize<OutgoingEmail>(json);
           
           _channel.BasicAck(result.DeliveryTag, false);
           
           return await Task.FromResult(email);
       }
   }
   ```

3. **Configuration** (appsettings.json):
   ```json
   {
     "RabbitMq": {
       "HostName": "localhost",
       "Port": 5672,
       "UserName": "guest",
       "Password": "guest"
     }
   }
   ```

4. **Dependency Injection** (Program.cs):
   ```csharp
   // Replace InMemoryEmailQueue with RabbitMqEmailQueue
   // builder.Services.AddSingleton<IEmailQueue, InMemoryEmailQueue>();
   builder.Services.AddSingleton<IEmailQueue, RabbitMqEmailQueue>();
   ```

**Benefits**:
- Message persistence (survives application restarts)
- Scalability (multiple worker instances can consume from queue)
- Reliability (automatic retries, dead letter queues)
- Monitoring (RabbitMQ management UI)

**Migration Path**:
1. Deploy RabbitMQ server (Docker recommended)
2. Test RabbitMqEmailQueue in staging environment
3. Switch DI registration from InMemoryEmailQueue to RabbitMqEmailQueue
4. Monitor queue depth and worker performance

---

## Pending Implementation Tasks

### Phase 2: Application Layer (IN PROGRESS)
**Status**: Partially complete - DTOs created, Commands/Queries needed

**TODO**:
- [ ] Create MediatR Commands (CreateEmailProfileCommand, ProcessEmailCommand, etc.)
- [ ] Create MediatR Queries (GetEmailProfilesQuery, GetEmailHistoryQuery, etc.)
- [ ] Create Command/Query Handlers
- [ ] Create FluentValidation Validators
- [ ] Create AutoMapper Profiles
- [ ] Create Application Interfaces (IEmailService, IPdfProcessingService, IDmsService, etc.)

**Example Command**:
```csharp
// src/DigitalData.EmailProfiler.Application/EmailProfiles/Commands/CreateEmailProfileCommand.cs
public record CreateEmailProfileCommand(string ProfileName, int EmailAccountId) : IRequest<int>;

public class CreateEmailProfileCommandHandler : IRequestHandler<CreateEmailProfileCommand, int>
{
    private readonly IEmailProfileRepository _repository;
    
    public async Task<int> Handle(CreateEmailProfileCommand request, CancellationToken cancellationToken)
    {
        var profile = new EmailProfile
        {
            ProfileName = request.ProfileName,
            EmailAccountId = request.EmailAccountId,
            IsActive = true
        };
        
        await _repository.AddAsync(profile, cancellationToken);
        return profile.Id;
    }
}
```

### Phase 3: Infrastructure Layer
**Status**: Not started

**TODO**:
- [ ] Create EmailProfilerDbContext with DbSet<T> for all entities
- [ ] Create Entity Configurations (Fluent API) for all entities
- [ ] Create Repositories implementing Application interfaces
- [ ] Create MailKitEmailService (IMAP/SMTP with OAuth2)
- [ ] Create PdfSharpProcessingService
- [ ] Create WindreamDmsService (COM Interop)
- [ ] Create EncryptionService (Data Protection API)
- [ ] Create initial EF Core migration

**DbContext Example**:
```csharp
public class EmailProfilerDbContext : DbContext
{
    public DbSet<EmailAccount> EmailAccounts { get; set; }
    public DbSet<EmailProfile> EmailProfiles { get; set; }
    // ... other DbSets
    
    protected override void OnModelCreating(ModelBuilder modelBuilder)
    {
        modelBuilder.ApplyConfigurationsFromAssembly(Assembly.GetExecutingAssembly());
        
        // Important: Check for triggers
        modelBuilder.Entity<EmailHistory>().ToTable(tb => tb.HasTrigger("TR_TBEMLP_HISTORY_AUDIT"));
    }
}
```

### Phase 4: API Layer
**Status**: Minimal structure exists

**TODO**:
- [ ] Create Controllers (ProfilesController, EmailAccountsController, HistoryController)
- [ ] Create Background Workers (EmailPollingWorker, EmailSenderWorker)
- [ ] Configure Serilog
- [ ] Configure Scalar (OpenAPI documentation)
- [ ] Add Exception Handling Middleware
- [ ] Configure DI for all layers
- [ ] Support both IIS and Windows Service hosting

**Worker Configuration** (appsettings.json):
```json
{
  "Workers": {
    "EmailPolling": {
      "Enabled": true,
      "IntervalSeconds": 60
    },
    "EmailSender": {
      "Enabled": true,
      "IntervalSeconds": 5
    }
  },
  "Hosting": {
    "Mode": "IIS"  // or "WindowsService"
  }
}
```

### Phase 5: Testing
**Status**: Not started

**TODO**:
- [ ] Unit tests for Domain entities
- [ ] Unit tests for Application handlers (using FakeItEasy)
- [ ] Integration tests for Repositories (using Testcontainers)
- [ ] API tests (using WebApplicationFactory)
- [ ] Generate fake test data (using Bogus)

**Test Example**:
```csharp
public class MessageIdGeneratorTests
{
    [Fact]
    public void Generate_ShouldProduceSameHashAsLegacy()
    {
        // Arrange
        var generator = new MessageIdGenerator();
        var original = "msg-123";
        var sender = "test@example.com";
        var date = new DateTime(2026, 1, 1, 12, 0, 0);
        var subject = "Test Subject";
        
        // Act
        var messageId = generator.Generate(original, sender, date, subject);
        
        // Assert
        messageId.Hash.Should().NotBeNullOrEmpty();
        // TODO: Verify against known legacy hash
    }
}
```

---

## Architecture Decisions

### Clean Architecture Layers
1. **Domain**: Core business logic, no dependencies
2. **Application**: Use cases, depends on Domain
3. **Infrastructure**: External concerns, depends on Domain + Application
4. **API**: Entry point, depends on all

### CQRS Pattern with MediatR
- **Commands**: Modify state (Create, Update, Delete)
- **Queries**: Read data (Get, List)
- Separate models for read and write operations

### Repository Pattern
- Interface in Application layer
- Implementation in Infrastructure layer
- One repository per Aggregate Root

---

## Known Issues and Limitations

### 1. PdfSharp Embedded File Extraction
PdfSharp has limited support for embedded file extraction from PDFs. If advanced PDF processing is needed, consider:
- **iText7** (AGPL or commercial license)
- **Aspose.PDF** (commercial license)
- Custom PDF parsing using PDF specification

### 2. windream COM Interop
The windream DMS integration uses COM Interop which is Windows-only. The application cannot be fully cross-platform unless windream provides a REST API alternative.

### 3. OAuth2 Token Refresh
Current implementation acquires new tokens on each request. Consider implementing token caching:
- Use `Microsoft.Identity.Web` for automatic token management
- Cache tokens in memory or distributed cache (Redis)

---

## Development Guidelines

### 1. Code Style
- All code and comments: **English**
- README.md and user documentation: **German**
- Follow C# naming conventions (PascalCase, camelCase)
- Use nullable reference types (`#nullable enable`)

### 2. Logging
Use Serilog with structured logging:
```csharp
_logger.LogInformation("Processing email {MessageId} from profile {ProfileId}", messageId, profileId);
```

### 3. Configuration
- Development: `appsettings.Development.json` + User Secrets
- Production: `appsettings.json` + Environment Variables + Azure Key Vault

### 4. Error Handling
- Domain: Throw `DomainException` for business rule violations
- Application: Use `FluentValidation` for input validation
- API: Use exception handling middleware to return proper HTTP status codes

---

## Deployment Scenarios

### IIS Hosting (Default)
```json
{
  "Hosting": {
    "Mode": "IIS"
  }
}
```

### Windows Service Hosting
```json
{
  "Hosting": {
    "Mode": "WindowsService"
  }
}
```

In `Program.cs`:
```csharp
var builder = WebApplication.CreateBuilder(args);

if (builder.Configuration["Hosting:Mode"] == "WindowsService")
{
    builder.Host.UseWindowsService();
}
```

Install as Windows Service:
```bash
sc create EmailProfiler binPath="C:\Path\To\DigitalData.EmailProfiler.API.exe"
```

---

## Contact and Support

For questions about this implementation, consult:
- Legacy system analysis: `legacy/PROJECT_ANALYSIS.md`
- Migration plan: `MIGRATION_PLAN.md` (if created)
- This document: `agents.md`

---

**Last Updated**: 2026-07-07
**Version**: 1.0
**Status**: Phase 1 Complete (Domain Layer), Phase 2-8 Pending