Documentation

Journal store

In software development, journal stores are mechanisms or databases used to temporarily record persistent states or transactions to ensure data resilience and prevent loss. They are particularly valuable in systems requiring high reliability, allowing operations to proceed safely even if a failure occurs. Journals log changes, retry failed processes, and ensure consistency by holding data until it is processed successfully.

Examples include distributed storage or key-value stores like Etcd, Redis, and databases such as PostgreSQL or SQL Server. These are often used in combination with application-level processors for custom handling and retries, enhancing system durability and fault tolerance.

To use journal store, you need to add an attribute on the method. You also need to implement an interface as that is used to wire everything up.

public interface IInvoice
{
   bool Processing(Guid key, Invoice value);
}

public class Invoicer: IInvoice
{
    [AddJournal("key", typeof(Guid), "Invoicer", typeof(CustomJournalEntryProcessor))]
    public virtual bool Processing(Guid key, Invoice value)
    {
        Console.WriteLine($"{obj.InvoiceId}, {obj.Name}, {obj.Value}");
        return true;
    }
}

Let’s break it down.

[AddJournal("key", typeof(Guid), "Invoicer", typeof(CustomJournalEntryProcessor))] contains 4 properties:

  • KeyName with value "key"
  • KeyType with value typeof(Guid)
  • CollectionName with value "Invoicer"
  • JournalEntryProcessor with value typeof(CustomJournalEntryProcessor)

KeyName and KeyType defines the name and type of identifier key, that is used to store the entry to the journal.

The CollectionName is a key prefix or table name in a journal. In example above we use a GUID type with name of the property key. That is used in the method signature: Processing(Guid key, Invoice value).

If the key value is 100, then the key stored to the Etcd or Redis is Invoicer/100. This way we can have grouping in the Etcd as it does not support tables or collections. When we use SQL databases for journal stores, CollectionName is a table name.

The Second parameter in the method is the object, which is stored as the key value in Etcd. In this case it’s Invoice value which is serialized to JSON and stored.

If all is wired up, then this method state is stored in the journal and if the method Processing succeeds then it’s removed from the journal. Otherwise, it stays in the journal, when the CustomJournalEntryProcessor retries the process.

  • note that process will fail at runtime if you do not specify the correct id and the value.
  • note that method must be virtual

1. Implement Journal

1.1. Register the Etcd Journal store

To register and Etcd a journal, you need to:

services.AddTransient<IInvoice, Invoicer>();
        
services.AddEtcdJournal(new EtcdOptions
{
    ConnectionString = "http://localhost:2379",
    Username = null,
    Password = null
});
1.2. Create custom JournalEntryProcessor

Processor will be defined as a last parameter in the AddJournal attribute. In the ProcessAsync you can define your retry process or mechanism.

public class CustomJournalEntryProcessor : IJournalEntryProcessor
{
    private readonly ILogger<CustomJournalEntryProcessor> _logger;

    public CustomJournalEntryProcessor(ILogger<CustomJournalEntryProcessor> logger)
    {
        _logger = logger;
    }

    public async Task ProcessAsync(string key, object data, CancellationToken cancellationToken)
    {
        // Example: Add your custom processing logic here
        _logger.LogInformation($"Processing journal entry with key '{key}' and data: {JsonSerializer.Serialize(data)}");

        // Simulate processing delay
        await Task.Delay(100, cancellationToken);

        _logger.LogInformation($"Successfully processed journal entry with key '{key}'.");
    }
}
1.3. Add journal attribute to your method
public interface IInvoice
{
   bool Processing(Guid key, Invoice value);
}

public class Invoicer: IInvoice
{
    [AddJournal("key", typeof(Guid), "Invoicer", typeof(CustomJournalEntryProcessor))]
    public virtual bool Processing(Guid key, Invoice value)
    {
        Console.WriteLine($"{obj.InvoiceId}, {obj.Name}, {obj.Value}");
        return true;
    }
}

2. Different types journal stores

2.1 Etcd Journal implementation

You need to use AddEtcdJournal and change the journal registration to con

services.AddEtcdJournal(new EtcdOptions
{
    ConnectionString = "http://localhost:2379",
    Username = null,
    Password = null
});
2.2. Redis Journal implementation

You only need to change the journal registration to include the connection string to the Redis database:

services.AddRedisJournal("localhost:6379");
2.3. Postgresql Journal implementation

You only need to change the journal registration to include the connection string to the Postgresql database:

 services.AddPostgreSQLJournal("connectionstring");
2.4. SQL Server Journal implementation

You only need to change the journal registration to include the connection string to the Microsoft SQL server database:

services.AddSqlServerJournal("connectionstring");

2025 Nomirun©