Diagnostic Monad For Operation Outcomes: Is It Overkill?

by Alex Johnson 57 views

In our processing workflow, we aim to gather diverse information in the form of OperationOutcome issues. These issues are written into the job at each job update, providing a comprehensive log of the process. One potential solution for this involves a monadic structure, which we've termed acc for accumulator. This approach offers a way to manage and collect these outcomes efficiently. Let's delve deeper into what this monadic structure entails and how it can be implemented.

Understanding the Acc Monad

The Acc monad, as we've defined it, serves as a container that holds both a value of type T and a list of OperationOutcome.OperationOutcomeIssueComponent objects. This dual nature allows us to carry along informational, warning, or error messages alongside the primary value being processed. Here’s a breakdown of its key components:

public record Acc<T>(
        T value,
        List<OperationOutcome.OperationOutcomeIssueComponent> issues
) {

    public static <T> Acc<T> ok(T value) {
        return new Acc<>(value, List.of());
    }

    public Acc<T> addInfo(String msg) {
        return addIssue(issue(OperationOutcome.IssueSeverity.INFORMATION, msg));
    }

    public Acc<T> addWarning(String msg) {
        return addIssue(issue(OperationOutcome.IssueSeverity.WARNING, msg));
    }

    public Acc<T> addError(String msg) {
        return addIssue(issue(OperationOutcome.IssueSeverity.ERROR, msg));
    }

    private static OperationOutcome.OperationOutcomeIssueComponent issue(
            OperationOutcome.IssueSeverity severity,
            String msg
    ) {
        return new OperationOutcome.OperationOutcomeIssueComponent()
                .setSeverity(severity)
                .setDiagnostics(msg);
    }

    public <R> Acc<R> map(Function<T, R> f) {
        return new Acc<>(f.apply(value), issues);
    }

    public <R> Acc<R> flatMap(Function<T, Acc<R>> f) {
        Acc<R> next = f.apply(value);
        List<OperationOutcome.OperationOutcomeIssueComponent> merged =
                new ArrayList<>(issues);
        merged.addAll(next.issues());
        return new Acc<>(next.value(), merged);
    }
}

Core Components of the Acc Monad

  1. Value (T): This is the primary data being processed. It can be any type, allowing the Acc monad to be versatile across various operations.
  2. Issues (List<OperationOutcome.OperationOutcomeIssueComponent>): This list stores the operational outcomes, such as informational messages, warnings, or errors, encountered during processing. Each issue is an instance of OperationOutcome.OperationOutcomeIssueComponent, which includes details like severity and diagnostic messages.

Key Methods for Operation Outcome Accumulation

  • ok(T value): A static factory method that creates an Acc instance with a given value and an empty list of issues. This is typically used to start a chain of operations.
  • addInfo(String msg): Adds an informational message to the issues list. This is useful for logging progress or notable events during processing.
  • addWarning(String msg): Adds a warning message to the issues list. Warnings indicate potential problems that don't necessarily halt processing but should be noted.
  • addError(String msg): Adds an error message to the issues list. Errors signify issues that may have caused a failure in processing.
  • issue(OperationOutcome.IssueSeverity severity, String msg): A private helper method that constructs an OperationOutcome.OperationOutcomeIssueComponent with the specified severity and message.
  • map(Function<T, R> f): Applies a function f to the value inside the Acc monad, transforming it from type T to type R. The issues list remains unchanged. This method is crucial for transforming the value without affecting the accumulated outcomes.
  • flatMap(Function<T, Acc<R>> f): This is the core of the monadic behavior. It applies a function f that transforms the value T into another Acc instance Acc<R>. The key difference from map is that flatMap allows chaining operations that themselves produce Acc instances. It merges the issues from the original Acc with those from the new Acc, ensuring that all outcomes are accumulated. This is essential for sequencing operations that may each produce their own set of issues.

Practical Application: Processing a Batch with Operation Outcomes

To illustrate the utility of the Acc monad, let's consider a practical example: processing a batch of patient data. This involves several steps, including loading patient compartments, resolving references, applying cascading deletes, copying and redacting data, and writing the output bundle. Each of these steps can potentially encounter issues that we need to track. The Acc monad allows us to chain these operations while accumulating any informational, warning, or error messages that arise.

Example Scenario: processBatch Method

Here’s how the Acc monad can be used within a processBatch method:

private Mono<BatchResult> processBatch(
        PatientBatchWithConsent inputBatch,
        UUID jobID,
        GroupsToProcess groupsToProcess,
        BatchState batchState
) {
    UUID id = inputBatch.id();
    logMemory(id);

    return Mono.just(Acc.ok(inputBatch))
            // Add info that we started
            .map(acc -> acc.addInfo("Starting batch " + id))

            // 1) Load patient compartment
            .flatMap(acc ->
                    directResourceLoader.directLoadPatientCompartment(
                                    groupsToProcess.directPatientCompartmentGroups(),
                                    acc.value()
                            )
                            .map(Acc::ok)
                            .map(a -> a.addInfo("Loaded patient compartment for batch " + id))
                            .onErrorResume(e ->
                                    Mono.just(acc.addError("Failed loading patient compartment: " + e.getMessage()))
                            )
            )

            // 2) Reference resolution
            .flatMap(acc ->
                    referenceResolver.processSinglePatientBatch(
                                    acc.value(),
                                    groupsToProcess.allGroups()
                            )
                            .map(Acc::ok)
                            .map(a -> a.addInfo("Resolved references for batch " + id))
                            .onErrorResume(e ->
                                    Mono.just(acc.addError("Reference resolution failed: " + e.getMessage()))
                            )
            )

            // 3) Cascading delete
            .map(acc ->
                    acc.map(p -> cascadingDelete.handlePatientBatch(p, groupsToProcess.allGroups()))
                       .addInfo("Applied cascading delete for batch " + id)
            )

            // 4) Copy + Redact
            .map(acc ->
                    acc.map(p -> batchCopierRedacter.transformBatch(p, groupsToProcess.allGroups()))
                       .addInfo("Applied copy/redact pipeline for batch " + id)
            )

            // 5) Write output bundle
            .flatMap(acc ->
                    writeBatch(jobID, acc.value())
                            .thenReturn(acc.addInfo("Wrote NDJSON bundle for batch " + id))
                            .onErrorResume(e ->
                                    Mono.just(acc.addError("Writing NDJSON failed: " + e.getMessage()))
                            )
            )

            // 6) Convert Acc<T> → BatchResult
            .map(acc -> {
                boolean hasErrors = acc.issues().stream()
                        .anyMatch(i -> i.getSeverity() == OperationOutcome.IssueSeverity.ERROR);

                BatchState updatedState = hasErrors
                        ? batchState.updateStatus(WorkUnitStatus.ERROR)
                        : batchState.updateStatus(WorkUnitStatus.FINISHED);

                Set<ResourceGroupRelation> valid = hasErrors
                        ? Set.of()
                        : acc.value().coreBundle().getValidResourceGroups();

                return new BatchResult(
                        updatedState,
                        valid,
                        acc.issues()
                );
            });
}

Step-by-Step Breakdown

  1. Initialization: The process starts by creating an Acc instance with the inputBatch as the value and an empty list of issues.
  2. Adding Information: An informational message is added to indicate the start of the batch processing.
  3. Loading Patient Compartment: The directLoadPatientCompartment method is called to load the patient compartment. This operation is wrapped in a flatMap to handle the asynchronous nature of the operation and to merge any issues that arise. If the loading is successful, an informational message is added. If an error occurs, an error message is added to the Acc instance using onErrorResume.
  4. Reference Resolution: The referenceResolver.processSinglePatientBatch method resolves references within the patient batch. Similar to the previous step, flatMap is used to handle the asynchronous operation and merge issues. Success and failure scenarios are handled with appropriate messages.
  5. Cascading Delete: The cascadingDelete.handlePatientBatch method applies cascading deletes. This step uses map since the operation is synchronous and doesn't return a new Acc instance. An informational message is added to indicate the completion of this step.
  6. Copy and Redact: The batchCopierRedacter.transformBatch method transforms the batch by copying and redacting data. Like the cascading delete step, map is used, and an informational message is added.
  7. Writing Output Bundle: The writeBatch method writes the output bundle. This asynchronous operation is handled with flatMap, and success and failure scenarios are managed with appropriate messages.
  8. Final Result Conversion: The final map operation converts the Acc instance into a BatchResult. It checks for any error issues in the accumulated list and updates the BatchState accordingly. The accumulated issues are included in the BatchResult.

Benefits of Using the Acc Monad

  • Centralized Error Handling: The Acc monad provides a centralized way to collect and manage issues that occur during processing. This makes it easier to track and handle errors, warnings, and informational messages.
  • Chained Operations: The monadic nature of Acc allows for chaining operations in a clean and readable manner. Each step in the process can add its own issues without disrupting the flow.
  • Asynchronous Handling: The use of flatMap makes it easy to work with asynchronous operations, such as those returning Mono or Flux.
  • Clear Separation of Concerns: The Acc monad separates the core processing logic from the issue tracking, making the code more modular and maintainable.

Is the Acc Monad Overkill?

The question remains: is this approach overkill? The answer depends on the complexity and requirements of your processing workflow. For simpler processes with minimal error handling needs, the Acc monad might introduce unnecessary complexity. However, for more complex workflows, especially those involving asynchronous operations and multiple potential failure points, the Acc monad can be a valuable tool.

Considerations for Determining Necessity

  1. Complexity of Workflow: If your workflow involves multiple steps with various potential issues, the Acc monad can help manage this complexity.
  2. Asynchronous Operations: When dealing with asynchronous operations, the Acc monad simplifies error handling and issue accumulation.
  3. Error Handling Requirements: If you need detailed tracking of errors, warnings, and informational messages, the Acc monad provides a structured way to do so.
  4. Maintainability: The Acc monad can improve code maintainability by separating issue tracking from the core logic.

Alternatives to the Acc Monad

If the Acc monad seems too heavy for your needs, there are alternative approaches to consider:

  • Try-Catch Blocks: Traditional try-catch blocks can be used to handle exceptions, but they don't provide a way to accumulate multiple issues.
  • Logging: Logging can be used to record issues, but it doesn't provide a structured way to manage them within the processing flow.
  • Custom Issue Collection: A custom class or data structure could be created to collect issues, but this would require implementing the logic for chaining operations and handling asynchronous tasks.

Conclusion

The Acc monad offers a robust solution for collecting and managing operation outcomes in complex processing workflows. Its ability to chain operations, handle asynchronous tasks, and centralize error handling makes it a powerful tool. However, it's essential to consider the complexity of your workflow and the specific requirements of your application to determine if the Acc monad is the right fit. For simpler scenarios, alternative approaches may suffice, but for intricate processes requiring detailed issue tracking, the Acc monad can significantly enhance code clarity and maintainability.

For more information on monadic structures and functional programming in Java, you can explore resources like Functional Programming in Java. This will provide a deeper understanding of the concepts and techniques discussed in this article.