Announcing Transaction Orchestrator

Medusa is introducing a powerful new feature called Transaction Orchestrator (TO).

In a monolithic application with a single data source, managing transactions is an easy task using standard SQL transactions. However, Medusa is evolving its architecture to be more modular, allowing individual modules to run in isolation, potentially with their own data sources or even on separate servers.

This change in architecture brings new challenges for handling transactions, as one system can no longer control another's commit or rollback actions. That's where the Transaction Orchestrator (TO) steps in, offering an effective way to manage transactions within this increasingly complex environment.

Designed to enhance the control and management of transactions and workflows across multiple services or modules, the TO simplifies creating and executing distributed transactions. This allows developers to concentrate on feature implementation rather than dealing with the intricacies of managing complex distributed transactions. At Medusa, we build tools for developers to create rich commerce applications that scale, and adapting to these new challenges is a part of that mission.

The Need for Improved Transaction and Workflow Management

The TO supervises transaction flows and guarantees that successful transactions are executed fully or entirely rolled back in case of failure. With clearly defined steps for Invoke and Compensate actions, the TO follows the separation of concerns principle, allowing developers improved control over transactions and workflows.

Distributed transactions are necessary for scenarios where a given workflow involves different modules for several reasons:

1. Data consistency: In a distributed system, maintaining data consistency across different databases becomes challenging. A TO ensures that if any part of the transaction fails, all the previous steps are rolled back (compensated), keeping the data consistent across all involved databases.
2. Coordination: Acting as a coordinator between different modules and their respective servers, the TO manages the order of execution and communication between modules, ensuring that each transaction step is executed correctly and at the right time.
3. Simplifying complex workflows: In a distributed environment, transactions can become complex due to the need to coordinate between different services and databases. A TO simplifies this process by abstracting away the complexities of managing distributed transactions, allowing developers to focus on implementing the business logic.
4. Scalability: As a system grows, managing transactions across multiple services becomes increasingly difficult. A TO helps with scalability by providing a robust framework for managing distributed transactions, making it easier to maintain and expand the system.

In the context of digital commerce, this is an important addition to Medusa’s toolbox for the following reasons:

• Composable architectures: A transaction orchestrator supports composable architectures, allowing developers to easily combine and reuse modules as needed. This enables the creation of highly customizable commerce applications, tailored to specific business requirements.
• Adoption in legacy systems: The TO can also facilitate the gradual transition of legacy systems to more modern, distributed architectures. This makes it easier for businesses to adopt and integrate new technologies without having to rebuild their entire infrastructure from scratch.
• Unlocking infrastructure technologies: With a transaction orchestrator, developers can leverage advanced infrastructure technologies, such as serverless and edge computing. This can lead to improved performance, reduced latency, and increased reliability for commerce applications, resulting in better user experiences and higher customer satisfaction.

Here's an example of how a transaction orchestrator would help in a distributed system involving three modules:

123456789
Module A creates an order (Server 1, Database 1)
Module B manages inventory (Server 2, Database 2)
Module C handles payments (Server 3, Database 3)

A typical transaction flow would be:

1. Create order in Module A
2. Deduct inventory in Module B
3. Process payment in Module C

With a TO, you can define the steps in the transaction and specify compensation actions for each step. If any step fails, the orchestrator will automatically execute the corresponding compensation actions to revert previous steps, ensuring data consistency across services.

In this example, if the payment processing in Module C fails, the TO would trigger compensation actions to add back the deducted inventory in Module B and remove the created order in Module A, preventing data inconsistency.

Achieving Cleaner Code with the Transaction Orchestrator

Utilizing a transaction orchestrator results in cleaner code and single-responsibility functions compared to manually managing distributed transactions. This is due to several factors:

1. Abstraction: A TO abstracts the complexity of managing distributed transactions by providing a standardized framework for defining transaction steps, their corresponding compensation actions, and the flow of execution. This allows developers to focus on implementing the core business logic of each step rather than managing the complexities of distributed transactions.
2. Separation of concerns: By clearly delineating the responsibilities of each function, the TO enforces the separation of concerns. Each function is responsible for either the "Invoke" (execution) or "Compensate" (rollback) action, ensuring they perform a single, specific task. This makes the code more readable, maintainable, and testable.
3. Modularity: By using a TO, the code becomes more modular as each step in the transaction is encapsulated within its own function. This allows developers to easily modify, add, or remove transaction steps without affecting the overall structure of the transaction.
4. Error Handling: When handling distributed transactions manually, the code can become convoluted due to intricate error handling logic. The TO simplifies this by automatically managing errors and retries, allowing developers to create cleaner code without the need to address error handling for each step.
5. Reusability: The TO allows you to define reusable functions for both the "Invoke" and "Compensate" actions, which can be easily reused across different transaction scenarios. This reduces code duplication and ensures that changes to a specific action only need to be made in one place.

Here is an example used in the Multi-Warehouse Project that coordinates the flow to create a product variant, creating an inventory item and finally linking both together.

Here is definition of the flow:

1234567891011121314
const createVariantFlow: TransactionStepsDefinition = {
    next: {
        action: "createVariantStep",
        saveResponse: true,
        next: {
            action: "createInventoryItemStep",
            saveResponse: true,
            next: {
                action: "attachInventoryItemStep",
                noCompensation: true,
            },
        },
    },
}

The actions are handled by a single function called by the TO respecting the Transaction Definition above. Note that the implementation of each function was omitted to keep the example short; however, their names are self-explanatory.

12345678910111213141516171819202122232425262728293031323334
async function transactionHandler(
    actionId: string,
    type: TransactionHandlerType,
    payload: TransactionPayload
) {
    const command = {
        createVariantStep: {
            invoke: async (data: CreateProductVariantInput) => {
                return await createProductVariant(data) // omitted
            },
            compensate: async (data: CreateProductVariantInput, { invoke }) => {
                await removeProductVariant(invoke.createVariantStep) // omitted
            },
        },
        createInventoryItemStep: {
            invoke: async (data: CreateProductVariantInput, { invoke }) => {
                return await createInventoryItem(invoke.createVariantStep) // omitted
            },
            compensate: async (data: CreateProductVariantInput, { invoke }) => {
                await removeInventoryItem(invoke.createInventoryItemStep) // omitted
            },
        },
        attachInventoryItemStep: {
            invoke: async (data: CreateProductVariantInput, { invoke }) => {
                return await attachInventoryItem( // omitted
                    invoke.createVariantStep,
                    invoke.createInventoryItemStep
                )
            },
        },
    }

    return command[actionId][type](payload.data, payload.context)
}

Finally, the TO is instantiated and a new transaction initialized.

1234567891011
const strategy = new TransactionOrchestrator(
    "create-variant-with-inventory", // transaction name
    createVariantFlow // transaction steps definition
)

const transaction = await strategy.beginTransaction(
    ulid(), // unique id
    transactionHandler, // handler
    createProductVariantInput // input
)
await strategy.resume(transaction)

Here is an animated diagram of the transaction described above:

This marks just the beginning of Transaction Orchestration in Medusa. In the near future, we'll introduce numerous other flows that function similarly, enabling you to customize and expand existing workflows with your unique steps.

Handling Complex Workflows with the Transaction Orchestrator

The Transaction Orchestrator enables developers to create complex workflows for synchronous and long-running tasks that may take a while to receive a response (asynchronous). These workflows can be organized in a way that may not necessarily be transactional but can still be effectively orchestrated.

There are several scenarios where asynchronous workflows with long-running steps can be applied using the TO:

1. E-commerce order fulfillment: In an e-commerce system, a workflow might involve creating an order, reserving inventory, charging the customer, generating shipping labels, and updating the shipping status. Some of these steps, such as generating shipping labels or charging the customer, could take a considerable amount of time due to external dependencies like payment gateways and shipping services.
2. Fraud detection and prevention: In an e-commerce system, fraud detection and prevention workflows might involve analyzing customer data, order patterns, and payment information to identify potential fraudulent activities. These workflows may include time-consuming tasks like querying external fraud detection APIs or applying machine learning models to analyze data.
3. Returns and refunds processing: In an e-commerce returns and refunds workflow, several steps may involve long-running tasks, such as receiving returned products, inspecting their condition, updating inventory, and processing refunds. Some of these steps may take a considerable amount of time, especially when dealing with external payment gateways or waiting for products to be returned and inspected.

In all these examples, workflows are valuable for handling intricate, multi-step processes that include tasks taking considerable time or involving external dependencies without immediate responses. By leveraging the Transaction Orchestrator, developers can proficiently manage these workflows, ensuring proper execution order and handling errors and compensations as needed.

What’s Next

The Transaction Orchestrator in Medusa enhances control over transactions and workflows in complex, multi-service scenarios. As Medusa's architecture becomes more modular, many flows will be refactored to leverage the Orchestrator. This allows developers to focus on business logic while the Orchestrator handles consistency and infrastructure concerns. In the future, support for asynchronous tasks will be added, enabling the handling of long-running tasks or non-REST API approaches, making Medusa even more adaptable and developer-friendly.