Introducing magic-di: A framework-agnostic dependency injector for building maintainable Python platforms & applications

As Python applications expand in size and complexity, developers often grapple with managing intricate codebases filled with tangled dependencies and convoluted logic. These sprawling services make implementing changes or introducing new features challenging without inadvertently affecting unrelated parts of the application. The complexity slows development and hampers collaboration among team members who may need help navigating and understanding the codebase. Adopting maintainable and scalable architectures that promote modularity is essential to address these issues.

Dependency Injection (DI) offers a solution by promoting loose coupling and modularity within the codebase. DI enhances flexibility and testability by supplying a class with its required dependencies from external sources rather than creating them internally. This decoupling allows components to be developed, modified, and tested independently, reducing the risk of unintended side effects and making the application more robust and easier to maintain. Given Python’s versatility and use across backend services, data science, and machine learning, we believe a DI solution must be framework-agnostic.

Introducing Magic DI, a novel dependency injection library developed at Wolt. Magic DI leverages Python’s type hinting to provide a zero-configuration, type-driven approach to dependency injection. This approach minimizes boilerplate code and eliminates reliance on global variables, empowering developers to construct a Python ecosystem akin to LEGO blocks. With Magic DI, you can easily install and use clients without delving deep into their implementations, simplifying integration with internal libraries and allowing engineers to focus on building features rather than managing dependencies.

Magic DI manages application lifecycle logic within each dependency through  __connect__ and __disconnect__ methods. For example, when integrating with a Kafka client, developers do not need to understand the startup and shutdown logic—Magic DI automatically handles it by invoking the relevant methods during application startup and shutdown.

Furthermore, Magic DI allows developers to write framework- and DI-agnostic code. By implementing the Connectable protocol using Python’s duck typing, you can make your library compatible with Magic DI without including it as a dependency. The Connectable protocol is a set of rules a class can follow to be compatible with Magic DI. 

At Wolt, we are building a robust ecosystem of reusable Python components—“Lego blocks”—that engineers can leverage without creating solutions from scratch. These libraries have built-in configurations, error handling, and scalability, allowing engineers to focus on core business needs. We aim to make these components easy to integrate, whether or not a service uses dependency injection. Maintaining a framework-agnostic design ensures that this ecosystem supports DI while remaining accessible and adaptable across diverse domains, from backend services to data science and ML.

This article explores how Magic DI simplifies dependency management, promotes cleaner and more maintainable code, and addresses the challenges faced in large-scale Python applications. We’ll delve into its key features, demonstrate its practical applications, and show how it fosters a reusable component ecosystem that transforms Python development into a more modular and efficient process. Additionally, this article describes how Magic-DI can be used to build a flexible Python platform and ecosystem, as illustrated by Wolt’s adoption experience.

You can find magic-di on its Github Repository here

Understanding Dependency Injection

Dependency Injection (DI) is a design pattern in which a class gets its required components from external sources rather than instantiating them internally. This inversion of control leads to more modular and flexible code, separating the creation of dependencies from their usage. Dependency Injection (DI) makes testing and maintenance more manageable by allowing for the swapping out of dependencies without modifying the using class [1].

In large-scale applications, managing dependencies becomes increasingly complex. DI helps reduce this complexity by clearly defining how components interact, promoting a modular architecture where each part has a single responsibility. This modularity improves testability and maintainability by allowing developers to independently develop, test, and refactor individual components, enhancing code quality and team collaboration.

Implementing DI in Python presents challenges, such as overreliance on global variables, which can lead to unpredictable behavior and testing difficulties. Manual dependency management often results in verbose code that’s hard to maintain. Additionally, without a standardized DI framework, replacing actual dependencies with mocks during testing becomes cumbersome, complicating efforts to refactor or extend the application.

The Challenges with DI in Python

Global Variables and Hidden Dependencies

A common practice in Python applications is using global variables to manage dependencies. While this approach might offer a quick fix, it often leads to unpredictable behavior and side effects that take time to trace. Global variables can create hidden dependencies, making understanding how different parts of the codebase interact challenging. This visibility issue becomes increasingly problematic in concurrent or asynchronous environments, where the global state can lead to race conditions and data inconsistencies[2]. Additionally, relying on global variables complicates testing, as it becomes harder to isolate components[3] and mock dependencies effectively.

Boilerplate Code and Framework Lock-In

Many existing dependency injection solutions in Python require extensive boilerplate code and configuration. Developers often need to write numerous constructors and configuration files or use complex patterns to manage dependencies, which can clutter the codebase and detract from the core business logic. Furthermore, certain data integration tools are tightly coupled with specific frameworks like FastAPI. While these tools can be practical within their ecosystems, they limit flexibility, making switching frameworks or using components independently challenging. This framework lock-in can impede the adoption of new technologies and constrain the application's ability to evolve.

Magic DI: A New Approach to Dependency Injection

We created Magic DI to address the shortcomings of traditional dependency injection methods in Python. Its core philosophy revolves around simplicity and elegance, aiming to make dependency management as straightforward as possible. By leveraging Python's built-in features, Magic DI minimizes the need for extra configuration and reduces boilerplate code. This simplicity not only makes the codebase cleaner but also lowers the barrier to entry for developers new to the project.

A fundamental design principle of Magic DI is the minimization of reliance on global variables. Global state can lead to unpredictable behavior, making code difficult to understand and test. Magic DI encourages explicit dependencies to be injected into classes and functions, promoting a decoupled architecture where components are independent and easily interchangeable. This decoupling enhances modularity, making it easier to refactor code and implement new features without affecting unrelated parts of the application.

Leveraging Python Type Hints

One of Magic DI's standout features is its use of Python type hints to facilitate dependency injection. Type hints specify the expected types for variables, function parameters, and return values. Magic DI uses these annotations to automatically resolve and inject dependencies, eliminating the need for manual wiring or extensive configuration files.

Conventional dependency injection without type hints:

Conventional dependency injection without type hints
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
class Database:
    def __init__(self):
        self.connected = False

    def connect(self):
        # Logic to connect to the database
        self.connected = True


class Service:
    def __init__(self, db):
        self.db = db

    def perform_action(self):
        if not self.db.connected:
            self.db.connect()
        # Perform some action using the database

# Manually creating and passing dependencies
db = Database()
service = Service(db)
db.connect()
service.perform_action()
db.disconnect()

In this example, dependencies are manually managed and passed around, which can become cumbersome in larger applications.

Using Magic DI with type hints:

Magic DI dependency injection
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
from magic_di import Connectable, DependencyInjector


class Database:
    async def __connect__(self):
        # Logic to connect to the database
        self.connected = True

    async def __disconnect__(self):
        # Logic to disconnect from the database
        self.connected = False


class Service(Connectable):
    def __init__(self, db: Database):
        self.db = db

    def perform_action(self):
        # No need to check or manage the database connection
        # Magic DI ensures that the db is connected
        pass


# Automatic dependency injection
injector = DependencyInjector()
service = injector.inject(Service)()

async with injector:  # <- connects and disconnects dependencies
    service.perform_action()

Above, type hints indicate that `Service` depends on Database, and Magic DI automatically injects a connected Database instance into Service. There's no need for manual instantiation or wiring of dependencies.

Zero-Config

Magic DI embraces a zero-configuration approach. It uses type hints to resolve dependencies without additional configuration files or complex setup. This method relies on the conventions established in the code, making dependency management more intuitive.

Traditional DI frameworks often require extensive configuration to map interfaces to implementations, define scopes, and specify lifecycle management.

Traditional DI configuration
1
2
3
4
5
6
7
8
9
10
11
12
13
14
# config.py
dependencies = {
    'db': Database(),
    'service': Service(),
}

# main.py
from config import dependencies

service = dependencies['service']
db = dependencies['db']
db.connect()

service.perform_action()

This approach separates the configuration from the code, which can lead to synchronization issues and increased complexity.

With Magic DI, external configuration files are unnecessary. Dependencies are defined and resolved directly in the code using type hints.

Magic DI dependency injection
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
from magic_di import inject_and_run


class Database(Connectable):
    # (same as before)


class Service(Connectable):
    def __init__(self, db: Database):
        self.db = db
    # (same as before)


async def main(service: Service):
    service.perform_action()

if __name__ == '__main__':
    inject_and_run(main)

By calling inject_and_run(main), Magic DI automatically resolves the dependencies required by main, instantiates them, and injects them where needed. Before executing the main function, Magic DI calls all asynchronous __connect__ methods so you can be confident that everything is fully initialized and ready to use within the main function. It also manages the application lifecycle by ensuring that dependencies are properly disconnected during shutdown, even if an exception occurs. This approach simplifies the application’s entry point and guarantees the correct initialization and cleanup of all dependencies.

Automatic Resolution and Injection

Magic DI scans the type hints and uses them to build a dependency graph. It then instantiates each dependency, respecting its lifecycles and initialization logic.

Magic DI scans the type hints
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
class Repository(Connectable):
    def __init__(self):
        pass # Data access logic


class Service(Connectable):
    def __init__(self, repo: Repository):
        self.repo = repo


class Controller(Connectable):
    def __init__(self, service: Service):
        self.service = service

    async def run():
           ...


# Magic DI automatically resolves:
# Controller -> Service -> Repository
injector = DependencyInjector()
controller = injector.inject(Controller)()

async with injector:  # <- connects and disconnects dependencies
    await controller.run()

In this example, Magic DI understands that the Controller depends on Service, which depends on the Repository. It resolves these dependencies recursively without any manual intervention.

In most actual applications, dependency graphs can become quite complicated, with services depending on multiple other components at different levels. Magic DI easily handles this complexity, automatically resolving dependencies in the correct order without requiring sophisticated configurations.

As illustrated, DeliveryOrderService depends on DeliveryFeeService, PaymentsClient, and Repository. The Repository depends on a Database. Magic DI examines these relationships and automatically orders the initialization, starting with the lowest-level dependencies (like Database) and building upwards, ensuring that each component receives fully initialized dependencies without manual intervention.

Magic DI disconnects components in the reverse order on shutdown, ensuring that high-level dependencies are closed first, followed by the lower-level components. This straightforward approach allows Magic DI to handle even complex dependency structures reliably, letting developers focus on their application’s logic without worrying about the intricacies of dependency management.

Contrasting with Anti-Patterns

Global Variables

Anti-Pattern:

Global variables anti-pattern
1
2
3
4
5
6
7
8
9
10
11
12
13
# global_vars.py
db = Database()

# service.py
from global_vars import db


class Service:
    def perform_action(self):
        if not db.connected:
            db.connect()
            # Perform action
            db.query(...)

This approach relies on a global state, which introduces several significant issues, such as hidden dependencies, reduced maintainability, and difficulties in testing. Global variables can be accessed or modified by any part of the program, making it challenging to understand and reason about the codebase. This non-locality means that to comprehend the behavior of a global variable, a developer must consider every part of the program that interacts with it, which increases cognitive load and the potential for errors. There is also no access control or constraint checking, so any part of the code can change a global variable's value without restriction, leading to unpredictable behavior and side effects. This implicit coupling between different parts of the program can hinder refactoring efforts and make the system fragile, as changes in one area may have unintended consequences elsewhere[2].

Global variables can also cause namespace pollution, increasing the risk of naming conflicts and making it harder to manage the codebase as it grows. In concurrent or multi-threaded applications, relying on global state can lead to synchronization issues, race conditions, and data inconsistencies, as multiple threads may attempt to read or write to the same variable simultaneously. From a testing perspective, the global state complicates the isolation of components[3], making it harder to write unit tests that are independent and repeatable. Tests may inadvertently depend on or alter global variables, leading to flaky tests and making it difficult to set up a clean testing environment. Overall, using global variables as a means of dependency management can significantly impair an application's maintainability, scalability, and reliability.

Magic DI Solution:

Magic DI alternative to global variables
1
2
3
4
5
6
class Service(Connectable):
    def __init__(self, db: Database):
        self.db = db
   
    def perform_action(self):
        db.query(...)

Dependencies are explicitly declared and injected, avoiding global state.

Manual Dependency Management

Anti-Pattern:

Manual dependency management anti-pattern
1
2
3
class Controller:
    def __init__(self):
        self.service = Service(Database())

In this example, the Controller directly instantiates the Service and Database within its constructor. This instantiation is a form of manual dependency management where each class creates its dependencies. This approach introduces several issues:

• Tight Coupling: The controller is now tightly coupled to specific implementations of the Service and Database. If you need to swap the Database with another data source or modify the Service, you must change the Controller. Tight coupling goes against the Single Responsibility Principle from SOLID design principles, as the Controller is now responsible for managing its own behavior and configuring its dependencies.

• Reduced Testability: Manual dependency management makes testing more difficult. Since the Controller directly creates Service and Database, you cannot easily substitute mock or fake versions of these dependencies for unit testing. This lack of flexibility complicates tests, as you have to rely on actual implementations of Service and Database, potentially introducing unwanted side effects and making the tests slower and harder to isolate.

• Limited Reusability: When components are tightly coupled, their reusability in different contexts is limited. For example, reusing the Controller with an alternative Service or Database would require modifying the code directly, leading to more maintenance and a higher risk of introducing errors.

Magic DI Solution:

Magic DI automated dependency management
1
2
3
class Controller(Connectable):
    def __init__(self, service: Service):
        self.service = service

With Dependency Injection, dependencies are injected rather than created manually within the Controller. This decouples Controller from specific implementations of Service and Database, allowing it to work with any compatible version of Service passed in as an argument. By injecting Service, Controller achieves several benefits:

• Improved Testability: Mocks or stubs can be easily passed into the Controller for testing, enabling isolated unit tests without the side effects of actual dependencies.

• Enhanced Flexibility and Reusability: Since the Controller no longer depends on specific implementations, it can be used with different variations of Service, promoting reusability and ease of configuration.

Benefits

By leveraging type hints and embracing a zero-config, type-driven approach, Magic DI simplifies dependency injection in Python. It eliminates the need for configuration files, reduces boilerplate code, and promotes a clean, maintainable codebase. Transparently managing dependencies allows developers to focus on writing business logic instead of dealing with plumbing code.

Key Features of Magic DI

Magic DI introduces several innovative features that redefine dependency injection in Python applications. By embracing Python's strengths and addressing the limitations of traditional DI methods, Magic DI offers developers a powerful and easy-to-use tool.

Duck-Typing: Independence from Magic DI

Magic DI stands out by leveraging Python’s Protocols and duck typing to detect dependencies in a way that doesn’t tightly couple components to the DI framework. This approach allows Magic DI to maintain framework independence, giving components the flexibility to function in any Python service with or without DI.

What is Duck Typing in Python?

Duck typing is implemented by Python’s Protocols, allowing classes to fulfill an interface without the need for inheritance. In other words, a class can implement specific behaviors simply by defining methods that align with a Protocol—without inheriting from a formal base class. Magic DI uses this approach to find injectable dependencies, making them independent from Magic DI.

How Magic DI Uses Duck Typing to Detect Dependencies

Magic DI identifies injectable dependencies by checking if a class implements the Connectable Protocol, which defines specific methods: __connect__ and __disconnect__ for managing lifecycle events. By defining these methods in a class, the component becomes compatible with Magic DI’s dependency injection, even if it does not explicitly inherit from a base class associated with Magic DI. This protocol allows for seamless integration with the DI framework while maintaining independence from it. For example, the Database class can be compatible with Magic DI simply by implementing the connect and disconnect methods specified by the Connectable protocol.

Similarly, Service can either implicitly follow the Connectable protocol by defining connect and disconnect methods or explicitly inherit from Connectable. By specifying Database as a dependency in its constructor, Service allows Magic DI to inject and manage Database automatically, ensuring both components are initialized and managed according to the unified lifecycle defined by Connectable.

Connectable protocol example
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
from magic_di import Connectable


class Database:
    # Follows the Connectable protocol without inheritance
    def __connect__(self):
        print("Connecting to the database...")

    def __disconnect__(self):
        print("Disconnecting from the database...")


class Service(Connectable):
    # Explicitly inherits from Connectable,
    # ensuring compliance with the protocol
    def __init__(self, db: Database):
        self.db = db

    def __connect__(self):
        # Handle Service connection logic

    def __disconnect__(self):
        # Handle Service disconnection logic

Singleton Pattern for Consistency

By default, Magic DI treats each class as a singleton within the context of an injector instance. This architecture means only one instance of each dependency is created and shared across the application per injector. Importantly, Magic DI does not maintain any global state; with every new instance of an injector, you get a fresh dependency container without any prior state. This approach ensures consistency within each injector’s scope and avoids the pitfalls of having multiple instances of the same class with different configurations or states. It also provides flexibility, as different parts of an application can use separate injectors if needed, each with its singletons.

Without Singletons:

Anti-pattern without singleton
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
class Config:
    def __init__(self):
        self.settings = load_settings()


class ServiceA:
    def __init__(self):
        self.config = Config()


class ServiceB:
    def __init__(self):
        self.config = Config()

# Each service has its own Config instance

In this example, ServiceA and ServiceB each have their own Config instances, which might lead to inconsistencies if the settings change during runtime.

With Magic DI Singletons:

Magic DI singleton pattern
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
from magic_di import DependencyInjector, Connectable


class Config(Connectable):
    def __init__(self):
        self.settings = load_settings()


class ServiceA(Connectable):
    def __init__(self, config: Config):
        self.config = config


class ServiceB(Connectable):
    def __init__(self, config: Config):
        self.config = config


injector = DependencyInjector()
service_a = injector.inject(ServiceA)()
service_b = injector.inject(ServiceB)()
# Both services share the same Config instance

Magic DI ensures that both services receive the same instance, reflecting changes to the settings everywhere.

Framework-Agnostic Design

Magic DI works independently of any specific web framework. It can integrate seamlessly using FastAPI, Flask, Celery, or a custom application. This framework-agnostic approach provides flexibility and allows developers to adopt Magic DI without being locked into a particular ecosystem.

Integration with FastAPI

Integration with FastAPI
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
from fastapi import FastAPI
from magic_di import Provide
from magic_di.fastapi import inject_app

app = inject_app(FastAPI())


class Database(Connectable):
    # Database connection logic


class Service(Connectable):
    def __init__(self, db: Database):
        self.db = db


@app.get("/items")
def get_items(service: Provide[Service]):
    return service.get_all_items()

Custom Application

Custom application integration
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
from magic_di import inject_and_run, Connectable


class Logger(Connectable):
    def log(self, message):
        print(message)


class Application(Connectable):
    def __init__(self, logger: Logger):
        self.logger = logger

    def run(self):
        self.logger.log("Application is running")


def main(app: Application):
    app.run()

if __name__ == "__main__":
    inject_and_run(main)

Early Dependency Validation

Magic DI instantiates all dependencies during application startup. This proactive approach ensures that any issues with dependencies—such as misconfigurations or missing components—are detected immediately. By catching these problems upfront, developers can prevent runtime errors that might occur later when the application handles requests or processes data.

Without Early Validation:

Without early validation
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
class PaymentGateway:
    def __init__(self, api_key = None):
        if not api_key:
            raise ValueError("API key is required")
   
    async def pay(self): ...


def get_payment_gateway():
    return PaymentGateway()

app = FastAPI()

@app.post("/payment")
def make_payment(pg: Annotated[PaymentGateway, Depends(get_database)]):
    return await pg.pay()

# Error occurs only when calling /payment endpoint if the API key doesn't exist

In this scenario, the error surfaces only when make_payment is invoked, which might be during a critical operation.

With Magic DI's Early Validation:

Magic DI early validation
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
class PaymentGateway:
    def __init__(self, api_key = None):
        if not api_key:
            raise ValueError("API key is required")

    async def __connect__(self) ...

    async def __disconnect__(self) ...

    async def pay(self): ...


app = inject_app(FastAPI())

@app.post("/payment")
def make_payment(pg: Provide[PaymentGateway]):
    return await pg.pay()

# Application fails to start due to missing API key

Magic DI forces the resolution of dependencies at startup, immediately catching the "missing API key" error.

Reduced Boilerplate and Cleaner Code

Magic DI significantly reduces the boilerplate code required by utilizing type hints and automatic injection. Developers don't need to write repetitive constructors or configuration methods for dependency management, leading to cleaner code that is easier to read and maintain.

Traditional DI with Boilerplate:

Traditional DI with boilerplate
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
class EmailService:
    def __init__(self, smtp_server):
        self.smtp_server = smtp_server


def get_email_service():
    smtp_server = os.getenv("SMTP_SERVER")

    return EmailService(smtp_server)


class NotificationService:
    def __init__(self, email_service):
        self.email_service = email_service


email_service = get_email_service()
notification_service = NotificationService(email_service)

This approach requires explicit wiring of dependencies and additional factory functions.

Magic DI simplification:

Magic DI simplified definition
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
from pydantic import Field
from pydantic_settings import BaseSettings


class EmailServiceConfig(BaseSettings, Connectable):
    smtp_server: str = Field(validation_alias="SMTP_SERVER")


class EmailService(Connectable):
    def __init__(self, config: EmailServiceConfig):
        self.config = config


class NotificationService(Connectable):
    def __init__(self, email_service: EmailService):
        self.email_service = email_service


injector = DependencyInjector()
notification_service = injector.inject(NotificationService)()

Magic DI automatically resolves the dependencies using type hints, eliminating the need for extra wiring code.

Improved Testing Capabilities

Dependency injection also improves code testability. By allowing dependencies to be easily overridden or mocked, developers can write unit tests focusing on specific components without worrying about the entire dependency graph. You can configure Magic DI's injector during testing to provide mock implementations.

Magic DI improved testing
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
rom magic_di.testing import get_injectable_mock_cls

# services.py
class DataFetcher(Connectable):
    def fetch(self):
        # Fetch data from external API


class Processor(Connectable):
    def __init__(self, fetcher: DataFetcher):
        self.fetcher = fetcher

    def process(self):
        data = self.fetcher.fetch()
        # Process data


# test_services.py
from unittest.mock import MagicMock
from magic_di import DependencyInjector


def test_processor():
    mock_fetcher = MagicMock()
    mock_fetcher.fetch.return_value = {"key": "value"}
    injector = DependencyInjector()

    injector.bind({
        DataFetcher: get_injectable_mock_cls(mock_fetcher),
    })

    processor = injector.inject(Processor)()
    processor.process()

 mock_fetcher.fetch.assert_called_once()

We can test Processor in isolation by binding DataFetcher to a mock implementation.

Encouraging Reusable Components

Magic DI's standardized approach to dependency injection promotes the creation of reusable components. Developers can build libraries or modules that others can easily integrate into projects without modification. This reusability enhances collaboration and accelerates development.

Creating a Reusable HTTP Client:

Creating a reusable HTTP client
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
# http_client.py
from magic_di import Connectable
import requests


class HTTPClient(Connectable):
    def __connect__(self):
        self.session = requests.Session()

    def get(self, url):
        return self.session.get(url)


# service.py
class ExternalAPIService(Connectable):
    def __init__(self, http_client: HTTPClient):
        self.http_client = http_client

    def fetch_data(self):
        response = self.http_client.get("https://api.example.com/data")
        return response.json()

# Another project can reuse HTTPClient without changes

Defining HTTPClient as a reusable component allows it to be shared across multiple services and projects, ensuring consistency and reducing duplication.

Magic DI's features collectively contribute to a more efficient and enjoyable development experience. Magic DI helps developers build robust Python applications with less effort by simplifying dependency management, ensuring consistency, and enhancing testability.

Comparing Magic DI with Existing Solutions

Several dependency injection solutions exist in the Python ecosystem, each with different features and trade-offs. Understanding how Magic DI compares with these solutions can help developers decide which tool best suits their needs.

Magic DI vs. FastAPI's Dependency Injection

FastAPI is a popular web framework that includes a dependency injection system (DI). While effective for many use cases, DI has limitations that Magic DI aims to overcome.

Framework Coupling vs. Framework Agnosticism

FastAPI's DI tightly integrates with the framework. Dependencies are declared using the Depends function and injected into endpoint functions.

Comparing with FastAPI Depends
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
from fastapi import FastAPI, Depends

app = FastAPI()

def get_database():
    db = Database()
    try:
        yield db
    finally:
        db.close()


@app.get("/items")
def read_items(db: Database = Depends(get_database)):
    return db.fetch_items()

The get_database function provides the db dependency in this example. While this approach works within FastAPI, it ties the code closely to the framework's constructs using Depends, making it less portable.

Magic DI is framework-agnostic. It defines dependencies using type hints and handles injection externally, decoupling the business logic from the development framework.

Magic DI is framework-agnostic
1
2
3
4
5
6
7
8
9
10
11
12
13
14
from fastapi import FastAPI
from magic_di import Provide
from magic_di.fastapi import inject_app


app = inject_app(FastAPI())

class Database(Connectable):
    # Database connection logic


@app.get("/items")
def read_items(db: Provide[Database]):
    return db.get_items()

Here, DatabaseConnection is not dependent on FastAPI. Magic DI injects the dependencies, allowing the code to remain flexible and reusable across different contexts.

Boilerplate Code and Configuration

FastAPI requires developers to write dependency provider functions for each dependency, which can lead to additional boilerplate.

FastAPI boilerplate
1
2
3
4
5
6
def get_config():
    return Config()


def get_service(config: Config = Depends(get_config)):
    return Service(config)

Every dependency chain necessitates a new function, increasing the amount of code and the potential for errors.

Magic DI reduces boilerplate by leveraging type hints and automatic resolution, eliminating the need for explicit provider functions. Moreover, each dependency contains logic for managing its lifecycle through the standardized methods __connect__ and __disconnect__. This standardized interface means that every dependency is responsible for its initialization and cleanup, following a consistent pattern across all dependencies. This standardized lifecycle management further simplifies the codebase, enhances maintainability, and ensures we appropriately manage resources without additional boilerplate code.

Magic DI automatically injects Config
1
2
3
4
5
6
7
class Config:
    # Configuration logic


class Service:
    def __init__(self, config: Config):
        self.config = config

Lazy vs. Early Dependency Resolution

When you call an endpoint, FastAPI lazily instantiates dependencies. This late instantiation can delay the discovery of configuration errors until runtime. If a dependency, like a database connection, fails to initialize, the error will not surface until the application accesses the specific endpoint that requires that dependency.

Magic DI resolves all dependencies at application startup. If there's a misconfiguration, the application fails to start, allowing developers to catch errors before the application is live.

Early dependency resolution
1
2
3
4
# Misconfigured DatabaseConnection will cause an error at startup
injector = DependencyInjector()

app = inject_app(FastAPI(), injector=injector)

By failing fast, Magic DI promotes robustness and reliability in applications.

Magic DI vs. Other Dependency Injection Frameworks

Other DI frameworks like python-dependency-injector offer robust features but can introduce complexity and require more boilerplate code.

Configuration Complexity

Many DI frameworks require explicit configuration of containers and providers, which can become verbose.

Configuration complexity
1
2
3
4
5
6
7
8
9
10
from dependency_injector import containers, providers


class Container(containers.DeclarativeContainer):
    config = providers.Configuration()
    database = providers.Singleton(
        DatabaseConnection, 
        db_url=config.db.url,
    )
    service = providers.Factory(ItemService, db=database)

This approach separates configuration from business logic but adds layers of abstraction.

Magic DI embeds configuration within the components using type hints and defaults, reducing the need for external configuration.

Magic DI embeds config within components
1
2
3
4
5
6
7
8
9
10
11
12
class DatabaseConfig(BaseSettings, Connectable):
    db_url: str = Field(validation_alias="DB_URL")


class DatabaseConnection:
    def __init__(self, config: DatabaseConfig):
        # Initialization logic


class ItemService:
    def __init__(self, db: DatabaseConnection):
        self.db = db

Dependencies are managed directly in the code, enhancing clarity.

Ease of Use and Learning Curve

Frameworks with complex configurations can have a steep learning curve, potentially slowing down development.

Magic DI emphasizes simplicity, utilizing familiar Python constructs like type hints and context managers.

Magic DI uses familiar Python concepts
1
2
3
4
5
6
7
8
9
from magic_di import inject_and_run

async def main(service: ItemService):
    items = service.get_items()
    print(items)


if __name__ == "__main__":
    inject_and_run(main)

Developers can focus on writing business logic without the overhead of managing DI infrastructure.

Flexibility and Adaptability

While powerful, some frameworks may enforce strict patterns that limit flexibility.

Magic DI allows for dynamic binding and easy overrides, making it adaptable to various use cases.

Dynamic binding and easy overrides
1
2
3
4
5
injector = DependencyInjector()

injector.bind({DatabaseConnection: MockDatabaseConnection})

# Now, any component requiring DatabaseConnection will receive MockDatabaseConnection

This flexibility is beneficial for testing, prototyping, or swapping out implementations.

Building a Reusable and Framework-Agnostic Python Ecosystem at Wolt

At Wolt, we aim to establish a rich ecosystem of reusable Python components—essentially “Lego blocks”—that engineers across teams can leverage without creating solutions from scratch. The platform team is building these libraries and tools with robust, out-of-the-box features, including necessary configurations, error handling, and scalability support so that engineers can focus on core business needs rather than infrastructure details. This approach saves time and promotes consistency by enabling engineers to plug in pre-built, high-quality components.

A crucial goal is to make these libraries easy to integrate, regardless of whether a service uses dependency injection. While DI is valuable for managing complex dependencies, we aim to avoid any framework or vendor lock-in. This approach is critical given Python’s broad use across backend services, machine learning, data science, and other domains. Our North Star aims to deliver a versatile, framework-agnostic ecosystem that supports DI but remains accessible to engineers, regardless of the tooling or frameworks they choose.

How Magic DI Handles Multi-Library Dependency Graphs

In real-world applications, dependency graphs can be intricate. Services often depend on multiple other components, which rely on additional dependencies. For example, let’s look at a typical scenario involving various libraries:

• wolt-queues depends on wolt-kafka-client for streaming messages.

• wolt-kafka-client also relies on wolt-statsd to collect metrics and wolt-logging.

• Each component requires specific initialization, such as wolt-statsd for metrics flushing and wolt-kafka-client for committing offsets.

In this setup, the real challenge isn’t just initializing components in the correct order but understanding the specific requirements for each dependency’s startup and shutdown. For example, when using wolt-queues, an engineer would typically need to know how to initialize and manage all its underlying dependencies—such as wolt-kafka-client for message streaming, wolt-logging for logging, and wolt-statsd for metrics collection. Each dependency may require specific methods to be called on startup, such as connecting to a message broker or setting up a logging configuration, and on shutdown, such as committing offsets in Kafka or flushing metrics in StatsD.

Magic DI simplifies this process by introducing standard protocols and automatically handling initialization and teardown. It scans the dependencies, resolves them, and ensures it calls the correct connect and disconnect methods for each component, following a unified protocol for startup and shutdown.

This automatic dependency resolution supports asynchronous operations, which is critical for components like Kafka that may rely on asynchronous message handling.

Defining Wolt service graph
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
from magic_di import inject_and_run, Connectable


class WoltLogger(Connectable):
    ...


class WoltStatsd(Statsd):
    async def __connect__(self):
        await self.start()

    async def __disconnect__(self):
        await self.flush()


class WoltKafkaClient(KafkaClient):
    def __init__(logger: WoltLogger, statsd: WoltStatsd):
        self.logger = logger
        self.statsd = statsd

   async def __connect__(self):
        await self.connect()

   async def __disconnect__(self):
        await self.commit()


class WoltQueue(Connectable):
    def __init__(kafka: KafkaClient):
        self.kafka = kafka

    async def consume(): ...


async def main(queue: WoltQueue):
    await queue.consume()

if __name__ == "__main__":
    inject_and_run(main)

Beyond simplified dependency injection, our vision is to make Magic DI the backbone of a unified Wolt Python ecosystem, fostering reusable, framework-agnostic libraries. By standardizing protocols and embedding dependency handling, Magic DI empowers engineers to effortlessly integrate components, ensuring scalability and maintainability without added complexity.

For platform engineers, this unification is crucial. Platform engineers standardize initialization and shutdown methods across the organization, ensuring that every component follows the same protocol, regardless of whether they use DI. These protocols are extendable, allowing the introduction of new standardized methods, such as a ping method for automatic health checks. For example, platform engineers can create general health checks that automatically detect and validate dependencies implementing this protocol. It sets a standard interface across the ecosystem, making it straightforward for engineers to integrate and manage internal libraries.

With this approach, engineers using internal libraries gain consistency and predictability. If they’re working without dependency injection, they automatically know they need to call standardized connect and disconnect methods to initialize and shut down components properly. If they’re using our SDK with Magic DI, the integration becomes effortless—the SDK handles everything behind the scenes, making the process almost invisible to the engineer. Dependencies are automatically set up and torn down without manual intervention, allowing engineers to focus on their core application logic.

Building Framework-Agnostic Components with Magic DI

One fundamental principle in our ecosystem is that each component should be framework-agnostic. While Magic DI provides seamless DI capabilities, we design our libraries to work independently. Engineers can choose to use Magic DI or any DI framework—or even no DI framework. This flexibility is essential for Python’s diverse ecosystem, as engineers may use the language in various fields, including data science, machine learning, and backend development.

For instance, consider a library for connecting to external services, such as message queues. This library should work effortlessly in DI and non-DI environments, allowing engineers to plug it into their projects regardless of architecture. If they’re using Magic DI, it will automatically inject the required dependencies. If they’re not using DI, they can still initialize the library manually, ensuring compatibility with any service.

Avoiding Vendor Lock-In and Dependency Bloat

While dependency injection (DI) can streamline dependency management, it often brings the risk of vendor lock-in. Many DI frameworks tightly coupled services with the DI container, making switching DI solutions or using components independently challenging. Magic DI takes a different approach by leveraging Python’s duck typing and protocols, ensuring it handles dependencies without embedding the DI framework within each library.

For engineers using our SDK with Magic DI, integration is automatic: Magic DI reads the type hints, resolves dependencies, and manages setup and teardown behind the scenes, seamlessly handling the entire dependency graph. This flexibility allows engineers to build components that work effortlessly across various contexts, with or without DI.

By avoiding framework-specific dependencies, Magic DI promotes a versatile and resilient Python platform, supporting varied use cases across Wolt. This approach ensures that our libraries remain lightweight, extensible, and compatible with different frameworks, fostering a flexible ecosystem that engineers can rely on without vendor lock-in or unnecessary dependency bloat.

A North Star for Wolt’s Python Ecosystem

We aim to establish a comprehensive Python ecosystem that empowers engineers to quickly build reliable, scalable applications. By providing reusable, framework-agnostic libraries supported by Magic DI, we reduce the cognitive load on engineers, allowing them to focus on building innovative solutions rather than reinventing the wheel.

Magic DI helps us achieve this by enabling seamless integration between components and supporting complex dependency graphs without manual setup. But our North Star is broader: we aim to make every component in the ecosystem easy to adopt and reuse, regardless of the technology stack or DI solution an engineer chooses.

As our Python platform evolves, we invest in libraries and tools that promote a modular, cohesive ecosystem.

Summary

As Python applications become complex, developers often need help managing dependencies and maintaining clean, modular codebases. Traditional dependency injection (DI) methods in Python can introduce issues like overreliance on global variables, boilerplate code, and tight coupling with specific frameworks, all hindering scalability and maintainability. We introduce Magic DI as a solution to these challenges.

Developed at Wolt, Magic DI is a framework-agnostic dependency injection library that leverages Python’s type hints and duck typing to provide a zero-configuration, type-driven approach to DI. By automatically resolving dependencies using type annotations, Magic DI reduces boilerplate code and eliminates the need for global variables, promoting a cleaner and more maintainable architecture.

Key features of Magic DI include:

• Leveraging Python Type Hints: Magic DI uses type annotations to automatically resolve and inject dependencies without manual wiring or configuration files.

• Zero-Configuration: eliminates the need for external configuration, relying on code conventions to manage dependencies.

• Duck Typing and Protocols: Magic DI utilizes Python’s duck typing and the Connectable protocol to detect dependencies, allowing components to remain independent of the DI framework.

• Singleton Pattern: It treats each class as a singleton within an injector instance, ensuring consistency and avoiding multiple instances of the same dependency.

• Framework-Agnostic Design: Magic DI integrates seamlessly with various frameworks, such as FastAPI, Celery, or custom applications, without being tightly coupled to any of them.

• Early Dependency Validation: It instantiates all dependencies at startup, catching configuration issues early and preventing runtime errors.

• Reduced Boilerplate: Magic DI minimizes repetitive code by automating dependency management through type hints.

• Improved Testing Capabilities: It simplifies testing by allowing easy overriding and mocking of dependencies.

• Encouraging Reusable Components: Magic DI promotes the creation of modular, reusable components for integration into different projects.

At Wolt, Magic DI is crucial in building a reusable, framework-agnostic Python ecosystem. By standardizing initialization and shutdown methods across components and using protocols, Magic DI simplifies the integration of internal libraries and tools, fostering consistency and scalability. This approach allows engineers to focus on business logic rather than infrastructure details, enhancing productivity and collaboration.

Conclusion

Magic DI offers a streamlined, efficient approach to dependency injection in Python, addressing the limitations of existing solutions by:

• Reducing boilerplate code through automatic injection using type hints.

• Remaining independent of any specific framework, thus promoting flexibility.

• Catching configuration errors early by resolving dependencies at startup.

• Simplifying testing with centralized dependency management and easy overrides.

• Enhancing developer productivity through familiar Python constructs.

Magic DI empowers developers to build scalable Python applications more effectively by facilitating a modular and maintainable codebase.

Find more about magic-di on its Github Repository here.

References

1. Hong Yul Yang, Ewan Tempero, Hayden Melton. An Empirical Study into Use of Dependency Injection in Java. Department of Computer Science, University of Auckland

2. C2 Wiki: Global Variables Are Bad

3. Robert Martin. The Open-Closed Principle: No Global Variables - Ever.

Does the work we do sound interesting? We’re hiring! Check out our open roles! 🚀