Decorating bound-methods in Python — a general and scalable solution
Decorators in Python can quickly become a complicated matter when dealing with bound-methods (i.e. class-instance bound function calls). In some cases a basic decorator is required. In other cases, we would like to supply parameters to the decorator itself. We also would like to extend/modify decorator definitions and not have to implement the complexities each time. Lastly we would like to control execution before, during and after a wrapped method call and having each such phase be configurable.
After experimenting with a variety of approaches, I have come up with a generalized solution that enables highly configurable and extensible bound-method decorators which use the elegance of object-oriented (OO) mechanics and the Python descriptor protocol (optional reading and not for the feint of heart).
In this article I describe the solution and the features it provides. It will also become quite clear why this is in fact a somewhat tricky problem that requires a deep understanding of the Python interpreter data-model dynamics. Despite these complexities, the solution presented is very easy to use. Besides providing a re-usable solution, I hope this article also goes some way to deepen your understanding of the Python language.
In order to properly appreciate the challenges involved and to better define the problem statement, it’s important to elaborate the underlying semantics of function decorators and how things quickly get complicated when we require variability in the declarations and thus dynamics.
Consider the following decorator usage styles:
Let’s compare and contrast these styles to comprehend what actually happens:
- The statement
Decorator(method1)must return a callable so that invocations of
method1follow the semantics of a bound method-call. This syntax seems consistent with initializing a Decorator object with the wrapped function object reference as the initializer argument. This suggests that the Decorator object (let’s call it
decorator) is in fact itself a callable, but even more-so, it must be callable in a way that captures the bound object of the method-call (more on this complication later)
- The statement
Decorator(param1="foo")(method2)must also return a callable, but this syntax looks more compounded, in the sense that the Decorator object initialization takes some parameter(s) other than the wrapped function and this is followed by a call on this object with the wrapped function object reference, i.e. the call occurs when this line in the class definition is interpreted. It suffices to say however, that this call only happens once, and subsequent calls are on the returned callable. Such subsequent calls must also be on a bound object
The point here is that despite the syntactic sugar of the decorator syntax, the underlying dynamics of the effective assignments could not be more different and this is what creates trouble in trying to support both formats.
Now that we’ve shown how a subtle difference in high-level syntax creates completely different mechanisms under the hood, lets formalize what makes for a complete and general solution for bound-method decorators. Consider a class with methods we may wish to decorate. We wish to do the following:
- Decorate specific bound-methods with a decorator
- In some cases, the decorator will have no parameters and can use a typical syntax without parentheses, e.g.
- In other cases, we would like parameters to be passed to the decorator, e.g.
Each decorator should have the option to do the following (let’s call this the decorator functionality):
- Optionally perform actions before calling the wrapped method with the ability to operate on the bound object instance
- Optionally override the default calling of the wrapped method in a particular way (e.g. fixing some argument) as a bound method call
- Optionally perform actions after calling the wrapped method with the ability to operate on the bound object instance. This should also allow the return value of the wrapped method to be modified for example
We would also like to create specializations of the decorator because we may want variations in the behaviors of the decorator functionality. The wrapped calling sub-phases are shown in the figure below. Any such sequence must be defined in a base class so that it can be overridden.
The solution requires handling two underlying mechanisms with some common structures. It turns out that the second case (decorator with parameters) is in fact the simpler case. First we will discuss the essence of the solution for each case.
Using a dedicated class
It perhaps goes without saying that using a dedicated class is the best approach to solving this problem. The main reasons are the following:
- Classes are great for problems that require stateful behaviors
- Classes provide construction semantics (via
- Classes provide callability semantics (via
- Classes provide the ability to extend and override
- The descriptor protocol of the Python interpreter is only available via a dedicated class
Decorators with parameters
This case (case #2 in the previous code example), can be implemented with the simplest approach. The reason is that it can be done completely without the descriptor protocol. Let’s look again at the following line which is the effective representation of the decorator assignment:
method2 is a bound method (i.e. one that takes the bound object reference typically denoted as
self as the first argument). Three things happen when this line is executed by the interpreter during what I would refer to as the decoration phase (i.e., the point at which the interpreter is executing this line as it reads the class definition):
- Reading from left to right, we first initialize a new
Decoratorobject which calls the
Decorator.__init__(param1="foo")method and returns a reference to a new
Decoratorobject. Let’s refer to this anonymous object as
decorator. It’s anonymous because we never assign its reference directly to any variable
- Continuing to read the statement, we now have an object reference
decoratorwhich we subsequently call, which under the hood will look like
decorator.__call__(method2). Again, this call occurs during the decoration phase. We’re not referring to any bound method calls here
decoratorobject call method returns a callable which represents the wrapped method call matching the previous figure showing the decorated call broken out in sub-phases
There is nothing especially complicated here, except that we must make sure to understand that when the now wrapped
method2 is invoked as a bound-method call (e.g.,
some_object.method2()), that the first argument in the callable is a reference to the bound instance. It turns out that the interpreter takes care of providing this reference which is passed into the callable call because this callable is local to each object of the class which defines
method2. This is important to understand because the other case acts differently.
Decorators without parameters
This is where things get a a bit more tricky. Let’s dissect the mechanics of what such a decorator assignment actually does:
method1 = Decorator(method1) # equivalent to @Decorator
This seems harmless enough. But what is it actually doing? Clearly it is just a an initializer call of the
Decorator class which passes the method reference (
method1) in this case as the only argument. This means that
method1 is assigned a reference to a
Decorator object (let’s call it
decorator). There is no subsequent call to get a callable. So, once the interpreter has processed this line, we effectively have:
method1 = decorator # A class-variable reference to a Decorator object
Bound-method calls of
method1 thus will result in calling the
decorator class-level object (i.e. this object is bound to the class which defines
What makes this different is that now all bound calls are actually bound to the decorator object
The difference is that in the previous case (parameterized-decorator), a function object (callable) is provided to
method2 during the decoration phase, which is always called as a bound-method call, such that calls to this callable follow regular bound-method call semantics.
In the parameter-less case, the mechanics resulted in a
Decorator initializer method being called. There is no way to return a function object from a class initializer call. An initializer always returns a reference to the initialized object. In this case, the returned
decorator object is just an associated object (via composition), so when
method1() is called (during a bound-method call),
decorator.__call__() is called. The call is bound, but it’s bound to the
decorator object, not the client object. This may appear to be an inconvenience, but any other behavior would in fact be nonsense.
How do we provide the bound object instance reference to this
The solution to this problem requires leveraging the Python “descriptor protocol”
To get into all the background and mechanics of the descriptor protocol would fill several posts, so let’s not go there. Instead, we can focus on the relevant dynamics which allow us to intercept and capture the caller’s instance reference before the
decorator.__call__() method is invoked.
To make this work, we add the
__get__() method in the
Decorator class, which, via Python’s wonderful data-model type-emulation system (affectionately known as duck-typing), turns a
Decorator class into a descriptor.
A descriptor allows attributes to be implemented via a class-bound object which has access to the instance reference of any object of that client class.
You may have already used descriptors, albeit indirectly if you have leveraged properties. Properties are just a very common pattern implemented upon the descriptor protocol framework
Decorator class becomes a descriptor, it turns the
decorator object into an attribute of the client class . Descriptors are always class-bound (because attribute definitions and method definitions are in fact class-bound). When a descriptor is accessed as a callable via an object attribute,
some wonderful things happen under the hood:
__get__()method is called in the first phase. This is called when the attribute is read. It is during this time that the caller’s instance reference is passed to our descriptor object. We save this information for the subsequent call phase (this is the crux of the magic)
- Following that, there is a function call attempt which will target the
__call__()method of the descriptor object. At this point in time, we have the bound object(instance) reference (this was saved in the last step)
So one can see that it is via the
__get__() method, that we can capture the calling object reference and then we use this captured reference during the call phase to implement the decorator call-phases (with full access to the calling object reference) to implement the earlier diagram. Just to be clear, both
__call__() are called by the interpreter in sequence when performing a calling action on an attribute implemented by a descriptor. I hope you’re still with me because that was the heavy part.
Example Code and Source
The source-code for the bound-method decorator base-class
BMDecorate and an associated test are available in my personal GitHub project here:
It is best to demonstrate the solution via an example. The example below creates a specialization of
my_decorator which overrides all three phases (this is equivalent to the
test.py code in the above GitHub link:
This example demonstrates that specializations of
BMDecorate can be used with our without parameters. Running the above example generates the following output:
The output above shows in detail all the phases and steps that occur, including the actual decoration phase which occurs when the interpreter is processing the A and B class definitions (which declare the decorators).
In this article I presented a general solution for bound-method decorators via a specialized class which can be extended for custom use. Providing the flexibility for both bare and parametrized decorators along with multi-phased execution required getting into the guts of the descriptor protocol which is the fundamental mechanism in Python to implement properties and other intelligent attributes.
I mostly wrote this article so that anyone who is wishing to decorate bound-methods can find and re-use this solution. Lastly, I hope that it provided you some increased depth in your Python OO knowledge.