__new__() in python

By : Akshar Raaj

Lately I started looking into Django code and wish to write about internals of Django. I started with Django models and will be writing about it soon. For understanding how Django models work, I had to understand what metaclasses are and how metaclasses work. Metaclasses use method "__new__" and so I looked at what "__new__" does.

As __new__ is a static method, we will see a lttle bit about static methods and then __new__ in detail.

  1. Understanding static methods.

  2. Understanding method "__new__" of any class. We will see how to override method __new__ in a class.

Also, I will be trying all the code we write here on Ipython and I suggest you to try everything on Ipython as well.

Static methods

A little bit about instance methods first. Let's write a class.

In [1]: class A(object):
   ...:     def met(self, a, b):
   ...:         print a, b
   ...:

In this case, met() is an instance method. So, it is expected that we pass an instance of A as the first argument to met.

Let's create an object and call met() on the created object and pass two arguments to met().

In [4]: obj = A()

In [5]: obj.met(1,2)
1 2                #output

What happened here?

When we called met(), we passed two arguments although met() expects three argument as per its definition. When we wrote obj.met(1, 2), interpreter took care of sending instance obj as the first argument to met() and 1 and 2 were passed as second and third arguments respectively.

Let's try calling met() without an instance or in other words let's call the method using class.

In [6]: A.met(1,2)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
/home/akshar/branding_git/netconference/<ipython-input-6-e8b323dba928> in <module>()
----> 1 A.met(1,2)

TypeError: unbound method met() must be called with A instance as first argument (got int instance instead)

We defined met() as an instance method of class A, so it expected an instance of A as the first argument. As its very clear from the error, met() expected an instance of A as the first argument but got an 'int' instead.

If we pass an instance of A as the first argument, it will work as expected.

In [7]: A.met(obj, 3, 4)
3 4                  #output

Notice that we called the method on class A and not on an instance of A. But we took care of sending an instance of A as the first argument to met() and it worked as expected.

Let's see static method now.

In [8]: class B(object):
   ...:     @staticmethod
   ...:     def met(a, b):
   ...:         print a,b
   ...:

What does @staticmethod above the method definition do?

Its a decorator which changes a method to static method. It means the method is no longer an instance method, which means that the method does not expect its first argument to be an instance. So, for our method definition, the method does not expect its first argument to be an instance of B. Even if we call the method on an instance of B, current instance will not be passed as the first argument to this method, since its a static method. For instance method that we saw earlier the current instance was passed as the first argument.

In [9]: B.met(5,6)
5 6                     #output

Here we were able to call the method on the class and were not required to pass an instance of B as the first argument.

Let's call this method on an instance.

In [10]: b = B()

In [11]: b.met(5,6)
5 6

Here we called the method on an instance and passed two arguments. Since its a static method, the current instance i.e b was not passed as the first argument to met(). Had it not been a static method, current instance would have been passed as first argument, 5 as second argument and 6 as third argument.

Understanding method __new__ of a class. We will also see how to override method __new__ in a class.

First let's see a little bit about __init__

In [13]: class A(object):
   ....:     def __init__(self, a, b):
   ....:         print "init gets called"
   ....:         print "self is", self
   ....:         self.a, self.b = a,b
   ....:

In __init__, we print something as we enter the method which is to validate that __init__ has been called. Then we print the first argument which is self and then we perform some assignments. __init__ is an instance method and expects the first argument to be an instance.

Let's call the class passing it two arguments. Keep in mind the part call the class, we are going to again use it in next few lines. If you have any confusion about the part calling the class, it will be clear in next few lines.

In [16]: a = A(1,2)
init gets called                                    #output
self is <__main__.A object at 0x3357210>            #output

Notice the second line of output which is "self is <__main__.A object at 0x3357210>". As apparent from second line of output, when __init__ is entered, object/instance has already been created by that time. Its only the assignment which is done in __init__, althought you could do some other stuff as well. But __init__ doesn't create the instance. __init__ receives the created instance as the first argument.

What creates the object?

Method __new__() creates the object.

What is __new__?

  1. __new__ is a static method which creates an instance. We will see the method signature soon. One reason i could think of having __new__ as a static method is because the instance has not been created yet when __new__ is called. So, they could not have had it as an instance method.
  2. __new__ gets called when you call the class. Call the class means issuing the statement "a=A(1,2)". Here A(1,2) is like calling the class. A is a class and we put two parenthesis in front of it and put some arguments between the parenthesis. So, its like "calling the class" similar to calling a method.
  3. __new__ must return the created object.
  4. Only when __new__ returns the created instance then __init__ gets called. If __new__ does not return an instance then __init__ would not be called. Remember __new__ is always called before __init__.
  5. __new__ gets passed all the arguments that we pass while calling the class. Also, it gets passed one extra argument that we will see soon.

How was the instance created in the last example when we didn't define __new__?

class A extends from object(Here we mean the class named object) i.e subclasses from object. object defines a method __new__, so A gets this method from object since its extending object. This inherited __new__ created the instance of A.

Method signature of __new__

__new__ receives the class whose instance need to be created as the first argument. This statement could be a little confusing, just continue reading and see the next example and again read it after seeing the example, it will be clear. The other arguments received by __new__ are same as what were passed while calling the class.

So, __new__ receives all the arguments that we pass while calling the class. Also, it receives one extra argument. This extra argument is the class whose instance need to be created and it will be received as first argument by __new__.

So, signature of __new__ could be written as:

__new__(cls, *args, **kwargs)

Let's see an example.

In [22]: class A(object):
   ....:     def __new__(cls, *args, **kwargs):
   ....:         print cls
   ....:         print "args is", args
   ....:         print "kwargs is", kwargs
   ....:

Here we override __new__ that we inherit from the superclass. We are printing all the arguments that this method receives so that we can check what gets passed to __new__. Let's try to create an instance of A by calling the class.

In [23]: a=A()
<class '__main__.A'>                              #output
args is ()                                        #output
kwargs is {}                                      #output

As we mentioned earlier, __new__ gets called when we call the class. As is apparent from the output __new__ was called and it printed three lines of output.

First line of output prints the first argument received by __new__. As we can see, it is class A itself. We tried to create an instance of A and __new__ of A received class A itself as the first argument. This is what we meant when we said "__new__ receives the class whose instance need to be created as the first argument". Now go back to the section "Method signature of __new__" and read it again.

While calling the class we did not pass any arguments. So our output shows that args and kwargs did not receive anything.

You can verify that all the arguments passed while calling the class gets sent to __new__. Just call the class passing it some arguments.

In [25]: a=A(1,2,named=5)
<class '__main__.A'>                              #output
args is (1, 2)                                    #output
kwargs is {'named': 5}                            #output

So, whatever arguments we passed while calling the class were passed to __new__ and were received by args and kwargs in __new__.

Let's check whether an object really gets created with how we have currently overridden __new__.

In [26]: a = A(1,2)
<class '__main__.A'>                              #output
args is (1, 2)                                    #output
kwargs is {}                                      #output

In [27]: print a
None                                              #output

We tried to create an instance and then tried printing the instance. But an instance of A was not created as apparent from the last print statement which printed None.

Why did this happen?

As we know if we don't return any value from a method, it implicitly returns None. Under the section "What is __new__", we mentioned that __new__ must return the created instance. Here we did not return the created instance from __new__, so None was implicitly returned and was assigned to a.

Let's combine __new__ and __init__.

In [29]: class A(object):
   ....:     def __new__(cls, *args, **kwargs):
   ....:         print cls
   ....:         print args
   ....:         print kwargs
   ....:     def __init__(self, a, b):
   ....:         print "init gets called"
   ....:         print "self is", self
   ....:         self.a, self.b = a, b
   ....:

Let's try to create an instance of A.

In [31]: a=A(1,2)
<class '__main__.A'>                              #output
(1, 2)                                            #output
{}                                                #output

As we mentioned earlier when a class gets called, first __new__ is called. Only when __new__ returns an instance then __init__ is called.

In our previous example __new__ did not return an instance. So __init__ was not called. Had __init__ been called we would have seen the print statements that we have inside __init__.

Also since __new__ did not return an instance, a will still be None. Verify that.

In [32]: print a
None                                              #output

Let's redefine the class to make it proper. We should return an instance from __new__, so that __init__ gets called and we get the desired behaviour.

In case we don't override __new__, __new__ of parent class creates the instance and then __init__ gets called. In case we are overriding __new__, we should call the __new__ of parent class to get the created instance. However if you know how object creation works at the low level and you can implement it in your overridden __new__, you don't need to call parent __new__ to get the created instance. I don't know such details of how object creation work and will use the parent __new__ to get the created instance.

Once we get the created instance we can perform any extra operations we wish before returning the instance from __new__ method.

For demonstration purpose, let us take a weird example where we need to add an attribute named 'created_at' to the created instance. For our case lets consider it needs to be done inside the __new__ method, althought we could have done it inside __init__.

In [33]: import datetime

In [35]: class A(object):
   ....:     def __new__(cls, *args, **kwargs):
   ....:         new_instance = object.__new__(cls, *args, **kwargs)
   ....:         setattr(new_instance, 'created_at', datetime.datetime.now())
   ....:         return new_instance
   ....:     def __init__(self, a, b):
   ....:         print "inside init"
   ....:         self.a, self.b = a, b
   ....:

In the first line of __new__, we called the __new__ of parent class to get the created instance. __new__ of parent class should be passed the same arguments that we received in the overridden __new__. __new__ of parent class i.e __new__ of class object knows how to create an instance and it returns the created instance.

In the next line, we used inbuilt method setattr() to set an attribute 'created_at' on the newly created instance. The value we set for this attribute is the current time. This line is equivalent to writing new_instance.created_at=datetime.datetime.now().

In the final line we returned the newly created instance. Since we are returning an instance from __new__, __init__ will be called passing it whatever arguments were used in the class call. Let's verify this.

In [36]: obj1 = A(1,2)
inside init                                    #output

This statement suggests that __init__ was called. Let's print the created instance.

In [37]: obj1
Out[37]: <__main__.A at 0x3357390>

Notice that earlier when we were not returning anything from __new__ and were trying to print it, we were getting output as None. But this time the output shows that obj1 refers to an instance of A.

We can verify that obj1 has an attribute 'created_at' and __init__ was properly executed by printing the three attributes of obj1.

In [37]: print obj1.created_at
2012-06-09 22:44:30.376914                    #output

In [38]: print obj1.a, obj1.b
1 2                                           #output

Let's see our final example.

In [60]: class B(object):
   ....:     pass
   ....:

In [61]: class A(object):
   ....:     def __new__(cls, *args, **kwargs):
   ....:         new_instance = object.__new__(B, *args, **kwargs)
   ....:         return new_instance
   ....:

Pay attention to first line of A's new. Instead of passing cls as the first argument to object.__new__, we pass class B as first argument. Let's see what happens in such case.

In [62]: a = A()

In [63]: print a
<__main__.B object at 0x7f912c036750>                    #Output. Tried creating an instance of A but got an instance of B

We tried to create an instance of A. But when we printed it, we realise that an instance of B has been created.

This happened because we passed class B as the first argument to object.__new__. This shows that whatever class we pass to superclass' __new__, an instance of that class will be created.

Remember __new__ receives the class whose instance need to be created as first argument. So for any __new__, the first argument (which is cls for our case) will always refer to the class inside which __new__ is defined. So, for our case, cls will be class A.

Here we wanted to create an instance of A. So, class A must be passed as first argument to object.__new__. Inside __new__ of class A, cls refers to class A. So, we need to pass cls as first argument to object.__new__.

That's why if we want proper behaviour we need to pass the same arguments to the superclass' __new__ as it was received by the overridden __new__.

We can make that single line change in A's __new__ and our code will behave as expected.

In [64]: class A(object):
   ....:     def __new__(cls, *args, **kwargs):
   ....:         new_instance = object.__new__(cls, *args, **kwargs)
   ....:         return new_instance
   ....:

In [65]: a=A()

In [66]: print a
<__main__.A object at 0x7f912c0368d0>                    #Output. We got an instance of A

That was all about method __new__. Hopefully next post would be about metaclasses and there we can see some more useful uses of __new__.

Hope you liked the post.


Related Posts


Can we help you build amazing apps? Contact us today.

Comments

Matthew Schinckel

Note: `__new__` is not a static method, it is a classmethod.

This is somewhere between a static method and an instance method. It means the first argument would be the class object.

It also (IIRC) means you can use `super` within `__new__`, and things will work as expected.

commmenttor
Akshar Raaj 11th June, 2012

@Matthew:

As i could understand it, __new__ is a static method.
Please see:
http://docs.python.org/reference/datamodel.html#object.__new__

http://stackoverflow.com/questions/9092072/why-isnt-new-in-python-new-style-classes-a-class-method

http://www.python.org/download/releases/2.2/descrintro/#__new__

We can use super within new. We could have written "new_instance = super(A, cls).__new__(cls, *args, **kwargs)"
in place of
"new_instance = object.__new__(cls, *args, **kwargs)".

For class methods, calling the method on the class implicitly sends the current class as the first argument. An example:

class A(object):
@classmethod
def met(cls, a):
print a

We can call this method on the class passing it only one argument and the class will implicitly be sent as the first argument.

A.met(5)

This works.

But when we override __new__, we have to explicitly send cls received in the overridden __new__ as the first argument. I mean we need to write:

super(A, cls).__new__(cls, *args, **kwargs)

Had it been a class method, this would have worked:

super(A, cls).__new__(*args, **kwargs)

because the class on which it were called would have been implicitly passed as first argument since it were a class method. But it doesn't work.

That's why i feel its a static method.

Let me know if i am wrong somewhere.

commmenttor
navaneethan

This is the fantastic explanation..I admired..Keep going :)

commmenttor
Matthew Schinckel

@Akshar: Yes, it would seem I was wrong :)

The funny thing is I had been doing stuff with `__new__` (and `__metaclass__`) recently, and had written code which uses the correct syntax (see http://schinckel.net/2012/06/11/metaclass-magic-registry-pattern/), where you need to explicitly pass in the class object.

Regards.

commmenttor
Oskar 16th June, 2012

Great article! Can't wait for the next one.

commmenttor
Luke Lee

Great post. I linked/commented on it here, http://codrspace.com/durden/good-discussion-of-python-instance-static-methods/.

Is there a reason you chose to use setattr() instead of the 'normal' assignment in your overridden __new__? I know you mentioned these are equivalents in the article. I was just curious of there was some type of advantage to using setattr at this point in object creation etc.?

commmenttor
Akshar Raaj 20th June, 2012

@Luke

No, at this point there is no advantage in using setattr() instead of normal assignments. We could have used normal assignment.

However you may find cases where its not possible to get done with normal assignment and you will be forced to use setattr.

commmenttor
Vishal

Painstakingly detailed. Loved it.

In the following code, I don't see __init__ of class A and class B working.

Could you help me understand this problem?

#############################################

class B(object):
def __new__(cls, *args, **kwargs):
new_instance = object.__new__(cls, *args, **kwargs)
print "in Class B:"
print new_instance
return new_instance

def __init__(self, a):
print "Init of B"

def fun(self):
print "B: hello"

class A(object):
def __new__(cls, *args, **kwargs):
new_instance = object.__new__(B, *args, **kwargs)
print "in Class A:"
print new_instance
return new_instance

def __init__(self, a):
print "Init of A"

def fun(self):
print "A: hello"

class C(object):
def __new__(cls, *args, **kwargs):
new_instance = object.__new__(cls, *args, **kwargs)
return new_instance

def __init__(self):
print "Init of C"

obj = A(1)
obj.fun()

obj2 = C()

commmenttor
Akshar Raaj 7th July, 2012

@Vishal: Glad you liked it.

First, __init__ of B should be working fine. Please check it once more.

__init__ of A doesn't work. As you can see, inside __new__ of A, you are passing B as the first argument to object.__new__. So, the created instance i.e 'new_instance' is an instance of B. So, from __new__ of A, you are returning an instance of B. So, __init__ of A is not called. If you want __init__ of A to be called, you MUST return an instance of A from __new__
of A.
As you would be already knowing, if you change object.__new__(B, *args, **kwargs) inside __new__ of A to object.__new__(A, *args, **kwargs) it would work as you expect. You can also change it to object.__new__(cls, *args, **kwargs) since cls refers to A inside __new__ of A.

Since __new__ of A did not return an instance of A so __init__ of A was not called. Still an instance of B was created and returned from __new__ of A. So this created instance of B was assigned to variable obj. So, in next line when you try obj.fun(), fun() of B is called since obj refers to an instance of B.

Hope it helps you understand the problem.

commmenttor
foo 21st July, 2012

excellent explanation!

commmenttor
Zev Griner 3rd Oct., 2012

Thanks for your step-by-step explanation of the __new__ mechanics. Other docs keep talking about returning an instance but never explain where the instance comes from, nor how it gets associated with the subclass.

Is it correct to say that Python's way of creating a subclass instance is to simply create the instance at the highest super class level, associate it with the lowest subclass, assign attributes and methods at each super level and return the instance to the caller, so that as the instance filters downward, attributes and methods can get overridden at each level?

Thanks.

commmenttor
shiva 6th Dec., 2012

Pretty Good stuff and very useful to understand the basics

commmenttor
Kripasindhu Sarkar 20th Dec., 2012

Same thing happened to me. Started learning Django; got confused how the Models are working; bumped into metaclass; and them bumped into __new__.

Nice explanation dude!

commmenttor
testdomain

That's interesting, Youre an excessively professional blogger Ive joined your rss feed and sit up for in search of extra of your fantastic post Additionally, Ive shared your site at my internet sites!

commmenttor
Melissa 6th Aug., 2013

The exact explanation I was looking for! Thank you for taking the time to write this with such great detail and use of simple examples :)

commmenttor
Sahil Jamwal 3rd Oct., 2013

Ultimate explanation! That is what i was looking for.I am totally impressed.:)

commmenttor
Chris Kavanagh 25th Dec., 2013

Thanks so much for this post, it helped me greatly in understanding several points I was unclear of.

commmenttor
Yasin Gürhan

hi Akshar;

thank you for this case..

but i dont understand it :
codes :

>>> class A(object):
def __new__(cls,*args,**kwargs):
new_instance = object.__new__(cls,*args,**kwargs)
setattr(new_instance, 'created_at', datetime.datetime.now())
return new_instance
def __init__(self, a, b):
print("inside init")
self.a, self.b = a, b


>>> b=A(1,2)
Traceback (most recent call last):
File "<pyshell#15>", line 1, in
b=A(1,2)
File "<pyshell#14>", line 3, in __new__
new_instance = object.__new__(cls,*args,**kwargs)
TypeError: object() takes no parameters

or

b=A()
Traceback (most recent call last):
File "<pyshell#16>", line 1, in
b=A()
TypeError: __init__() missing 2 required positional arguments: 'a' and 'b'

what this reasons ?
please help me..

commmenttor
Yasin Gürhan

it is work but why?
what is different __new__ and _new_
i try _new_ in code and it work.. why?

class B(object):
def _new_(cls,*a,**b):
ne = object._new_(cls,*a,**b)
setattr(ne,"created at",datetime.datetime.now())
return ne
def __init__(self,x,y):
print("initteyiz")
self.x=x
self.y=y


>>> e=B(1,2)
initteyiz
>>> print(e)
<__main__.B object at 0x000000000337A518>

commmenttor
rajiv 10th March, 2014

nice one :) Good work!!

commmenttor
satish goda

Really well written tutorial. Thank you :)

commmenttor
Nilesh Sutar 12th May, 2014

Very well explained !!!
Just one query As __new__ method takes first argument as class whose instance needs to be created but you have wrote __new__ is static method. This is just confusing me :(.

commmenttor
© Agiliq, 2009-2012