Using metaclasses

type()

The biggest difference between dynamic and static languages is the definition of functions and classes, which is not defined at compile time, but created dynamically at run time.

For example, if we want to define a Hello class, we write a hello.py module:

class Hello(object):
    def hello(self, name='world'):
        print('Hello, %s.' % name)

When the Python interpreter loads the Hello module, it executes all the statements of the module in turn. The result of execution is to create a Hello class object dynamically. The test is as follows:

>>> from hello import Hello
>>> h = Hello()
>>> h.hello()
Hello, world.
>>> print(type(Hello))
<class 'type'>
>>> print(type(h))
<class 'hello.Hello'>

The type() function can look at the type of a type or variable, Hello is a class, its type is type, and h is an instance, its type is class Hello.

We say that the definition of class is created dynamically at runtime, and the way to create class is to use the type() function.

The type() function can return the type of an object and create new types. For example, we can create Hello classes through the type() function without class Hello(object)... Definition:

>>> def fn(self, name='world'): # Define function first
...     print('Hello, %s.' % name)
...
>>> Hello = type('Hello', (object,), dict(hello=fn)) # Create Hello class
>>> h = Hello()
>>> h.hello()
Hello, world.
>>> print(type(Hello))
<class 'type'>
>>> print(type(h))
<class '__main__.Hello'>

To create a class object, the type() function passes in three parameters in turn:

1. The name of the class;
2. Inheritance of the parent class set, note that Python supports multiple inheritance, if there is only one parent class, do not forget tuple single element writing;
3. The method name of class is bound to the function. Here we bind the function fn to the method name hello.

The class created by the type() function is exactly the same as writing the class directly, because when the Python interpreter encounters the class definition, it simply scans the grammar of the class definition and then calls the type() function to create the class.

Normally, we use class Xxx... To define classes, however, the type() function also allows us to create classes dynamically. That is to say, the dynamic language itself supports the dynamic creation of classes at runtime, which is very different from the static language. To create classes at runtime of the static language, we must construct source code strings and then call the compiler, or use some tools. Generating bytecode implementations, essentially dynamic compilation, can be very complex.

metaclass

In addition to using type() to create classes dynamically, metaclass can also be used to control the creation behavior of classes.

Metaclass, literally translated into metaclass, is simply interpreted as:

When we define a class, we can create an instance based on that class, so we define the class first, and then create the instance.

But what if we want to create classes? Then you have to create classes based on metaclasses, so first define metaclasses, and then create classes.

Connecting is: first define metaclass, you can create classes, and finally create instances.

So metaclass allows you to create or modify classes. In other words, you can think of classes as "instances" created by metaclasses.

Metaclass is the most difficult to understand and use magic code in Python object-oriented. Normally, you don't have to use metaclass, so it doesn't matter if you don't understand the following, because basically you won't use it.

Let's start with a simple example. This metaclass can add an add method to our custom MyList:

Define ListMetaclass. By default, the class name of metaclass always ends with Metaclass to make it clear that it is a metaclass:

# metaclass is a template for a class, so it must be derived from `type':
class ListMetaclass(type):
    def __new__(cls, name, bases, attrs):
        attrs['add'] = lambda self, value: self.append(value)
        return type.__new__(cls, name, bases, attrs)

With ListMetaclass, when we define a class, we also instruct to use ListMetaclass to customize the class and pass in the keyword parameter metaclass:

class MyList(list, metaclass=ListMetaclass):
    pass

The magic works when we pass in the keyword parameter metaclass, which instructs the Python interpreter to create MyList through ListMetaclass.new(). Here, we can modify the definition of the class, for example, by adding a new method, and then returning the modified definition.

The parameters received by the new() method are in turn:

1. The object of the class currently being created;

2. Class name;

3. A collection of parent classes inherited by a class;

4. A collection of methods of a class.

Test whether MyList can call the add() method:

>>> L = MyList()
>>> L.add(1)
>> L
[1]

The normal list has no add() method:

>>> L2 = list()
>>> L2.add(1)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'list' object has no attribute 'add'

What is the significance of dynamic modification? Isn't it easier to write the add() method directly in the MyList definition? Normally, it should be written directly, and metaclass modification is pure perversion.

However, there is always a need to modify class definitions through metaclass. ORM is a typical example.

ORM is called "Object Relational Mapping", that is, object-relational mapping, which maps a row of relational database into an object, that is, a class corresponds to a table. In this way, it is easier to write code without directly operating SQL statements.

To write an ORM framework, all classes can only be dynamically defined, because only users can define the corresponding classes according to the structure of the table.

Let's try to write an ORM framework.

The first step in writing the underlying module is to write the calling interface first. For example, if a user uses this ORM framework and wants to define a User class to operate on the corresponding database table User, we expect him to write such code:

class User(Model):
    # Define the mapping of class attributes to columns:
    id = IntegerField('id')
    name = StringField('username')
    email = StringField('email')
    password = StringField('password')

# Create an instance:
u = User(id=12345, name='Michael', email='test@orm.org', password='my-pwd')
# Save to the database:
u.save()

Among them, the parent class Model and attribute types StringField and IntegerField are provided by ORM framework, and the remaining magic methods such as save() are all automated by metaclass. Although metaclass is complex to write, ORM users are extremely simple to use.

Now, we implement the ORM according to the above interface.

First, define the Field class, which is responsible for saving the field names and field types of database tables:

class Field(object):

    def __init__(self, name, column_type):
        self.name = name
        self.column_type = column_type

    def __str__(self):
        return '<%s:%s>' % (self.__class__.__name__, self.name)

On the basis of Field, we further define various types of Field, such as String Field, IntegerField and so on.

class StringField(Field):

    def __init__(self, name):
        super(StringField, self).__init__(name, 'varchar(100)')

class IntegerField(Field):

    def __init__(self, name):
        super(IntegerField, self).__init__(name, 'bigint')

The next step is to write the most complex Model Metaclass:

class ModelMetaclass(type):

    def __new__(cls, name, bases, attrs):
        if name=='Model':
            return type.__new__(cls, name, bases, attrs)
        print('Found model: %s' % name)
        mappings = dict()
        for k, v in attrs.items():
            if isinstance(v, Field):
                print('Found mapping: %s ==> %s' % (k, v))
                mappings[k] = v
        for k in mappings.keys():
            attrs.pop(k)
        attrs['__mappings__'] = mappings # Save mapping relationships between attributes and columns
        attrs['__table__'] = name # Assume that the table and class names are identical
        return type.__new__(cls, name, bases, attrs)

And the base class Model:

class Model(dict, metaclass=ModelMetaclass):

    def __init__(self, **kw):
        super(Model, self).__init__(**kw)

    def __getattr__(self, key):
        try:
            return self[key]
        except KeyError:
            raise AttributeError(r"'Model' object has no attribute '%s'" % key)

    def __setattr__(self, key, value):
        self[key] = value

    def save(self):
        fields = []
        params = []
        args = []
        for k, v in self.__mappings__.items():
            fields.append(v.name)
            params.append('?')
            args.append(getattr(self, k, None))
        sql = 'insert into %s (%s) values (%s)' % (self.__table__, ','.join(fields), ','.join(params))
        print('SQL: %s' % sql)
        print('ARGS: %s' % str(args))

When a user defines a class User(Model), the Python interpreter first looks for metaclass in the definition of the current class User, and if it does not find it, it continues to look for metaclass in the parent class Model. When it finds it, it uses the ModelMetaclass defined in the Model to create the User class. That is to say, metaclass can be implicitly created. Inheritance to subclasses, but subclasses themselves do not feel.

In ModelMetaclass, several things have been done:

1. Exclude the modification of Model class.

2. Find all attributes of the defined class in the current class (e.g. User). If a Field attribute is found, save it in a dict Of _mappings_ and delete the Field attribute from the class attribute. Otherwise, runtime errors are likely to occur (the attributes of instances will obscure the same-name attributes of the class).

3. Save the table name in _table_, which is simplified as table name defaults to class name.

In the Model class, you can define various methods to manipulate the database, such as save(), delete(), find(), update, and so on.

We implement the save() method to save an instance to a database. Because of the table name, the mapping of attributes to fields, and the set of attribute values, INSERT statements can be constructed.

Write code to try:

u = User(id=12345, name='Michael', email='test@orm.org', password='my-pwd')
u.save()

The output is as follows:

Found model: User
Found mapping: email ==> <StringField:email>
Found mapping: password ==> <StringField:password>
Found mapping: id ==> <IntegerField:uid>
Found mapping: name ==> <StringField:username>
SQL: insert into User (password,email,username,id) values (?,?,?,?)
ARGS: ['my-pwd', 'test@orm.org', 'Michael', 12345]

As you can see, the save() method has printed out executable SQL statements and parameter lists, and it only needs to connect to the database and execute the SQL statement to complete the real function.

Less than 100 lines of code, we implemented a streamlined ORM framework through metaclass, is it very simple?

Summary

metaclass is a very magical object in Python, which can change the behavior of class creation. This powerful function must be used with caution.

Reference source code

create_class_on_the_fly.py

#!/usr/bin/env python3
# -*- coding: utf-8 -*-

def fn(self, name='world'): # Define function first
    print('Hello, %s.' % name)

Hello = type('Hello', (object,), dict(hello=fn)) # Create Hello class

h = Hello()
print('call h.hello():')
h.hello()
print('type(Hello) =', type(Hello))
print('type(h) =', type(h))

use_metaclass.py

#!/usr/bin/env python3
# -*- coding: utf-8 -*-

# metaclass is the creation class, so it must be derived from `type':
class ListMetaclass(type):
    def __new__(cls, name, bases, attrs):
        attrs['add'] = lambda self, value: self.append(value)
        return type.__new__(cls, name, bases, attrs)

# Indicates the use of ListMetaclass to customize classes
class MyList(list, metaclass=ListMetaclass):
    pass

L = MyList()
L.add(1)
L.add(2)
L.add(3)
L.add('END')
print(L)

orm.py

#!/usr/bin/env python3
# -*- coding: utf-8 -*-

' Simple ORM using metaclass '

class Field(object):

    def __init__(self, name, column_type):
        self.name = name
        self.column_type = column_type

    def __str__(self):
        return '<%s:%s>' % (self.__class__.__name__, self.name)

class StringField(Field):

    def __init__(self, name):
        super(StringField, self).__init__(name, 'varchar(100)')

class IntegerField(Field):

    def __init__(self, name):
        super(IntegerField, self).__init__(name, 'bigint')

class ModelMetaclass(type):

    def __new__(cls, name, bases, attrs):
        if name=='Model':
            return type.__new__(cls, name, bases, attrs)
        print('Found model: %s' % name)
        mappings = dict()
        for k, v in attrs.items():
            if isinstance(v, Field):
                print('Found mapping: %s ==> %s' % (k, v))
                mappings[k] = v
        for k in mappings.keys():
            attrs.pop(k)
        attrs['__mappings__'] = mappings # Save mapping relationships between attributes and columns
        attrs['__table__'] = name # Assume that the table and class names are identical
        return type.__new__(cls, name, bases, attrs)

class Model(dict, metaclass=ModelMetaclass):

    def __init__(self, **kw):
        super(Model, self).__init__(**kw)

    def __getattr__(self, key):
        try:
            return self[key]
        except KeyError:
            raise AttributeError(r"'Model' object has no attribute '%s'" % key)

    def __setattr__(self, key, value):
        self[key] = value

    def save(self):
        fields = []
        params = []
        args = []
        for k, v in self.__mappings__.items():
            fields.append(v.name)
            params.append('?')
            args.append(getattr(self, k, None))
        sql = 'insert into %s (%s) values (%s)' % (self.__table__, ','.join(fields), ','.join(params))
        print('SQL: %s' % sql)
        print('ARGS: %s' % str(args))

# testing code:

class User(Model):
    id = IntegerField('id')
    name = StringField('username')
    email = StringField('email')
    password = StringField('password')

u = User(id=12345, name='Michael', email='test@orm.org', password='my-pwd')
u.save()