Sort Model Advancement-Wide&Deep

Rimeng Society

AI: Keras PyTorch MXNet TensorFlow PaddlePaddle Deep Learning Actual Warfare (Update from time to time)

8.5 Sort Model Advancement-Wide&Deep

Learning Objectives

  • target
    • nothing
  • application
    • nothing

8.5.1 wide&deep

  • Input characteristics of the Wide section:
    • Discrete characteristics
    • Combining discrete features
      • It is used in the paper of W&D without input having continuous value characteristics.
  • Input characteristics of the Deep section:

    • raw input+embeding processing
    • Embedding features other than discontinuous values is the policy feature, which is multiplied by an embedding-matrix.The interface inside TensorFlow is:tf.feature_column.embedding_column, default trainable=True.
    • The treatment of the continuous value feature is to compress it into [0,1] according to the cumulative distribution function P(X < x).

      Note: Training: notice: Wide part uses FTRL to train; Deep part uses AdaGrad to train.

  • The API interface of Wide&Deep in TensorFlow is:tf.estimator.DNNLinearCombinedClassifier

    • estimator = tf.estimator.DNNLinearCombinedClassifier()
      • model_dir="",
      • linear_feature_columns=wide_columns,
      • dnn_feature_columns=deep_columns
      • Dnn_Hidden_Units=[]: network structure of DNN layer

Tf.estimatorIncoming parameter principle

  • LinearClassifier and LinearRegressor: Accept all types of feature columns.
  • DNNClassifier and DNNRegressor: Only dense columns are accepted.Other types of columns must be encapsulated in indicator_column or embedding_column.
  • DNNLinearCombinedClassifier and DNNLinearCombinedRegressor:
    • Linear_Feature_The columns parameter accepts any type of feature column.
    • Dnn_Feature_The columns parameter accepts only dense columns.

Code:

import tensorflow as tf

class WDL(object):
    """wide&deep Model
    """
    def __init__(self):
        pass

    @staticmethod
    def read_ctr_records():
        # Define transformation functions, serialized at input
        def parse_tfrecords_function(example_proto):
            features = {
                "label": tf.FixedLenFeature([], tf.int64),
                "feature": tf.FixedLenFeature([], tf.string)
            }
            parsed_features = tf.parse_single_example(example_proto, features)

            feature = tf.decode_raw(parsed_features['feature'], tf.float64)
            feature = tf.reshape(tf.cast(feature, tf.float32), [1, 121])
            # Feature Order 1 channel_Id, 100 article_Vector, 10 user_Weights, 10 article_Weights
            # 1 channel_id-type features, 100-dimensional article vectors mean continuous features, 10-dimensional user weights mean continuous features
            channel_id = tf.cast(tf.slice(feature, [0, 0], [1, 1]), tf.int32)
            vector = tf.reduce_sum(tf.slice(feature, [0, 1], [1, 100]), axis=1)
            user_weights = tf.reduce_sum(tf.slice(feature, [0, 101], [1, 10]), axis=1)
            article_weights = tf.reduce_sum(tf.slice(feature, [0, 111], [1, 10]), axis=1)

            label = tf.cast(parsed_features['label'], tf.float32)

            # Construct Dictionary Name-tensor
            FEATURE_COLUMNS = ['channel_id', 'vector', 'user_weigths', 'article_weights']
            tensor_list = [channel_id, vector, user_weights, article_weights]

            feature_dict = dict(zip(FEATURE_COLUMNS, tensor_list))

            return feature_dict, label

        dataset = tf.data.TFRecordDataset(["./train_ctr_201905.tfrecords"])
        dataset = dataset.map(parse_tfrecords_function)
        dataset = dataset.batch(64)
        dataset = dataset.repeat()
        return dataset

    def build_estimator(self):
        """Modeling
        :param dataset:
        :return:
        """
        # Discrete Classification
        article_id = tf.feature_column.categorical_column_with_identity('channel_id', num_buckets=25)
        # Continuous type
        vector = tf.feature_column.numeric_column('vector')
        user_weigths = tf.feature_column.numeric_column('user_weigths')
        article_weights = tf.feature_column.numeric_column('article_weights')

        wide_columns = [article_id]

        # Embedding_A variable used by column to represent a type of category
        deep_columns = [tf.feature_column.embedding_column(article_id, dimension=25),
                        vector, user_weigths, article_weights]

        estimator = tf.estimator.DNNLinearCombinedClassifier(model_dir="./ckpt/wide_and_deep",
                                                             linear_feature_columns=wide_columns,
                                                             dnn_feature_columns=deep_columns,
                                                             dnn_hidden_units=[1024, 512, 256])

        return estimator


if __name__ == '__main__':
    wdl = WDL()
    estimator = wdl.build_estimator()
    estimator.train(input_fn=wdl.read_ctr_records)
    eval_result = estimator.evaluate(input_fn=wdl.read_ctr_records)
    print(eval_result)

8.5.2 Feature Data Processing Effect Comparison of Three Versions

Processing effects with different characteristics baseline 1 Separate Three Features, Mean Article Vector, Mean User Weight, Mean Article Weight 1 Discrete Feature, 1 111 Continuous Feature 1 Discrete Feature, 100 Continuous Article Vectors, 10 Article Weights, 10 User Weights
accuracy 0.9051438053097345 0.9046435 0.9046435 0.9046435
auc 0.719274521004087 0.57850575 0.5896939 0.62383443

Effect comparison summary:

  • The number of discrete data for black horse headlines is too small, so the underlying model can solve the problem
  • If the discrete or continuous characteristics increase, the use of WDL models will result in a certain degree of accuracy or AUC improvement.

Three Version Features Handle Data Functions and Build Models

  • First version:
@staticmethod
    def read_ctr_records_v1():
        # Define transformation functions, serialized at input
        def parse_tfrecords_function(example_proto):
            features = {
                "label": tf.FixedLenFeature([], tf.int64),
                "feature": tf.FixedLenFeature([], tf.string)
            }
            parsed_features = tf.parse_single_example(example_proto, features)

            feature = tf.decode_raw(parsed_features['feature'], tf.float64)
            feature = tf.reshape(tf.cast(feature, tf.float32), [1, 121])
            # Feature Order 1 channel_Id, 100 article_Vector, 10 user_Weights, 10 article_Weights
            # 1 channel_id-type features, 100-dimensional article vectors mean continuous features, 10-dimensional user weights mean continuous features
            channel_id = tf.cast(tf.slice(feature, [0, 0], [1, 1]), tf.int32)
            vector = tf.reduce_mean(tf.slice(feature, [0, 1], [1, 100]), axis=1)
            user_weights = tf.reduce_mean(tf.slice(feature, [0, 101], [1, 10]), axis=1)
            article_weights = tf.reduce_mean(tf.slice(feature, [0, 111], [1, 10]), axis=1)

            label = tf.cast(parsed_features['label'], tf.float32)

            # Construct Dictionary Name-tensor
            FEATURE_COLUMNS = ['channel_id', 'vector', 'user_weights', 'article_weights']
            tensor_list = [channel_id, vector, user_weights, article_weights]

            feature_dict = dict(zip(FEATURE_COLUMNS, tensor_list))

            return feature_dict, label

        dataset = tf.data.TFRecordDataset(["./ctr_train_20190706.tfrecords"])
        dataset = dataset.map(parse_tfrecords_function)
        dataset = dataset.batch(64)
        dataset = dataset.repeat(100)
        return dataset



    def build_estimator(self):
        """
        //Build a feature column to input into the model
        :return:
        """
        # Specify column characteristics
        channel_id = tf.feature_column.categorical_column_with_identity('channel_id', num_buckets=25)

        vector = tf.feature_column.numeric_column('vector')
        user_weights = tf.feature_column.numeric_column('user_weights')
        article_weights = tf.feature_column.numeric_column('article_weights')

        # wide side
        wide_columns = [channel_id]

        # deep side
        deep_columns = [
            tf.feature_column.embedding_column(channel_id, dimension=25),
            vector,
            user_weights,
            article_weights
        ]

        # Construct Model
        estimator = tf.estimator.DNNLinearCombinedClassifier(model_dir="./tmp/ckpt/wide_and_deep",
                                                             linear_feature_columns=wide_columns,
                                                             dnn_feature_columns=deep_columns,
                                                             dnn_hidden_units=[256, 128, 64])
        return estimator
  • Version 2:
@staticmethod
    def read_ctr_records_v2():
        # Define transformation functions, serialized at input
        def parse_tfrecords_function(example_proto):
            features = {
                "label": tf.FixedLenFeature([], tf.int64),
                "feature": tf.FixedLenFeature([], tf.string)
            }
            parsed_features = tf.parse_single_example(example_proto, features)

            feature = tf.decode_raw(parsed_features['feature'], tf.float64)
            feature = tf.reshape(tf.cast(feature, tf.float32), [1, 121])
            channel_id = tf.cast(tf.slice(feature, [0, 0], [1, 1]), tf.int32)

            label = tf.cast(parsed_features['label'], tf.float32)

            # Construct Dictionary Name-tensor
            FEATURE_COLUMNS = ['channel_id', 'feature']

            tensor_list = [channel_id, feature]

            feature_dict = dict(zip(FEATURE_COLUMNS, tensor_list))

            return feature_dict, label

        dataset = tf.data.TFRecordDataset(["./ctr_train_20190706.tfrecords"])
        dataset = dataset.map(parse_tfrecords_function)
        dataset = dataset.batch(64)
        dataset = dataset.repeat(100)
        return dataset

def build_estimator_v2(self):
        """
        //Build a feature column to input into the model
        :return:
        """
        # Specify column characteristics
        channel_id = tf.feature_column.categorical_column_with_identity('channel_id', num_buckets=25)

        feature = tf.feature_column.numeric_column('feature', shape=[1, 121])

        # wide side
        wide_columns = [channel_id]

        # deep side
        deep_columns = [
            tf.feature_column.embedding_column(channel_id, dimension=25),
            feature
        ]

        # Construct Model
        estimator = tf.estimator.DNNLinearCombinedClassifier(model_dir="./tmp/ckpt/wide_and_deep_v2",
                                                             linear_feature_columns=wide_columns,
                                                             dnn_feature_columns=deep_columns,
                                                             dnn_hidden_units=[256, 128, 64])
        return estimator
  • Third Version
@staticmethod
    def read_ctr_records_v3():
        # Define transformation functions, serialized at input
        def parse_tfrecords_function(example_proto):
            features = {
                "label": tf.FixedLenFeature([], tf.int64),
                "feature": tf.FixedLenFeature([], tf.string)
            }
            parsed_features = tf.parse_single_example(example_proto, features)

            feature = tf.decode_raw(parsed_features['feature'], tf.float64)
            feature = tf.reshape(tf.cast(feature, tf.float32), [1, 121])
            channel_id = tf.cast(tf.slice(feature, [0, 0], [1, 1]), tf.int32)
            vector = tf.slice(feature, [0, 1], [1, 100])
            user_weights = tf.slice(feature, [0, 101], [1, 10])
            article_weights = tf.slice(feature, [0, 111], [1, 10])

            label = tf.cast(parsed_features['label'], tf.float32)

            # Construct Dictionary Name-tensor
            FEATURE_COLUMNS = ['channel_id', 'vector', 'user_weights', 'article_weights']

            tensor_list = [channel_id, vector, user_weights, article_weights]

            feature_dict = dict(zip(FEATURE_COLUMNS, tensor_list))

            return feature_dict, label

        dataset = tf.data.TFRecordDataset(["./ctr_train_20190706.tfrecords"])
        dataset = dataset.map(parse_tfrecords_function)
        dataset = dataset.batch(64)
        dataset = dataset.repeat(100)
        return dataset


def build_estimator_v3(self):
        """
        //Build a feature column to input into the model
        :return:
        """
        # Specify column characteristics
        channel_id = tf.feature_column.categorical_column_with_identity('channel_id', num_buckets=25)

        vector = tf.feature_column.numeric_column('vector', shape=[1, 100])
        user_weights = tf.feature_column.numeric_column('user_weights', shape=[1, 10])
        article_weights = tf.feature_column.numeric_column('article_weights', shape=[1, 10])

        # wide side
        wide_columns = [channel_id]

        # deep side
        deep_columns = [
            tf.feature_column.embedding_column(channel_id, dimension=25),
            vector,
            user_weights,
            article_weights
        ]

        # Construct Model
        estimator = tf.estimator.DNNLinearCombinedClassifier(model_dir="./tmp/ckpt/wide_and_deep_v3",
                                                             linear_feature_columns=wide_columns,
                                                             dnn_feature_columns=deep_columns,
                                                             dnn_hidden_units=[256, 128, 64])
        return estimator

 

Keywords: network

Added by dsaba on Sun, 21 Jun 2020 06:35:34 +0300