Exploratory Data Analysis EDA (Exploratory Data Analysis) analysis with python

Exploratory Data Analysis EDA (Exploratory Data Analysis) analysis with python


show holy respect to python community, for there dedication and wisdom


Dataset related:

First, UCL wine dataset:

UCI data set is a commonly used standard test data set for machine learning. It is a database for machine learning proposed by the University of California Irvine. UCI data sets are mostly used in the testing of machine learning algorithms. The important thing is the word "standard". The newly compiled machine learning programs can be tested with UCI data sets, and similar machine learning algorithms can also be compared. Its official website address is as follows:
website: UCI Machine Learning Repository

Field related:

Fixed acidity: most wine related acids are either fixed or nonvolatile (not easy to evaporate)
Volatile sour taste: the content of acetic acid in wine is too high, which will produce unpleasant vinegar taste
Citric acid: a small amount of citric acid can increase the freshness and flavor of wine
Residual sugar: the amount of residual sugar after fermentation. There is little wine below 1g per liter, and wine above 45g is considered sweet
Chloride: the content of salt in wine
Free sulfur dioxide: SO2 exists in the equilibrium state between SO2 molecule (as dissolved gas) and bisulfite ion in free form; It can prevent microbial growth and oxidation in wine
Total sulfur dioxide: amount of free and bound SO2; At low concentration, SO2 can hardly be detected in wine, but when the concentration of free SO2 exceeds 50ppm, SO2 will become obvious in the sense of smell and taste of wine
Density of water and sugar
PH value: describes the degree of acidity or alkalinity of wine, from 0 (very acidic) to 14 (very alkaline); Most wines have a pH between 3 and 4
Sulfate: a wine additive that increases the level of sulfur dioxide (SO2) and acts as an antibacterial and antioxidant
Alcohol: the percentage of alcohol in wine
Quality: output variable (score 0 - 10 according to sensory data), with the occupation of special wine appraiser and bartender

Second, the Kaggle Titanic dataset:

The sinking of the Titanic is one of the most well-known shipwrecks in history. On April 15, 1912, during her maiden voyage, the Titanic sank after colliding with an iceberg, killing 1502 of the 2224 passengers and crew on board. This sensational tragedy shocked the international community and promoted the improvement of ship safety regulations.

One of the reasons for the shipwreck is that the passengers and crew did not have enough lifeboats. Although there are some luck factors in surviving the shipwreck, some people are more likely to survive than others, such as women, children and upper class society.

In this challenge, it is required to complete the analysis of who is likely to survive. In particular, machine learning tools are required to predict which passengers will survive the tragedy.


Field related:

passengerid: Passenger ID
Class: class (1 = 1st, 2 = 2nd, 3 = 3rd)**
Name: passenger name
sex: Gender
Age: age
sibsp: number of siblings / spouses on board
parch: number of parents / children on board
Ticket: ticket information
fare: fare
Cabin: cabin
embarked: port of embarkation (C = Cherbourg, Q = Queenstown, S = Southampton)
survived: the variable is predicted to be 0 or 1 (where 1 means survival and 0 means death)



Related to drawing tools:








#Import related packages

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

#Draw nominal variable diagram

def plot_categoricals(x, y, data, annotate = True):
    """Plot counts of two categoricals.
    Size is raw count for each grouping.
    Percentages are for a given value of y."""
    # dict vectorizer
    # Raw counts 
    raw_counts = pd.DataFrame(data.groupby(y)[x].value_counts(normalize = False))
    raw_counts = raw_counts.rename(columns = {x: 'raw_count'})
    # Calculate counts for each group of x and y
    counts = pd.DataFrame(data.groupby(y)[x].value_counts(normalize = True))
    # Rename the column and reset the index
    counts = counts.rename(columns = {x: 'normalized_count'}).reset_index()
    counts['percent'] = 100 * counts['normalized_count']
    # Add the raw count
    counts['raw_count'] = list(raw_counts['raw_count'])
    plt.figure(figsize = (14, 10))
    # Scatter plot sized by percent
    plt.scatter(counts[x], counts[y], edgecolor = 'k', color = 'lightgreen',
                s = 100 * np.sqrt(counts['raw_count']), marker = 'o',
                alpha = 0.6, linewidth = 1.5)
    if annotate:
        # Annotate the plot with text
        for i, row in counts.iterrows():
            # Put text with appropriate offsets
            plt.annotate(xy = (row[x] - (1 / counts[x].nunique()), 
                               row[y] - (0.15 / counts[y].nunique())),
                         color = 'navy',
                         s = f"{round(row['percent'], 1)}%")
    # Set tick marks
    # Transform min and max to evenly space in square root domain
    sqr_min = int(np.sqrt(raw_counts['raw_count'].min()))
    sqr_max = int(np.sqrt(raw_counts['raw_count'].max()))
    # 5 sizes for legend
    msizes = list(range(sqr_min, sqr_max,
                        int(( sqr_max - sqr_min) / 5)))
    markers = []
    # Markers for legend
    for size in msizes:
        markers.append(plt.scatter([], [], s = 100 * size, 
                                   label = f'{int(round(np.square(size) / 100) * 100)}', 
                                   color = 'lightgreen',
                                   alpha = 0.6, edgecolor = 'k', linewidth = 1.5))
    # Legend and formatting
    plt.legend(handles = markers, title = 'Counts',
               labelspacing = 3, handletextpad = 2,
               fontsize = 16,
               loc = (1.10, 0.19))
    plt.annotate(f'* Size represents raw count while % is for a given y value.',
                 xy = (0, 1), xycoords = 'figure points', size = 10)
    # Adjust axes limits
    plt.xlim((counts[x].min() - (6 / counts[x].nunique()), 
              counts[x].max() + (6 / counts[x].nunique())))
    plt.ylim((counts[y].min() - (4 / counts[y].nunique()), 
              counts[y].max() + (4 / counts[y].nunique())))
    plt.xlabel(f"{x}"); plt.ylabel(f"{y}"); plt.title(f"{y} vs {x}");

#Import data and view it

df = pd.read_csv('winequality-white.csv', sep=';')


#Check for missing values:

# Check if any of the following is NULL

#Use the heat map to see how worthwhile it really is

sns.heatmap(df.isnull(), cbar=False, yticklabels=False, cmap='viridis')

#Check the number and total number of unique values of some nominal variables


#View data type and column name


#Get statistics of continuous variables


#Draw histogram:

df['fixed acidity'].plot(kind = 'hist',figsize=(20, 7), )

#Draw density map

#Draw box diagram

import seaborn as sns
plt.figure(figsize=(10, 7))


#Draw the box diagram and erect the image

import seaborn as sns
plt.figure(figsize=(10, 7))
sns.boxplot(data = df,y='alcohol',)

#scatter plot

import matplotlib.pyplot as plt
fig, ax = plt.subplots(figsize=(16,8))
ax.scatter(df['volatile acidity'] , df['citric acid'])
ax.set_xlabel('volatile acidity')
ax.set_ylabel('citric acid')

#Regression mapping

plt.figure(figsize=(25, 7))
sns.regplot(x="alcohol", y="density", data=df);

#Histogram drawing

def bar_plot(df,key):
    df[key].value_counts().sort_index().plot.bar(figsize = (12, 5),
                      edgecolor = 'k', linewidth = 2)

    # Formatting
    plt.xticks(rotation = 60)
    plt.title('BAR PLOT for ' + key);


#Multi box drawing

def plot_box_plot2(df,key,value):
    import copy
    box = copy.deepcopy(df)
    box[value] = box[value].astype('float')
    fig,axes=plt.subplots(1,1,figsize = (25,7))
    sns.boxplot(data = box, x=key, y=value)

plot_box_plot2(df,'quality','citric acid')

plt.figure(figsize=(25, 7))
# plt.style.use('seaborn-white')
ax = sns.boxplot(x="quality", y="free sulfur dioxide", data=df)

#Drawing violin

plt.figure(figsize=(25, 7))

# Draw a nested violinplot and split the violins for easier comparison
sns.violinplot(data=df, x="quality", y="density", 
               split=True, inner="quart", linewidth=1,)


#Draw correlation diagram:


#Draw a correlation diagram and display the correlation value

sns.heatmap(df.corr(), color='b', annot=True)

#Draw the statistical chart of nominal variables

sns.countplot(x='quality', data = df)

#Draw a box diagram of all variables:


for i, col in enumerate(list(df.columns.values)):

#Draw a histogram of all variables


for i,col in enumerate(list(df.columns.values)):
    sns.distplot(df[col], color='b', kde=True, label='data')

# pair plot

sns.pairplot(data=df, kind='scatter',diag_kind='kde')


#Discretize the variables and construct the relationship diagram of discrete variables

df['alcohol_bin'] = pd.cut(df.alcohol,bins=[7,9,11,13,15],labels=['low','mid_low','mid_high','high'])
# df.insert(5,'Age Group',category)

df['alcohol_value'] = pd.cut(df.alcohol,bins=[7,9,11,13,15],labels=[1,2,3,4])
# df.insert(5,'Age Group',category)

plot_categoricals('alcohol_value', 'quality', df, annotate = True)



#Titanic dataset


#Load data



# df.tail()





#Get statistics


#By default, describe only outputs the information of continuous variables, so I want to see the statistics of data of other variable types:


#View statistics (single column)



df[(df['survived'] == 1) & (df['pclass'] == 1)]['age'].max()

df[df['name'].apply(lambda name: name[0] == 'A')].head()


df[df['name'].apply(lambda name: name[0] == 'A')].head()

#replace function:

x = {1 : 'Class I', 2 : 'Class II', 3:'Class III'}
df_new=df.replace({'pclass': x})

#Get aggregate information (group by)

#Contingency table information

pd.crosstab(df['survived'], df['pclass'])
pd.crosstab(df['survived'], df['sex'], margins=True)

#pivot table


#Data sorting (based on a specific field)

df.sort_values(by=["fare"], ascending=False).head()
df.sort_values(by=["fare"], ascending=False).tail()

#Visualization of missing values

import seaborn as sns
plt.rcParams['figure.dpi'] = 100# the dpi can be set to enhance the resolution of the image
# Congiguring retina format
%config InlineBackend.figure_format = 'retina'
sns.heatmap(df.isnull(), cmap='viridis',yticklabels=False)

#Survival statistics


#Survival of different genders

sns.countplot(data =df, x = 'survived',hue = 'sex')

#Different survival statistics of class registration

sns.countplot(data = df , x = 'survived', hue='pclass')

#Cross table of class and survival

pd.crosstab(df['survived'], df['pclass'], margins=True)

#Class counting statistics


#Draw age histogram and density map

plt.figure(figsize=(20, 7))
sns.distplot(df.age, color='purple')

#Draw age histogram and density map (remove missing values)

plt.figure(figsize=(20, 7))

#Cost density map

plt.figure(figsize=(20, 7))
sns.distplot(df.fare, color='green')

#Draw the box diagram and violin diagram of the cost:

plt.figure(figsize=(20, 7))

sns.boxplot(data = df, y='fare',orient = 'v')
sns.violinplot(data = df, y='fare',orient = 'v')
#(Q1−1.5⋅IQR, Q3+1.5⋅IQR) 

#Box diagram of cost and age relative to class

plt.figure(figsize=(20, 7))

sns.boxplot(x=df.pclass, y=df.age)

#Correlation visualization

# Considering only numerical variables
scatter_var = list(set(df.columns)-set(['name', 'survived', 'ticket','cabin','embarked','sex','sibsp','parch']))

# Creating heatmap
corr_matrix = df[scatter_var].corr()

#Scatter plot of age and cost

plt.scatter(df['age'], df['fare'])
plt.title("Age Vs Fare")

#Scatter chart of class and cost

plt.scatter(df['pclass'], df['fare'])
plt.title("pclass Vs fare")

# pair plot of variables

#Scatter relation between two variables


#Sex, boarding place, class and survival statistics

f, [ax1,ax2,ax3] = plt.subplots(1,3,figsize=(20,5))
sns.countplot(x='sex', hue='survived', data=df, ax=ax1)
sns.countplot(x='pclass', hue='survived', data=df, ax=ax2)
sns.countplot(x='embarked', hue='survived', data=df, ax=ax3)
ax1.set_title('sex feature analysis')
ax2.set_title('pclass feature analysis')
ax3.set_title('embarked feature analysis')
f.suptitle('categorical feature analysis', size=20, y=1.1)


#Cross statistics of boarding place, class and gender

grid = sns.FacetGrid(data = df, col='pclass', hue='sex', palette='seismic', size=4)
grid.map(sns.countplot, 'embarked', alpha=0.8)

#Plot the age density of survival:

f,ax = plt.subplots(figsize=(10,5))
sns.kdeplot(df.loc[(df['survived'] == 0),'age'] , color='gray',shade=True,label='not survived')
sns.kdeplot(df.loc[(df['survived'] == 1),'age'] , color='g',shade=True, label='survived')
plt.title('age feature distribution', fontsize = 15)
plt.xlabel("age", fontsize = 15)
plt.ylabel('frequency', fontsize = 15)

#Age density map under different gender and survival conditions:

def plot_distribution( df , var , target , **kwargs ):
    row = kwargs.get( 'row' , None )
    col = kwargs.get( 'col' , None )
    facet = sns.FacetGrid( df , hue=target , aspect=4 , row = row , col = col )
    facet.map( sns.kdeplot , var , shade= True )
    facet.set( xlim=( 0 , df[ var ].max() ) )

plot_distribution( df , var = 'age' , target = 'survived' , row = 'sex' )

#Cost map of people with different survival conditions

And calculate the variance and mean and conduct visual analysis

# Fill in missing values
df["fare"].fillna(df["fare"].median(), inplace=True)

df['fare'] = df['fare'].astype(int)

# Get the Fare of the surviving and dead passengers respectively
fare_not_survived = df["fare"][df["survived"] == 0]
fare_survived = df["fare"][df["survived"] == 1]

# The mean and variance of far are obtained
avgerage_fare = pd.DataFrame([fare_not_survived.mean(), fare_survived.mean()])
std_fare = pd.DataFrame([fare_not_survived.std(), fare_survived.std()])

df['fare'].plot(kind='hist', figsize=(15,3),bins=100, xlim=(0,50))

avgerage_fare.index.names = std_fare.index.names = ["survived"]

#Statistics of loneliness and starting with family

#The impact of being alone or with your family on survival

df['family'] =  df["parch"] + df["sibsp"]
df['family'].loc[df['family'] > 0] = 1
df['family'].loc[df['family'] == 0] = 0

# Delete Parch and SibSp
df_new = df.drop(['sibsp','parch'], axis=1)

# mapping
fig, (axis1,axis2) = plt.subplots(1,2,sharex=True,figsize=(10,5))

sns.countplot(x='family', data=df, order=[1,0], ax=axis1)

# It can be divided into two situations: taking a boat with your family and taking a boat alone
family_perc = df_new[["family", "survived"]].groupby(['family'],as_index=False).mean()
sns.barplot(x='family', y='survived', data=family_perc, order=[1,0], ax=axis2)

axis1.set_xticklabels(["With Family","Alone"], rotation=0)

# pair plot

g = sns.pairplot(df[[u'survived', u'pclass', u'sex', u'age', u'family', u'fare', u'embarked']], hue='survived', palette = 'seismic',
                 size=4,diag_kind = 'kde',diag_kws=dict(shade=True),plot_kws=dict(s=50) )

# pandas profiling

#pip install pandas-profiling

# from pandas_profiling import ProfileReport
# EDA_report = ProfileReport(df)
# EDA_report.to_file(output_file='EDA.html')

Reference: kaggle

Reference: Interview Questions on Exploratory Data Analysis (EDA)

Reference: Introduction to Exploratory Data Analysis (EDA)

Reference: How to do Exploratory Data Analysis (EDA) with python?

Reference: Kaggle entry level Title: prediction of Titanic survivors - Data Analysis

Reference: Kaggle entry level Title: prediction of Titanic survivors - Data Mining

Reference: UCI data set sorting (attached with common data sets of papers)

Reference: pandas

Reference: pandas profiling


Keywords: Python Big Data Machine Learning Data Analysis Data Mining

Added by iacataca on Thu, 10 Feb 2022 05:31:49 +0200