Please indicate the source for Reprint: simiam.com

Amway: py awesome Kit

Py awesome kit is a python toolkit I have been tossing about recently. It provides functions commonly used in the process of big data development and data analysis. I hope it can be gradually enriched and improved in the future.
Project address: https://github.com/monkeychen/py-awesome-kit

1. Install Python 3

# Backup installed python Libraries
pip3 freeze > python3-installed-libs.txt

# Remove installed python
# Linux dependent package download address: https://vault.centos.org/7.3.1611/os/x86_64/Packages/
mkdir /env/python3.7
cd /env/Python-3.7.9
./configure --prefix=/env/python3.7 --enable-shared --enable-optimizations
make&&make install

2. Download third-party packages and their dependencies using pip

• Take installing jupyterab as an example

mkdir jupyterlab
cd jupyterlab
# download
pip3 download jupyterlab -i https://pypi.tuna.tsinghua.edu.cn/simple

# install
cd ..
pip3 install jupyterlab/jupyterlab-3.0.1-py3-none-any.whl --no-index --find-links=./jupyterlab

3. Build and install py awesome kit from the source code

# Enter the project root directory
cd <project_root>

# Build / package
python3 setup.py bdist_wheel

# Install to python Library
pip install dist/py-awesome-kit-<version>-py3-none-any.whl

# The application development process will change frequently. Each installation needs to uninstall the old version first, which is very troublesome.
# If you install using the develop ment mode, the actual code will not be copied to the site packages, except for a link (*. Egg link) to the current application.
# In this way, the source code changes in the current location will be immediately reflected in the site packages. Use the following:
pip install -e .  # Or Python setup.py development

# To uninstall, use the following command:
pip uninstall fmcc-awesome-kit

Appendix: common Linux commands

Install rpm package and dependent package commands using yum or up2date

• View dependency packages: you can use the yum deplist command to find the dependency list of rpm packages. For example, to find the rpm dependent package of git:

yum deplist git

• Scheme 1 (recommended): repertrack command

# Install Yum utils
yum -y install yum-utils

# Download git full dependency package
repotrack git

• Scheme 2: yumdownloader command

# Install Yum utils
yum -y install yum-utils

# Download git dependency package
yumdownloader --resolve --destdir=/tmp git

Parameter Description:

– destdir: specify the rpm package download directory (if not specified, the default is the current directory)

– resolve: download the dependent rpm package.

Note: only the main package and the dependency package based on your current operating system will be downloaded.

• Scheme 3: download only plug-in from yum

# Download git dependency package
yum -y install git --downloadonly --downloaddir=/tmp

Note: like the yumdownloader command, it will only download the main package and the dependency package based on your current operating system.

• Install all downloaded rpm packages offline

rpm -Uvh --force --nodeps *.rpm

• Other commands

repoquery git

yum provides git

git usage related

• The project was previously cloned based on https. Now, after configuring SSH-KEY, you want to change to SSH submission

It was before https Link, now want to use SSH What about submitting?
Directly modify the project directory .git Under folder config File, just modify the address.

Common system configuration problems

1. After writing the shell script, there is no problem to execute it alone under bash, but an error is reported after it cannot be executed in crontab

# Problem: after writing the shell script, there is no problem to execute it alone under bash, but it cannot be executed in crontab. The following error message is reported:
# [psql: error while loading shared libraries: libpq.so.5: cannot open shared object file: No such file or directory]
# Cause analysis:
# A bad problem with crontab is that it always does not read environment variable parameters from the user profile file by default, which often leads to success when manually executing a script, but errors will occur when trying to execute it regularly in crontab.

# Solution 1 (recommended):
# ------------------------------
sudo cp ./bin/greenplum-x86_64.conf /etc/ld.so.conf.d/greenplum-x86_64.conf
sudo ldconfig

# Check whether it takes effect with the following command
ldconfig -p | grep green

# ------------------------------
# Solution 2: manually set LD before each shell file called by crontab_ LIBRARY_ Path environment variable.
export PATH="$PATH:/opt/mssql-tools/bin"
source /env/greenplum/clients/greenplum_clients_path.sh
source /env/greenplum/loaders/greenplum_loaders_path.sh

2. Problems caused by not setting PYTHONPATH environment variable

If the PYTHONPATH environment variable is not set in the system, python some will be caused_ When module.py is started, the current directory cannot be added to sys.path,
Finally, the module in other directories in the import project reports an error: the specified module is not found.

The solution is as follows:

# vim ~/.bashrc or vim ~/.bash_profile or sudo vim /etc/profile
export PYTHONPATH=":$PYTHONPATH"

PS: even if PYTHONPATH is set, if you call python script through shell in crontab, you will still find that the current directory is not added to sys.path, resulting in module loading error.

The reason is that ~ /. bashrc or ~ /. Bash will not be executed when crontab is started_ Profile or / etc/profile,
This will result in the absence of PYTHONPATH in the execution context of crontab, that is, the essential reason is still that PYTHONPATH is not set.
There are many solutions:

1. Set PYTHONPATH in the shell that crontab executes.
2. Modify the global configuration file of crontab (not recommended).