Please indicate the source for Reprint: simiam.com
Amway: py awesome Kit
Py awesome kit is a python toolkit I have been tossing about recently. It provides functions commonly used in the process of big data development and data analysis. I hope it can be gradually enriched and improved in the future.
Project address: https://github.com/monkeychen/py-awesome-kit
1. Install Python 3
# Backup installed python Libraries pip3 freeze > python3-installed-libs.txt # Remove installed python # Linux dependent package download address: https://vault.centos.org/7.3.1611/os/x86_64/Packages/ mkdir /env/python3.7 cd /env/Python-3.7.9 ./configure --prefix=/env/python3.7 --enable-shared --enable-optimizations make&&make install
2. Download third-party packages and their dependencies using pip
- Take installing jupyterab as an example
mkdir jupyterlab cd jupyterlab # download pip3 download jupyterlab -i https://pypi.tuna.tsinghua.edu.cn/simple # install cd .. pip3 install jupyterlab/jupyterlab-3.0.1-py3-none-any.whl --no-index --find-links=./jupyterlab
3. Build and install py awesome kit from the source code
# Enter the project root directory cd <project_root> # Build / package python3 setup.py bdist_wheel # Install to python Library pip install dist/py-awesome-kit-<version>-py3-none-any.whl # The application development process will change frequently. Each installation needs to uninstall the old version first, which is very troublesome. # If you install using the develop ment mode, the actual code will not be copied to the site packages, except for a link (*. Egg link) to the current application. # In this way, the source code changes in the current location will be immediately reflected in the site packages. Use the following: pip install -e . # Or Python setup.py development # To uninstall, use the following command: pip uninstall fmcc-awesome-kit
Appendix: common Linux commands
Install rpm package and dependent package commands using yum or up2date
- View dependency packages: you can use the yum deplist command to find the dependency list of rpm packages. For example, to find the rpm dependent package of git:
yum deplist git
- Scheme 1 (recommended): repertrack command
# Install Yum utils yum -y install yum-utils # Download git full dependency package repotrack git
- Scheme 2: yumdownloader command
# Install Yum utils yum -y install yum-utils # Download git dependency package yumdownloader --resolve --destdir=/tmp git
– destdir: specify the rpm package download directory (if not specified, the default is the current directory)
– resolve: download the dependent rpm package.
Note: only the main package and the dependency package based on your current operating system will be downloaded.
- Scheme 3: download only plug-in from yum
# Download git dependency package yum -y install git --downloadonly --downloaddir=/tmp
Note: like the yumdownloader command, it will only download the main package and the dependency package based on your current operating system.
- Install all downloaded rpm packages offline
rpm -Uvh --force --nodeps *.rpm
- Other commands
repoquery git yum provides git
git usage related
- The project was previously cloned based on https. Now, after configuring SSH-KEY, you want to change to SSH submission
It was before https Link, now want to use SSH What about submitting? Directly modify the project directory .git Under folder config File, just modify the address.
Common system configuration problems
1. After writing the shell script, there is no problem to execute it alone under bash, but an error is reported after it cannot be executed in crontab
# Problem: after writing the shell script, there is no problem to execute it alone under bash, but it cannot be executed in crontab. The following error message is reported: # [psql: error while loading shared libraries: libpq.so.5: cannot open shared object file: No such file or directory] # Cause analysis: # A bad problem with crontab is that it always does not read environment variable parameters from the user profile file by default, which often leads to success when manually executing a script, but errors will occur when trying to execute it regularly in crontab. # Solution 1 (recommended): # ------------------------------ sudo cp ./bin/greenplum-x86_64.conf /etc/ld.so.conf.d/greenplum-x86_64.conf sudo ldconfig # Check whether it takes effect with the following command ldconfig -p | grep green # ------------------------------ # Solution 2: manually set LD before each shell file called by crontab_ LIBRARY_ Path environment variable. export PATH="$PATH:/opt/mssql-tools/bin" source /env/greenplum/clients/greenplum_clients_path.sh source /env/greenplum/loaders/greenplum_loaders_path.sh
2. Problems caused by not setting PYTHONPATH environment variable
If the PYTHONPATH environment variable is not set in the system, python some will be caused_ When module.py is started, the current directory cannot be added to sys.path,
Finally, the module in other directories in the import project reports an error: the specified module is not found.
The solution is as follows:
# vim ~/.bashrc or vim ~/.bash_profile or sudo vim /etc/profile export PYTHONPATH=":$PYTHONPATH"
PS: even if PYTHONPATH is set, if you call python script through shell in crontab, you will still find that the current directory is not added to sys.path, resulting in module loading error.
The reason is that ~ /. bashrc or ~ /. Bash will not be executed when crontab is started_ Profile or / etc/profile,
This will result in the absence of PYTHONPATH in the execution context of crontab, that is, the essential reason is still that PYTHONPATH is not set.
There are many solutions:
- Set PYTHONPATH in the shell that crontab executes.
- Modify the global configuration file of crontab (not recommended).