I summary
In the last [crawler] article on deploying Scrapy to k8s, it was mentioned that errors were reported in the custom ， spiderkeeper ， image, so I submitted a ， python ， package called ， spiderkeeper new ， based on its source code to the ， pypi ， official website. This article will record the whole construction process and hope to provide some help to small partners with the same needs. Click to get programming materials for free
II Upload Python package to pypi official website
First, check Official documents , the official documents are written in great detail. After reading the official documents, you will know the whole construction process. Here is mainly a summary of the read documents, omitting a lot of detailed introduction, which is convenient for quick start-up experience.
1. Configuration items
Create a {setup. EXE file under the root directory of the project Py ， file. Here, take the contents of ， spiderkeeper ， as an example, setup The contents of the PY file are as follows:
#!/usr/bin/env python
from os import path
from setuptools import setup, find_packages
here = path.abspath(path.dirname(__file__))
with open(path.join(here, 'requirements.txt'), 'r', encoding='utf-8') as f:
all_reqs = f.read().split('\n')
with open(path.join(here, 'README.md'), 'r', encoding='utf-8') as f:
long_description = f.read()
install_requires = [x.strip() for x in all_reqs if 'git+' not in x]
setup(
name='SpiderKeeper-new', # Required. The name of the project. Users can install according to this name. PIP install spiderkeeper NEW
version='1.0.0', # Required, the version of the project. It is recommended to follow the semantic version specification
author='edisonwd', # Author of the project
description='Admin ui for spider service, update info: upgrade dependent package', # A brief description of the project
long_description=long_description, # The detailed description of the project, usually read readme Contents of MD file
long_description_content_type='text/markdown', # Description format, optional values: text/plain, text/x-rst, and text/markdown
author_email='2388100489@qq.com', # Valid email address of the author
url='https://github.com/edisonwd/SpiderKeeper ', # project source address
license='MIT',
include_package_data=True,
packages=find_packages(), # Required. Specify the packaging directory. The default is the current directory. If it is another directory, such as SRC, you can use find_packages(where='src')
install_requires=install_requires, # Specify the python package that your project depends on. Read the requirements directly here txt
# The following settings will provide a command called spiderkeeper on the command line to execute the main method of run under the spiderkeeper package to start the project
entry_points={
'console_scripts': {
'spiderkeeper = SpiderKeeper.run:main'
},
},
# The classifier helps users find items by classifying items
classifiers=[
'Development Status :: 4 - Beta',
'License :: OSI Approved :: MIT License',
'Operating System :: OS Independent',
'Programming Language :: Python :: 2.7',
'Programming Language :: Python :: 3.7',
],
)
Copy codeSetup Py ， requirements are referenced in the ， file Txt and readme MD # these two files. By default, the source distribution package that packages python only contains py files. If these two files are not introduced during installation, an error will be reported. To package other files into the distribution package, you need to install them in {manifest In ￠ file, specify ￠ manifest. Of spiderkeeper The contents are as follows:
recursive-include SpiderKeeper/app/templates * recursive-include SpiderKeeper/app/static * include requirements.txt include README.md Copy code
2. Packaged items
Before building sdists and wheel of the project, you need to install the build package. Use the following command:
pip install build Copy code
In setup Py file, use the following command to build the source package and wheel package of the project:
python -m build Copy code
-The content behind the m option is module, which is used to run the module as a script.
If you only want to build the source package, you can execute the following command:
python -m build --sdist Copy code
If you only want to build the wheel package, you can execute the following command:
python -m build --wheel Copy code
After the build is completed, a {dist / directory will be generated in the root directory of the project to store the built package.
3. Upload project to pypi
Twin is a tool for uploading python packages to pypi. Use the following command to install:
pip install twine Copy code
Before uploading, use the following command to check whether the built package has problems:
twine check dist/* Copy code
First, you need a PyPI User account. You can use the forms on the PyPI website Create an account.
A PyPI will now be created API token In order to be able to upload items safely.
go to pypi.org/manage/acco... API token ； Do not limit its scope to a specific project because a new project is being created.
Do not close the page until you copy and save the token - you will no longer see the token.
To avoid having to copy and paste tokens every time you upload, you can create a $home / Pypirc file:
[pypi] username = __token__ password = <the token value, including the `pypi-` prefix> Copy code
Note that this stores the token in clear text.
For more details, please see. pypirc.
Once you have an account, you can use it twine upload your distribution toPyPI.
The process of uploading a version is the same whether or not the project already exists on PyPI - if it does not already exist, it will be created automatically when the first version is uploaded.
For the second and subsequent versions, PyPI only requires that the version number of the new version is different from any previous version.
twine upload dist/* Copy code
You can navigate to the uploaded project name https://pypi.org/project/ The URL of < sampleproject > to check whether the package has been successfully transferred. The uploaded project may take a minute or two to appear on the website.
III Problems encountered
When you use pip install to install the package, you will be prompted to upgrade. Use the following command:
pip install --upgrade pip setuptools wheel Copy code
Errors are reported as follows:
ERROR: Could not install packages due to an OSError: [WinError 5] Access denied.: 'e:\\anaconda3\\scripts\\pip.exe' Consider using the `--user` option or check the permissions. Copy code
Execute the pip command again, and the error is as follows:
> pip Script file 'E:\anaconda3\Scripts\pip-script.py' is not present Copy code
Cause analysis
When upgrading with pip install --ugrade pip command, the original pip will be uninstalled first, and then the new version of pip will be installed. However, after successful uninstallation, the new version of pip cannot be installed due to permission problems, so pip cannot be used.
resolvent
Use the following methods to install:
$ curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py # download installation script $ python get-pip.py # Run the installation script Copy code
**Note: * * pip is associated with the version of Python to which the installation script is run. If Python 3 is used, execute the following command:
$ sudo python3 get-pip.py # Run the installation script. Copy code
Generally, pip corresponds to Python 2.7 and pip3 corresponds to Python 3.0 x.
Some Linux distributions can install pip directly with the package manager, such as Debian and Ubuntu:
sudo apt-get install python-pip Copy code
Reference documents
packaging.python.org/en/latest/g...
packaging.python.org/en/latest/g...
twine.readthedocs.io/en/stable/
