Rimeng Society
AI AI:Keras PyTorch MXNet TensorFlow PaddlePaddle deep learning real combat (irregular update)
Intelligent dialogue system: Unit dialogue API
linux Installation neo4j,Installing Redis on linux,supervisor installation
neo4j graph database: structured data pipeline, unstructured data pipeline
Named entity audit task: BERT Chinese pre training model
Named entity audit task: build RNN model
Named entity audit task: model training
Named entity recognition task: BiLSTM+CRF part1
Named entity recognition task: BiLSTM+CRF part2
Named entity recognition task: BiLSTM+CRF part3
System joint commissioning test and deployment
9.1 system joint commissioning and test
- Learning objectives:
- Master how to start the service of the online part of the system
- Master the process of testing running services separately
- System architecture diagram:
- Main services of online part of the system:
- werobot service
- Main logical services
- Sentence related model service
- redis service (Session Management)
- neo4j service (figure data query)
- Note: system joint commissioning and testing is the joint commissioning and testing of online services of the system, excluding offline services
- Start the service of the online part of the system:
- Start the werobot service in a suspended manner
- Use supervisor to start the main logical service and its redis service
- Start the child dependent model service in a suspended manner
- Start and view neo4j services (started by default)
- Start the werobot service in a suspended manner:
nohup python /data/wr.py &
- Run in: / dara / directory
- View the suspended service process through the port:
# Install the lsof command using yum or up2date sudo yum install lsof -y # View processes on port 80 lsof -i:80
- Use supervisor to start the main logical service and its redis service
- Brief analysis of supervisor configuration file:
... ; use gunicorn Start based Flask Main logical services of the framework [program:main_server] command=gunicorn -w 1 -b 0.0.0.0:5000 app:app ; the program (relative uses PATH, can take args) stopsignal=QUIT ; signal used to kill process (default TERM) stopasgroup=false ; send stop signal to the UNIX process group (default false) killasgroup=false ; SIGKILL the UNIX process group (def false) stdout_logfile=./log/main_server_out ; stdout log path, NONE for none; default AUTO stdout_logfile_maxbytes=1MB ; max # logfile bytes b4 rotation (default 50MB) stderr_logfile=./log/main_server_error ; stderr log path, NONE for none; default AUTO stderr_logfile_maxbytes=1MB ; max # logfile bytes b4 rotation (default 50MB) ; start-up redis Service to manage the database as a session [program:redis] command=redis-server
- Start service:
# Use the supervisor command to read the files in the specified directory supervisord -c /data/doctor_online/main_serve/supervisor.conf
- View started service status
supervisorctl status all
- Output effect:
main_server RUNNING pid 31609, uptime 0:32:20 redis RUNNING pid 31613, uptime 0:32:18
- Start the sentence related model service in a suspended manner:
- Write the startup script start sh
# In the script is the command to start the service using gunicorn gunicorn -w 1 -b 0.0.0.0:5001 app:app
- Start the service in a suspended manner
nohup sh /data/doctor_online/bert_serve/start.sh &
- Viewing pending service processes through ports
# Install the lsof command using yum or up2date sudo yum install lsof -y # View the process of port 5001 lsof -i:5001
- Start and view neo4j service (figure data query):
# The neo4j service should be started until neo4j start # View service startup status: neo4j status
- Test:
- Step 1: specify the test instructions
- Step 2: add and print test content
- Step 3: restart the main logical service
- Step 4: conduct data flow test and check the print log
- Step 1: Test Description
- Because the main logical service is the central service of all online services (the service will receive or send requests to other services), our test printing information is carried out in the main logical service
- Step 2: add and print test content
class Handler(object):
"""Processing class of main logical service"""
def __init__(self, uid, text, r, reply):
"""
:param uid: User unique identifier uid
:param text: Text entered by the user this time
:param r: redis Database connection object
:param reply: The object that the Rule dialog template is loaded into memory(Dictionaries)
"""
self.uid = uid
self.text = text
self.r = r
self.reply = reply
def non_first_sentence(self, previous):
"""
description: Non first sentence processing function
:param previous: Current sentence of the user(Enter text)Previous sentence text
:return: Return the output statement in the case of non first sentence according to the logic diagram
"""
# Try to request the model service. If it fails, the error result will be printed
###################################################
# It can print this information, indicating that it has entered the non first sentence processing function
print("Prepare to request sentence related model services!")
###################################################
try:
data = {"text1": previous, "text2": self.text}
result = requests.post(model_serve_url, data=data)
if not result.text: return unit_chat(self.text)
###################################################
# This information can be printed to indicate that the sentence related model service request is successful
print("Sentence related model service request succeeded, The returned result is:", result.text)
###################################################
except Exception as e:
print("Model service exception:", e)
return unit_chat(self.text)
###################################################
# The ability to print this message indicates that the request neo4j query service is ready
print("After requesting model services, Prepare request neo4j Query service!")
###################################################
# Continue to query the graph database and get the results
s = query_neo4j(self.text)
###################################################
# If the information can be printed, the neo4j query is successful
print("neo4j Query service request succeeded, Return results:", s)
###################################################
# If the judgment result is an empty list, it will be returned directly using the UnitAPI
if not s: return unit_chat(self.text)
# If the result is not empty, get the last recovered disease old_disease
old_disease = self.r.hget(str(self.uid), "previous_d")
if old_disease:
# new_disease is the disease to be stored this time and the union of the stored disease and the disease found this time
new_disease = list(set(s) | set(eval(old_disease)))
# res is the disease to be returned. It is the difference set between the queried disease and the stored disease
res = list(set(s) - set(eval(old_disease)))
else:
# If old_ If disease is empty, they are the same, which is the query result s
res = new_disease = list(set(s))
# Store new_ Old before disease overwrites_ disease
self.r.hset(str(self.uid), "previous_d", str(new_disease))
# Set expiration time
self.r.expire(str(self.uid), ex_time)
# Convert the list into a string and add it to the Rule dialog template to return
res = ",".join(s)
###################################################
# Be able to print this information to indicate that neo4j query results are available and the Rule dialog template will be used
print("Use the Rule dialog template to generate the return dialog!")
###################################################
print("###################################################")
return self.reply["2"] %res
def first_sentence(self):
"""First sentence processing function"""
# Directly query the graph database and obtain the results
###################################################
# The ability to print this information indicates that the first sentence processing function is entered and the neo4j query is performed immediately
print("The user spoke for the first time recently, You do not have to request model services, Prepare request neo4j Query service!")
###################################################
s = query_neo4j(self.text)
###################################################
# Being able to print this information indicates that the neo4j query has been completed
print("neo4j Query service request succeeded, The returned result is:", s)
###################################################
# If the judgment result is an empty list, it will be returned directly using the UnitAPI
if not s: return unit_chat(self.text)
# Store s as "last returned disease"
self.r.hset(str(self.uid), "previous_d", str(s))
# Set expiration time
self.r.expire(str(self.uid), ex_time)
# Convert the list into a string and add it to the Rule dialog template to return
res = ",".join(s)
###################################################
# Be able to print this information to indicate that neo4j query results are available and the Rule dialog template will be used
print("Use the Rule dialog generation template to generate the return dialog!")
###################################################
print("###################################################")
return self.reply["2"] %res
# Set the routing and request methods of main logical services
@app.route('/v1/main_serve/', methods=["POST"])
def main_serve():
###################################################
# This information can be printed, indicating that the werobot service sent the request successfully
print("Has entered the main logical service, werobot The service is running normally!")
###################################################
# Receive fields from the werobot service
uid = request.form['uid']
text = request.form['text']
# Get an active connection from the redis connection pool
r = redis.StrictRedis(connection_pool=pool)
# Get his last sentence according to the uid (may not exist)
previous = r.hget(str(uid), "previous")
# Sets the currently entered text to the previous sentence
r.hset(str(uid), "previous", text)
###################################################
# This information can be printed, indicating that redis can read and write data normally
print("Initial session management has been completed, redis Normal operation!")
###################################################
# Read the contents of the Rule dialog template into memory
reply = json.load(open(reply_path, "r"))
# Instantiate main logical processing objects
handler = Handler(uid, text, r, reply)
# If previous exists, it is not the first sentence
if previous:
# Call non_ first_ Sentince method
return handler.non_first_sentence(previous)
else:
# Otherwise, call first_ Sentince() method
return handler.first_sentence()
- Step 3: restart the main logical service
supervisorctl restart all
- Step 4: perform data flow test and view the print log
- Test request 1: sends the first symptom description after the user pays attention to the official account.
- Test request 2: after the first sending, the user continues to send some symptom descriptions
- Test request 1:
- Users first send some symptom descriptions after they are concerned about the official account.
- Corresponding data flow:
- werobot Service -- > request primary logical Service -- > request redis service in primary logical Service -- > request neo4j query service -- > use Rule dialog template / UnitAPI
- Corresponding operation:
- Pay attention to the official account (using new users) and send out "I have some abdominal pains recently".
- To view the main logical service logs:
cat /data/doctor_online/main_serve/log/main_server_out
- Log print results:
## Print the following results to show that the test is successful! Has entered the main logical service, werobot The service is running normally! Initial session management has been completed, redis Normal operation! The user spoke for the first time recently, You do not have to request model services, Prepare request neo4j Query service! neo4j Query service request succeeded, The returned result is: ['epilepsy', 'Childhood diabetes mellitus', 'Adrenal crisis', 'Ectopic acute appendicitis', 'acute cholecystitis'] Use the Rule dialog template to return the generated dialog!
- Test request 2:
- After the first sending, the user continues to send some symptom descriptions
- Corresponding data flow:
- werobot Service -- > request main logical Service -- > request redis service in main logical Service -- > request sentence related model service -- > request neo4j query service -- > use Rule dialog template / UnitAPI
- Corresponding operation:
- After sending "I have some abdominal pain recently", continue to send "and there are some red dots on my left abdomen"
- Log print results:
## Print the following results to show that the test is successful! (use the UnitAPI to return results) Has entered the main logical service, werobot The service is running normally! Initial session management has been completed, redis Normal operation! Prepare to request sentence related model services! Sentence related model service request succeeded, The returned result is: 1 After requesting model services, Prepare request neo4j Query service! neo4j Query service request succeeded, Return results: []
- Note: if the print log cannot appear immediately, restart the main logical service
-
Summary of this chapter:
- Learned how to start the service of the online part of the system:
- Start the werobot service in a suspended manner
- Use supervisor to start the main logical service and its redis service
- Start the child dependent model service in a suspended manner
- Start and view neo4j services (started by default)
- Learned how to test:
- Step 1: specify the test instructions
- Step 2: add and print test content
- Step 3: restart the main logical service
- Step 4: conduct data flow test and check the print log
- Step 1: Test Description
- Because the main logical service is the central service of all online services (the service will receive or send requests to other services), our test printing information is carried out in the main logical service
- Step 2: add and print test content
- Step 3: restart the main logical service
- Step 4: perform data flow test and view the print log
- Test request 1: sends the first symptom description after the user pays attention to the official account.
- Test request 2: after the first sending, the user continues to send some symptom descriptions
- Learned how to start the service of the online part of the system:
- matters needing attention:
- This installation manual only applies to the operating system: CentOS 7
- Installation and deployment steps:
- 1: Copy the required files
- 2: Install Anaconda scientific computing environment, including Python 2, Pip, panda, numpy, matplotplib and other scientific computing packages
# Install the environment package in the / root / directory cd /root curl -O https://repo.anaconda.com/archive/Anaconda3-5.2.0-Linux-x86_64.sh sh Anaconda3-2019.07-Linux-x86_64.sh # Configure ~ / bashrc, add a line: export PATH=/root/anaconda/bin/:$PATH
- 3: Several independent tools required for the installation project
# Install flash pip install Flask==1.1.1 # Install Redis database yum install redis -y # Install redis driver in python pip install redis # Installing gunicorn pip install gunicorn==20.0.4 # Install supervisor yum install supervisor -y # Install lsof yum install lsof -y
- 4: Pytoch. Is an important tool for installing the project
# Install pytorch pip install pytorch
- 5: Install diagram database neo4j
# Step 1: load the neo4j installation information into the yum search list cd /tmp wget http://debian.neo4j.org/neotechnology.gpg.key rpm --import neotechnology.gpg.key cat <<EOF> /etc/yum.repos.d/neo4j.repo # Write the following [neo4j] name=Neo4j RPM Repository baseurl=http://yum.neo4j.org/stable enabled=1 gpgcheck=1 # Step 2: use the yum install command to install yum install neo4j-3.3.5 # Step 3: use your own configuration file cp /data/neo4j.conf /etc/neo4j/neo4j.conf
- 6: Start neo4j diagram database and check status
# Start neo4j command neo4j start # View status command neo4j status
- 7: Use scripts to generate maps
# Execute the written script code and write the data to the neo4j database python /data/doctor_offline/neo4j_write.py
- 8: Use script to train the model
# The online part has only one model, Bert Chinese cd /data/doctor_online/bert_server/ python train.py
- 9: Start the werobot service in a suspended manner
# Start the werobot service so that users can complete a dialogue with AI doctors through the wechat interface nohup python /data/wr.py &
- 10: Use supervisor to start the main logical service and its redis service
# Brief analysis of supervisor configuration file # File path location / data/doctor_online/main_serve/supervisor.conf ; use gunicorn Start based Flask Main logical services of the framework [program:main_server] command=gunicorn -w 1 -b 0.0.0.0:5000 app:app ; the program (relative uses PATH, can take args) stopsignal=QUIT ; signal used to kill process (default TERM) stopasgroup=false ; send stop signal to the UNIX process group (default false) killasgroup=false ; SIGKILL the UNIX process group (def false) stdout_logfile=./log/main_server_out ; stdout log path, NONE for none; default AUTO stdout_logfile_maxbytes=1MB ; max # logfile bytes b4 rotation (default 50MB) stderr_logfile=./log/main_server_error ; stderr log path, NONE for none; default AUTO stderr_logfile_maxbytes=1MB ; max # logfile bytes b4 rotation (default 50MB) ; start-up redis Service to manage the database as a session [program:redis] command=redis-server
# Use the supervisor command to read the files in the specified directory supervisord -c /data/doctor_online/main_serve/supervisor.conf
# View started service status supervisorctl status
- 11: Start the sentence related model service in a suspended manner
# Start the service in a suspended manner, and the code has been pre written # Script start The content in SH is gunicorn -w 1 -b 0.0.0.0:5001 app:app nohup sh /data/doctor_online/bert_serve/start.sh &
- 12: Start and view neo4j service (figure data query):
# The neo4j service should be started until neo4j start # View service startup status: neo4j status
- 13: Perform the test
Test 1: Official account(new user), send out"I have some abdominal pain recently". Test 2: (Old users)send out"I have some abdominal pain recently"after, Continue sending"And there are some red spots on the left abdomen".