90.1 presentation environment

Implementation process:

• CM and CDH version: 5.13.1
• The CDSW and CDH cluster operating systems are RedHat: 7.2
• Nginx server: redhat6 four
• Livy version: 0.4
• Nginx version: 1.8.6

90.2 operation demonstration

1. Install DNS service and configure pan domain name resolution

• The DNS service is mainly used to resolve the IP address of Nginx service. Because CDSW uses a pan domain name, it uses multiple domain names, such as:
  • Note that the domain names are dynamically generated. In order to facilitate the resolution of the domain names listed above to the Nginx service, choose to install the DNS service to resolve the Nginx service

livelog.cdsw1.fayson.com
consoles.cdsw1.fayson.com
tty-{xxx}.cdsw1.fayson.com

• Fayon's / etc / dnsmasq Conf configuration fragment information:
  • The DNS service installed here is not configured on the Nginx service, but on your client

conf-dir=/etc/dnsmasq.d
resolv-file=/etc/resolv.Dnsmasq.conf
strict-order
listen-address=99.6.xxx.xxx
addn-hosts=/etc/hosts
address=/cdsw1.fayson.com/99.6.xxx.xxx
address=/cdsw1/99.6.xxx.xxx

• Configure DNS on clients accessing Nginx
  • Test whether the domain name can be normally resolved to the Nginx service on the CMD command line of the client

2.Nginx configuration

• Modify nginx Conf file, add the following in http:

http {
    include       mime.types;
    include       reverse-proxy.conf;
    default_type  application/octet-stream;
    sendfile        on;
    keepalive_timeout  65;
    resolver 99.13.31.205;
    ...
}

• include and resolver configuration:
  • Reverse proxy. Of the include configuration item The conf file mainly configures the reverse proxy for accessing the CDSW service
  • The resolver configuration item is mainly used to prevent "no resolver defined to resolve", because the CDSW service uses pan domain name resolution in the proxy_ If the access address is set directly by pass, this problem will not occur
• reverse-proxy.conf configuration file, as follows

server {
   listen 80;
   server_name cdsw1.fayson.com;
   location / {
     proxy_cookie_path / /;
     proxy_pass http://cdsw1.fayson.com;
   }
}
#Configure reverse proxy
server {
   listen 80;
   server_name consoles.cdsw1.fayson.com;
   location / {
     add_header 'Access-Control-Allow-Origin' 'http://cdsw1.fayson.com';
     add_header 'Access-Control-Allow-Credentials' 'true';
     add_header 'Access-Control-Allow-Methods' 'GET';
     proxy_pass http://consoles.cdsw1.fayson.com/;
   }
}
server {
   listen 80;
   server_name livelog.cdsw1.fayson.com;
   location / {
     proxy_http_version 1.1;
     proxy_set_header Upgrade $http_upgrade;
     proxy_set_header Connection "upgrade";
     proxy_pass http://livelog.cdsw1.fayson.com;
     proxy_redirect off;
   }
}
#Configure the reverse proxy of WebSocket
server {
   listen 80;
   server_name *.cdsw1.fayson.com;
   location ~* /*/ws {
     proxy_http_version 1.1;
     proxy_set_header Upgrade $http_upgrade;
     proxy_set_header Connection "upgrade";
     proxy_pass http://$host;
     proxy_redirect off;
   }
   location / {
     add_header 'Access-Control-Allow-Origin' 'http://cdsw1.fayson.com';
     add_header 'Access-Control-Allow-Credentials' 'true';
     add_header 'Access-Control-Allow-Methods' 'GET';
     proxy_pass http://$host;
   }
}
server {
   listen 80;
   server_name cdh.fayson.com;
   location / {
      proxy_pass http://99.13.31.205:7180;
   }
}
server {
   listen 80;
   server_name ads.tableau.cmbchina.cn;
   location / {
      proxy_pass http://99.6.136.193;
   }
}

• Use the following command to verify that the configuration file is correct and reload the configuration

nginx -t 
nginx -s reload

• Nginx – s reload command reloads the configuration of nginx without restarting the service

3. Summary

• CDSW uses the WebSocket service. The Nginx version must be greater than 1.3, otherwise the WebSocket reverse proxy will be abnormal
• When configuring DNS, it should be noted that the reverse proxy domain name of Nginx is consistent with the domain name accessed by CDSW. Because some connections in CDSW service use the HostName of CDSW Master, we use the same domain name as the reverse proxy to avoid the loss of access rights caused by cross domain access Session

4. Frequently asked questions

• Error when accessing Nginx The following exceptions are found in the log:

2018/01/17 17:08:21 [error] 11065#0: *2952 tty-u5wgknkxfa2315i1.cdsw1.fayson.com could not be resolved
 (110: Operation timed out), client: 99.7.42.25, server: *.cdsw1.fayson.com, request: "GET /5kg4v7tey3
1x4x7y/ HTTP/1.1", host: "tty-u5wgknkxfa2315i1.cdsw1.fayson.com", referrer: "http://cdsw1.itc.cmbchina
.cn/admin/demo/engines/u5wgknkxfa2315i1"

• resolvent:
  • Need to be in nginx Add DNS configuration in http of conf configuration: resolver 99.13.31.205;

Big data video recommendation:
CSDN
Big data voice recommendation:
Application of enterprise big data technology
Recommendation system for big data machine learning cases
natural language processing
Big data foundation
Artificial intelligence: introduction to deep learning to mastery