1. Systematic overview
1. Description
- Code hosting system is an indispensable tool in development. Code hosting system facilitates collaborative development, code permission control, code disaster preparedness, version management, problem tracking, project document management and other necessary tools for project management. Code hosting systems always play an important role in any process of a project.
2. Code Management Tools
At present, there are two main systems of open source code management tools on the market:
- SVN: Centralized Code Version Control System
- GIT: Distributed Code Version Control System
Centralized and distributed have their advantages and disadvantages. Simply speaking, centralized requires a central server, and the central server failure service is not available. However, centralized control is strict and granular. On the contrary, distributed does not have a strict central server concept. Hosted system server failures can still be used, but the permission control is relatively weak.
Generally speaking, open source projects prefer git
Gitlab and Gogs are one of the many management systems derived from git and svn, such as github, Ali cloud code, code cloud, etc. The difference is that gitlab and gogs need to be built by themselves, while GitHub are provided to the public in the form of services.
3. Why gogs?
- When it comes to self-built code management systems, the first thing to think about is gitlab, which is a rich code management system based on ruby. However, gitlab consumes a lot of resources, can't support a large number of users on a low-configuration server, and installation and maintenance can be very cumbersome. Gogs is an open source control system based on the go language. The go language is naturally characterized by high concurrency and high performance. It can also support large numbers of users to play on servers with low configurations. Gogs starts late and does not have gitlab-rich functions, but provides basic functions and is easier to install and maintain than gitlab. After evaluating the function of gogs, we conclude that it is fully in line with our requirements. So choose gogs
2. Install Gogs
1. Install git
- Before installing gogs, you must install git, which can be installed by the yum install git command
2. Environment Configuration
Prepare the server first.
- Supplier: Aliyun
- CPU:2 Core
- Memory: 4G
- Hard Disk: 40G System Disk, 100G Data Disk (hanging point/data)
- Bandwidth: 100M (by volume)
- System: Centos 7.4
- Gogs: 0.11.43
- Download address: https://dl.gogs.io/0.11.43/gogs_0.11.43_linux_amd64.tar.gz
- GITHUB: https://github.com/gogits/gogs
3. Install Gogs
1. Download gogs:
[root@Gogs ~]# wget https://dl.gogs.io/0.11.43/gogs_0.11.43_linux_amd64.tar.gz --2018-04-03 15:31:02-- https://dl.gogs.io/0.11.43/gogs_0.11.43_linux_amd64.tar.gz Resolving dl.gogs.io (dl.gogs.io)... 138.68.27.161 Connecting to dl.gogs.io (dl.gogs.io)|138.68.27.161|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 19214809 (18M) [application/x-gzip] Saving to: 'gogs_0.11.43_linux_amd64.tar.gz' 100%[===========================================================>] 19,214,809 341KB/s in 31s 2018-04-03 15:31:34 (607 KB/s) - 'gogs_0.11.43_linux_amd64.tar.gz' saved [19214809/19214809] [root@Gogs ~]#
2. Unzip and Copy
[root@Gogs ~]# tar -zxvf gogs_0.11.43_linux_amd64.tar.gz
Copy the unzipped folder to the destination path:
[root@Gogs ~]# cp -rf gogs /usr/local/gogs
The extracted files are as follows:
[root@Gogs ~]# cd /usr/local/gogs/ [root@Gogs gogs]# ls gogs LICENSE public README.md README_ZH.md scripts templates [root@Gogs gogs]#
- Where the gogs file is the startup command
- Public is the public file directory
- Scripts are script files
- Templates is a template file for the web
3. Prepare user and warehouse catalogs
It is not recommended to use root user to run gogs, we prepare a user to run gogs
[root@Gogs ~]# groupadd gogs [root@Gogs ~]# useradd -g gogs -s /bin/bash gogs [root@Gogs ~]# passwd gogs Changing password for user gogs. New password: Retype new password: passwd: all authentication tokens updated successfully. [root@Gogs ~]#
Prepare warehouse and log directories
[root@Gogs ~]# mkdir -p /data/gogs/repositories [root@Gogs ~]# mkdir -p /data/gogs/logs
Modify related file permissions
[root@Gogs ~]# chown -R gogs:gogs /data/gogs/ [root@Gogs ~]# chown -R gogs:gogs /usr/local/gogs/
4. Startup
Switch to gogs user before starting
[root@Gogs ~]# su gogs [gogs@Gogs root]$ cd /usr/local/gogs/ [gogs@Gogs gogs]$ ./gogs web 2018/04/04 10:22:41 [ WARN] Custom config '/usr/local/gogs/custom/conf/app.ini' not found, ignore this if you're running first time 2018/04/04 10:22:41 [TRACE] Custom path: /usr/local/gogs/custom 2018/04/04 10:22:41 [TRACE] Log path: /usr/local/gogs/log 2018/04/04 10:22:41 [TRACE] Log Mode: Console (Trace) 2018/04/04 10:22:41 [ INFO] Gogs 0.11.43.0330 2018/04/04 10:22:41 [ INFO] Cache Service Enabled 2018/04/04 10:22:41 [ INFO] Session Service Enabled 2018/04/04 10:22:41 [ INFO] SQLite3 Supported 2018/04/04 10:22:41 [ INFO] Run Mode: Development 2018/04/04 10:22:41 [ INFO] Listen: http://0.0.0.0:3000
As mentioned above, gogs has been started, and the startup information is printed directly on the screen. You can get all kinds of information about gogs, versions, paths, startup modes, listening ports, and so on. Of course, the above information is the default of gogs and is not suitable for formal use. We're changing later.
As mentioned above, gogs listens on a 3000 port through which we can access gogs web services. Visit http://ip:3000 perhaps http://domain:3000 Yes, the first visit will guide us through the installation.
Installation Page Options Description:
Be careful:
1. If there is no administrator account configured on the installation page, the user with user ID 1 will be the administrator when the registration button is clicked to register the user after the installation is completed.
2. Special attention should be paid to the issue of permissions. Full permissions should be given to the gogs directory, warehouse directory and log directory.
3. Gogs Configuration
1. Official Configuration Items
2. Current Use Configuration
We use the following configuration
APP_NAME = GOGS Code Hosting Platform RUN_USER = gogs RUN_MODE = prod [database] DB_TYPE = mysql HOST = 127.0.0.1:3306 NAME = gogs USER = gogs PASSWD = gogs@123 SSL_MODE = disable PATH = data/gogs.db [admin] DISABLE_REGULAR_ORG_CREATION = true [repository] ROOT = /data/gogs/repositories MAX_CREATION_LIMIT = 0 ENABLE_LOCAL_PATH_MIGRATION = true FORCE_PRIVATE = true DISABLE_HTTP_GIT = false [server] DOMAIN = git.example.com HTTP_PORT = 3000 ROOT_URL = https://git.example.com/ DISABLE_SSH = false SSH_PORT = 2222 START_SSH_SERVER = true OFFLINE_MODE = false ENABLE_GZIP = true [mailer] ENABLED = false [service] REGISTER_EMAIL_CONFIRM = false ENABLE_NOTIFY_MAIL = false DISABLE_REGISTRATION = true ENABLE_CAPTCHA = false REQUIRE_SIGNIN_VIEW = false [picture] DISABLE_GRAVATAR = true ENABLE_FEDERATED_AVATAR = false [session] PROVIDER = file [log] MODE = file LEVEL = Trace ROOT_PATH = /data/gogs/logs [security] INSTALL_LOCK = true SECRET_KEY = QSL4Bv87UrgzZsK LOGIN_REMEMBER_DAYS = true COOKIE_USERNAME = true COOKIE_REMEMBER_NAME = true REVERSE_PROXY_AUTHENTICATION_USER = true [webhook] TYPES = gogs DELIVER_TIMEOUT = 300 SKIP_TLS_VERIFY = true PAGING_NUM = true [cron] ENABLED = true
In the configuration above, we started the built-in SSH service and did not allow users to register or create warehouses. Please refer to the usage conventions.
4. System Service Configuration
1. Description
- By running. / gogs web can start gogs, but this startup method is not run as daemon, gogs will exit after the terminal exits, obviously this is not what we want. We need gogs to run in the background.
2. gogs.service Configuration
[Unit] Description=Gogs After=network.target [Service] Type=simple User=gogs Group=gogs WorkingDirectory=/usr/local/gogs ExecStart=/usr/local/gogs/gogs web Restart=always [Install] WantedBy=multi-user.target
The above configuration is written to/usr/lib/systemd/system/gogs.service file.
If you are using a local database, it is recommended that you start gogs after the After configuration database is started.
Because we use the RDS service of Ali Cloud, we can ignore it.
Note that the user and group configurations must be consistent with the running user of the gogs, or the startup will fail. Also note whether git is included in the environment variable, or if it is not possible to pass in the environment variable through Environment=
3. Enable gogs through systemctl
[root@Gogs gogs]# systemctl enable gogs.service
Created symlink from /etc/systemd/system/multi-user.target.wants/gogs.service to /usr/lib/systemd/system/gogs.service.
[root@Gogs gogs]# systemctl start gogs.service
[root@Gogs gogs]# systemctl status gogs.service
● gogs.service - Gogs
Loaded: loaded (/usr/lib/systemd/system/gogs.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2018-04-04 11:20:35 CST; 5s ago
Main PID: 2445 (gogs)
CGroup: /system.slice/gogs.service
└─2445 /usr/local/gogs/gogs web > /data/gogs/logs/output.log 2>&1 &
Apr 04 11:20:35 Gogs systemd[1]: Started Gogs.
Apr 04 11:20:35 Gogs systemd[1]: Starting Gogs...
Apr 04 11:20:35 Gogs nohup[2445]: 2018/04/04 11:20:35 [TRACE] Custom path: /usr/local/gogs/custom
Apr 04 11:20:35 Gogs nohup[2445]: 2018/04/04 11:20:35 [TRACE] Log path: /data/gogs/logs
Apr 04 11:20:35 Gogs nohup[2445]: 2018/04/04 11:20:35 [TRACE] Log Mode: File (Trace)
Apr 04 11:20:35 Gogs nohup[2445]: 2018/04/04 11:20:35 [ INFO] The most compelling code management platform 0.11.43.0330
As mentioned above, gogs ran successfully as daemon.
4. Nginx Proxy
Gogs defaults to 3000 as the web's listening port, because of permission issues, gogs can't create ports below 1024. For specific reasons, it can search by Baidu Google itself, which is no longer expanded here. If you need to access it directly from the domain name, we also need to use nginx as a proxy. Nginx proxy brings us two benefits:
- 1. You can use the domain name to access it directly without adding a port number.
- 2. You can configure an SSL certificate.
After installing nginx, add the following configuration:
server {
listen 80;
server_name git.example.com;
rewrite ^(.*)$ https://$host$1 permanent;
}
# HTTPS server
server {
listen 443 ssl;
server_name git.example.com;
ssl_certificate /etc/letsencrypt/live/git.example.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/git.example.com/privkey.pem;
ssl_session_cache shared:SSL:1m;
ssl_session_timeout 5m;
ssl_ciphers HIGH:!aNULL:!MD5;
ssl_prefer_server_ciphers on;
location / {
index index.php index.html index.htm;
autoindex off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_pass http://localhost:3000;
access_log logs/git.access.log;
}
}
As mentioned above, all access to http will jump to access to https.
Nginx installation configuration and certificate visa will not be expanded here, please refer to the "Server SSL Certificate Generation, Configuration, Renewal Manual" for details.
5. API interface of Gogs
Gogs provides an API interface to help us manage gogs. As the API interface is very informative, please move to wiki
6. Migrating warehouses on other platforms
1. Description
Before gogs, we used CODE s from Ali Cloud because we migrated to gogs for a variety of reasons.
The migration of Aliyun CODE has been completed. Here we record how to migrate and the migration scripts for reference in the next similar situation.
Note: The following two things must be done before starting the migration:
- 1. Configure ENABLE_in the [repository] configuration item in the configuration file first LOCAL_ PATH_ MIGRATION = true to turn on the ability to run migrated external warehouses.
- 2. Administrator account must generate tokens in authorization applications
2. Get all the warehouse addresses of Aliyun
Ali Cloud CODE does not provide API interface, can not get all the warehouses under the account number through API interface, there are two ways to solve, one is to copy manually, the other is to crawl through python crawler, we migrated nearly 100 warehouses, so I used crawlers, the specific crawl code is not detailed here, the HTTPS link of the crawled warehouse is stored in https_ Git. In the TXT file, one warehouse per line.
3. Write python migration scripts
The Python script is as follows:
# -*- coding: utf-8 -*-
import urllib2
import urllib
f = open("/root/https_git.txt")
for git_https in f.readlines():
clone_addr = git_https.replace("\n","").replace("\r","")
repo_name = git_https.split("/")[4].split(".")[0]
data='''{"clone_addr" : "%s",
"uid" : 1,
"repo_name" : "%s",
"auth_username" : "linuxops",
"auth_password" : "linuxops@2018",
"private" : true }'''%(clone_addr,repo_name)
url="http://localhost:3000/api/v1/repos/migrate?token=36d125718c911a4afcbbd409ac9d0ff0a6c44e4c"
headers={"User-Agent":"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)",'Content-Type':'application/json'}
req=urllib2.Request(url,headers=headers,data=data)
resul=urllib2.urlopen(req)
print resul.read()
This script only uses Ali Cloud CODE, modify clone_if you want to migrate other systems Addr and repo_ Variable of name
4. Perform migration
Run the python script, and the following successful creation will return information in json format.
[root@Gogs ~]# python git_migrate.py
{"id":15,"owner":{"id":1,"login":"root","full_name":"","email":"linuxops@foxmail.com","avatar_url":"https://secure.gravatar.com/avatar/ba7fb301e420c8ea24fd9f942b817100?d=identicon","username":"root"},"name":"linuxops_admin_php","full_name":"root/linuxops_admin_php","description":"","private":false,"fork":false,"parent":null,"empty":false,"mirror":false,"size":275456,"html_url":"http://git.test.com:3000/root/linuxops_admin_php","ssh_url":"ssh://gogs@git.test.com:2222/root/linuxops_admin_php.git","clone_url":"http://git.test.com:3000/root/linuxops_admin_php.git","website":"","stars_count":0,"forks_count":0,"watchers_count":1,"open_issues_count":0,"default_branch":"master","created_at":"0001-01-01T00:00:00Z","updated_at":"0001-01-01T00:00:00Z","permissions":{"admin":true,"push":true,"pull":true}}
Since then, the gogs migration has been basically successful.
5. Follow-up work
Gogs, like other hosting systems, provides a wiki functionality, which is also a repository, named in the format repository name plus wiki. For example, a warehouse is named admin and the directory where the warehouse is stored is named admin.git, the wiki directory for this repository is admin.wiki.git, every wiki is also a repository.
Migrate libraries from other platforms by migrating functionality. If other platform libraries do not have wikies or do not provide wikies functionality, gogs will make an error on wikies. There may be no wiki libraries generated, or they may be generated because no content causes access failures, which will be reflected in 500 errors on the web side.
To solve this problem, you can simply iterate through all libraries through python, create one if there are no wiki libraries, and inhale a file submission. If there are wiki libraries, create a file submission.
The specific python code is no longer shown here.
7. Use Agreement
1. Permission Provisions
The Gogs Code Hosting Platform is used for code management within the enterprise, but we have agreed on the following for better management:
Service configuration:
- User registration is prohibited.
- Warehouse creation is prohibited and only super administrators are allowed to create warehouses.
- User name and nickname are not allowed (by requiring a page)
Code base management:
- All code libraries are organized under one organization.
- No authorization to the organization, only super administrators within the organization
- For developers who need permissions, add to collaborators, default writable permissions (writable permissions include read and write)
- The default branch in branch management is master.
- Turn on branch protection for maste s, and turn on submitting code through merge requests.
- Turn on branch protection for master, turn on restricted push code members, and allow only team leaders to push to master
Code release:
- Before publishing, the responsible person needs to submit the merge to master and is responsible for code conflict resolution and code review.
- Relevant version changes require the person in charge to label them.
- Whether it's a test or a formal environment, just publish the master branch's code or tag Tags