Official documents
Version upgrade route and dependencies
Operation steps
1. Backup the machine
Create an image of the machine in the background of alicloud.
According to Alibaba cloud's documentation, you need to comment it out first /etc/fstab Mount the data disk inside, and then mirror it. Note that after the mirror image is finished, the /etc/fstab Remove the notes from the. Time consuming statistics: 3 T Data disk printing image It took less than 60 minutes.
To be on the safe side, the upgrade here is divided into two days to advance, leaving one day for observation.
day1, upgraded from 12.9.2 to 12.10.6 [backup once before operation]
day2, upgrade from 12.10.6 to 12.10.14, and then upgrade from 12.10.14 to 13.0.14 [backup once before operation]
2. Upgrade from 12.9.2 to 12.10.6
The gitlab version produced here is 12.9.2. First upgrade to 12.10.6, then upgrade to 12.10.14, and then upgrade to 13.0.14
12. The 10 major changes are: upgrade the version of PG to pg11.
# Stop the current operation of gitlab (this step can be omitted, and it is also possible to stop gitlab without stopping. Of course, it is recommended to stop if conditions permit)
# gitlab-ctl stop
# Pull the latest warehouse information
# apt update
# List the versions we want to install
apt-cache madison gitlab-ce | grep '12.10.6'
result:
gitlab-ce | 12.10.6-ce.0 | https://packages.gitlab.com/gitlab/gitlab-ce/ubuntu xenial/main amd64 Packages
# Installing gitlab CE in 12.10.6
apt-get install gitlab-ce=12.10.6-ce.0
The log is as follows:
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer required:
geoip-database-extra libjs-openlayers libwireshark8 libwiretap6 libwscodecs1 libwsutil7 linux-headers-4.4.0-93 linux-headers-4.4.0-93-generic
linux-image-4.4.0-93-generic linux-image-extra-4.4.0-93-generic
Use 'apt autoremove' to remove them.
The following packages will be upgraded:
gitlab-ce
1 upgraded, 0 newly installed, 0 to remove and 93 not upgraded.
Need to get 799 MB of archives.
After this operation, 84.2 MB of additional disk space will be used.
WARNING: The following packages cannot be authenticated!
gitlab-ce
Install these packages without verification? [y/N] y
Get:1 https://packages.gitlab.com/gitlab/gitlab-ce/ubuntu xenial/main amd64 gitlab-ce amd64 12.10.6-ce.0 [799 MB]
Fetched 799 MB in 1min 12s (11.1 MB/s)
(Reading database ... 243001 files and directories currently installed.)
Preparing to unpack .../gitlab-ce_12.10.6-ce.0_amd64.deb ...
gitlab preinstall:
gitlab preinstall: This node does not appear to be running a database
gitlab preinstall: Skipping version check, if you think this is an error exit now
gitlab preinstall:
Unpacking gitlab-ce (12.10.6-ce.0) over (12.9.2-ce.0) ...
Setting up gitlab-ce (12.10.6-ce.0) ...
Checking PostgreSQL executables:Starting Chef Client, version 14.14.29
resolving cookbooks for run list: ["gitlab::config", "postgresql::bin"]
Synchronizing Cookbooks:
- gitlab (0.0.1)
- postgresql (0.1.0)
- redis (0.1.0)
- package (0.1.0)
- monitoring (0.1.0)
- registry (0.1.0)
- mattermost (0.1.0)
- gitaly (0.1.0)
- consul (0.1.0)
- praefect (0.1.0)
- letsencrypt (0.1.0)
- nginx (0.1.0)
- runit (4.3.0)
- acme (4.1.1)
- crond (0.1.0)
Installing Cookbook Gems:
Compiling Cookbooks...
Converging 3 resources
Recipe: postgresql::bin
* ruby_block[check_postgresql_version] action run (skipped due to not_if)
* ruby_block[check_postgresql_version_is_deprecated] action run (skipped due to not_if)
* ruby_block[Link postgresql bin files to the correct version] action run
- execute the ruby block Link postgresql bin files to the correct version
Running handlers:
Running handlers complete
Chef Client finished, 1/3 resources updated in 04 seconds
Checking PostgreSQL executables: OK
Checking if Grafana needs to be reset:
Status file found. Grafana is not vulnerable. Skipping reset.
Checking if Grafana needs to be reset: OK
Found /etc/gitlab/skip-auto-reconfigure, exiting...
_______ __ __ __
/ ____(_) /_/ / ____ _/ /_
/ / __/ / __/ / / __ `/ __ \
/ /_/ / / /_/ /___/ /_/ / /_/ /
\____/_/\__/_____/\__,_/_.___/
Upgrade complete! If your GitLab server is misbehaving try running
sudo gitlab-ctl restart
before anything else.
If you need to roll back to the previous version you can use the database
backup made during the upgrade (scroll up for the filename).
=== WARNING ===
Note that PostgreSQL 11 will become the minimum required PostgreSQL version in GitLab 13.0 (May 2020).
PostgreSQL 9.6 and PostgreSQL 10 will be removed in GitLab 13.0.
To upgrade, please see: https://docs.gitlab.com/omnibus/settings/database.html#upgrade-packaged-postgresql-server
=== WARNING ===
# Restart gitlab
gitlab-ctl restart
ok: run: alertmanager: (pid 15738) 0s
ok: run: gitaly: (pid 15749) 0s
ok: run: gitlab-exporter: (pid 15768) 1s
ok: run: gitlab-pages: (pid 15770) 0s
ok: run: gitlab-workhorse: (pid 15800) 1s
ok: run: grafana: (pid 15809) 0s
ok: run: logrotate: (pid 15818) 1s
ok: run: nginx: (pid 15824) 0s
ok: run: node-exporter: (pid 15830) 0s
ok: run: postgres-exporter: (pid 15912) 1s
ok: run: postgresql: (pid 15918) 0s
ok: run: prometheus: (pid 15926) 1s
ok: run: redis: (pid 15937) 0s
ok: run: redis-exporter: (pid 15942) 1s
ok: run: registry: (pid 15948) 0s
ok: run: sidekiq: (pid 15957) 0s
ok: run: unicorn: (pid 15963) 1s
# View status
gitlab-ctl status
run: alertmanager: (pid 15738) 14s; run: log: (pid 1815) 2647s
run: gitaly: (pid 15749) 14s; run: log: (pid 1839) 2647s
run: gitlab-exporter: (pid 15768) 14s; run: log: (pid 1809) 2647s
run: gitlab-pages: (pid 15770) 13s; run: log: (pid 1808) 2647s
run: gitlab-workhorse: (pid 15800) 13s; run: log: (pid 1820) 2647s
run: grafana: (pid 15809) 12s; run: log: (pid 1810) 2647s
run: logrotate: (pid 15818) 12s; run: log: (pid 1816) 2647s
run: nginx: (pid 15824) 11s; run: log: (pid 1812) 2647s
run: node-exporter: (pid 15830) 11s; run: log: (pid 1833) 2647s
run: postgres-exporter: (pid 15912) 11s; run: log: (pid 1835) 2647s
run: postgresql: (pid 15918) 10s; run: log: (pid 1804) 2647s
run: prometheus: (pid 15926) 10s; run: log: (pid 1805) 2647s
run: redis: (pid 15937) 9s; run: log: (pid 1821) 2647s
run: redis-exporter: (pid 15942) 9s; run: log: (pid 1806) 2647s
run: registry: (pid 15948) 8s; run: log: (pid 1803) 2647s
run: sidekiq: (pid 15957) 8s; run: log: (pid 1807) 2647s
run: unicorn: (pid 15963) 8s; run: log: (pid 1802) 2647s
# Start upgrading pg's version
gitlab-ctl pg-upgrade
The results are as follows:
Checking for an omnibus managed postgresql: OK
Checking if postgresql['version'] is set: OK
Checking if we already upgraded: NOT OK
Checking for a newer version of PostgreSQL to install
Upgrading PostgreSQL to 11.7
Checking if PostgreSQL bin files are symlinked to the expected location: OK
Waiting 30 seconds to ensure tasks complete before PostgreSQL upgrade.
See https://docs.gitlab.com/omnibus/settings/database.html#upgrade-packaged-postgresql-server for details
If you do not want to upgrade the PostgreSQL server at this time, enter Ctrl-C and see the documentation for details
Please hit Ctrl-C now if you want to cancel the operation.
There are many logs here, and most of the contents are omitted....
Running handlers:
Running handlers complete
Chef Client finished, 30/812 resources updated in 47 seconds
Running reconfigure: OK --- There's an automatic reconfigure
Waiting for Database to be running.
Database upgrade is complete, running analyze_new_cluster.sh
Toggling deploy page:rm -f /opt/gitlab/embedded/service/gitlab-rails/public/index.html
Toggling deploy page: OK
Toggling services:ok: run: alertmanager: (pid 17631) 0s
ok: run: gitaly: (pid 17643) 1s
ok: run: gitlab-exporter: (pid 17654) 0s
ok: run: gitlab-pages: (pid 17667) 1s
ok: run: grafana: (pid 17686) 0s
ok: run: logrotate: (pid 17705) 1s
ok: run: node-exporter: (pid 17719) 0s
ok: run: postgres-exporter: (pid 17728) 0s
ok: run: prometheus: (pid 17741) 1s
ok: run: redis-exporter: (pid 17753) 0s
ok: run: registry: (pid 17601) 5s
ok: run: sidekiq: (pid 17766) 1s
Toggling services: OK
==== Upgrade has completed ====
Please verify everything is working and run the following if so
sudo rm -rf /var/opt/gitlab/postgresql/data.9.6
sudo rm -f /var/opt/gitlab/postgresql-version.old
# View the current version of PG
cd /opt/gitlab/embedded/bin
./psql --version
psql (PostgreSQL) 11.7
3. Upgrade from 12.10.6 to 12.10.14
Here is a small version upgrade without major changes. We have finished the version upgrade of pg.
Upgrade steps:
# Stop the current gitlab gitlab-ctl stop # Pull the latest warehouse information # apt update # List the versions we want to install apt-cache madison gitlab-ce | grep '12.10.13' apt-get install gitlab-ce=12.10.14-ce.0 # Start gitlab again gitlab-ctl start gitlab-ctl status
4. Upgrade from 12.10.14 to 13.0.14
13. There are many changes at the beginning, but the details are very specific changes log
13.0 changes;
1. puma replaces unicorn as the default web server
2. pg11 is standard and support for pg9 and 10 is removed
3. The configuration ns of alertmanager is moved to monitoring
# Stop the current gitlab
gitlab-ctl stop
# Pull the latest warehouse information
# apt update
# List the versions we want to install
apt-cache madison gitlab-ce | grep '13.0.14'
apt-get install gitlab-ce=13.0.14-ce.0
# Start gitlab again
gitlab-ctl start
gitlab-ctl status
# Note that if there is a problem, we need to handle it with reconfigure
gitlab-ctl reconfigure
As follows:
Recipe: gitlab::gitlab-workhorse
* runit_service[gitlab-workhorse] action restart (up to date)
Recipe: nginx::enable
* runit_service[nginx] action restart (up to date)
Recipe: registry::enable
* runit_service[registry] action restart (up to date)
Recipe: monitoring::gitlab-exporter
* runit_service[gitlab-exporter] action restart (up to date)
Recipe: monitoring::prometheus
* execute[reload prometheus] action run
- execute /opt/gitlab/bin/gitlab-ctl hup prometheus
Running handlers:
Running handlers complete
Chef Client finished, 85/890 resources updated in 01 minutes 15 seconds
gitlab Reconfigured!Possible problems:
Question 1: gitlab appears to be working properly. However, if a CI is triggered, it will be found that the status of the CI is abnormal (the process neither reports an error nor goes down)
At this time, we still need to execute
gitlab-ctl reconfigure
Some logs are as follows:
Recipe: gitlab::gitlab-workhorse
* runit_service[gitlab-workhorse] action restart (up to date)
Recipe: nginx::enable
* runit_service[nginx] action restart (up to date)
Recipe: registry::enable
* runit_service[registry] action restart (up to date)
Recipe: monitoring::gitlab-exporter
* runit_service[gitlab-exporter] action restart (up to date)
Recipe: monitoring::prometheus
* execute[reload prometheus] action run
- execute /opt/gitlab/bin/gitlab-ctl hup prometheus
Running handlers:
Running handlers complete
Chef Client finished, 85/890 resources updated in 01 minutes 15 seconds
gitlab Reconfigured!Question 2: after the project is changed, you can see that these steps are completed in seconds from the console log after executing CICD (there seems to be something wrong), But in fact, the steps of packaging -- > pushing to the warehouse -- > publishing to K8s are not done
resolvent: Clean up the items under CI Cache and run down again CICD Just.
Rollback step
If you find a problem that cannot be solved for a period of time, you can consider starting a new gitlab instance through the previously created machine image.
1. Start a backup gitlab new ECS instance from image, mount the disk, and execute gitlab CTL restart to start the process
2. Change the backend node of SLB of gitlab machine to gitlab new
3. The CI/CD process is tested internally, and services can be provided externally when there are no problems