preface

In project development, it is sometimes necessary to use another project (third-party or independent project). At this time, you can use Git tools - sub modules To manage. When a sub module (GIT sub module add < project gitlab URL >) is added locally and pushed to gitlab, we expect gitlab CI to automatically clone or pull the corresponding dependent projects and build them normally. However, we may encounter:

fatal: could not read Username for 'https://gitserver.com/ ': No such device or address

This article will describe how to correctly use submables on Gitlab.

The first step is configuration gitmodules file

Method 1: provided on the official website

Using submodules on Gitlab CI/CD, the official website provides the following methods: Using Git submodules with GitLab CI/CD , about two points:

1. If the main project and sub module are under the same gitlab server, for example, the address of the main project: https://gitlab.com/android/android-main , subproject address: https://gitlab.com/android/android-sub Or https://gitlab.com/android/common-project/android-sub1 , then The gitmodules file can be configured using relative url addresses:

[submodule "android-sub"]
  path = android-sub
  url = ../../android-sub.git

1. If the main project and sub module are not in the same gitlab server, they can be in The gitmodules file can be configured using the absolute url address:

[submodule "android-sub"]
  path = android-sub
  url = https://gitserver.com/group/android-sub.git

Mode 2: actual project needs (recommended)

For the method provided on the official website, whether the main project and sub module are in or not in the same gitlab server, the open members permissions of sub projects are usually convergent. In a project group, only some people may have read and write permissions, while others only have read permissions.

In this case, you need to use the information provided by Gitlab Deploy tokens To configure gitmodules file.

There are two points about Deploy tokens:

1. Members with project permission Maintainer or Owner are responsible for the management (creation and deletion) of Deploy tokens;
2. Deploy tokens allows you to clone, push or pull items, but no account and password are required.

Create and configure Deploy tokens

Enter the sub project, the left menu bar, and open settings/repository/deploy_tokens:

After creation, copy username and deploy_token to configure gitmodules file:

[submodule "android-sub"]
 path = android-sub
 url = https://<username>:<deploy_token>@gitserver.com/android/android-sub.git

In this way, you can avoid Gitlab CI/CD permission problems.

Step 2: configure gitlab-ci.yml file

Using submodules in Gitlab jobs, the official website provides the following methods: Use Git submodules in CI/CD jobs , roughly speaking:

1. Ensure that the sub project and the main project are on the same gitlab server;
2. Yes gitlab-ci.yml file configuration GIT_SUBMODULE_STRATEGY: normal or recursive:

variables:
  GIT_SUBMODULE_STRATEGY: recursive

Configure GIT_SUBMODULE_STRATEGY: normal is equivalent to executing:

# Update the new URL to the file git/config
git submodule sync
# Update submodule from new URL
git submodule update --init

Configure GIT_SUBMODULE_STRATEGY: recursive is equivalent to executing:

# Copy the new URL to the local configuration
git submodule sync --recursive
# Update submodule from new URL
git submodule update --init --recursive

The third step is to ensure permissions

In the above two steps, you need to ensure that the project members who execute the above commands have the permission of Maintainer or Owner of the project in the main project and sub project.

Problems encountered

1.fatal: repository not found

If configured The gitmodules file uses the relative url address configuration method, which Gitlab CI encounters

Synchronizing submodule url for 'android-sub '
Entering 'android-sub'
Entering 'android-sub'
HEAD is now at 5b2e725 Feat. ci submodule test
remote: The project you were looking for could not be found.
fatal: repository 'https://gitlab-ci-token:xxxxxxxxxxxxxxxxxxxx@gitlab/android-sub.git/' not found
Unable to fetch in submodule path 'android-sub'
stdin: is not a tty
ERROR: Job failed: exit status 1

This is because the relative url address of the sub project and the main project is incorrect. For example, a company has two projects: https://gitlab.com/android/project1/android-main ， https://gitlab.com/android/project2/android-main , there is now a common subproject: https://gitlab.com/android/common-project/android-sub1 , the relative url address used when introducing public subprojects into the main project should be:

[submodule "android-sub"]
  path = android-sub1
  url = ../../../android-sub.git

2. Build failed, unable to find the relevant files (classes) of the subproject

After integrating the submodule, the project is built in Gitlab for the first time. The construction fails because the relevant files (classes) of the sub project cannot be found. View the job console log display:

HEAD is now at 691bc88845 Feat. ci Pull submodule pack
From https://gitlab.com/android/android-main
 + 691bc88845...246c8cfd6f feature/2.3.0-wangjiang -> origin/feature/2.3.0-wangjiang  (forced update)
Checking out 246c8cfd as feature/2.3.0-wangjiang...
Updating/initializing submodules recursively...
Synchronizing submodule url for 'android-sub'
Cloning into '/home/gitlab-runner/builds/aWHpgegt/0/studio/android/android-sub'...
Submodule path 'android-sub': checked out 'd26c5afc724dde0b78a64c45b3333900b32a6cf9'

At this time, it only means that the sub project has been cloned to the main project, but the main project after clone only contains the root folder of the sub project, and there are no files of sub modules downloaded in the folder.

To solve this problem, you can write a script (double check) in the main project to execute the command: git submodule update --init --recursive, such as (Android project script):

    def repositoryPath = "android-sub"
    def repositoryFile = file(repositoryPath)
    def childFiles = repositoryFile.listFiles()
    if (!repositoryFile.exists() || childFiles == null || childFiles.length == 0) {
        def cmd = 'git submodule update --init --recursive'
        exec {
            ExecSpec execSpec ->
                executable 'bash'
                args '-c', cmd
        }
    }

In addition, when the build is successful, the job console log should display:

HEAD is now at 0b26b59181 Feat. ci Pull submodule pack test
Checking out 0b26b591 as feature/2. 3.0-wangjiang...
Updating/initializing submodules recursively...
Synchronizing submodule url for 'android-sub'
Entering 'android-sub'
Entering 'android-sub'
HEAD is now at c8914ee Feat. Android todo list

Indicates that git has been executed at this time_ SUBMODULE_ STRATEGY: recursive.

3. Authority issues

Execute pipeline in Gitlab and read submodule s. The following problems are encountered:

fatal: could not read Username for 'https://gitserver.com/': No such device or address
fatal: clone of 'https://gitserver.com/android/android-sub.git' into submodule path '/home/gitlab-runner/builds/hF55UpTC/0/studio/android/android-sub' failed
Failed to clone 'android-sub'. Retry scheduled
Cloning into '/home/gitlab-runner/builds/hF55UpTC/0/studio/android/android-sub''...
fatal: could not read Username for 'https://gitserver.com/'': No such device or address
fatal: clone of 'https://gitserver.com/android/android-sub.giit' into submodule path '/home/gitlab-runner/builds/hF55UpTC/0/studio/android/android-sub' failed
Failed to clone 'android-sub' a second time, aborting
stdin: is not a tty
ERROR: Job failed: exit status 1

This is the problem of project permission. Add the permission to execute the main project to the corresponding sub project. The permission is at least Maintainer or Owner. Or use the first step, configuration There are two pairs of modes in the gitmodules file Gitmodules file for configuration.