Lightweight timed task tool Cronicle: Previous

This article will introduce a lightweight and simple Web UI, which is suitable for small and medium-sized teams and individuals: Cronicle.

This is the first article about Cronicle. It mainly talks about the common problems of this software under container packaging and the ideas of container packaging.

Write in front

It has been more than five years since Cronicle officially started open source in 2016. The first time I noticed this software was in 2018, when I was selecting the right timing tool for my HomeLab. In the past few years, we can see that the software has been optimized in detail functions. At present, what has been done has been relatively perfect. Especially in the past two years, there are basically no major functional revisions and changes.

In addition to supporting basic scheduled tasks, the software also includes many useful functions:

Support multiple instances to build a distributed timed task system
It has the ability of fault self-healing and automatic service migration
Support service discovery and automatic networking
Allows you to view task execution status in real time
It has a basic plug-in system and supports the use of any language and method to expand the capability
You can create scheduled tasks for different time zones
You can queue tasks that take a long time to be degraded
Basic task performance icons and statistics
Have open API and support application API key
Web Hook notification capability

If you only use it as a task trigger, its memory resource consumption will be very small. In my re encapsulated image, running programs for more than 20 hours, the panel shows that the memory consumption is only over 80MB, while the result of docker stats is less than 50MB.

CONTAINER ID   NAME                                    CPU %     MEM USAGE / LIMIT     MEM %     NET I/O           BLOCK I/O    PIDS
ec4d5ab18b68   docker_cronicle_1                       5.56%     46.09MiB / 15.64GiB   0.29%     4.71MB / 20.6MB   0B / 0B      11

Compared with the traditional way of running tasks, like most software, it comes with a task execution list. The default minimum accuracy is 10 seconds, which is enough for use in most scenarios.

At the same time, it also supports the output of simple statistical summaries for tasks.

In order to use it better, we need to encapsulate it in containers first.

If you can't wait to start using it, you can jump to the next chapter "using Cronicle in containers".

Encapsulating cronicles with containers

The first time to use containers to encapsulate this software should be in 2018 This PR Before and after. Later, although many netizens encapsulated the software, there are some imperfections. For example, if you migrate or rebuild the container, the software will not run unless you Repair manually ； For example, when the software is used for the first time, it needs to wait at least one minute to make the software network successfully, and then it can be used Start using...

So, in this article, let's solve these two problems first.

Reduce Cornicle startup latency

stay jhuckaby/Cronicle/blob/master/lib/engine.js In, there is a function called startup, about more than 100 lines, which records what needs to be done after Cronicle is started. The logic that we need to wait 60 seconds to use the software is mainly this part:

...

startup: function(callback) {
    // start cronicle service
    var self = this;
    this.logDebug(3, "Cronicle engine starting up");

    // create a few extra dirs we'll need

    ...

    // archive logs daily at midnight
    this.server.on('day', function () {
        self.archiveLogs();
    });

    // determine master server eligibility
    this.checkMasterEligibility(function () {
        // master mode (CLI option) -- force us to become master right away
        if (self.server.config.get('master') && self.multi.eligible) self.goMaster();

        // reset the failover counter
        self.multi.lastPingReceived = Tools.timeNow(true);

        // startup complete
        callback();
    });
}
...

Because the default software running environment is assumed to be a multi machine distributed environment, there is a long process of service discovery and registration.

In this article, we mainly run in stand-alone mode, so we can make some minor modifications to it, let self.goMaster(); The action of registering the current running instance can be executed before this.checkmasterability.

Considering that the Cronicle team takes a long time to receive PR, in order to quickly realize this function, it can adopt self-made patches and "patch application" in the process of container construction.

After making appropriate adjustments to the program, execute diff - U lib / engine.js / TMP / engine. JS > engine.patch to easily create a program patch similar to the following.

--- lib/engine.js	2021-06-17 19:03:36.000000000 +0000
+++ /tmp/engine.js	2021-12-04 09:50:13.000000000 +0000
@@ -152,7 +152,8 @@
 		this.server.on('day', function() {
 			self.archiveLogs();
 		} );
-		
+		// for docker env
+		self.goMaster();
 		// determine master server eligibility
 		this.checkMasterEligibility( function() {
 			// master mode (CLI option) -- force us to become master right away

It is also very simple to apply a patch. You only need to execute patch - P3 < engine.patch lib / engine.js.

We will save the patch file and use it later.

Write a startup script suitable for use in the container

The above clearly mentioned the problems that are easy to occur and the root causes of the problems, so it is very simple to write a program to solve these problems:

#!/usr/bin/env node

const { existsSync, unlinkSync } = require('fs');
const { dirname } = require('path');
const { hostname, networkInterfaces } = require('os');
const StandaloneStorage = require('pixl-server-storage/standalone');

if (existsSync("./logs/cronicled.pid")) unlinkSync("./logs/cronicled.pid");

process.chdir(dirname(__dirname));

const config = require('../conf/config.json');

const storage = new StandaloneStorage(config.Storage, function (err) {
    if (err) throw err;

    const dockerHostName = (process.env['HOSTNAME'] || process.env['HOST'] || hostname()).toLowerCase();

    const networks = networkInterfaces();
    const [ip] = Object.keys(networks).
        filter(eth => networks[eth].
            filter(addr => addr.internal === false && addr.family === "IPv4").length).
        map(eth => networks[eth])[0];

    const data = {
        "type": "list_page",
        "items": [{ "hostname": dockerHostName, "ip": ip.address }]
    };

    const key = "global/servers/0";
    storage.put(key, data, function () {
        storage.shutdown(function () {
            console.log("Record successfully saved: " + key + "\n");
            storage.get(key, function (_, data) {
                if (storage.isBinaryKey(key)) {
                    console.log(data.toString() + "\n");
                } else {
                    console.log(((typeof (data) == 'object') ? JSON.stringify(data, null, "\t") : data) + "\n")
                }
                storage.shutdown(function () {
                    console.log("Docker Env Fixed.");
                    require('../lib/main.js');
                });
            });
        });
    });
});

The above less than 50 lines of code mainly do several things:

Check whether there are PID files left over from the previous running program. If so, clean them up to avoid affecting the program startup.
Update the IP and host name in the container environment that is actually running to the program configuration to prevent the program from starting correctly.
Run the program in the foreground mode to avoid handling other programs and ensure that the container is simple enough.

Write container image file

Because the shell will be used in the actual operation of Cronicle, it is not recommended to use the previous one Using language centric container based mirroring of destroy The method mentioned in the article can be used to minimize image construction, and only ordinary two-phase construction can be used:

FROM node:16 AS Builder
ENV CRONICLE_VERSION=0.8.62
WORKDIR /opt/cronicle
COPY Cronicle-${CRONICLE_VERSION}.tar.gz /tmp/
RUN tar zxvf /tmp/Cronicle-${CRONICLE_VERSION}.tar.gz -C /tmp/ && \
    mv /tmp/Cronicle-${CRONICLE_VERSION}/* . && \
    rm -rf /tmp/* && \
    npm install --registry=https://registry.npm.taobao.org
COPY ./patches /tmp/patches
RUN patch -p3 < /tmp/patches/engine.patch lib/engine.js


FROM node:16-alpine
COPY --from=builder /opt/cronicle/ /opt/cronicle/
WORKDIR /opt/cronicle

ENV CRONICLE_foreground=1
ENV CRONICLE_echo=1
ENV CRONICLE_color=1
ENV debug_level=1

ENV HOSTNAME=main-server

RUN node bin/build.js dist && \
    bin/control.sh setup
COPY docker-entrypoint.js ./bin/
CMD ["node", "bin/docker-entrypoint.js"]

Because even ordinary two-phase construction and basic image switching can reduce the image volume of the software from 1G to less than 150M, which is more suitable for distribution and storage.

cronicle                                      latest                              c0575a5b900b   22 hours ago    1.04GB
cronicle                                      latest                              e31626eac385   3 seconds ago        146MB

Using Cronicle in containers

To make Cronicle run quickly, you can use my pre built container image. In order to make this image run normally, we need two orchestration files for program initialization and normal operation. First, write the normal operation files:

version: "3.6"

services:

  cronicle:
    image: soulteary/cronicle:0.8.62
    restart: always
    hostname: cronicle
    ports:
      - 3012:3012
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /etc/timezone:/etc/timezone:ro
      - ./data/data:/opt/cronicle/data
      - ./data/logs:/opt/cronicle/logs
      - ./data/plugins:/opt/cronicle/plugins
    extra_hosts:
      - "cronicle.lab.io:0.0.0.0"
    environment:
      - TZ=Asia/Shanghai
    healthcheck:
      test: ["CMD-SHELL", "wget --no-verbose --tries=1 --spider localhost:3012/api/app/ping || exit 1"]
      interval: 5s
      timeout: 1s
      retries: 3
    logging:
        driver: "json-file"
        options:
            max-size: "10m"

After saving the above file as docker-compose.yml, continue to write the configuration for initialization run:

version: "3.6"

services:

  cronicle:
    image: soulteary/cronicle:0.8.62
    hostname: cronicle
    command: /opt/cronicle/bin/control.sh setup
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /etc/timezone:/etc/timezone:ro
      - ./data/data:/opt/cronicle/data
      - ./data/logs:/opt/cronicle/logs
      - ./data/plugins:/opt/cronicle/plugins
    environment:
      - TZ=Asia/Shanghai

Save the above file as docker-compose.init.yml, and then execute docker-compose -f docker-compose.init.yml up. Not surprisingly, you will get something similar to the following:

cronicle_1  | 
cronicle_1  | Setup completed successfully!
cronicle_1  | This server (main) has been added as the single primary master server.
cronicle_1  | An administrator account has been created with username 'admin' and password 'admin'.
cronicle_1  | You should now be able to start the service by typing: '/opt/cronicle/bin/control.sh start'
cronicle_1  | Then, the web interface should be available at: http://main:3012/
cronicle_1  | Please allow for up to 60 seconds for the server to become master.
cronicle_1  | 
docker-cronicle_cronicle_1 exited with code 0

Then use docker compose up - D to start the service. After a few seconds, use docker compose PS to check the service, and you can see the result that the service is running normally.

           Name                         Command                  State               Ports         
---------------------------------------------------------------------------------------------------
docker-cronicle_cronicle_1   docker-entrypoint.sh node  ...   Up (healthy)   0.0.0.0:3012->3012/tcp

At this point, we can start using the software by opening localhost:3012 in the browser. The default account and password of the software are admin.

Because the software function interface is very intuitive, I won't elaborate on the basic use of the software here.

last

Distributed use, in container disaster recovery and transfer, and plug-in writing may be suitable for the next article on Cronicle. I have the relevant code in the text Upload to GitHub , you can take it yourself if you need it.

I would like to dedicate this article to the small partners in the newly established technology discussion group.

–EOF

We have a small tossing group, which gathers hundreds of little friends who like tossing.

Without advertising, we will talk about some problems in software and hardware, HomeLab and programming, and occasionally share some information about the technology salon in the group.

Friends who like to toss are welcome to scan the code to add friends. (if you add a friend, please note your real name and indicate the source and purpose, otherwise it will not pass the review)

About tossing into the group

If you think the content is practical, you are welcome to like it and share it with your friends. Thank you here.

This article uses the "signature 4.0 International (CC BY 4.0)" license agreement. You are welcome to reprint, modify or use it again, but you need to indicate the source. Signature 4.0 International (CC BY 4.0)

Author: Su Yang

Created on: December 5, 2021
Statistics: 4412 words
Reading time: 9 minutes
Link to this article: https://soulteary.com/2021/12/05/cronicle-a-lightweight-tool-for-timed-tasks-part-1.html

Keywords: crontab Docker cron

Added by nosher on Mon, 06 Dec 2021 03:31:43 +0200

Programming VIP