Hello, I'm a fisherman from Go school.

Everyone is familiar with caching. Baidu Encyclopedia is defined as:

Cache refers to the memory that can exchange high-speed data. It exchanges data with the CPU before the memory, so the speed is very fast.

Therefore, cache is used to improve the speed of data exchange. The cache we are going to talk about today is not the cache in the CPU, but the cache of the database in the application. The application program reads data from the cache before the database to reduce the pressure on the database and improve the reading performance of the application program.

In the actual project, I believe everyone has encountered a similar situation: the amount of data is small, but access is frequent (such as national standard administrative region data), and you want to store it completely in local memory. This can avoid direct access to mysql or redis, reduce network transmission and improve access speed. How should it be realized?

This paper introduces a method often used in Go project: load the data from the database to the local file, and then load the data in the file into memory. The data in memory is directly used by the application. As shown in the figure below:

This article will ignore the process from the database to the local file, because this link is a process of uploading and downloading files to the local file. So we will focus on how to load data from local files into memory.

01 objectives

In the Go language project, the data of the local file is loaded into the memory of the application for direct use by the application.

Let's break down the goal into two goals:

1. When the program starts, initialize the data of the local file into memory, that is, cold start

2. During the program running, when the local file is updated, the data is updated to memory.

02 code implementation

The main purpose of this article is to explain the realization of the goal, so it will not take you step by step analysis, but to provide you with a reference implementation by explaining the implemented code.

Therefore, we first give the class diagram we designed:

As can be seen from the class diagram, there are two main structures: FileDoubleBuffer and LocalFileLoader. Let's explain the properties and method implementation of these two structures one by one.

2.1 scenario assumptions

Taking the weather conditions of the city as an example, we store the real-time temperature and wind force of each city in the file in json format, and update the file when the temperature or wind force of the city changes. As follows:

{
    "beijing": {
        "temperature": 23,
        "wind": 3
    },
    "tianjin": {
        "temperature": 20,
        "wind": 2
    },
    "shanghai": {
        "temperature": 20,
        "wind": 20
    },
    "chongqing": {
        "temperature": 30,
        "wind": 10
    }
}

2.2 calling main

Here, we first give an example of calling the main function. According to the implementation of the main function, we look at the implementation of the two main structures in the figure step by step. The code is as follows:

//The first step is to define the structure of the data in the loading file
type WeatherContainer struct {
    Weathers map[string]*Weather //Actual weather for each city
}
//Weather conditions of each city in the file data
type Weather struct {
    Temperature int //Current temperature ` json:"temperature"`
    Wind        int //Current wind ` json:"wind"`
}
func main() {
    pwd, _ := os.Getwd()
    //Loaded file path
    filename := pwd + "/cache/cache.json"
    //Initialize local file loader
    localFileLoader := NewLocalFileLoader(filename)
    //Initialize the file buffer instance and use localFileLoader as the underlying file buffer
    fileDoubleBuffer := NewFileDoubleBuffer(localFileLoader)

    // Starting to load the contents of the file into the buffer variable is essentially loading the file data through load and reload    
    fileDoubleBuffer.StartFileBuffer()

    //get data
    weathersConfig := fileDoubleBuffer.Data().(*WeatherContainer)
    fmt.Println("weathers:", weathersConfig.Weathers["beijing"])

    blockCh := make(chan int)
    //This channel is used to block the process from ending, so that the reload process can be executed
    <-blockCh
}

2.3 FileDoubleBuffer structure and Implementation

The function of this structure is mainly oriented to the application (here we are the main function) for the application to directly obtain data from memory, that is, bufferData. The structure is defined as follows:

// The main application mainly obtains data for this structure
type FileDoubleBuffer struct {
    Loader     *LocalFileLoader
    bufferData []interface{}
    curIndex   int32
    mutex      sync.Mutex
}

First, look at the properties of the structure:

Loader: is a LocalFileLoader type (the structure will be defined later), which is used to load data from specific files into bufferData.

bufferData slice: the variable that receives the data in the file. On the one hand, the data in the file will be loaded into the variable. On the other hand, the application gets the desired data information directly from the variable, not the file or database. The data type of this variable is interface {}, which indicates that any type of data structure can be loaded. In addition, we note that the variable is a slice, which has only two elements. The two elements have the same data structure and are used in combination with the curIndex attribute.

curIndex: this attribute specifies which index data is being used by the current bufferData. The value of this attribute circulates between 0 and 1 for switching between new and old data. For example, the data of the index element curIndex=1 is currently used externally. When there is new data in the file, first load the data of the file into the index 0 element. When the data of the file is completely loaded, then point the value of curIndex to 0. In this way, when there is new data in the file to refresh the data in memory, it will not affect the application's use of old data.

Let's look at the functions in FileDoubleBuffer:

Data() function

The application uses this function to obtain the dataBuffer data in FileDoubleBuffer. The specific implementation is as follows:

func (buffer *FileDoubleBuffer) Data() interface{} {
    // bufferData actually stores two elements with the same structure for switching between new and old data
    index := atomic.LoadInt32(&buffer.curIndex)
    return buffer.bufferData[index]
}

load function

This function is used to load the data in the file into bufferData. The code implementation is as follows:

func (buffer *FileDoubleBuffer) load() {
  buffer.mutex.Lock()
  defer buffer.mutex.Unlock()
  //Determine which element of the bufferData array is currently used
  // Since there are only two elements in bufferData, it is either 0 or 1
  curIndex := 1 - atomic.LoadInt32(&buffer.curIndex)

  err := buffer.Loader.Load(buffer.bufferData[curIndex])
  if err == nil {
    atomic.StoreInt32(&buffer.curIndex, curIndex)
  }
}

reload function

Used to load new data from the file into bufferData. In fact, it is a for loop, which executes the load function every certain time. The code is as follows:

func (buffer *FileDoubleBuffer) reload() {
  for {
    time.Sleep(time.Duration(5) * time.Second)
    fmt.Println("Start loading...")
    buffer.load()
  }
}

StartFileBuffer function

This function is used to start data loading and updating. The code is as follows:

func (buffer *FileDoubleBuffer) StartFileBuffer() {
  buffer.load()
  go buffer.reload()
}

NewFileDoubleBuffer(loader *LocalFileLoader) *FileDoubleBuffer function

This function initializes the FileDoubleBuffer instance. The code is as follows:

func NewFileDoubleBuffer(loader *LocalFileLoader) *FileDoubleBuffer {
  buffer := &FileDoubleBuffer{
    Loader:   loader,
    curIndex: 0,
  }

  //Memory space is allocated here to load the values in the file into the variable for use by the application
  buffer.bufferData = append(buffer.bufferData, loader.Alloc(), loader.Alloc())
  return buffer
}

2.4 structure and implementation of localfileloader
Because we first load the data from the database to the local file, and then load the data of the file into the memory buffer, we have the LocalFileLoader structure. The function of this structure is to load specific file data and detect file updates. LocalFileLoader is defined as follows:

type LocalFileLoader struct {
  filename       string //File to be loaded, full path
  lastModifyTime int64  //Last modification time of the file
}

First, let's look at the properties of the structure:

filename: specify a specific file name to load data from the file

modifyTime: the time when the file was last loaded. If the update time of the file is greater than this time, it indicates that the file has been updated

Let's look at the functions in LocalFileLoader:

Load (filename, string, I interface) function

This function loads the data in the filename file into the variable i. This variable i is actually an element in bufferData passed in from FileDoubleBuffer. The code is as follows:

// Here, the i variable is actually an element in the dataBuffer passed in from the load method of the filedoublebauffer structure
func (loader *LocalFileLoader) Load(i interface{}) error {
    // The WeatherContainer structure is defined based on the data stored in the file, which will be described later
    weatherContainer := i.(*WeatherContainer)
    fileHandler, _ := os.Open(loader.filename)
    defer fileHandler.Close()
    body, _ := ioutil.ReadAll(fileHandler)
    _ := json.Unmarshal(body, &weatherContainer.Weathers)
    // Here we omit those err judgments
    return nil
}

DetectNewFile() function

This function is used to check whether the filename file is updated. If the file modification time is greater than modifyTime, FileDoubleBuffer will load the new data into the dataBuffer. The code is as follows:

// This function checks whether the file has been updated. If so, it returns true; otherwise, it returns false
func (loader *LocalFileLoader) DetectNewFile() bool {
    fileInfo, _ := os.Stat(loader.filename)
    //The modification time of the file is longer than the last modification time, indicating that the file has been updated
    if fileInfo.ModTime().Unix() > loader.lastModifyTime {
        loader.lastModifyTime = fileInfo.ModTime().Unix()
        return true
    }
    return false
}

*Alloc() interface{} *

It is used to allocate specific variables for loading the data in the file. The variables allocated here will eventually be stored in the dataBuffer data in FileDoubleBuffer. The code is as follows:

// Assign specific variables to carry the specific contents in the file. The variable structure needs to be consistent with the structure in the file
func (loader *LocalFileLoader) Alloc() interface{} {
    return &WeatherContainer{
        Weathers: make(map[string]*Weather),
    }
}

You also need a function to initialize the LocalFileLoader instance:

//Specify the file path path to load
func NewLocalFileLoader(path string) *LocalFileLoader {
    return &LocalFileLoader{
        filename: path,
    }
}

summary

This method is generally suitable for scenarios with small amount of data and frequent reading. In the figure at the beginning of the article, we can see that because the server is often a cluster, the file content on each machine may be temporarily different, so this implementation is not suitable for scenarios with strong consistency requirements for data.