Python - mmap shared memory

In the process of program running, you may encounter the need for information interaction between processes or languages of different platforms. The existence of hard disk is a solution, but the speed is too slow. python's mmap library provides a practical scheme of shared memory, which can complete the interaction of information between memory.

brief introduction

Shared memory

Memory sharing means that two different processes share memory: the same piece of physical memory is mapped to the respective process address space of two processes. The physical memory has been specified in size (the size must be larger than what is actually written) and name. When you need to write, find the memory name, and then write to the memory. When you need to read, first know how big you want to read (because the physical memory is larger than what you want to read, and some "empty" things will be read if you read all), and then find the physical block with the corresponding name, and then read. It's that simple.

mmap

mmap is a method of virtual memory mapping file, that is, a file or other object is mapped to the process address space to realize the one-to-one mapping relationship between the file disk address and a virtual address in the process virtual address space. The theoretical explanation of mmap in the system can be seen Baidu Encyclopedia and Wikipedia Description and Introduction to mmap function , the instructions here are for the use of mmap block in Python.

  • Official website documents: https://docs.python.org/2/library/mmap.html

usage method

Create: creates and returns an mmap object

m = mmap.mmap(fileno, length[, flags[, prot[, access[, offset]]]])
  • Fileno: file descriptor, which can be the fileno() method of file object or from OS Open(), open the file before calling mmap(), and close it when it is no longer needed.
os.O_RDONLY   Open as read-only Read only
os.O_WRONLY   Open in write only mode Write only
os.O_RDWR     Open in read-write mode Read and write
os.O_APPEND  Open in append mode  
os.O_CREAT   Create and open a new file
os.O_EXCL     os.O_CREAT| os.O_EXCL If the specified file exists, an error is returned
os.O_TRUNC    Open a file and truncate it to zero length (must have write permission)
os.O_BINARY          Open file in binary mode (no conversion)
os.O_NOINHERIT        Prevent creation of a shared file descriptor
os.O_SHORT_LIVED
os.O_TEMPORARY        And O_CREAT Create temporary files together
os.O_RANDOM         Cache optimization,However, random access from disk is not limited
os.O_SEQUENTIAL   Cache optimization,But it does not restrict sequential access from disk
os.O_TEXT           Open file in text mode (conversion)

  • **length: * * the size of the file part to be mapped (in bytes). If this value is 0, the whole file will be mapped. If the size is greater than the current size of the file, the file will be expanded.

  • flags: MAP_PRIVATE: this memory mapping is only available to this process; mmap.MAP_SHARED: share the memory mapping with other processes. All processes mapping the same file can see the changes made by one of them;

  • **prot: **mmap.PROT_READ, mmap.PROT_WRITE and MMAP PROT_ WRITE | mmap. PROT_ READ. The last one means that it can be read and written at the same time.

  • **Access: * * there are optional parameters in mmap. The values of access are:

ACCESS_READ: Read access.
ACCESS_WRITE: Write access, default.
ACCESS_COPY: Copy access, do not write changes to files, use flush Write changes to file.

Object method

  • m.close()

Close m the corresponding file;

  • m.find(str, start=0)

Starting from the start subscript, find the subscript of the substring str from left to right in m;

  • m.flush([offset, n])

Brush n bytes from offset in m into the corresponding file;

  • m.move(dstoff, srcoff, n)

Equal to m[dstoff:dstoff+n] = m[srcoff:srcoff+n], copy n bytes from srcoff to n bytes from dstoff, which may cover the overlapping part.

  • m.read(n)

Return a string, read up to n bytes from the file corresponding to m, and move the position pointer of the file corresponding to m backward;

  • m.read_byte()

Return a 1-byte string, read 1 byte from the file corresponding to m, and call read if EOF is reached_ Byte(), an exception ValueError is thrown;

  • m.readline()

Returns a string from the current location of m corresponding file to the next '\ n', and returns an empty string if the file is in EOF when calling readline();

  • m.resize(n)

Change the length of m to n, and the length of M is independent of the length of the corresponding file of M;

  • m.seek(pos, how=0)

The seek operation of the same file object changes m the current location of the corresponding file;

  • m.size()

Return the length of the file corresponding to m (not the length len(m)) of the m object;

  • m.tell()

Return m the current location of the corresponding file;

  • m.write(str)

Write str to the current location of the m corresponding file. If there is not enough space left from the current location of the m corresponding file to the end of m, len(str), throw ValueError;

  • m.write_byte(byte)

Write a byte (corresponding to a character) to the current location of the M corresponding file, in fact m.write_byte(ch) equals m.write(ch). If the current position of the file corresponding to m is at the end of M, that is, the space left from the current position of the file corresponding to m to the end of M is less than 1 byte, write() throws an exception ValueError and write_byte() does nothing.

Use example

Write data into shared memory

import ctypes
import mmap   # Core library
import os
import struct
import numpy as np

# Create memory mapped file handle
fd = os.open('tmp/mmaptest', os.O_CREAT | os.O_TRUNC | os.O_RDWR)

# Create memory buffer
# not win32
buf = mmap.mmap(fd, mmap.PAGESIZE, mmap.MAP_SHARED, mmap.PROT_WRITE)
# win32
buf = mmap.mmap(fd, 67108864, access = mmap.ACCESS_WRITE)

# Write files to buf (binary format)
f = open('tmp/test2.bmp', 'rb').read()
buf.write(b'abc') # Write string
buf.write(f) # write file

# Current pointer position
buf.tell()

# Move pointer to 11th byte
buf.seek(10)

# Writes information in memory to a file
buf.flush(0, 100)

# close
buf.close()

Read data from shared memory

import mmap
import os
import struct
import cv2
import numpy as nps

# Create memory mapped file handle
fd = os.open('share_memory/tmp/mmaptest', os.O_RDONLY)

# Create memory buffer
# not win32
buf = mmap.mmap(fd, mmap.PAGESIZE, mmap.MAP_SHARED, mmap.PROT_READ)
# win32
buf = mmap.mmap(fd, 67108864, access = mmap.ACCESS_READ)

# Reads and prints the memory of the specified area in the buffer
string_length = 136
string, = struct.unpack('{}s'.format(string_length), buf[:string_length])
print(string)

# Convert some areas to the desired format and use (image)
np_str = np.fromstring(buf[:string_length], dtype='uint8')
img = cv2.imdecode(np_str, flags=-1)

# Pure image pixel data (just convert the string to uint8)
data = np.fromstring(buf[:string_length], dtype='uint8')
img = data.reshape([10,10])

The code will not run smoothly, only providing ideas for use

reference material

  • https://docs.python.org/2/library/mmap.html
  • https://zhuanlan.zhihu.com/p/166330573
  • https://www.cnblogs.com/zhoujinyi/p/6062907.html

Keywords: Python Back-end

Added by Threehearts on Thu, 03 Feb 2022 22:06:14 +0200

Popular Keywords

©2024 Programming VIP