Restoration and enhancement of specified images
Original image 1-1
The experimental process is as follows:
According to the observation, it is not difficult to find that there is impulse noise in the image, so the median filter is used to filter and remove impulse noise.
The filtering of impulse noise is not limited to the filtering method. The author has tried other ideas, but because the effect is slightly rough compared with the median value, it is only described below as an additional scheme.
For the bright lines in the image, the method of truncated Fourier Transform for frequency domain filtering is adopted to try to restore.
The method is described as follows:
According to the shape of bright lines and the cognition of space waves, we can think that these bright lines are composed of space waves of a certain frequency, that is, we may filter them in the frequency domain. However, a question should be raised here: can the existing FT deal with the space wave frequency constituting the bright line?
As we all know, Fourier Transform is an important means for us to understand the frequency domain information, but this method is still not perfect. We must note that the frequency domain information obtained by Fourier Transform does not contain the spatial information of the original image (for this, the shift invariance of FT is a good example), and emphasizes the global frequency information. FT decomposes the image into a series of periodic and uniform space waves in the image. Therefore, the change of the amplitude of any component is bound to affect the whole image. Due to the continuity of space waves, we can't specify the amplitude change in only one region. To sum up, we need to optimize ft: add spatial information and obtain spectrum.
Fortunately, although it is difficult for us to introduce spatial information into the exquisite ft, we can change the image, that is, the image is intercepted and FT is performed. At this time, our processing in the frequency domain will not affect the position outside the intercepted area, and the introduction of spatial information is indirectly realized.
Truncated FT results 1-2
Other schemes for impulse noise treatment:
For impulse noise, median filter has this good effect, but it also inevitably causes blur effect on the image. In the process of filtering, we can find that there are many unnecessary operations in median filtering: even non noise points are still replaced by median! So we optimize with this point: can we determine the position of noise in the image?
According to the characteristics of impulse noise, it has an extremely significant difference from the pixels in the neighborhood, or has an obvious edge with normal pixels. Therefore, we can try to judge the noise by edge detection. Note here that the edge detection results in not only noise, but also the edge of the normal image. Although the edge should not be processed, it has reduced the range of error processing relative to the median. Therefore, the image edge is denoised here
After we get the noise position, we need to know the pixel value of the original image at this point. According to information theory, we can make the following assumption: there is a linear combination in a neighborhood, so that the pixel value of the point can be represented. Based on this assumption, we can use other pixels in the neighborhood to speculate. Due to the author's limited level, the effect of adaptive window and nonlinear fitting will be better.
See the experimental results attached for the experimental results
After consulting relevant materials, it is found that bright lines can be treated by corrosion. The general process is as follows:
Firstly, the boundary to be corroded needs to be determined in the corrosion operation. Therefore, the edge detection of the image is still carried out
Edge detection 1-3
The back edge information divides the image into several connected areas
Connected set 1-4
Take the two connected sets with the largest area as the image mask, and reverse the mask to make the Boolean value including the area to be processed true
Image mask 1-5
According to the observation, the part to be removed is rectangular. Assuming that the mask of half of the pixels in a row is true, it is determined as the area to be processed. The corrosion area can be repaired after confirming the line
1. Program code
import cv2 as cv
import numpy as np
import matplotlib.pyplot as plt
import tool as tk # See the end of the text for the original code
def showimg(name, scr):
length = len(scr)
for i in range(length):
cv.imshow(name[i], scr[i])
def w(name, src):
cv.imwrite('../img/'+name+'.jpg', src)
def different(img1, img2):
cv.imshow('different', np.array((img1 - img2), dtype=np.uint8))
cv.waitKey(0)
def solution1(img):
recover = np.array(img, dtype=np.uint8)
mask = np.empty_like(recover, dtype=np.uint8)
time = 3
for i in range(time):
mask = tk.detectedge(recover, 80)
recover = tk.interpolation(recover, mask, method='bilinear')
name = ['img', 'mask', 'res']
showimg(name, [img, mask, recover])
cv.waitKey(0)
cv.destroyAllWindows()
def solution2(img):
img = cv.cvtColor(img, cv.COLOR_RGB2GRAY)
length, width = img.shape
slice_length, slice_width = 40, width
mask = np.ones((slice_length, slice_width))
for i in range(slice_length):
if i in range(np.int(slice_length / 2) - 1, np.int(slice_length / 2) + 2):
continue
mask[i, np.int(slice_width / 2)] = 0
res = tk.ppf(img, slice_length, mask, begin=5, size=1)
res = tk.score(res, method='mid')
img_back = np.array(res, dtype=np.uint8)
eq_img_back = cv.equalizeHist(img_back)
tk.histogram([img_back, eq_img_back])
cv.imshow('res' ,eq_img_back)
cv.waitKey(0)
cv.destroyAllWindows()
def get_area(edge):
vis = np.zeros_like(edge, dtype=np.bool)
area_mark = np.zeros_like(edge, dtype=np.int)
height, width = edge.shape
area_kind, area = (1, [0])
move = np.array([[0, 1], [0, -1], [1, 0], [-1, 0]], dtype=np.int)
check = lambda x, y: x >= 0 and y>= 0 and x < height and y < width and not vis[x, y] and edge[x, y]==0
for x in range(height):
for y in range(width):
if vis[x, y] or edge[x, y] > 0:
continue
queue, vis[x, y], area_mark[x, y], amount = (np.array([ [x, y] ]), True, area_kind, 1)
while queue.shape[0] != 0:
pos = np.array(queue[0])
queue = np.delete(queue, 0, 0)
for i in range(4):
next_pos = np.array([ [pos[0] + move[i, 0], pos[1] + move[i, 1]] ])
if check(next_pos[0, 0], next_pos[0,1]):
queue = np.append(queue, next_pos, axis=0)
vis[next_pos[0, 0], next_pos[0, 1]], area_mark[next_pos[0, 0], next_pos[0, 1]] = (True, area_kind)
amount = amount + 1
area_kind = area_kind + 1
area.append(amount)
interval = 31
visual = np.zeros((height, width, 3), dtype=np.uint8)
for x in range(height):
for y in range(width):
kind = area_mark[x, y]
for i in [2, 1, 0]:
visual[x, y, i] = 255 - interval * (kind % (interval + 1))
kind = np.int(kind / (interval + 1))
return area_mark, np.array(area, dtype=np.int), visual
def find_rect(mask):
height, width = mask.shape
limit = np.int(width / 2)
for x in range(height):
num = 0
for y in range(width):
num = num + 1 if mask[x, y] else num
if num < limit:
mask[x] = np.full((1, width), False)
return mask
def remove_contain_edge(area_mark, mark, RGB=(156, 93, 139)):
height, width = area_mark.shape
mask = np.zeros_like(area_mark, dtype=np.bool)
move = np.array([[0, 1], [0, -1], [1, 0], [-1, 0]], dtype=np.int)
check = lambda x : x[0] >= 0 and x[1] >= 0 and x[0] < height and x[1] < width
for x in range(height):
for y in range(width):
if area_mark[x, y] != mark:
continue
num = 0
for i in range(4):
next_pos = [x + move[i, 0], y + move[i, 1]]
if check(next_pos):
num = num + np.int(area_mark[next_pos[0], next_pos[1]] == mark)
if num >= 3:
mask[x, y] = True
to_visual = np.zeros((height, width, 3), dtype=np.uint8)
mask_index = np.nonzero(mask)
index_length = len(mask_index[0])
for i in range(index_length):
x, y = (mask_index[0][i], mask_index[1][i])
to_visual[x, y] = RGB
return mask, to_visual
def dilate(img, mask):
SE = np.array([[1,1,1],[1,1,1],[1,1,1]])
res = np.array(img, dtype=np.uint8)
check_legal = lambda x, y: x - 2 >= 0 and y - 2 >= 0
mask_index = np.nonzero(mask)
index_length = len(mask_index[0])
for i in range(index_length):
x, y = (mask_index[0][i], mask_index[1][i])
if check_legal(x, y) :
tmp = img[x - 2 : x + 1, y - 2 : y + 1] * SE
# res[x, y] = np.sort(np.reshape(img[x - 2 : x + 1, y - 2 : y + 1], (1,9)))[0, 2]
res[x ,y] = np.min(tmp)
# res[x, y] = np.max(tmp)
return res
if __name__ == '__main__':
img = cv.imread('../img/girl_noise.bmp')
cv.imshow('ori', img)
img = cv.cvtColor(img, cv.COLOR_RGB2GRAY)
height, width = img.shape
edge = tk.detectedge(img, 38)
# cv.imshow()
w('edge', edge)
section_mark, area,visual_section = get_area(edge[:height - 2, :width - 2])
# cv.imshow('area', visual_section)
# w('section_visual_thre38.5', visual_section)
mask_max, visual_mask_max = remove_contain_edge(section_mark, np.argmax(area))
mask_second, visual_mask_second = remove_contain_edge(section_mark,
np.argmax(
np.hstack((area[:np.argmax(area)],
0,
area[np.argmax(area) + 1:]))
),
(123, 34,193))
# cv.imshow('max_area', visual_mask_max)
# cv.imshow('seconde_area', visual_mask_second)
# w('max_area', visual_mask_max), w('seconde_area', visual_mask_second), w('1plus2', visual_mask_max + visual_mask_second)
cv.imshow('mask', visual_mask_max + visual_mask_second)
mask = np.logical_not(np.logical_or(mask_second, mask_max))
rect_area= find_rect(mask)
dilate_img_back = dilate(img, rect_area)
# cv.imshow('dilate_img_back', dilate_img_back)
w('dilate_img_back', dilate_img_back)
recover = tk.score(dilate_img_back, method='mid')
# cv.imshow('recover', recover)
w('recover', recover)
cv.waitKey(0)
cv.destroyAllWindows()
2. Experimental results and analysis
First, after the truncated FT operation
Truncate FT 2-1 original image 2-2
According to the experimental results, truncated FT does play a certain role. At the same time, the size of different windows and the selection of filters also have varying degrees of influence on the experimental results. The following are the comparison results of different window lengths
Length 20 2-3
Length 10 2-4
Length 5 2-5
After median filtering
Median filtering 2-6
Histogram equalization
Restored image 2-7
Attachment:
Edge detection to remove impulse noise
Edge detection and denoising 2-8
In practice, one denoising process can not achieve good results, so the author has carried out iterative denoising on the image for many times.
Iteration 1 2-9
2 iterations 2-10
We can see that compared with median filter, the edge detection method has better clarity, but it can not effectively remove the noise, only suppress the noise to a certain extent, and blur the flowers under the picture to a great extent
In the actual experiment, it is found that there are limitations in choosing the largest two connected sets. When selecting different thresholds, even the largest two parts may not be able to surround the area where the bright line is located.
threshold: 15
threshold: 25
threshold: 30
threshold: 36
threshold: 40
Connecting domains under different thresholds 2-11
Therefore, it can be changed to exclude the minimum connection set so that the remaining part accounts for 90% of the image, which can make the method more widely applicable.
The processed image is shown below:
Restored image 2-12
It can be seen that this method can effectively remove the bright lines of the image compared with the first two methods. However, after processing, the image part is stepped, and there is still much room for improvement. In view of the author's limited ability, I hope to improve on this issue in the future
tool.py
import cv2 as cv
import numpy as np
import matplotlib.pyplot as plt
def histogram(imgs):
"""
show the histogram of image in input
Displays the histogram of each image in the input image group
:param imgs: tuple of images
:return: none
"""
num = len(imgs)
statics = np.zeros((num, 256), dtype=np.int)
for i in range(num):
for (x, y), ele in np.ndenumerate(imgs[i]):
statics[i, ele] += 1
plt.plot(statics[i])
plt.show()
def score(img, **param):
"""
this function use noLinear core, instead of sort core
more detail in param method
:param img: input image
:param method: midcore, maxcode, mincore
:return: image pass by filter using assigned core
"""
switch = {
'mid': midcore,
'max': maxcore,
'min': mincore
}
return switch.get(param['method'], default)(img)
def default(img):
print('error:no this method core')
def midcore(img):
imgInfo = img.shape
height = imgInfo[0]
width = imgInfo[1]
dst = np.zeros((height, width), dtype=np.uint8)
for i in range(height - 2):
for j in range(width - 2):
tmp = img[i:i+3, j:j+3].reshape(1, 9)
dst[i,j] = np.sort(tmp)[0, 4]
return dst
def mincore(img):
imgInfo = img.shape
height = imgInfo[0]
width = imgInfo[1]
dst = np.zeros((height, width), dtype=np.uint8)
for i in range(height - 2):
for j in range(width - 2):
tmp = img[i:i + 3, j:j + 3].reshape(1, 9)
dst[i, j] = np.min(tmp)
return dst
def maxcore(img):
imgInfo = img.shape
height = imgInfo[0]
width = imgInfo[1]
dst = np.zeros((height, width), dtype=np.uint8)
for i in range(height - 2):
for j in range(width - 2):
tmp = img[i:i + 3, j:j + 3].reshape(1, 9)
dst[i, j] = np.max(tmp)
return dst
def interpolation(src, mask, **param):
"""
calculate interpolation on some designed point or area we interesting
:param src: input image
:param mask: to decide where need to calculate interpolation
:param method: bilinear, medium
:return:
"""
switch = {
'bilinear': bilinear,
'medium': medium
}
return switch.get(param['method'], default)(src, mask)
def bilinear(src, mask):
pos = np.array(np.nonzero(mask[1:mask.shape[0], 1:mask.shape[1]]))
num = pos.shape[1]
dst = np.array(src, dtype=np.uint8)
for i in range(num):
y, x = (pos[1, i] + 1, pos[0, i] + 1)
val = (np.int(src[x - 1, y]) + np.int(src[x, y - 1])) / 2
dst[x, y] = val if val <= 255 else 255
return dst
def medium(src, mask):
pos = np.array(np.nonzero(mask[2:mask.shape[0], 2:mask.shape[1]]))
num = pos.shape[1]
dst = np.array(src, dtype=np.uint8)
for i in range(num):
y, x = (pos[1, i] + 2, pos[0, i] + 2)
tmp = np.array(src[x - 2:x + 1, y - 2:y + 1])
tmp = tmp.reshape(1, 9)
dst[x, y] = np.sort(tmp)[0, 4]
return dst
def ppf(img, segment_size, mask, **param):
"""
part pass filter, add some space information to fourier transform
:param img: input
:param segment_size: slice length(aix=0)
:param mask: filter
:param begin: star index of filtering
:param size: the length of filtering
:return:
"""
length, width = img.shape
splitime = np.int(length / segment_size)
begin = param['begin'] if param.get('begin', 0) else 0
size = param['size'] if param.get('size', 0) else splitime - begin
# print('begin:', begin, '\nsize:', size)
if begin >= splitime:
print('error:check begin position')
return
if size + begin > splitime:
print('error:check size')
return
mask_range = range(begin, begin + size)
img_mask_back = [img[0 : begin * segment_size, :]]
for i in range(begin, begin + size):
data = np.array(img[i * segment_size: (i + 1) * segment_size, :])
f = np.fft.fft2(data)
fshift = np.fft.fftshift(f)
if i in mask_range:
fshift = fshift * mask
fishift = np.fft.ifftshift(fshift)
img_mask_back.append(np.abs(np.fft.ifft2(fishift)))
img_mask_back.append(img[(begin + size) * segment_size : length, :])
return np.array(np.vstack(img_mask_back), dtype=np.int)
def detectedge(img, threshold= 120):
length, width = img.shape
nn = np.array([1, 0, -1], dtype=np.int)
H = np.zeros((length, width ), dtype=np.uint8)
for x in range(width - 2):
for y in range(length - 2):
tmp = np.array(img[y, x:x+3])
dx = np.sum(tmp * nn)
tmp = np.array(img[y:y+3, x]).T
dy = np.sum(tmp * nn)
H[y, x] = np.sqrt(dx * dx +dy * dy)
return bianrization(H, threshold)
def bianrization(img, val):
img[img < val] = 0
img[img >= val] = 255
return img