Work sharing - (demand: save the pictures locally for three months)
Demand profile
Recently, I have encountered such a demand in my work: save the image data of a business locally for three months. In our work, such a similar demand is common, so I share my practice this time as a reference for you.
Specific needs
There is a table in xxx business, which contains such a picture field. You need to download and save the pictures locally. This saving requirement is to establish a corresponding date folder in the root directory of xxx, and then create 24 folders in the corresponding date folder to store the picture data generated every half an hour of the day in turn.
My thoughts
First of all, it is similar to this demand. During the day, users are considered to be using it, so as not to affect the performance and other factors. One of my ideas is to do a regular task, delete the pictures saved three months ago at 0:05 every night, cycle query and save the picture data every half an hour.
·············································································································································
The date of 0:00 of the current day is named by API instead of mine.
·············································································································································
For Scheduled tasks, I use SpringBoot's Scheduled directly. One disadvantage of such Scheduled tasks is that if your service goes down during the execution of Scheduled tasks, it can't be re executed. For this requirement, the pictures of the day will not be saved. So how to solve it? In fact, you can use Quartz.
·············································································································································
Quartz is another open source project of OpenSymphony open source organization in the field of Job scheduling. It is completely developed by Java and can be used to perform scheduled tasks. One advantage of this framework is that all scheduled tasks are stored in the database. If your service goes down and the scheduled tasks are not executed, you can retry the execution when your service is restarted.
Realize function
Before writing the scheduled task, we first write two methods, one for downloading pictures and the other for deleting the specified folder. Then start the scheduled task and write the scheduled task method.
Start scheduled task
1.Start class add annotation to start scheduled task@EnableScheduling 2.Scheduled task class addition@Compoment Annotations include classes spring bean Administration 3.Create method add@Scheduled Annotation configuration cron Timing attribute
cron expressions from left to right (separated by spaces): seconds, minutes, hours, dates, months, weeks, years
0 0 10,14,16 * * ? Every day at 10 a.m., 2 p.m., 4 p.m., every day at 10 a.m., 2 p.m., 4 p.m
0 0/30 9-17 * * ? Every half an hour during nine to five working hours
0 0 12 ? * WED means 12 noon every Wednesday
0 0 12 * * ? Triggered at 12 noon every day
0 15 10 ? * * Triggered every morning at 10:15
0 15 10 * * ? Triggered every morning at 10:15
0 15 10 * * ? * Triggered every morning at 10:15
0 15 10 * * ? Triggered at 10:15 am every day in 2019
0 * 14 * * ? Triggered every 1 minute between 2 p.m. and 2:59 p.m. every day
0 0/5 14 * * ? Triggered every 5 minutes between 2 p.m. and 2:55 p.m. every day
0 0/5 14,18 * * ? Triggered every 5 minutes between 2 p.m. and 2:55 p.m. and between 6 p.m. and 6:55 p.m
0 0-5 14 * * ? Triggered every 1 minute between 2 p.m. and 2:05 p.m. every day
0 10,44 14 ? 3 WED is triggered at 2:10 PM and 2:44 pm on Wednesday in March every year
0 15 10 ? * MON-FRI is triggered at 10:15 am from Monday to Friday
0 15 10 15 * ? Triggered at 10:15 am on the 15th of each month
0 15 10 L * ? Triggered at 10:15 am on the last day of each month
0 15 10 ? * 6L triggered at 10:15 am on the last Friday of each month
0 15 10 ? * 6L 2018-2019 triggered at 10:15 am on the last Friday of each month from 2018 to 2019
0 15 10 ? * Triggered at 10:15 am on the third Friday of 6#3 each month
Download the picture and store it according to the specified path
Small pit encountered: the timeout must be set after opening the connection and before getting the input stream, otherwise it will not take effect.
/**
* Download the picture and store it according to the specified path
*
* @param url Picture path
* @param folderName Folder name
* @param fileName file name
*/
public static void downloadImages(String url, String folderName, String fileName) throws IOException {
try {
URL url1 = new URL(url);
URLConnection uc = url1.openConnection();
//Connection timeout
uc.setConnectTimeout(3000);
//Read timeout
uc.setReadTimeout(3000);
InputStream inputStream = uc.getInputStream();
FileOutputStream out = new FileOutputStream(folderName + "\\" + fileName + ".png");
int j;
while ((j = inputStream.read()) != -1) {
out.write(j);
}
log.info("folder:{} - Picture:{} - Download complete...: ", folderName, fileName);
inputStream.close();
} catch (SocketTimeoutException e) {
log.info("folder:{} - Picture:{} - Download failed(Access timeout)...: ", folderName, fileName);
} catch (ConnectException e) {
log.info("folder:{} - Picture:{} - Download failed(Connection exception)...: ", folderName, fileName);
}
}
Delete the specified directory folder
/**
* Delete the specified directory folder
*
* @param file
*/
public static void delTempChild(File file) {
if (file.isDirectory()) {
//Get all subfolders under the folder
String[] children = file.list();
//Recursively delete subdirectories in the directory
for (int i = 0; i < children.length; i++) {
delTempChild(new File(file, children[i]));
}
}
// The directory is empty. Please delete it
file.delete();
}
Scheduled task
/**
* Scheduled task: the bayonet picture is saved locally for three months
* Save the picture of the day before yesterday at 0:05 every day
* Delete pictures three months ago
*
* @throws ParseException
*/
@Scheduled(cron = "0 05 0 * * ?")
public void scheduleTask() throws ParseException {
//Delete pictures three months ago
String threeMonthsAgo = LocalDate.now().minusMonths(3).format(DateTimeFormatter.ofPattern("yyyyMMdd"));
File deleteFilePath = new File("D:\\xxx\\imgs\\" + threeMonthsAgo);
delTempChild(deleteFilePath);
log.info("Delete folder path:{}", deleteFilePath);
//Create a folder for the day based on the date
String data = LocalDate.now().minusDays(1).format(DateTimeFormatter.ofPattern("yyyyMMdd"));
//Image storage root directory
File file = new File("D:\\xxx\\imgs\\" + data);
//If not, create
if (!file.exists()) {
file.mkdir();
//Query required time parameters
SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");
long current = System.currentTimeMillis();
long zero = current / (1000 * 3600 * 24) * (1000 * 3600 * 24) - TimeZone.getDefault().getRawOffset();
Date date = new Date(zero - (24 * 60 * 60 * 1000));
Calendar calendar1 = Calendar.getInstance();
calendar1.setTime(date);
Calendar calendar2 = Calendar.getInstance();
calendar2.setTime(date);
calendar2.add(Calendar.MINUTE, 30);
//Query the data every half an hour and put it into the folder in turn
int len = 24;
for (int i = 0; i < len; i++) {
//Create folder under current day folder
File file1 = new File(file + "\\" + i);
file1.mkdir();
log.info("Newly generated folder path:{}", file1);
List<JSONObject> xxxService;
if (i != 1) {
calendar1.add(Calendar.MINUTE, 30);
calendar2.add(Calendar.MINUTE, 30);
}
//Query data every half an hour
lists = xxxService.selectxxx(sdf.format(calendar1.getTimeInMillis()), sdf.format(calendar2.getTimeInMillis()));
if (!lists.isEmpty()) {
lists.forEach(t -> {
try {
downloadImages(t.getString("imagepath"), file1.toString(), t.getString("id"));
} catch (IOException e) {
e.printStackTrace();
}
});
}
}
}
}
summary
The above is all the content I shared, as well as the small pits I encountered. If there are mistakes in the article or you have better practices, please give more advice in the comment area.