Important things are at the top. Please look carefully:
This paper aims to learn and exchange crawler technology, especially JS reverse technology. We should focus on learning the ideas and process. Please don't imitate the website exemplified in the article. If you want to crawl, please crawl no more than 3 pages, otherwise you will bear the consequences.
Crawl It!
The first step of crawler technology in ten thousand years -- Analyzing web page structure and data loading method:
As a result, as soon as you open the web page, it seems that the website maintenance personnel have gone to the festival. Then come back from vacation and continue writing..
This article is suspended at 10:25 a.m. on February 11, 2021
The Spring Festival holiday was over in the blink of an eye. I came back to "do things" again!
- Open the url and display the data page.
- You can know from the Network tab of Chrome browser that page flipping is realized by ajax.
The response data are as follows:
"95780ba0943730051dccb5fe3918f9fe1b6f2130681f99d5620c5497aa480f133ef544506ed439f 4bddb1ca63d7c066964c776ea098085c5cd9112e7cbcf533c5823e8b4bb0d542e5e2591fb6a46134 7bbb39b1ed0de45bce0d2512d6a0834f5c1d80d738d51820fb78c1aa2481fd43c825924bc101ad71 917930074d902df9764fe7f29f5a6cb917ea3e617ca76a348186f29d547a8b2bd4044cc9d9036fe2 b22e953f4aeea13705d9d1583300371749d5a882f6c50176739ca1171a50db15aae8c6550591dd8a e62ad68dc69935e3fcf57379cc625c94acabb2effc4d5505f214f94a2a0e50110d9647d20bb189f4 4acad6184304fa46dff0048b0f58dd1b5d8a945794bbe9e66898b8e363192a79b84cde408f90a02c 9c9b85df2b6d5de973fc00874b223acd2da7df8642ce1e1e64aa6ff550e7b96db782d348f8209d5f d59b9e26a2aa7ee9c4621cfc5a6d757d91e3862469c39b3b5152054a50667e0f8dfad00fafe55917 e938398c09fc682becfd9aa54f68ce6fd7e023cc7db416727a1572389c798d85f1cd348ff918a4d0 9e67c41a719ac57b6c06a05e954b84a6f0303192b740d3d5ed9c9cc4fe8f0c4de71aa12edc33069d 55f311894109c8c686745fcd22ed2cb33f5deeb6ca9af98c0097e86b450e98a9b1011587011014a3 5b63a3d55c17d92849f56d5e20e76847238c37620d2098b96c153accc9df0df7722868ff86028415 1bf7d7d0aa78ffeb89c882d592994ff09a235180f6f1ff6092fd60ce7eae7c69fc30ba47239f513e 08b9eb3e530cf08647933ea18d2c34c4cb9515b82d4153172311afb9af5d8bdd770d9d9410e75381 cf0f2aa2ca5cb33ccf3276ed0c7dfeda58b7c953bd7225b4395b07e922b75c26c3b5ee302ac3f0ae 40dc31a55ef92afdc7909c7f2e900e1ae8bf92764571c5a5ff27671fbce00c27fc379d64ca61d84f 95558bd2af9b83dd58349922543b8c8e0dc43dead3a102025ce19f4db76f5346b5bbd8784cefa854 51ef22ccd3967f1f6b2eb6f3a9114cc7f26ce107b25985c35ac9a82f11b637fd4a52c2c4f6966f05 3ec48eea3d056268304e58540c3eac82dd70e668e2b24707daeb5d1026b6ad7e7efb638bd51838f7 18c690d0ca4f421246e09432169a0c84ee96adcd4082f5796f474c636ed6d19702e8e5c45421e98a 56d94324160196673363f729ec7cc207f478fbaa865af04ab243cd1a5d3cf54b22393a04aeff51fd 42d33a3d76d11814a540332db0d9f61172e99b3f08f75b081367da26edae0ee7ff7dcab16883ab28 82e48eb95177cdf3d337a1836c949182e7bd5b155aef25c87f399664a367fbe961f1e532295dbce5 0c5a1f7acd891849bae2e6169eb8b9bfbe62e85e740686bc4d3a1ff47cc224507f7be3578751d571 6f006eebdced3c82eabbd500e55b8ff610cef3d947a317f103bf2cbde3fdfc2a0bfdc7e26bdcfee7 9a6967fd66eb01fc84e5c07718747c603e52719d045fdd19ff01f4a00921db26c1761fb744eebf73 262688e1802da1c1a0320196dc721f1a604512589fd23cc01895760881ba4a7af28e5e66b921a14b 85b7f86cd317e691aadf81a8d2b463318058490b19293a35d2fe87ffdecd220cd24fbee8d4073c26 bc61a3a0ad31377db1e660bb5b15c397d15ff820832ffb46f4008ced73bb1133dc957e6e3ec3d829 309addb1aa1e3a431d06b1c8aef6c43ed040f5ca311910bc0d70c34d8aca94d5b7084cfe46fb793d e87a027f7bfe4d60489d05d3a0e661e9405a768548cdb69bb8a8e52aa0c775684b3e30f18cc059c9 38b451b3376a1736a21df7809791ed8144a893cd831245c64321064f45c67721413018e6bfef3ebe 572cda88723a970ae924cdde1b0a205dbf80b2e5aa55e6baac55b4a30232ee3bad287a40d4916a42 414f6805955ad4cb32bc1c7a4946d77cc9aa4582add2e68976a9a85003acec58f21327c30750e7c1 0615b94a66912b93237e5cc5568a5a53b7214801d3f9903bff3f445e33cc9fda9d19137917abbee4 c08d178095e5956acbb22dcd91639943ace2c24510b7206d92d011f44c055973c7e25c3de3553a5d ad8457f0b0023e5cfef2f66ed0288b320f37e8313d061b9b75add7c7848ad722e616cad7c7c12359 018c34f723e37afaf21d466654312495a08beb81cb77e2c2bae6d0151fdaa68df623a70618cd6d3c b66c1475c8eb24829742f10b90bfb9b5794f5eeba7226f7530ba86a69804cf8fbb4c6985c843c0c4 0a3f5bf88bdf55437b7bfcb0bb52f3bae7ba8ee035923b0dcceb144a30f7d2559103d512b4c786c1 710ebba3cedde63e99eee1c5f6ff7991d4b013ba8f2b223bd791baa5e6962a687415b947efb5dea9 07e3dc7d3eb19d9adbc4e9498c5362460ceb945533e3c4c049e93ce9711ca776d54279a01a226998 c5160efd85151618f565c38e9beed091441d2bd1e7677e032d13a25817f051241ca97995188f45cb be1602636e855f827f35122b6b8cc887f013402b032fc4d1e24939c7cf2c5e686fdb4c4565022ba6 11726a8b73b5c7f8e77502d73b56b58f14239eebe1fcf356b20367e41a7434ce793b738a6d2b7f7d 3a6753a950641bce3fe01428fd777ddadab33fdde61b9fc56c527d89f52ec4cce77fca7ff693ba1d 7e45225eff225fded5e6708e73c20f6fb72b8ce8b080f3f1f055d5b4b624d0b4020a599b9dbebd9e 88c567cd894a94d846022de3298b171556155bec89323dc12a49137f06acb0f3fbad45aa55f94ebc 31458ccb0289dc403540b98e3a36d9b48a975bbb055dccfc57d87a6213f1fc57db70919dab450c3f 21da6daec9d0df7a847fd632a3e9a882b0e14e15565d06c7f437e7e69e9648f1cb9b022d6118d08e 7a0407d15a820b3249bff0ce986c8dc05ca7eca74b6302080a24c898437e1bbc5d380eb93a783710 407fa1d5389c127ab201e2508cdbc63b51a9d5c546d6bcee9f5e0b3a4b92239509884edc05c27aa1 76f64a21f4a7d4bfe997b4a4fb1b95c9f7e1cf1a85c33df25848be1ae83fe4a948095c38d66dbe10 73d8fa6e1bda1fc8a2913d8731d78b1bd6ed8dda5597da4e63d9dadbb8eadb1612198fc88342afd6 dd2442170683a5d55d27dea2475bf57fec14663906dbb6beec6c91a012f909e01626bf9a1a2a452f c696b52e4edabe499f349cfeff364b28d8bf82aa9a5cfaae80c4b92e22f03c58b3cca43363f1a64e cfa7c652de4e676b3d18029e0fb606e8ca0e5f2960b16705eeb7d34f48ef3d6f1a6f2e9dd2ce32fb 44af7cfff512488d7bfe6f57f40b5681a18bf2b6a60827b8aa888c81afbcaff7063fc4aeed89b137 5d35f4f7af1ce3cf9c5b3c81c4175651a981f755f422c30713d3e6ca4f3763f95c0f0b1d1f51c32b 769157a7c0a25f050560bb851fbf581a7186d515bd88e100fc91a56b39b86da71e8d028485358e58 31b00d2e653485ee7f404cf8848a681f3fafceb06138e090fc2caf2b03a4978173e1c6e71821ada8 9ceba3302cc16cfaa4b9bf472fa79156772e5ffdab8b44606e258176fa651d13e7388c13ee17d965 4b3e5564c24ccf478015014da5b0dcc8deb213cb0558c185339b22c490a332b4d4f9cf700cf2662d ec0f815c8418e2573621713717463e4606b5903f66d7bc0581369a1586e84ebd7ba27277aecbd67f 737e35ab87ac78c428588bd7a1a26e4bc2834779d3399e06790bf1cb1ea77f665a449b9e6b3f459d bd14bbf1534b4f19292f35fbb5942ec505fcf654235d8e1b556843c87b6b868b4e049733e85f551f 78e3587dd6223183f038be2482259638a0650cf6eff6f1bf5383d4774e5b1e5bd20dc839df78ae23 cca22d23d30f2fa1a49307220474a371666908619339f83cc7ae3fcd37a37db640a06970b39db055 a2a8a0d067fe72516b9b9eccc51a69dcd19e961bfe3b131e8130ec2e648d20aafafaceb7f05f03e5 a4e75cd3dadece5d15cdb26c7e6c39a4fcf400490a668e4221f4c0129cfeb46078775392218c65db 37935c172fc5c10f2b0079a195698acd00f736071593604a1e088db94b29319b75f759cc0e6fb3a4 e226a2305dfb7ad0f0f3b9b02b229953e030ac50fe75560f7a98ccfff5a3dd06617349b4fd1ee5ac db33789b5724e5c3e4441d13a11974b11406bca931d7077b0b2e1e9c4c7a6eb7245fc5779c1cf922 0f64e3bbd4c1fc1cd5000c879700dcb35e5fd77508fa96063de2c7531ae658bf048156b7678b4b41 bf66f00eb6198503671668115bcfcea46ab1a07a7c077d339bb24f65757fe8cfc944cbc1ab7aae39 b19c40619f28d86e58b1f39e52202ff220092ded48323531264e8271129c644ca7e6c4ee03ccbae9 02b990257b1a55daec1b919f33fff441198653032d1026ac1b404de3562c6f851664fd534b04e544 c43dd82ac7ebeaccc7a9a479678c2371665e99815e6a0c60b8b66a21fb0275dd5ee2c26cd69ddff0 b173b40b90e6a602a30c5f0b33e8b933de6f13520af43adf5b87e22ee5af4c307a27b2f788feaefc 668d87af90bf442df137fd26c8e29a6d40c5cb5fbd551752ef0660477013614528661901067c6d64 a95917b9b419180c4d3d3f0a1edb8521d90700ad70a2f4d0717d5bdc24766802ece3be32dc45bfa0 79cd7ff464608926728f9b395f609f53e42c7ba4136abd566d53d0b1819fbef45358df803eadd071 a250c3ca1abd7c2426975dbdfc5dc8369c79e785a4f6c0ed5a7152ec4912df7bf97914abca3f0b4f 616905ed128809a5ead9d9fe57425a16a32fb0ff3e81e409718814eca8226676d3cc8df61b8b0827 c5c54696a5c5c25f697fb842ca316a36a09dfb48b6fc9f5fdc3717affb3477d98259126821e2fedb 292daba634bff50e1e6136dfc83a4441bc8f2ed5350792b6c4668e76f5db7d64b4f40e94bb749471 19881f0077536a67422278bf0cc97614c2f7800ad08fc3e04e5ea3b010a6c16af0a8e16dfb15b780 e6eb284f197ce9d3d370536bb10c1f48762e7842a651875c0d47c530fc9fd5705fad0b0f3a533500 153e810ad751ff15af761011399a162029a77b4fe937edd5b8987ac01e2c89f1642716fe83627124 7f26c983bea9c09ae415a7d367f5a3bfd1d5cf3e3eafd235d0a222869a6d2f3403937a2f6bb0b7eb 6568c743a39ee90bb7b8efd6d1b5e863da2cae9642eedb758d44f2dd749a6327ee586de842042c7c ae67b09fd95c57aca18b6a25321440c04bdb056ae12b593fc2f96a45f608d0d0a96e15ad6fff2bbb bf906b02451af38cfce206ec1004de390f3fb724ddafd903dd31d15fabe97d7a540fda85467014d2 0ea9e3a5a92e88c9718ac6ebd35b7047c333a4a8dcf56b66039b2b77fda17f9e6a741801a43410b6 d4db887d713b83a5db7a1dae485312f452d805da5cf8e385c656b10754359ef3031222fcb3fa3f79 de5acd68630ebc075a86b63d100c05f472057401e9bbc70e931550337f6536b69bda901e8503de8a b9b0216a0edaea8117381f7e0a91e1abd1e4c3f8009ce0eda0802fc896248754b35d85b76b8ec569 917451a3ec2dadc3a2d07b7f5fb0bcc6c5a9fc5e1280310fbf7dc4bf060228ed3c997f35ab16d44c 0c5dceec43281120738882cc22b1ee5172d41613723c8da59fa6a70438868e2aa216e152435ed196 53866ebcc82721b1a524ebe0dde2a60647924e0d9ea4e391e2fc042f46b93658dc73087360b60969 cf50615daf3a7fe37f24691aad9114b8b8b2c010115236100c007d2e0754aff82e380d9147b18b5d 1489bdfb5753d0ad58854212935801080ad49054b9a599ecaafaa29c2194ef5cf98515e60dfdbce4 61f190b7ea166946b7d3b584b0f137ece880feb9431c5d6732bd3302c1147dbd03edf1aedc8c509e 0a6238a74aae81acd9cab9c1c4d8aeed9a0ac8f9a8a32a93217600e5a43c807677caa2954fb1e065 51b74336c8f37c83edd238f3b2b9434949300a6dc5f00f08ecc2fc8b4ba329c2217bc03e8e6e3457 e92d72c824ef97a8e35ebe395bd4e8944074fd64d6f59e672daba63bb7dcf416ec1d47400d186a4f 882721cbc6a6809b92aab8e0e66f6a6f82b35fc254558afadfa857179c245c626672b8a82c99b194 8761c98707206cbe7ecab8ffee47e9a7577504f78a59059b232a8ed9600a9516cb1619a42e471b23 0b4074d1df1acbb07834ece68090ca8f242fb126e688881004e7751c3f47a18c9b62994e4bbe2aff fdc007f125234cdddc007b138a4eadaab2bc94f4d645e3adf00d490e4e2283d7dfff39bd3b0c5759 4ca2696c549281887f82776d98e1bab01f031d5767a71053122976a6e59fb2a13f07c98b5e45d530 1bb11d75bb83fb0c8c1090cd5089ff73eb07b6ffc775e5da3194702c2bf8b01f139457158475a8a2 dab9657e4321ebf05f52b64c604e6338fe1801c81db8120c553b641e3b3344d700a91d25412c90fc 981abe075450ed73fd9145d5b9626983272a47d60b087d30a76ede332fdaef790e14583d81dfc89b 8206a120badd4db7b11d73bc2ce882fea80d0ebb86d835c786bf81c63c3855d873535ad17ab183cd 4"
The messy characters returned by the request are the encrypted data. (of course, your server gives me encrypted data. If you want to display it on the browser page, you must decrypt the data locally. You won't display this pile of garbled code.)
- Then we need to find the corresponding js decryption process.
Through the efforts of nine cattle and two tigers, we finally found the place to decrypt. This step is the most time-consuming and bald step.
The main methods to find the reverse entry point are:
- Keyword search, break point debugging
- Set breakpoints with XHR;
- View the call stack.
On this page, I found the encryption place by typing XHR breakpoint + search keyword decrtpt (decryption).
Let's take a look at the debugging data~
{"code":200,"data":
{"list":[{"QY_FR_NAME":"Miao Cunxu","QY_REGION":"350200","QY_NAME":"Xiamen Xinghai Bay Engineering Management Co., Ltd","QY_REGION_NAME":"Fujian Province-Xiamen","QY_ORG_CODE":"913502112601177740","COLLECT_TIME":1477106448000,"RN":31,"QY_ID":"616160676166636361616466606865656369","QY_SRC_TYPE":"0","OLD_CODE":"260117774"},
{"QY_FR_NAME":"You Liang Chen","QY_REGION":"330800","QY_NAME":"Zhejiang Zhoudao Engineering Management Co., Ltd","QY_REGION_NAME":"Zhejiang Province-Quzhou City","QY_ORG_CODE":"91330802330012039P","COLLECT_TIME":1477106529000,"RN":32,"QY_ID":"616160676166636361616466606865656261","QY_SRC_TYPE":"0","OLD_CODE":"330012039"},
{"QY_FR_NAME":"Xu Bingshui","QY_REGION":"370100","QY_NAME":"Shandong Zhengda Construction Supervision Co., Ltd","QY_REGION_NAME":"Shandong Province-Jinan City","QY_ORG_CODE":"913701022671825366","COLLECT_TIME":1477106425000,"RN":33,"QY_ID":"616160676166636361616466606865656260","QY_SRC_TYPE":"0","OLD_CODE":"267182536"},
{"QY_FR_NAME":"Wang Zhong","QY_REGION":"510100","QY_NAME":"KaiBang Construction Management Group Co., Ltd","QY_REGION_NAME":"Sichuan Province-Chengdu","QY_ORG_CODE":"91510105725354703M","COLLECT_TIME":1477106469000,"RN":34,"QY_ID":"616160676166636361616466606865656263","QY_SRC_TYPE":"0","OLD_CODE":"725354703"},
{"QY_FR_NAME":"Liu Yinuo","QY_REGION":"371300","QY_NAME":"Linyi Hongli Construction Engineering Supervision Consulting Co., Ltd","QY_REGION_NAME":"Shandong Province-Linyi City","QY_ORG_CODE":"913713005543587173","COLLECT_TIME":1477106454000,"RN":35,"QY_ID":"616160676166636361616466606865656262","QY_SRC_TYPE":"0","OLD_CODE":"554358717"},
{"QY_FR_NAME":"Wang Hongjuan","QY_REGION":"410100","QY_NAME":"Henan Zhongjian Engineering Technology Co., Ltd","QY_REGION_NAME":"Henan Province-Zhengzhou City","QY_ORG_CODE":"914101057919311268","COLLECT_TIME":1477106526000,"RN":36,"QY_ID":"616160676166636361616466606865656265","QY_SRC_TYPE":"0","OLD_CODE":"791931126"},
{"QY_FR_NAME":"Lin Jin","QY_REGION":"350100","QY_NAME":"Fujian Jianxin Engineering Management Group Co., Ltd","QY_REGION_NAME":"Fujian Province-Fuzhou","QY_ORG_CODE":"913501001544135960","COLLECT_TIME":1477106407000,"RN":37,"QY_ID":"616160676166636361616466606865656264","QY_SRC_TYPE":"0","OLD_CODE":"154413596"},
{"QY_FR_NAME":"Yang gaobao","QY_REGION":"421200","QY_NAME":"Jiayu Jiatai Construction Supervision Co., Ltd","QY_REGION_NAME":"Hubei province-Xianning ","QY_ORG_CODE":"91421221737145124M","COLLECT_TIME":1477106486000,"RN":38,"QY_ID":"616160676166636361616466606865656267","QY_SRC_TYPE":"0","OLD_CODE":"737145124"},
{"QY_FR_NAME":"Jiang Yabin","QY_REGION":"410800","QY_NAME":"Henan synergy Engineering Supervision Co., Ltd","QY_REGION_NAME":"Henan Province-Jiaozuo City","QY_ORG_CODE":"91410800555731456W","COLLECT_TIME":1477106405000,"RN":39,"QY_ID":"616160676166636361616466606865656266","QY_SRC_TYPE":"0","OLD_CODE":"555731456"},
{"QY_FR_NAME":"Xiang Jun Huang","QY_REGION":"450100","QY_NAME":"Guangxi Nanning Jianke Engineering Supervision Co., Ltd","QY_REGION_NAME":"Guangxi Zhuang Autonomous Region-Nanning","QY_ORG_CODE":"914501031983159639","COLLECT_TIME":1477106406000,"RN":40,"QY_ID":"616160676166636361616466606865656269","QY_SRC_TYPE":"0","OLD_CODE":"198315963"},
{"QY_FR_NAME":"Liao Yuan","QY_REGION":"510100","QY_NAME":"Chengdu Hailong Engineering Project Management Co., Ltd","QY_REGION_NAME":"Sichuan Province-Chengdu","QY_ORG_CODE":"91510100201986152W","COLLECT_TIME":1477106437000,"RN":41,"QY_ID":"616160676166636361616466606865656268","QY_SRC_TYPE":"0","OLD_CODE":"201986152"},
{"QY_FR_NAME":"Zou Yun","QY_REGION":"360500","QY_NAME":"Xinyu YONGGU Engineering Supervision Co., Ltd","QY_REGION_NAME":"Jiangxi Province-Xinyu City","QY_ORG_CODE":"91360500723935352F","COLLECT_TIME":1477109337000,"RN":42,"QY_ID":"616160676166636361616466606865656561","QY_SRC_TYPE":"0","OLD_CODE":"723935352"},
{"QY_FR_NAME":"Wang Wenquan","QY_REGION":"371300","QY_NAME":"Shandong zhenghang Engineering Consulting Co., Ltd","QY_REGION_NAME":"Shandong Province-Linyi City","QY_ORG_CODE":"91371302092199458A","COLLECT_TIME":1477106426000,"RN":43,"QY_ID":"616160676166636361616466606865656560","QY_SRC_TYPE":"0","OLD_CODE":"092199458"},
{"QY_FR_NAME":"Hong Jian Zhang","QY_REGION":"530100","QY_NAME":"Yunnan jinji'an construction consulting and Supervision Co., Ltd","QY_REGION_NAME":"Yunnan Province-Kunming","QY_ORG_CODE":"91530000719414685K","COLLECT_TIME":1479833531000,"RN":44,"QY_ID":"616160676166636361616466606865656567","QY_SRC_TYPE":"0","OLD_CODE":"719414685"},
{"QY_FR_NAME":"Zhang Zhiming","QY_REGION":"610700","QY_NAME":"Hanzhong Zhicheng Construction Engineering Supervision Co., Ltd","QY_REGION_NAME":"Shaanxi Province-Hanzhong City","QY_ORG_CODE":"91610725667992132H","COLLECT_TIME":1477106417000,"RN":45,"QY_ID":"616160676166636361616466606865656568","QY_SRC_TYPE":"0","OLD_CODE":"667992132"}],
"pageNum":2,"pageSize":15,"total":450},"message":"","success":true}
From the above picture, we can know that the decryption process is implemented by an h function. The h function decrypts the parameter t (that is, a large string of encrypted characters) and returns a json format string through AES, which is the data we want.
Let's start to simulate this process with python:
OK, we can get the encrypted data through requests. The next step is to decrypt the characters.
This is the second time-consuming step in the reverse process. Pick the js code in the web page, and then use the pyexecjs module to obtain the decrypted data; Or use the js code in the web page
Rewrite python (that is, change the logic of js code into Python).
According to my knowledge, this is AES encryption. The mode is CBC and the filling mode is Pkcs7. As long as we know the offset iv and the key, the decryption process can be completely broken through. According to the position parameters of the function, f corresponds to the secret key.
It takes no time ~ ~ ~ just next to it. After you get the key and vi, you can use the Crypto module in Python for AES decryption. The detailed code is as follows:
import requests
from Crypto.Cipher import AES
from binascii import b2a_hex, a2b_hex
# Send a request to the url, and the parameter is page_num is the number of page numbers requested
def request(page_num):
headers = {
'Connection': 'keep-alive',
'Pragma': 'no-cache',
'Cache-Control': 'no-cache',
'Accept': 'application/json, text/plain, */*',
'timeout': '30000',
'accessToken': '',
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.104 Safari/537.36',
'Referer': 'http://jzsc.mohurd.gov.cn/data/company',
'Accept-Language': 'zh-CN,zh;q=0.9',
}
params = (
('pg', page_num),
('pgsz', '15'),
('total', '450'),
)
response = requests.get('http://jzsc.mohurd.xxx.cn/api/webApi/dataservice/query/comp/list', headers=headers, params=params, verify=False)
return response.text
# Decrypt the response data returned from the request. The parameter text is the response data
def decrypt(text):
key = 'jo8j9wGw%6HbxfFn'.encode('utf-8') //secret key
iv = '0123456789ABCDEF'.encode('utf-8') //Encryption offset
mode = AES.MODE_CBC //Encryption mode
cryptos = AES.new(key, mode, iv)
plain_text = cryptos.decrypt(a2b_hex(text))
return bytes.decode(plain_text).rstrip('')
# After obtaining the data, we can observe whether the printed data is decrypted data
def get_data():
for i in range(3):
print(decrypt(request(str(i))))
if __name__ == '__main__':
get_data()
After executing the code, we get the data on the web page:
{"code":200,"data":{"list":[{"QY_FR_NAME":"Liao Luyun","QY_REGION":"360500","QY_NAME":"Jiangxi Tiangong water and land engineering design Co., Ltd","QY_REGION_NAME":"Jiangxi Province-Xinyu City","QY_ORG_CODE":"91360500079012847X","COLLECT_TIME":1477106416000,"RN":1,"QY_ID":"616160676166636361616466606865626860","QY_SRC_TYPE":"0","OLD_CODE":"079012847"},{"QY_FR_NAME":"Shan Jun","QY_REGION":"360500","QY_NAME":"Jiangxi Zhongyu architectural design and Research Institute Co., Ltd","QY_REGION_NAME":"Jiangxi Province-Xinyu City","QY_ORG_CODE":"91360500159864134W","COLLECT_TIME":1477106400000,"RN":2,"QY_ID":"616160676166636361616466606865626863","QY_SRC_TYPE":"0","OLD_CODE":"159864134"},{"QY_FR_NAME":"Lin Maoqing","QY_REGION":"420100","QY_NAME":"Wuhan Maoye Construction Engineering Co., Ltd","QY_REGION_NAME":"Hubei province-Wuhan","QY_ORG_CODE":"91420106074476087K","COLLECT_TIME":1477106413000,"RN":3,"QY_ID":"616160676166636361616466606865626862","QY_SRC_TYPE":"0","OLD_CODE":"074476087"},{"QY_FR_NAME":"Tu Ming","QY_REGION":"360100","QY_NAME":"Nanchang water conservancy planning and Design Institute","QY_REGION_NAME":"Jiangxi Province-Nanchang City","QY_ORG_CODE":"91360100MA35J54Y86","COLLECT_TIME":1477106440000,"RN":4,"QY_ID":"616160676166636361616466606865626865","QY_SRC_TYPE":"0","OLD_CODE":"491102609"},{"QY_FR_NAME":"Guo Linjiang","QY_REGION":"410100","QY_NAME":"Bowei Zhucheng Design Co., Ltd","QY_REGION_NAME":"Henan Province-Zhengzhou City","QY_ORG_CODE":"914101025651418126","COLLECT_TIME":1477106411000,"RN":5,"QY_ID":"616160676166636361616466606865626864","QY_SRC_TYPE":"0","OLD_CODE":"565141812"},{"QY_FR_NAME":"Gu Yahong","QY_REGION":"440100","QY_NAME":"Guangdong Boya Information Technology Co., Ltd","QY_REGION_NAME":"Guangdong Province-Guangzhou City","QY_ORG_CODE":"914401017163873948","COLLECT_TIME":1477106545000,"RN":6,"QY_ID":"616160676166636361616466606865626867","QY_SRC_TYPE":"0","OLD_CODE":"716387394"},{"QY_FR_NAME":"Hong Xichang","QY_REGION":"440300","QY_NAME":"Shenzhen Dasheng Construction Group Co., Ltd","QY_REGION_NAME":"Guangdong Province-Shenzhen City","QY_ORG_CODE":"91440300564201676L","COLLECT_TIME":1477106522000,"RN":7,"QY_ID":"616160676166636361616466606865626869","QY_SRC_TYPE":"0","OLD_CODE":"564201676"},{"QY_FR_NAME":"Zheng Li","QY_REGION":"440300","QY_NAME":"Shenzhen Qisheng decoration design Engineering Co., Ltd","QY_REGION_NAME":"Guangdong Province-Shenzhen City","QY_ORG_CODE":"91440300087044558G","COLLECT_TIME":1477106425000,"RN":8,"QY_ID":"616160676166636361616466606865626868","QY_SRC_TYPE":"0","OLD_CODE":"087044558"},{"QY_FR_NAME":"Li Sheng","QY_REGION":"440100","QY_NAME":"Guangdong Beisi Control Technology Co., Ltd","QY_REGION_NAME":"Guangdong Province-Guangzhou City","QY_ORG_CODE":"91440000735024820F","COLLECT_TIME":1477106400000,"RN":9,"QY_ID":"616160676166636361616466606865656160","QY_SRC_TYPE":"0","OLD_CODE":"735024820"},{"QY_FR_NAME":"Zhang XiaoCong","QY_REGION":"440600","QY_NAME":"Guangdong Tianyuan Architectural Design Co., Ltd","QY_REGION_NAME":"Guangdong Province-Foshan City","QY_ORG_CODE":"91440606725095377Y","COLLECT_TIME":1477106406000,"RN":10,"QY_ID":"616160676166636361616466606865656163","QY_SRC_TYPE":"0","OLD_CODE":"725095377"},{"QY_FR_NAME":"Chen Fen","QY_REGION":"440100","QY_NAME":"Shenzhen Zhidao Landscape Design Co., Ltd","QY_REGION_NAME":"Guangdong Province-Guangzhou City","QY_ORG_CODE":"91440300758618904A","COLLECT_TIME":1477106842000,"RN":11,"QY_ID":"616160676166636361616466606865656162","QY_SRC_TYPE":"0","OLD_CODE":"758618904"},{"QY_FR_NAME":"Huang Ronghui","QY_REGION":"440500","QY_NAME":"Shantou Zhenqiao Decoration Engineering Corporation","QY_REGION_NAME":"Guangdong Province-Shantou","QY_ORG_CODE":"914405001927423079","COLLECT_TIME":1477106439000,"RN":12,"QY_ID":"616160676166636361616466606865656165","QY_SRC_TYPE":"0","OLD_CODE":"192742307"},{"QY_FR_NAME":"Li Jian","QY_REGION":"440300","QY_NAME":"Shenzhen Senlei Yiming Design Consulting Co., Ltd
Ask Co., Ltd","QY_REGION_NAME":"Guangdong Province-Shenzhen City","QY_ORG_CODE":"914403007663577081","COLLECT_TIME":1477106424000,"RN":13,"QY_ID":"616160676166636361616466606865656164","QY_SRC_TYPE":"0","OLD_CODE":"766357708"},{"QY_FR_NAME":"Zhang Yumei","QY_REGION":"450100","QY_NAME":"Guangxi Xipu Engineering Design Co., Ltd","QY_REGION_NAME":"Guangxi Zhuang Autonomous Region-Nanning","QY_ORG_CODE":"669733898","COLLECT_TIME":1477106445000,"RN":14,"QY_ID":"616160676166636361616466606865656167","QY_SRC_TYPE":"0","OLD_CODE":"669733898"},{"QY_FR_NAME":"Peng Wei","QY_REGION":"440100","QY_NAME":"Guangzhou tap water engineering Co., Ltd","QY_REGION_NAME":"Guangdong Province-Guangzhou City","QY_ORG_CODE":"91440101190437587Y","COLLECT_TIME":1477106420000,"RN":15,"QY_ID":"616160676166636361616466606865656061","QY_SRC_TYPE":"0","OLD_CODE":"190437587"}],"pageNum":0,"pageSize":15,"total":513826},"message":"","success":true}
{"code":200,"data":{"list":[{"QY_FR_NAME":"Wu fan","QY_REGION":"510100","QY_NAME":"Four seasons wonderful Design Group Co., Ltd","QY_REGION_NAME":"Sichuan Province-Chengdu","QY_ORG_CODE":"915100000858139239","COLLECT_TIME":1477106410000,"RN":16,"QY_ID":"616160676166636361616466606865656060","QY_SRC_TYPE":"0","OLD_CODE":"085813923"},{"QY_FR_NAME":"Wang Hongwu","QY_REGION":"510100","QY_NAME":"Chengdu chengtiandi Engineering Design Consulting Co., Ltd","QY_REGION_NAME":"Sichuan Province-Chengdu","QY_ORG_CODE":"91510105762288418C","COLLECT_TIME":1477106422000,"RN":17,"QY_ID":"616160676166636361616466606865656063","QY_SRC_TYPE":"0","OLD_CODE":"762288418"},{"QY_FR_NAME":"Yuan Liang","QY_REGION":"610100","QY_NAME":"Shaanxi shouchuang Tiancheng Engineering Technology Co., Ltd","QY_REGION_NAME":"Shaanxi Province-Xi'an","QY_ORG_CODE":"91610000684783500F","COLLECT_TIME":1477106808000,"RN":18,"QY_ID":"616160676166636361616466606865656062","QY_SRC_TYPE":"0","OLD_CODE":"684783500"},{"QY_FR_NAME":"Bai Jinhan","QY_REGION":"510100","QY_NAME":"Sichuan Taixing Decoration Engineering Co., Ltd","QY_REGION_NAME":"Sichuan Province-Chengdu","QY_ORG_CODE":"915100002018730434","COLLECT_TIME":1477106445000,"RN":19,"QY_ID":"616160676166636361616466606865656065","QY_SRC_TYPE":"0","OLD_CODE":"201873043"},{"QY_FR_NAME":"Yellow Chimonanthus","QY_REGION":"510100","QY_NAME":"Chengdu Sanyuan Architectural Survey and Design Co., Ltd","QY_REGION_NAME":"Sichuan Province-Chengdu","QY_ORG_CODE":"9151018479216087X2","COLLECT_TIME":1477106448000,"RN":20,"QY_ID":"616160676166636361616466606865656064","QY_SRC_TYPE":"0","OLD_CODE":"202625084"},{"QY_FR_NAME":"Duan Li","QY_REGION":"510100","QY_NAME":"Sichuan ruishizhongzheng Construction Engineering Co., Ltd","QY_REGION_NAME":"Sichuan Province-Chengdu","QY_ORG_CODE":"91510115592094097F","COLLECT_TIME":1477106696000,"RN":21,"QY_ID":"616160676166636361616466606865656067","QY_SRC_TYPE":"0","OLD_CODE":"592094097"},{"QY_FR_NAME":"Lou Yongjun","QY_REGION":"110000","QY_NAME":"Beijing Huaqing Technology Engineering Management Co., Ltd","QY_REGION_NAME":"Beijing","QY_ORG_CODE":"9111010872146477XF","COLLECT_TIME":1477106408000,"RN":22,"QY_ID":"616160676166636361616466606865656066","QY_SRC_TYPE":"0","OLD_CODE":"72146477X"},{"QY_FR_NAME":"Chen Jigang","QY_REGION":"210100","QY_NAME":"Shenyang Huasheng Engineering Construction Supervision Co., Ltd","QY_REGION_NAME":"Liaoning Province-Shenyang
","QY_ORG_CODE":"91210103242650807A","COLLECT_TIME":1477106423000,"RN":23,"QY_ID":"616160676166636361616466606865656069","QY_SRC_TYPE":"0","OLD_CODE":"242650807"},{"QY_FR_NAME":"Liu Haifu","QY_REGION":"230600","QY_NAME":"Daqing Daguang Engineering Construction Supervision Co., Ltd","QY_REGION_NAME":"Heilongjiang Province-Daqing City","QY_ORG_CODE":"912306047253128885","COLLECT_TIME":1477106451000,"RN":24,"QY_ID":"616160676166636361616466606865656361","QY_SRC_TYPE":"0","OLD_CODE":"725312888"},{"QY_FR_NAME":"Li Huaibin","QY_REGION":"211400","QY_NAME":"Huludao Haiwei project
Construction Supervision Co., Ltd","QY_REGION_NAME":"Liaoning Province-Huludao City","QY_ORG_CODE":"912114007164286430","COLLECT_TIME":1477106432000,"RN":25,"QY_ID":"616160676166636361616466606865656360","QY_SRC_TYPE":"0","OLD_CODE":"716428643"},{"QY_FR_NAME":"Wang Feng","QY_REGION":"120000","QY_NAME":"Tianjin Port Oil Engineering Construction Supervision Co., Ltd","QY_REGION_NAME":"Tianjin","QY_ORG_CODE":"9112011674668236XP","COLLECT_TIME":1477107655000,"RN":26,"QY_ID":"616160676166636361616466606865656363","QY_SRC_TYPE":"0","OLD_CODE":"74668236X"},{"QY_FR_NAME":"Lu Minjie","QY_REGION":"310000","QY_NAME":"Shanghai Shenyi Engineering Consulting Co., Ltd","QY_REGION_NAME":"Shanghai","QY_ORG_CODE":"913101181345318787","COLLECT_TIME":1477106405000,"RN":27,"QY_ID":"616160676166636361616466606865656362","QY_SRC_TYPE":"0","OLD_CODE":"134531878"},{"QY_FR_NAME":"Gao Shuangxuan","QY_REGION":"130100","QY_NAME":"Shijiazhuang Zhongtian Engineering Construction Supervision Co., Ltd","QY_REGION_NAME":"Hebei Province-Shijiazhuang City ","QY_ORG_CODE":"9113010060118700X3","COLLECT_TIME":1477106403000,"RN":28,"QY_ID":"616160676166636361616466606865656365","QY_SRC_TYPE":"0","OLD_CODE":"60118700X"},{"QY_FR_NAME":"Fan Guangyi","QY_REGION":"231000","QY_NAME":"Mudanjiang Longhua Engineering Construction Supervision Co., Ltd","QY_REGION_NAME":"Heilongjiang Province-Mudanjiang City","QY_ORG_CODE":"912310007026319900","COLLECT_TIME":1477106420000,"RN":29,"QY_ID":"616160676166636361616466606865656367","QY_SRC_TYPE":"0","OLD_CODE":"702631990"},{"QY_FR_NAME":"Guo Hua Wang","QY_REGION":"320100","QY_NAME":"Jiangsu traffic engineering consulting and Supervision Co., Ltd","QY_REGION_NAME":"Jiangsu Province-Nanjing City","QY_ORG_CODE":"913200001347697633","COLLECT_TIME":1477106402000,"RN":30,"QY_ID":"616160676166636361616466606865656366","QY_SRC_TYPE":"0","OLD_CODE":"134769763"}],"pageNum":1,"pageSize":15,"total":450},"message":"","success":true}
{"code":200,"data":{"list":[{"QY_FR_NAME":"Miao Cunxu","QY_REGION":"350200","QY_NAME":"Xiamen Xinghai Bay Engineering Management Co., Ltd","QY_REGION_NAME":"Fujian Province-Xiamen","QY_ORG_CODE":"913502112601177740","COLLECT_TIME":1477106448000,"RN":31,"QY_ID":"616160676166636361616466606865656369","QY_SRC_TYPE":"0","OLD_CODE":"260117774"},{"QY_FR_NAME":"You Liang Chen","QY_REGION":"330800","QY_NAME":"Zhejiang Zhoudao Engineering Management Co., Ltd","QY_REGION_NAME":"Zhejiang Province-Quzhou City","QY_ORG_CODE":"91330802330012039P","COLLECT_TIME":1477106529000,"RN":32,"QY_ID":"616160676166636361616466606865656261","QY_SRC_TYPE":"0","OLD_CODE":"330012039"},{"QY_FR_NAME":"Xu Bingshui","QY_REGION":"370100","QY_NAME":"Shandong Zhengda Construction Supervision Co., Ltd","QY_REGION_NAME":"Shandong Province-Jinan City","QY_ORG_CODE":"913701022671825366","COLLECT_TIME":1477106425000,"RN":33,"QY_ID":"616160676166636361616466606865656260","QY_SRC_TYPE":"0","OLD_CODE":"267182536"},{"QY_FR_NAME":"Wang Zhong","QY_REGION":"510100","QY_NAME":"KaiBang Construction Management Group Co., Ltd","QY_REGION_NAME":"Sichuan Province-Chengdu","QY_ORG_CODE":"91510105725354703M","COLLECT_TIME":1477106469000,"RN":34,"QY_ID":"616160676166636361616466606865656263","QY_SRC_TYPE":"0","OLD_CODE":"725354703"},{"QY_FR_NAME":"Liu Yinuo","QY_REGION":"371300","QY_NAME":"Linyi Hongli Construction Engineering Supervision Consulting Co., Ltd","QY_REGION_NAME":"Shandong Province-Linyi City","QY_ORG_CODE":"913713005543587173","COLLECT_TIME":1477106454000,"RN":35,"QY_ID":"616160676166636361616466606865656262","QY_SRC_TYPE":"0","OLD_CODE":"554358717"},{"QY_FR_NAME":"Wang Hongjuan","QY_REGION":"410100","QY_NAME":"Henan Zhongjian Engineering Technology Co., Ltd","QY_REGION_NAME":"Henan Province-Zhengzhou City","QY_ORG_CODE":"914101057919311268","COLLECT_TIME":1477106526000,"RN":36,"QY_ID":"616160676166636361616466606865656265","QY_SRC_TYPE":"0","OLD_CODE":"791931126"},{"QY_FR_NAME":"Lin Jin","QY_REGION":"350100","QY_NAME":"Fujian Jianxin Engineering Management Group Co., Ltd","QY_REGION_NAME":"Fujian Province-Fuzhou","QY_ORG_CODE":"913501001544135960","COLLECT_TIME":1477106407000,"RN":37,"QY_ID":"616160676166636361616466606865656264","QY_SRC_TYPE":"0","OLD_CODE":"154413596"},{"QY_FR_NAME":"Yang gaobao","QY_REGION":"421200","QY_NAME":"Jiayu Jiatai Construction Supervision Co., Ltd","QY_REGION_NAME":"Hubei province-Xianning ","QY_ORG_CODE":"91421221737145124M","COLLECT_TIME":1477106486000,"RN":38,"QY_ID":"616160676166636361616466606865656267","QY_SRC_TYPE":"0","OLD_CODE":"737145124"},{"QY_FR_NAME":"Jiang Yabin","QY_REGION":"410800","QY_NAME":"Henan synergy Engineering Supervision Co., Ltd","QY_REGION_NAME":"Henan Province-Jiaozuo City","QY_ORG_CODE":"91410800555731456W","COLLECT_TIME":1477106405000,"RN":39,"QY_ID":"616160676166636361616466606865656266","QY_SRC_TYPE":"0","OLD_CODE":"555731456"},{"QY_FR_NAME":"Xiang Jun Huang","QY_REGION":"450100","QY_NAME":"Guangxi Nanning Jianke Engineering Supervision Co., Ltd
","QY_REGION_NAME":"Guangxi Zhuang Autonomous Region-Nanning","QY_ORG_CODE":"914501031983159639","COLLECT_TIME":1477106406000,"RN":40,"QY_ID":"616160676166636361616466606865656269","QY_SRC_TYPE":"0","OLD_CODE":"198315963"},{"QY_FR_NAME":"Liao Yuan","QY_REGION":"510100","QY_NAME":"Chengdu
Hailong Engineering Project Management Co., Ltd","QY_REGION_NAME":"Sichuan Province-Chengdu","QY_ORG_CODE":"91510100201986152W","COLLECT_TIME":1477106437000,"RN":41,"QY_ID":"616160676166636361616466606865656268","QY_SRC_TYPE":"0","OLD_CODE":"201986152"},{"QY_FR_NAME":"Zou Yun","QY_REGION":"360500","QY_NAME":"Xinyu YONGGU Engineering Supervision Co., Ltd","QY_REGION_NAME":"Jiangxi Province-Xinyu City","QY_ORG_CODE":"91360500723935352F","COLLECT_TIME":1477109337000,"RN":42,"QY_ID":"616160676166636361616466606865656561","QY_SRC_TYPE":"0","OLD_CODE":"723935352"},{"QY_FR_NAME":"Wang Wenquan","QY_REGION":"371300","QY_NAME":"Shandong zhenghang Engineering Consulting Co., Ltd","QY_REGION_NAME":"Shandong Province-Linyi City","QY_ORG_CODE":"91371302092199458A","COLLECT_TIME":1477106426000,"RN":43,"QY_ID":"616160676166636361616466606865656560","QY_SRC_TYPE":"0","OLD_CODE":"092199458"},{"QY_FR_NAME":"Hong Jian Zhang","QY_REGION":"530100","QY_NAME":"Yunnan jinji'an construction consulting and Supervision Co., Ltd","QY_REGION_NAME":"Yunnan Province-Kunming","QY_ORG_CODE":"91530000719414685K","COLLECT_TIME":1479833531000,"RN":44,"QY_ID":"616160676166636361616466606865656567","QY_SRC_TYPE":"0","OLD_CODE":"719414685"},{"QY_FR_NAME":"Zhang Zhiming","QY_REGION":"610700","QY_NAME":"Hanzhong Zhicheng Construction Engineering Supervision Co., Ltd","QY_REGION_NAME":"Shaanxi Province-Hanzhong City","QY_ORG_CODE":"91610725667992132H","COLLECT_TIME":1477106417000,"RN":45,"QY_ID":"616160676166636361616466606865656568","QY_SRC_TYPE":"0","OLD_CODE":"667992132"}],"pageNum":2,"pageSize":15,"total":450},"message":"","success":true}
Here, the crawling data section is all over.
About Python technology reserve
It's good to learn Python well, whether in employment or sideline, but to learn python, you still need to have a learning plan. Finally, let's share a full set of Python learning materials to help those who want to learn Python!
1, Python learning routes in all directions
All directions of Python is to sort out the commonly used technical points of Python and form a summary of knowledge points in various fields. Its purpose is that you can find corresponding learning resources according to the above knowledge points to ensure that you learn more comprehensively.
2, Learning software
If a worker wants to do well, he must sharpen his tools first. The commonly used development software for learning Python is here, which saves you a lot of time.
3, Getting started video
When we watch videos, we can't just move our eyes and brain without hands. The more scientific learning method is to use them after understanding. At this time, the hand training project is very suitable.
4, Actual combat cases
Optical theory is useless. We should learn to knock together and practice, so as to apply what we have learned to practice. At this time, we can make some practical cases to learn.
5, Interview materials
We must learn Python in order to find a high paying job. The following interview questions are the latest interview materials from front-line Internet manufacturers such as Alibaba, Tencent and byte, and Alibaba boss has given authoritative answers. After brushing this set of interview materials, I believe everyone can find a satisfactory job.
