preface
In April, he joined a data-oriented artificial intelligence enterprise in Nanjing as a python crawler intern. In addition to writing scripts for Python crawlers, this position also has a job of text markup. Text marking can be regarded as a process of sorting and cleaning data manually. Although it is boring, it is also an essential link.
In the past half a month or so, I have come into contact with the crawler of json files. Before, I also compared dishes. I came into contact with the kind that can directly match html files with xpath. In my current work, I came into contact with the crawler of json files for the first time. Now I take the fund Minsheng Jiayin CSI 300ETF as an example and crawl its subscription and redemption list by analyzing its json files.
analysis
It is understood that generally, only the funds ending in the three letters of ETF will have the "purchase and redemption list", and here only the fund "Minsheng Canada Bank Shanghai and Shenzhen 300ETF" meets the requirements, as shown in the figure below:
Therefore, there is no need to write the page turning function this time, because there is only one that meets the requirements, so just climb this page alone, as shown in the figure:
What we need to crawl are the following fields: fund code, announcement date, name of position stock, code of position stock, number of position stock, cash substitution mark, subscription premium proportion, redemption discount proportion and substitution amount. You can see that there are all of them on this page.
So I wrote the following code:
import requests
headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.141 Safari/537.36',
'Referer': 'http://www.msjyfund.com.cn/osoa/views/funds/fundDetail.html?fund_code=mcaSesVXrHU=&fund_type=M0Uo0ORwvYw=',
}
base_url = "http://www.msjyfund.com.cn/osoa/views/funds/fundDetail.html?fund_code=mcaSesVXrHU=&fund_type=M0Uo0ORwvYw="
html = requests.get(url=base_url, headers=headers).content.decode()
print(html)The following results are obtained after operation:
C:\Users\ABC\AppData\Local\Programs\Python\Python38\python.exe C:/Users/ABC/PycharmProjects/pythonProject/1.py
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta http-equiv="X-UA-Compatible" content="IE=9" />
<meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1">
<link href="/osoa/css/style.css" rel="stylesheet" type="text/css">
<title>Minsheng Jiayin</title>
</head>
<body>
<!--header-->
<div id="headerContent"></div>
<!--banner-->
<div class="banner_box banner_box1"></div>
<!--content-->
<div id="bodyContent">
<div class="content bg_gray" id="funds_fundDetail">
<div class="wrapfix pt290">
<div class="prarea clearfix prarea1" style="background: #fcfcfe;">
<div class="fl">
<!-- Non monetary type -->
<div id="tabs02" class="fhp_fund">
<div class="jy_top clearfix">
<p class="active" data-value="1">Trend of unit net worth</p>
<p data-value="2">Cumulative net worth trend</p>
</div>
<div class="jy_body">
<div class="jy_bot" style="display: block;">
<div class="tab03">
<div class="jy_sub_tit clearfix">
<a href="javascript:;" class="active" data-value="1">1 month</a>
<a href="javascript:;" data-value="3">3 month</a>
<a href="javascript:;" data-value="6">6 month</a>
<a href="javascript:;" data-value="12">1 year</a>
<a href="javascript:;" data-value="36">3 year</a>
<a href="javascript:;" data-value="40">since this year on</a>
<a href="javascript:;" data-value="41">Since its establishment</a>
<span class="nv_block">Historical net worth query ></span>
</div>
<div class="jy_sub_body">
<div class="jy_sub_bot" style="display: block;">
<div class="jy_chart" id="fhp_fund_chat_yjbx">
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<!-- Currency type -->
<div id="tabs02" class="hp_fund" style="display: none;">
<div class="jy_top clearfix">
<p class="active" data-value="3">Seven day annualized yield</p>
<p data-value="4">10000 shares of income</p>
</div>
<div class="jy_body">
<div class="jy_bot" style="display: block;">
<div class="tab03">
<div class="jy_sub_tit clearfix">
<a href="javascript:;" class="active" data-value="3">3 Months</a>
<a href="javascript:;" data-value="12">1 year</a>
<a href="javascript:;" data-value="36">3 year</a>
<a href="javascript:;" data-value="40">since this year on</a>
<span class="nv_block">Historical net worth query ></span>
</div>
<div class="jy_sub_body">
<div class="jy_sub_bot" style="display: block;">
<div class="jy_chart" id="hp_fund_chat_yjbx">
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="fr" id="fund_base_info">
<script id="fund_info_02" type="text/template" defer="defer">
<div class="jy_tit">
<h2>{{= it.fund_short_name }}<span>({{= it.fund_code }})</span></h2>
<div class="clearfix">
<a href="javascript:;" class="blue_bor">{{= it.fund_type }}fund</a>
<a href="javascript:;" class="red_bor">{{= it.risk_level }}</a>
</div>
</div>
{{? it.fund_text == "Non monetary type" }}
<div class="jy_cover clearfix" id="show_fhb_content">
<div>
<h2><strong>{{= it.unit_nv }}</strong></h2>
<label>Average NAV<span>[ {{= it.nv_date.substr(5,5) }} ]</span></label>
</div>
<div class="nr_bor">
<p>Daily rise and fall:<span class="{{= it.daily_change_color }}">{{= it.daily_change }}</span></p>
<p class="nb_mar">
Change in net worth:<span class="{{= it.nv_change_color }}">{{= it.nv_change }}</span>
</p>
</div>
</div>
{{??}}
<!-- Currency type -->
<div class="jy_cover clearfix" id="show_hb_content" style="padding-bottom: 30px;">
<div>
<h2><strong>{{= it.serven_days_annual_profit }}</strong></h2>
<label>Seven day annualized yield <span>[ {{= it.nv_date.substr(5,5) }} ]</span></label>
<small>10000 shares of income <em>{{= it.profit_per_million }}</em></small>
</div>
<div class="nr_bor">
<p>Redemption duration:<span>{{= it.redeem_time || 0 }}</span>Trading days</p>
<p>Date of establishment:<span>{{= it.establish_date }}</span></p>
<p class="nb_mar">
Latest scale:{{= it.newest_scale ? '<span>'+it.newest_scale+'element</span><em>('+it.newest_scale_date+')</em>' : '--' }}
</p>
</div>
</div>
{{?}}
<div class="jy_bottom">
<h5>Investment amount:</h5>
<div class="clearfix">
<label>
<input type="text" value="{{= it.min_buy_amount_detail.indexOf('--') >= 0 ? '0.00' : it.min_buy_amount_detail }}" id="tz_amount"/>
<em>element</em>
</label>
<a href="javascript:;" class="lj_bug">Immediate subscription</a>
<span class="rate_calculation"></span>
</div>
<p class="bug_rate" style="{{= it.fund_text == 'Non monetary type' ? 'display: block;' : 'display: none;' }}">
rate <s>{{= it.buy_rate }}</s><em>{{= it.preferential_rate.indexOf('--') >= 0 ? '' : '1 Fold up' }}</em><b>(Deduction by using jiayinbao (except for China Merchants Bank card)</b>
</p>
</div>
</script>
</div>
</div>
<div class="infoarea clearfix">
<div class="fl" style="height: auto;min-height: 761px;">
<div class="in_tab clearfix" id="show_tag">
<a href="javascript:;" class="active">essential information</a>
<a href="javascript:;" style="display: none;">investment portfolio </a>
<a href="javascript:;">fund manager</a>
<a href="javascript:;">Fund announcement</a>
<a href="javascript:;">rate </a>
<a href="javascript:;">Dividend information</a>
<a href="javascript:;">Sales organization</a>
<a href="javascript:;" >Historical net worth</a>
<a href="javascript:;" style="margin-right: 0px;display: none;" id="etf_qd">Redemption list</a>
</div>
<div id="product_info">
<!-- essential information -->
<div class="in_block jbxx fund_info_01" style="display: block;">
<script id="fund_info_01" type="text/template" defer="defer">
<ul class="in_ul">
<li>
<span>Fund name</span>
<p class="p1">{{= it.fund_name }}(abbreviation "{{= it.fund_short_name }}")</p>
</li>
<li>
<span>Fund code</span>
<p class="p1">{{= it.fund_code }}</p>
</li>
<li>
<span>Fund type</span>
<p class="p1">{{= it.fund_type }}</p>
</li>
<li>
<span>Date of Incorporation</span>
<p class="p1">{{= it.establish_date.replace("-", "year").replace("-", "month").replace("Year 0", "year").replace("Month 0", "month")+"day" }}</p>
</li>
<li>
<span>Fund manager</span>
<p class="p1">{{= it.manager_name }}</p>
</li>
<li>
<span>Fund custodian</span>
<p class="p1">{{= it.custodian_name }}</p>
</li>
<li>
<span>Performance benchmark</span>
<p class="p1">{{= it.benchmark.replace(/style/ig, "data-style") }}</p>
</li>
<li>
<span>Investment objectives</span>
<p class="p1">{{= it.invest_objective.replace(/style/ig, "data-style") }}</p>
</li>
<li>
<span>scope of investment</span>
<p class="p1">{{= it.invest_scope.replace(/style/ig, "data-style") }}</p>
</li>
<li class="fund_strategy" style="{{= it.invest_strategy.replace(/<.*?>/ig, "") ? 'display: block;' : 'display: none;' }}">
<span>Investment proportion</span>
<p class="p1">{{= it.invest_strategy.replace(/style/ig, "data-style").replace(/span/ig, "p") }}</p>
</li>
<li>
<span>Risk return characteristics</span>
<p class="p1">{{= it.risk_feature.replace(/style/ig, "data-style") }}</p>
</li>
</ul>
</script>
</div>
<!-- investment portfolio -->
<div class="in_table tzzh" style="display: none;padding: 0;">
<div class="in_table ">
<div class="in_tit clearfix">
<h4>--</h4>
<p>--</p>
</div>
<div class="in_chart in_chart1">
<div class="in_img" id="tzzh_graph" style="width:550px;height:400px">
</div>
<div class="in_info clearfix" style="position: relative;">
<p>shares<i></i></p>
<p>bond<i class="p2"></i></p>
<p>cash<i class="p3"></i></p>
<p class="nr_mar">other<i class="p4"></i></p>
</div>
</div>
</div>
</div>
<!-- fund manager -->
<div class="in_table nice_tab_box zjjl" style="display: none;padding: 0;">
<ul class="nice_tab_tit clearfix manager_list">
<script id="manager_list" type="text/template" defer="defer">
{{ for(var i = 0; i < it.length; i++) { }}
<li class="{{= i == 0 ? 'active' : ''}}">
<div class="ntt_img">
<img src="{{= it[i].photo_url }}" height=100 alt=""/>
</div>
<h5>{{= it[i].member_name }}</h5>
<p>{{= it[i].education }}</p>
<span class="{{= i == 0 ? 'active' : ''}}"></span>
</li>
{{ } }}
</script>
</ul>
<div class="nice_tab_body">
<div class="nice_tab_children">
<div class="in_table ">
<div class="posit clearfix">
<div class="p_left">
<a href="javascript:;"><img></a>
</div>
<div class="p_right">
<h4>
--<span>--</span>
<div class="a_btns">
<!-- <a href="/osoa/views/newsView/fundManagerList.html" target="_blank">Approach the fund manager ></a> -->
</div>
</h4>
<span>Working time:<em>--</em></span>
<p>--</p>
</div>
</div>
</div>
<div class="in_table ">
<div class="in_tit clearfix">
<h4>Fund manager view</h4>
</div>
<div class="in_word">--</div>
</div>
<div class="in_table in_table4">
<div class="in_tit clearfix">
<h4>Product performance</h4>
</div>
<table width="100%" border="1" cellpadding="0" cellspacing="0">
<thead>
<tr>
<th>Fund code</th>
<th>Fund abbreviation</th>
<th>Date of Incorporation</th>
<th>Share net worth(element)</th>
<th>since this year on (%)</th>
<th>Since its establishment (%)</th>
</tr>
</thead>
<tbody class="manager_performance">
</tbody>
</table>
<div class="in_tit clearfix hbly_type" style="margin-top:20px;display:none">
</div>
<table class="hbly_type" width="100%" border="1" cellpadding="0" cellspacing="0" style="display:none">
<thead>
<tr>
<th>Fund code</th>
<th>Fund abbreviation</th>
<th>Date of Incorporation</th>
<!-- <th>Share net worth(element)</th> -->
<th>Seven day annualized yield (%)</th>
<th>10000 shares of income (element)</th>
</tr>
</thead>
<tbody class="manager_performance2">
</tbody>
</table>
<div class="fund_manger_page"></div>
</div>
</div>
</div>
</div>
<!-- Fund announcement -->
<div class="zjgg" style="display: none;">
<div class="in_table xxarea xxarea1" id="release_document">
<div class="in_tit clearfix">
<h4>legal document</h4>
<a href="javascript:;" class="a_more announcement_list" data-type="0"></a>
</div>
<div class="posit clearfix">
</div>
</div>
<div class="in_table xxarea xxarea1" id="periodic_announcement">
<div class="in_tit clearfix">
<h4>Regular announcement</h4>
<a href="javascript:;" class="a_more announcement_list" data-type="1"></a>
</div>
<div class="posit clearfix">
</div>
</div>
<div class="in_table xxarea xxarea1" id="interim_announcement">
<div class="in_tit clearfix">
<h4>Temporary announcement</h4>
<a href="javascript:;" class="a_more announcement_list" data-type="2"></a>
</div>
<div class="posit clearfix">
</div>
</div>
</div>
<!-- rate -->
<div class="zjfl" style="display: none;">
<div class="in_table rgfv">
<div class="in_tit clearfix">
<h4>Fund subscription rate</h4>
</div>
<table width="100%" border="1" cellpadding="0" cellspacing="0">
<thead>
<tr>
<th>Subscription amount(M,Company:element)</th>
<th>rate </th>
</tr>
</thead>
<tbody>
</tbody>
</table>
</div>
<div class="in_table sgfv">
<div class="in_tit clearfix">
<h4>Fund subscription rate</h4>
</div>
<table width="100%" border="1" cellpadding="0" cellspacing="0">
<thead>
<tr>
<th>Subscription amount(M,Company:element)</th>
<th>rate </th>
</tr>
</thead>
<tbody>
</tbody>
</table>
</div>
<div class="in_table shfv">
<div class="in_tit clearfix">
<h4>Fund redemption rate</h4>
</div>
<table width="100%" border="1" cellpadding="0" cellspacing="0">
<thead>
<tr>
<th>Holding period(T)</th>
<th>rate </th>
</tr>
</thead>
<tbody>
</tbody>
</table>
</div>
<div class="in_table in_table3 yzfv">
<div class="in_tit clearfix">
<h4>Fund operation fee</h4>
</div>
<table width="100%" border="1" cellpadding="0" cellspacing="0">
<thead>
<tr>
<th>Management rate</th>
<th>Custody fee</th>
<th>Annual sales service fee</th>
</tr>
</thead>
<tbody>
</tbody>
</table>
</div>
<div class="a_btns"><a href="/osoa/views/serviceCenter/rateDiscount.html" target="_blank">Online transaction rate discount ></a></div>
</div>
<!--Dividend information -->
<div class="fhxx" style="display: none;">
<div class="in_table in_table1">
<div class="in_tit clearfix">
<h4>Fund dividend record</h4>
</div>
<table width="100%" border="1" cellpadding="0" cellspacing="0">
<thead>
<tr>
<th>Dividend year</th>
<th>Equity registration date</th>
<!-- <th>Ex dividend day</th> -->
<th>Dividend payment date</th>
<th class="w29">Dividends are paid out every 10 points of income (element)</th>
</tr>
</thead>
<tbody>
</tbody>
</table>
<div class="in_no" style="display: none;">
<img src="/osoa/images/jp/img_03.png" />
No dividend record
</div>
</div>
<div class="in_table in_table2">
<div class="in_tit clearfix">
<h4>Fund split record</h4>
</div>
<table width="100%" border="1" cellpadding="0" cellspacing="0">
<thead>
<tr>
<th>Split date</th>
<th>Net value before split</th>
<th>Net value after split</th>
<th>Split scale (%)</th>
</tr>
</thead>
<tbody>
</tbody>
</table>
<div class="in_no" style="display: none;">
<img src="/osoa/images/jp/img_03.png" />
No split record
</div>
</div>
</div>
<!-- Sales organization -->
<div class="xsjg" style="display: none;">
<div class="in_table">
<table width="100%" border="1" cellpadding="0" cellspacing="0">
<thead>
<tr>
<th>Sales organization</th>
<th>Service hotline</th>
</tr>
</thead>
<tbody>
</tbody>
</table>
</div>
</div>
<!-- Historical net worth list -->
<div class="in_table in_table2 lsjz" style="display: none;">
<div class="in_tit clearfix">
<h4>Historical net worth list</h4>
<div class="date clearfix">
<div class="d_input">
<input type="text" id="start_time" readOnly onClick="WdatePicker({maxDate:'#F{$dp.$D(\'end_time\')}'})"/>
<i></i>
</div>
<span>one</span>
<div class="d_input">
<input type="text" id="end_time" readOnly onClick="WdatePicker({minDate:'#F{$dp.$D(\'start_time\')}'})"/>
<i></i>
</div>
<a href="javascript:;" id="search_data"></a>
</div>
</div>
<table width="100%" border="1" cellpadding="0" cellspacing="0">
<thead>
<tr>
<th>Release date</th>
<th>--</th>
<th>--</th>
<th>Net asset value (yuan)</th>
</tr>
</thead>
<tbody>
</tbody>
</table>
</div>
<!-- Redemption list start-->
<div class="redeem_list shqd" style="display: none;">
<div class="redeem_search">
<em></em>
<span>List date</span>
<input type="text" id="startDate" readonly="readonly" />
<a href="javascript:void(0);" class="btn" id="serchBtn">query</a>
</div>
<!-- Fund information -->
<div id="fund_info" >
</div>
<!-- Component stock information content -->
<div id="fund_list" >
</div>
</div>
<!-- End of redemption list-->
</div>
</div>
<div class="fr">
<div class="f_top">Hot selling fund</div>
<ul id="hot_fund_list">
</ul>
</div>
</div>
</div>
</div>
</div>
<!--footer-->
<div id="footerContent"></div>
<!--Right side suspension-->
<!--
<div class="right_bar">
<a href="javascript:;" class="bar_block kf"></a>
<a href="javascript:;" class="bar_block wei">
<div class="hide_wei">
<img src="/osoa/images/img_w3.png">
</div>
</a>
<a href="javascript:;" class="bar_block up"></a>
</div> -->
<!-- Template -->
<script id="manager_performance" type="text/template" defer="defer">
{{ for(var i = 0; i < it.length; i++) { }}
{{? !(it[i].fund_type == '6'||it[i].fund_type == '5') }}
<tr>
<td>{{= it[i].fund_code }}</td>
<td>{{= it[i].fund_short_name }}</td>
<td>{{= it[i].establish_date }}</td>
<td class="{{= it[i].unit_nv_color }}">{{= it[i].unit_nv }}</td>
<td class="{{= it[i].yieldratety_color }}">{{= it[i].yieldratety }}</td>
<td class="{{= it[i].yieldratestart_color }}">{{= it[i].yieldratestart }}</td>
</tr>
{{?}}
{{ } }}
</script>
<script id="manager_performance2" type="text/template" defer="defer">
{{ for(var i = 0; i < it.length; i++) { }}
{{? it[i].fund_type == '5'||it[i].fund_type == '6'}}
<tr>
<td>{{= it[i].fund_code }}</td>
<td>{{= it[i].fund_short_name }}</td>
<td>{{= it[i].establish_date }}</td>
<td class="{{= it[i].serven_days_annual_profit }}">{{= it[i].serven_days_annual_profit }}</td>
<td class="{{= it[i].profit_per_million }}">{{= it[i].profit_per_million }}</td>
</tr>
{{?}}
{{ } }}
</script>
<!-- Template -->
<script type="text/javascript">
//Project body page pageCode: page path code, jsonParam: parameter, isLoad: whether to load every time
var _defaultPage = {"pageCode": "funds/fundDetail", "jsonParam":{},"isLoad":false};
//pageCode of project header page: page path code, jsonParam: parameter, isLoad: whether to load every time. The default is include / header
var _headerPage = {"pageCode": "include/header", "jsonParam":{},"isLoad":false};
//pageCode of project footer page: page path code, jsonParam: parameter, isLoad: whether to load every time. The default is include / footer
var _footerPage = {"pageCode": "include/footer", "jsonParam":{},"isLoad":false};
</script>
<script type="text/javascript" src="/osoa/version.js"></script>
<!--Basic framework package -->
<script type="text/javascript" src="/thinkiveJs.min.js"></script>
</body>
</html>
Process finished with exit code 0
You can press Ctrl+F to search for keywords such as fund name, and you will find that there is no relevant HTML at all, so the data of this table should be rendered through Ajax. I pressed F12 in the browser to check the DOC item in the Network, and confirmed my idea at the same time. I searched XHR in Network in F12 and finally found that all the data I wanted was in this json file, as shown in the following two figures:
So now, just crawl the json file, so I wrote the following code:
import requests
headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.141 Safari/537.36',
'Referer': 'http://www.msjyfund.com.cn/osoa/views/funds/fundDetail.html?fund_code=mcaSesVXrHU=&fund_type=M0Uo0ORwvYw=',
}
base_url = "http://www.msjyfund.com.cn/servlet/json"
json = requests.get(url=base_url, headers=headers).json()
print(json)The operation was carried out with joy, but the following results were obtained:
C:\Users\ABC\AppData\Local\Programs\Python\Python38\python.exe C:/Users/ABC/PycharmProjects/pythonProject/1.py
{'error_no': '-10004', 'error_info': 'call BUS The function number of the interface cannot be empty'}
Process finished with exit code 0After reflection, I found that there was no parameter passed, so I found the parameter to be passed here, as shown in the figure below:
So the url is changed to
http://www.msjyfund.com.cn/servlet/json?funcNo=905505&fund_code=515350&tradingday=&curtPageNo=1&numPerPage=1000
Then follow this idea to climb to the data!
Complete code
import requests
data = {
"decel_date": "", # Announcement date
# "trade_date": "", # Transaction date
# "sub_redem_sec_type": "", # Category of redemption application components
"fund_code": "", # Fund transaction code
"sec_code": "", # Component securities publication code
"sec_name": "", # Securities abbreviation
"sec_quantity": "", # Number of Securities: shares / hands / gram
"cash_substitute_sign": "", # Cash substitution flag: 0 - prohibited; 1 - allowed; 2 - must; 3 - reimbursement
# "cash_substitute_ratio": "", # Cash substitution premium ratio
"fixed_substitute_amount": "", # Fixed substitution amount
"sub_replace_amt": "", # Subscription substitution amount
"redem_replace_amt": "", # Redemption substitution amount
}
headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.141 Safari/537.36',
'Referer': 'http://www.msjyfund.com.cn/osoa/views/funds/fundDetail.html?fund_code=mcaSesVXrHU=&fund_type=M0Uo0ORwvYw=',
}
base_url = "http://www.msjyfund.com.cn/servlet/json?funcNo=905505&fund_code=515350&tradingday=&curtPageNo=1&numPerPage=1000"
json = requests.get(url=base_url, headers=headers).json()
# print(json)
items = json.get('dataList')[0].get('data')
for item in items:
data['fund_code'] = item['fundcode']
data['decel_date'] = item['tradingday']
data['sec_code'] = item['stockcode']
data['sec_name'] = item['stockname']
data['sec_quantity'] = item['stocknum']
data['cash_substitute_sign'] = item['cashflag']
data['sub_replace_amt'] = item['redemptiondiscountrate']
data['redem_replace_amt'] = item['cashratio']
data['fixed_substitute_amount'] = item['substituteamount']
print(data)Operation results
END