Background:
IDC computer room built by the company, based on IDC computer room to build a large data cluster; need to monitor cluster resources, the cluster uses CDH cluster, collection is mainly divided into two parts:
Collection of indexes related to HDFS and YARN
IDC machine's own index collection
Note: Some may wonder why the CM interface already provides monitored diagrams and why you need to present them yourself.The reason is that this information needs to be integrated into the internal data platform to make corresponding data reports and visually displayed on their own data platform
The implementation ideas can be roughly divided into two types:
Use the Java API provided by CM to get
Use the REST API provided by CM to getIn fact, they are essentially the same, and the Java API s provided by CM are implemented in accordance with the REST API suite, and they are consistent
The core code is as follows:
public class IdcHostResource {
private static final Logger LOGGER = LoggerFactory.getLogger(IdcHostResource.class);
static RootResourceV18 apiRoot;
// TODO...dead, needs improvement
static {
apiRoot = new ClouderaManagerClientBuilder()
.withHost("cm ip")
.withPort(7180)
.withUsernamePassword("user", "passwd")
.build()
.getRootV18();
}
/**
* Fixed access to basic resource information for Host
*/
public static List<IdcHostBasicInfo> getAllHostResource() {
List<IdcHostBasicInfo> hosts = new ArrayList<IdcHostBasicInfo>();
HostsResourceV10 hostsResourceV10 = apiRoot.getHostsResource();
List<ApiHost> hostLists = hostsResourceV10.readHosts(DataView.SUMMARY).getHosts();
LOGGER.info("Total" + hostLists.size() + "Host");
for (ApiHost hostList : hostLists) {
IdcHostBasicInfo host = formatHost(hostsResourceV10.readHost(hostList.getHostId()));
LOGGER.info("Host Name: " + host.getHostName());
LOGGER.info("Host Health Summary: " + host.gethostHealthSummary());
LOGGER.info("Host Physical Memory: " + host.getTotalPhysMemBytes());
hosts.add(host);
}
return hosts;
}
public static IdcHostBasicInfo formatHost(ApiHost apiHost) {
IdcHostBasicInfo idcHostBasicInfo = new IdcHostBasicInfo();
idcHostBasicInfo.sethostHealthSummary(apiHost.getHealthSummary().toString());
idcHostBasicInfo.setHostName(apiHost.getHostname());
idcHostBasicInfo.setTotalPhysMemBytes(apiHost.getTotalPhysMemBytes());
return idcHostBasicInfo;
}
/**
* Get corresponding metrics info dynamically through tsquery
*
* @param query
* @param startTime
* @param endTime
* @return
*/
public static List<IdcMetricInfo> getHostMetrics(String query, String startTime, String endTime) throws ParseException {
TimeSeriesResourceV11 timeSeriesResourceV11 = apiRoot.getTimeSeriesResource();
ApiTimeSeriesResponseList responseList = timeSeriesResourceV11.queryTimeSeries(query, startTime, endTime);
List<ApiTimeSeriesResponse> apiTimeSeriesResponseList = responseList.getResponses();
List<IdcMetricInfo> metrics = formatApiTimeSeriesResponseList(apiTimeSeriesResponseList);
return metrics;
}
public static List<IdcMetricInfo> formatApiTimeSeriesResponseList(List<ApiTimeSeriesResponse> apiTimeSeriesResponseList) throws ParseException {
List<IdcMetricInfo> metrics = new ArrayList<IdcMetricInfo>();
DateUtils dateUtils = new DateUtils();
for (ApiTimeSeriesResponse apiTimeSeriesResponse : apiTimeSeriesResponseList) {
List<MetricData> dataList = new ArrayList<MetricData>();
List<ApiTimeSeries> apiTimeSeriesResponseLists = apiTimeSeriesResponse.getTimeSeries();
for (ApiTimeSeries apiTimeSeries : apiTimeSeriesResponseLists) {
LOGGER.info("query sql is: " + apiTimeSeries.getMetadata().getExpression());
IdcMetricInfo metric = new IdcMetricInfo();
metric.setMetricName(apiTimeSeries.getMetadata().getMetricName());
metric.setEntityName(apiTimeSeries.getMetadata().getEntityName());
metric.setStartTime(apiTimeSeries.getMetadata().getStartTime().toString());
metric.setEndTime(apiTimeSeries.getMetadata().getEndTime().toString());
for (ApiTimeSeriesData apiTimeSeriesData : apiTimeSeries.getData()) {
MetricData data = new MetricData();
// Insert EntityName into the Data to avoid duplicate data generation
data.seHostname(apiTimeSeries.getMetadata().getEntityName());
// CM defaults to EEE MMM DD HH:mm:ss'CST'yyyy and converts to yyyy-MM-dd HH:mm:ss
data.setTimestamp(dateUtils.parse(apiTimeSeriesData.getTimestamp().toString()));
data.setType(apiTimeSeriesData.getType());
data.setValue(apiTimeSeriesData.getValue());
dataList.add(data);
}
metric.setData(dataList);
metrics.add(metric);
}
}
return metrics;
}Be careful:
The DataUtils involved in the code need to be implemented on their own
This part of the code can get metric s information of the corresponding idc cluster by passing in tsquery; the next code only needs to get the code of the corresponding monitoring indicator through ServiceImpl.
If you want to integrate with spring boot through cm api, you will also encounter two problems:
Dependency conflict, mainly in the conflict between jackson and cxf; can be resolved by excluding jar packages
Regular parsing error, which is a pit in cm usage, is still under investigation as follows:
There is a space here, so regular parsing errors will be reported directly during compilation; however, we can see that this problem is no longer present in the api version of cm 6.x:
It is possible to upgrade the version of the api directly to solve this problem, but the problem is that it is inconsistent with the version of cm running online (online version 5.13.2), so it still needs to be considered how to solve this problem; however, after testing, it is found that using the cm 6.x version of the api refers to the version of the current online version.Labels do not affect