公司应用es有一段时间,今天接触了一个相对复杂的业务,针对每隔几分钟,几小时,几天进行统计折线图,具体逻辑如下图:
如图,系统必须要支持查询,每小时(每隔10分钟),每日(每隔4小时统计),每周(每隔1日),每月(每隔5日)进行统计,找到最大值显示到折线图上。
首先4张图像使用term聚合,每张图像上有两条线,表示cpu和内存,也属于term聚合方式,整个折线图采用dateHistogram聚合方式。
使用语句如下:
执行结果:
- GET /system-audit1/auditEvent/_search
- {
- "aggs": {
- "sales": {
- "terms": {
- "field": "psName.keyword"
- },"aggs": {
- "type": {
- "terms": {
- "field": "type.keyword"
- },"aggs": {
- "staticTime": {
- "date_histogram": {
- "field": "statisticTime","interval": "4h"
- },"aggs": {
- "maxValue": {
- "max": {
- "field": "value"
- }
- }
- }
- }
- }
- }
- }
- }
- }
- }
- "aggregations": {
- "sales": {
- "doc_count_error_upper_bound": 0,"sum_other_doc_count": 0,"buckets": [
- {
- "key": "192.168.1.241:es","doc_count": 7516,"type": {
- "doc_count_error_upper_bound": 0,"buckets": [
- {
- "key": "cpu","doc_count": 3763,"staticTime": {
- "buckets": [
- {
- "key_as_string": "2018-01-05T16:00:00.000Z","key": 1515168000000,"doc_count": 2067,"maxValue": {
- "value": 23.100000381469727
- }
- },{
- "key_as_string": "2018-01-05T20:00:00.000Z","key": 1515182400000,"doc_count": 132,"maxValue": {
- "value": 22.799999237060547
- }
- },{
- "key_as_string": "2018-01-06T00:00:00.000Z","key": 1515196800000,"doc_count": 0,"maxValue": {
- "value": null
- }
- }...
java代码实现:
获得时间类型:
- List<SystemDistribution> list = new ArrayList<>(); //统计最终的数据
- BoolQueryBuilder boolQueryBuilder=QueryBuilders.boolQuery();
- boolQueryBuilder.must(QueryBuilders.rangeQuery("createTime").lte(endTime).gt(startTime)); //createTime是YYYYMMDDHHMMSSSSS格式字符串
- DateHistogramInterval dateHistogramInterval=getDateHistogramInterval(timeType); //聚合时间类型
- TermsAggregationBuilder termAggregation=AggregationBuilders.terms("psName").field("psName.keyword"); //服务器名称聚合
- TermsAggregationBuilder typeAggregation=AggregationBuilders.terms("type").field("type.keyword");
- AggregationBuilder timeAggregation =
- AggregationBuilders
- .dateHistogram("agg")
- .field("statisticTime")//统计时间聚合
- .dateHistogramInterval(dateHistogramInterval);
- MaxAggregationBuilder maxAggregation = AggregationBuilders.max("maxValue").field("value");//最大值聚合
- timeAggregation.subAggregation(maxAggregation);
- typeAggregation.subAggregation(timeAggregation);
- termAggregation.subAggregation(typeAggregation);
- SearchResponse response = client.prepareSearch(INDEX_NAME).setTypes(TYPE)
- .setQuery(boolQueryBuilder).addAggregation(termAggregation).execute().actionGet();
- Terms genders = response.getAggregations().get("psName");
- for (Terms.Bucket entry : genders.getBuckets()) {
- SystemDistribution systemDistribution=new SystemDistribution();
- String psName=entry.getKey().toString();
- systemDistribution.setHostName(psName);
- Terms typeTerm = entry.getAggregations().get("type");
- List<RiskStatisticsVo> memRiskStatistics=new ArrayList<>();
- List<RiskStatisticsVo> cpuRiskStatisTics=new ArrayList<>();
- for (Terms.Bucket entry1 : typeTerm.getBuckets()) {
- String type = entry1.getKeyAsString(); // Key as String 2017-12-27T00:00:00.000Z
- Histogram histogram=entry1.getAggregations().get("agg");
- for(Histogram.Bucket entry2 : histogram.getBuckets()){
- RiskStatisticsVo riskStatisticsVo=new RiskStatisticsVo();
- riskStatisticsVo.setRiskType(type);
- String statisTime=entry2.getKeyAsString();
- Max max=entry2.getAggregations().get("maxValue");
- Double maxValue=max.getValue();
- if(maxValue.equals(Double.NEGATIVE_INFINITY)){ //如果为无穷大,赋值为0
- maxValue=0.0;
- }//-Infinity
- riskStatisticsVo.setStatisticTime(formatReturnTime(statisTime,timeType));//2018-01-08T11:00:00.000Z
- riskStatisticsVo.setCount(maxValue.toString());
- if("mem".equals(type)){
- memRiskStatistics.add(riskStatisticsVo);
- }else{
- cpuRiskStatisTics.add(riskStatisticsVo);
- }
- }
- }
- systemDistribution.setcpuStatisticList(cpuRiskStatisTics);
- systemDistribution.setEmeStatisticList(memRiskStatistics);
- list.add(systemDistribution);
- }
- return list;
格式化返回时间:
- private DateHistogramInterval getDateHistogramInterval(String dateType) {
- if(StatisticTimeTypeEnum.HOUR.getName().equals(dateType)){
- return DateHistogramInterval.minutes(10);//统计一个小时内数据,每隔10分钟一个显示
- }else if(StatisticTimeTypeEnum.Day.getName().equals(dateType)){
- return DateHistogramInterval.hours(4); //统计每日,每隔4小时统计
- }else if(StatisticTimeTypeEnum.WEEK.getName().equals(dateType)){
- return DateHistogramInterval.days(1); //每周,统计每天的数据统计
- }else{
- return DateHistogramInterval.days(5); //每月,每隔5天一个统计数据
- }
- }
- private String formatReturnTime(String time,String dateType){
- if(StatisticTimeTypeEnum.HOUR.getName().equals(dateType)){
- return time.substring(11,16);
- }else if(StatisticTimeTypeEnum.Day.getName().equals(dateType)){
- return time.substring(8,10)+"日"+time.substring(11,13)+"时";
- }else{
- return time.substring(8,10)+"日";
- }
- }