写一个简单的异常告警

叔牙

发布于 2023-09-07 09:39:12

1850

发布于 2023-09-07 09:39:12

文章被收录于专栏：一个执拗的后端搬砖工一个执拗的后端搬砖工

一、背景

在一些中小型团队,没有完善的监控告警平台,为了保证线上服务运行状况不是黑盒状态,我们需要手动写一些简单的基础工具,比如接口监控告警等能力,当然就算有监控告警平台,有时候也需要手动写一些告警工具,来支持一些自定义或者个性化的告警能力。

二、实现方案

通过拦截器或者切面,拦截服务接口
如果接口抛出异常,则拦截器或者切面捕获异常,并组装告警消息
拦截器或者切面调用办公协同平台的api发送告警消息,办公协同平台将告警消息推送到对应的告警群

三、编写告警组件

既然是告警组件,也就意味着要提供一个通用能力供业务使用,此处我们也写成一个starter组件,原理就是写一个自定义注解,和手动告警工具通过jar包的形式暴露出去。

1.编写自定义告警注解

告警注解:

@Target({ElementType.METHOD})
@Retention(RetentionPolicy.RUNTIME)
public @interface Alarm {


    /**
     * 抛出该异常集合里面的异常时，进行告警
     */
    Class<? extends Throwable>[] ex() default {};
    /**
     * 异常名字正则匹配
     */
    boolean allEX() default false;
    /**
     * 返回值转字符串后与当前值相等时，进行告警
     */
    String rValue() default "";
    /**
     * 告警内容
     */
    String aContent() default "";
    /**
     * 告警信息是否携带方法参数
     */
    boolean wParam() default false;
    /**
     * 告警信息方法参数字符长度
     */
    int pContentLen() default 600;
}

该注解定义告警的异常类型、告警内容等相关信息。

开启告警注解:

@Target({ElementType.TYPE})
@Retention(RetentionPolicy.RUNTIME)
@Import({AlarmAutoConfiguration.class, AlarmAspect.class})
public @interface EnableAlarm {
}

通过该注解开启告警能力,并导入了支撑告警能力的相关配置类和切面。

2.编写告警工具

编写一个支持多渠道发送告警的工具:

@Slf4j
public class AlarmUtil {
    /**
     * 服务端告警通知
     *
     * <description>
     *     <ul>
     *         <li>1.飞书告警</li>
     *         <li>2.dingTalk告警</li>
     *         <li>3.企业微信告警</li>
     *     </ul>
     * </description>
     *
     * @param param
     */
    public static final void report(AlarmParam param) {
        if(null == param
                || null == param.getAlarmType()
                || null == param.getWebhookUrl()) {
            log.warn("AlarmUtil.report param illegal,can't trigger report;param={}",param);
            return;
        }
        if(AlarmType.fs.getCode().equals(param.getAlarmType())) {
            reportFs(param);
        } else if(AlarmType.dingTalk.getCode().equals(param.getAlarmType())) {
            //todo
        } else if(AlarmType.wechat.getCode().equals(param.getAlarmType())) {
            //todo
        }
    }


    public static final void reportFs(AlarmParam param) {
        String webhookUrl = param.getWebhookUrl();
        //sendFsAlarm(webhookUrl,param.getTitle(), param.getE(), );
        String host = getLocalhost();
        String title = param.getTitle();
        Map<String,String> paramMap = new HashMap<>();
        paramMap.put("host",host);
        paramMap.put("exceptionName",title);
        paramMap.put("description",param.getDescription());
        paramMap.put("currentEnv",param.getCurrentEnv());
        paramMap.put("params",param.getWithParam());
        paramMap.put("applicationName",param.getApplicationName());
        StrSubstitutor substitute = new StrSubstitutor(paramMap);
        String msg = substitute.replace(fsAlarmTemplate);
        try {
            HttpClientUtil.sendPostRequest(webhookUrl,null,msg);
        } catch (HttpRequestException e) {
            log.error("send feishu alarm occur error;param={}",param,e);
        }
    }
}

3.编写告警配置和切面

写一个拦截自定义告警注解的切面,提供告警能力:

@Slf4j
public class AlarmAspect implements MethodInterceptor, EnvironmentAware {


    @Autowired
    private AlarmConfig alarmConfig;
    @Value("${spring.application.name:}")
    private String applicationName;
    protected CurrentEnv currentEnv;


    @Override
    public Object invoke(MethodInvocation methodInvocation) throws Throwable {
        Object result = null;
        Throwable ex = null;
        try {
            result = methodInvocation.proceed();
        } catch (Throwable e) {
            ex = e;
            throw e;
        } finally {
            //满足线上环境 或者 告警开关开启,都做告警处理
            if(Boolean.TRUE.equals(alarmConfig.getOpenAlarm())
                    || CurrentEnv.isProd(currentEnv)) {
                handleEX(methodInvocation, ex, result);
            }
        }
        return result;
    }


    private void handleEX(MethodInvocation methodInvocation, Throwable ex, Object result) {
        Alarm alarm = methodInvocation.getMethod().getAnnotation(Alarm.class);
        if (null == alarm) {
            pointcutAlarm(methodInvocation, ex);
            return;
        }
        String content = alarm.aContent();
        String returnValue = alarm.rValue();
        Class<? extends Throwable>[] exs = alarm.ex();
        if (StringUtils.isBlank(content) && ex == null) {
            log.debug("alarm content is empty, skip alarm");
            return;
        }
        if (ex != null) {
            if (alarm.allEX()) {
                annoAlarm(methodInvocation, ex);
                return;
            }
            for (Class<? extends Throwable> aClass : exs) {
                if (!ex.getClass().isAssignableFrom(aClass)) {
                    continue;
                }
                annoAlarm(methodInvocation, ex);
                break;
            }
        }
        if (result != null && StringUtils.equals(returnValue, result.toString())) {
            annoAlarm(methodInvocation, null);
        }
    }


    private void annoAlarm(MethodInvocation methodInvocation, Throwable ex) {
        Alarm alarm = methodInvocation.getMethod().getAnnotation(Alarm.class);
        Object[] arguments = methodInvocation.getArguments();
        doAlarm(alarm.aContent(), arguments, ex, alarm.wParam(), alarm.pContentLen());
    }
    private void pointcutAlarm(MethodInvocation methodInvocation, Throwable ex) {
        if (ex == null) {
            return;
        }
        if (StringUtils.isBlank(alarmConfig.getNoAlarmEX())){
            doAlarm(null, methodInvocation.getArguments(), ex, true, alarmConfig.getPContentLen());
        }
        ArrayList<String> noAlarmExs = Lists.newArrayList(alarmConfig.getNoAlarmEX().split(","));
        boolean alarm = true;
        for (String noAlarmEx : noAlarmExs) {
            try {
                Class<?> aClass = Class.forName(noAlarmEx);
                if(aClass.isAssignableFrom(ex.getClass())) {
                    alarm = false;
                    break;
                }
            } catch (ClassNotFoundException ignored) {
            }
        }
        if (alarm) {
            doAlarm(null, methodInvocation.getArguments(), ex, true, alarmConfig.getPContentLen());
        }
    }
    private void doAlarm(String content, Object[] arguments, Throwable ex, boolean wParam, int pContentLen) {
        StringBuilder stringBuilder = new StringBuilder();
        String pContent = null;
        if (wParam && ArrayUtils.isNotEmpty(arguments)) {
            pContent = transferArgs2String(arguments);
            if (pContent.length() > pContentLen) {
                pContent = pContent.substring(0, pContentLen);
            }
        }
        if (null != ex) {
            stringBuilder.append(ExceptionHelper.printStackTrace(ex,10));
        }
        AlarmParam alarmParam = new AlarmParam();
        alarmParam.setCurrentEnv(null != currentEnv ? currentEnv.getDesc() : null);
        alarmParam.setTitle(content);
        alarmParam.setWithParam(pContent);
        alarmParam.setAlarmType(alarmConfig.getAlarmType());
        alarmParam.setWebhookUrl(alarmConfig.getWebhookUrl());
        alarmParam.setApplicationName(this.applicationName);
        alarmParam.setDescription(stringBuilder.toString());
        AlarmUtil.report(alarmParam);
    }
    private String transferArgs2String(Object[] arguments) {
        if(null == arguments || arguments.length <= 0) {
            return null;
        }
        List<Object> args = Arrays.stream(arguments).filter(item -> !(item instanceof HttpServletRequest || item instanceof HttpServletResponse)).collect(Collectors.toList());
        return ListUtil.toPlaintString(args);
    }
    @Override
    public void setEnvironment(Environment environment) {
        String env = environment.getActiveProfiles()[0];
        log.info("AlarmAspect.setEnvironment env = {}",env);
        if (null == env) {
            log.error("AlarmAspect.setEnvironment environment.getActiveProfiles()[0] is null");
            return;
        }
        this.currentEnv = CurrentEnv.of(env.toLowerCase());
    }
}

提供了根据开关和服务部署环境开启告警能力。

编写自动配置类:

@Configuration
@EnableConfigurationProperties({AlarmConfig.class})
@Slf4j
public class AlarmAutoConfiguration {


    @Bean
    @Role(BeanDefinition.ROLE_INFRASTRUCTURE)
    public AspectJExpressionPointcutAdvisor alarmAspectAdvisor(AlarmAspect alarmAspect,AlarmConfig alarmConfig) {
        String pointCut = "@annotation(xxx.alarm.annotation.Alarm)";
        if (StringUtils.isNotBlank(alarmConfig.getPointCut())) {
            pointCut = "execution(* " + alarmConfig.getPointCut() + ")" + "||" + pointCut;
        }
        AspectJExpressionPointcutAdvisor advisor = new AspectJExpressionPointcutAdvisor();
        advisor.setOrder(Ordered.HIGHEST_PRECEDENCE);
        advisor.setExpression(pointCut);
        advisor.setAdvice(alarmAspect);
        return advisor;
    }


    @Bean
    @Role(BeanDefinition.ROLE_INFRASTRUCTURE)
    public AlarmAspect alarmAspect() {
        log.info("AlarmAutoConfiguration.alarmAspect init alarmAspect ...");
        AlarmAspect alarmAspect = new AlarmAspect();
        return alarmAspect;
    }


}

这样将组件打包后,业务中引入依赖就能使用相关能力了。组件功能结构大致如下:

四、业务引用告警

业务服务使用告警能力,需要将依赖引进来,然后在接口上使用自定义注解,或者在业务中捕获异常后手动发送告警。

1.引入告警组件

在业务项目中引入告警组件:

<dependency>
    <groupId>com.xxx.common</groupId>
    <artifactId>alarm</artifactId>
</dependency>

2.定义告警相关配置

在项目配置文件中添加告警用到的相关属性:

alarm:
  openAlarm: true
  alarmType: 1
  webhookUrl: https://open.feishu.cn/open-apis/bot/v2/hook/xxxx

3.引用告警注解

在业务接口上添加@Alarm注解,并填入基本的告警信息。

@PostMapping("/xxx-api")
@Alarm(aContent = "xxxApi调用异常", wParam = true, ex = {xxxException.class})
public CommonResult<> api(HttpServletRequest request, @RequestBody ApiReq req) {
    return service.callBuzz(req);
}

4.使用自定义告警

前边是基于注解的方式使用告警能力,有些时候我们在处理一些非接口调用业务的时候,也需要关注是否执行成功了,如果执行失败可以手动调用告警工具发送告警。

我们可以在告警组件告警工具添加自定义告警实现:

public static final void reportCustom(AlarmType alarmType,String webhookUrl, String title, Throwable ex,Object... arguments) {
    CommonConfig commonConfig = SpringContextUtil.getBean(CommonConfig.class);
    if(null == commonConfig) {
        log.warn("reportCustom commonConfig is null,abort alarm;content={}",title);
        return;
    }
    webhookUrl = null == webhookUrl ? commonConfig.getWebhookUrl() : webhookUrl;
    String pContent = null;
    if (ArrayUtils.isNotEmpty(arguments)) {
        pContent = Arrays.toString(arguments);
        if (pContent.length() > 500) {
            pContent = pContent.substring(0, 500);
        }
    }
    if(AlarmType.fs.equals(alarmType)) {
        sendFsAlarm(webhookUrl, title, ex, commonConfig, pContent);
    } else if(AlarmType.dingTalk.equals(alarmType)) {
        //todo
    } else if(AlarmType.wechat.equals(alarmType)) {
        //todo
    }
}
private static void sendFsAlarm(String webhookUrl, String title, Throwable ex, CommonConfig commonConfig, String pContent) {
    String host = getLocalhost();
    Map<String,String> paramMap = new HashMap<>();
    paramMap.put("host",host);
    paramMap.put("exceptionName", title);
    paramMap.put("description",ExceptionHelper.printStackTrace(ex,5));
    paramMap.put("currentEnv", commonConfig.getCurrentEnv());
    paramMap.put("applicationName", commonConfig.getApplicationName());
    paramMap.put("params", pContent);
    StrSubstitutor substitute = new StrSubstitutor(paramMap);
    String msg = substitute.replace(fsAlarmTemplate);
    try {
        HttpClientUtil.sendPostRequest(webhookUrl,null,msg);
    } catch (HttpRequestException e) {
        log.error("send custom alarm occur error;param={}",paramMap,e);
    }
}

这样我们就可以在捕获异常的地方手动发送告警信息了,使用如下:

try {
    result = HttpUtil.sendPostRequest(url,reqJson);
} catch (HttpRequestException e) {
    log.error("remote call failed;req={}",reqJson,e);
    AlarmUtil.reportCustom(AlarmType.fs,"webhookUrl",e,url,reqJson);
    throw new CommonRuntimeException("something occur err",e);
}

本文参与腾讯云自媒体分享计划，分享自微信公众号。

原始发表：2023-08-02，如有侵权请联系 cloudcommunity@tencent.com 删除

服务