java日志到ES索引的字段细化
目录
二、怎么办
1、JSON Message
2、Log Parser
3、JSON Layout
三、推荐用法
一、是什么
java日志收集到ES索引,由message字段对应一整行日志 拆分为多个字段对应一行日志的过程叫做java日志到ES索引的字段细化。
字段细化前
message一个字段对应一整行日志
字段细化后
使用action,appId,elapsed,level,logKey,loggerName等多个字段对应一行日志
字段分类说明。
引擎字段表示通过log4j或logback配置而来的。
线程字段表示在一个线程内记录多条日志,这些字段的值应该是一致的。
行字段表示一个线程内记录多条日志,这些字段的值不一定是一样的。
二、怎么办
1、JSON Message
原理
特点
日志记录过程中实现
不能保证所有的日志都是json格式
会有两种日志文件 普通+json
案例
1:定义一个logModel对象
public class LogModel implements Serializable {
/**
* 应用名称
*/
private String app;
/**
* 请求链路traceID
*/
private String traceId;
/**
* 执行时长
*/
private int elapse;
/**
* 日志来源,自定义 default、es、redis、scheduler、mq、db、等等
*/
private String source = DefaultDataEnums.Source.DEFAULT.getStatus();
/**
* 执行动作 add、del、update、query
*/
private String action;
......
}
2:定义LogUtil类
@Slf4j
public class XLogUtil {
public static void info(String message,String app,...){
LogModel logModel = new LogModel();
logModel.setApp(app);
......
log.info(JsonUtil.toJson(logModel))
}
......
}
3:修改日志配置文件,以log4j2为例
<?xml version="1.0" encoding="UTF-8"?>
<configuration >
<RollingRandomAccessFile name="jsonLogAsyncAppender"
fileName="json.log" ......>
<PatternLayout>
<Pattern>%m%n</Pattern>
</PatternLayout>
</RollingRandomAccessFile>
<!--分两种日志,json日志和普通日志-->
<loggers>
<AsyncLogger name="LogUtil" level="info" additivity="false">
<appender-ref ref="jsonLogAsyncAppender"/>
</AsyncLogger>
<AsyncRoot level="info">
<appender-ref ref="Console"/>
<appender-ref ref="stringLogAsyncAppender"/>
</AsyncRoot>
</loggers>
</configuration>
2、Log Parser
原理
特点
日志收集过程中实现
处理各种非结构化数据
3、JSON Layout
原理
特点
日志记录过程中实现
log4j2、logback内核支持
引擎字段不用关心、线程字段统一处理、行字段单独处理
案例
1:增加LoggerWrapper类实现日志记录前后业务处理
@Data
public class LoggerWrapper implements Logger {
protected Logger log;
public LoggerWrapper(Logger log) {
this.log = log;
}
protected void prepare() { }
protected void finish() { }
@Override
public String getName() {
return log.getName();
}
@Override
public boolean isInfoEnabled() {
return false;
}
@Override
public void info(String msg) {
try {
prepare();
log.info(msg);
} finally {
finish();
}
}
@Override
public void info(String format, Object arg) {
try {
prepare();
log.info(format, arg);
} finally {
finish();
}
}
//......覆盖所有方法
}
2:定义LogProxy存放行字段(也可以是线程字段)并继承LoggerWrapper
@Data
public class LogProxy extends LoggerWrapper {
private String traceId;
private String userId;
private String app;
private Object request;
private Object response;
private String source;
private String action;
private Long elapsed = 0L;
public LogProxy(Logger log) {
super(log);
}
@Override
protected void prepare() {
if (log == null) {
throw new RuntimeException("log 不能为空");
}
if (traceId != null) {
MDC.put("traceId", traceId);
}
if (userId != null) {
MDC.put("userId", userId);
}
if (app != null) {
MDC.put("appId", appId);
}
......
}
@Override
protected void finish() {
if (traceId != null) {
MDC.remove("traceId");
}
if (userId != null) {
MDC.remove("userId");
}
if (app != null) {
MDC.remove("appId");
}
......
}
public LogProxy app(String app) {
this.app = app;
return this;
}
public LogProxy request(Object request) {
this.request = request;
return this;
}
......
}
3:定义LogUtil类
@Slf4j
public class LogUtil {
public static LogProxy log(Logger log) {
LogProxy logProxy = new LogProxy(log);
return logProxy;
}
}
4:拦截器统一处理线程字段
public class LogInterceptor extends HandlerInterceptorAdapter {
@Override
public boolean preHandle(HttpServletRequest request, HttpServletResponse response, Object handler) throws Exception {
MDC.put("traceId", traceId);
MDC.put("userId", userId);
return true;
}
@Override
public void afterCompletion(HttpServletRequest request, HttpServletResponse response, Object handler, Exception ex) throws Exception {
MDC.clear();
}
}
5:xml配置(以log4j2为例)
<?xml version="1.0" encoding="UTF-8"?>
<configuration status="WARN" >
<RollingRandomAccessFile name="jsonLogAsyncAppender"
immediateFlush="false" append="true" ......>
<JsonLayout locationInfo="true" properties="false">
<KeyValuePair key="traceId" value="$${grace:${ctx:trace}}"/>
<KeyValuePair key="app" value="$${grace:${ctx:app}}"/>
<KeyValuePair key="userId" value="$${grace:${ctx:userId}}"/>
<KeyValuePair key="source" value="$${grace:${ctx:source}}"/>
<KeyValuePair key="action" value="$${grace:${ctx:action}}"/>
<KeyValuePair key="request" value="$${grace:${ctx:request}}"/>
<KeyValuePair key="response" value="$${grace:${ctx:response}}"/>
......
<Compact>true</Compact>
<EventEol>true</EventEol>
<StacktraceAsString>true</StacktraceAsString>
<Charset>UTF-8</Charset>
</JsonLayout>
.....
</RollingRandomAccessFile>
</configuration>
6:日志记录方式
log.info("message........");
LogUtil.log(log).source("mysql").action("query").....info("message....")