Log link tracking practice sharing based on Egg framework

Log link tracking practice sharing based on Egg framework

Quick navigation

Demand background

Realize full-link log tracking, which is convenient for log monitoring, troubleshooting, interface response time-consuming data statistics, etc. 1. the API interface service receives the caller's request, and according to the traceId passed by the caller, when the business is processed in the call chain, such as If the log needs to be printed, the log information is printed in accordance with the agreed specification, and the traceId is recorded to realize log link tracking.

  • Log path convention
/var/logs/${projectName}/bizLog/${projectName}-yyyyMMdd.log
 
  • Log format convention
 []traceId[] IP[] IP[] [] 
 

Using Egg.js framework egg-logger middleware, during the implementation process, it was found that printing in the above log format was not enough (at least I have not found a way to achieve it). If you want to implement it yourself, you may have to make your own wheels. Fortunately, the official egg-logger middleware provides custom log extension, reference advanced custom log , the log itself also provides segmentation, multi-process log processing and other functions.

egg-logger provides multiple transmission channels. Our requirement is mainly to store the requested business logs in a custom format. Two channels, fileTransport and consoleTransport, are mainly used to print logs to files and terminals respectively.

Custom log plugin development

Egg-logger custom development based on a plug-in project, refer to plug-in development , with the following egg-logger-custom project, write code to show the core

  • Write logger.js

egg-logger-custom/lib/logger.js

const moment = require('moment');
const FileTransport = require('egg-logger').FileTransport;
const utils = require('./utils');
const util = require('util');

/**
 *   FileTransport
 */
class AppTransport extends FileTransport {
    constructor(options, ctx) {
        super(options);

        this.ctx = ctx; // 
    }

    log(level, args, meta) {
        // 
        const customMsg = this.messageFormat({
            level,
        });

        //  Error  
        if (args[0] instanceof Error) {
            const err = args[0] || {};
            args[0] = util.format('%s: %s\n%s\npid: %s\n', err.name, err.message, err.stack, process.pid);
        } else {
            args[0] = util.format(customMsg, args[0]);
        }

        // 
        super.log(level, args, meta);
    }

    /**
     *  
     *  
     * @param { String } level
     */
    messageFormat({
        level
    }) {
        const { ctx } = this;
        const params = JSON.stringify(Object.assign({}, ctx.request.query, ctx.body));

        return [
            moment().format('YYYY/MM/DD HH:mm:ss'),
            ctx.request.get('traceId'),
            utils.serviceIPAddress,
            utils.clientIPAddress(ctx.req),
            level,
        ].join(utils.loggerDelimiter) + utils.loggerDelimiter;
    }
}

module.exports = AppTransport;
 
  • tool

egg-logger-custom/lib/utils.js

const interfaces = require('os').networkInterfaces();

module.exports = {

    /**
     *  
     */
    loggerDelimiter: '[]',

    /**
     *  IP
     */
    serviceIPAddress: (() => {
        for (const devName in interfaces) {
            const iface = interfaces[devName];

            for (let i = 0; i < iface.length; i++) {
                const alias = iface[i];

                if (alias.family === 'IPv4' && alias.address !== '127.0.0.1' && !alias.internal) {
                    return alias.address;
                }
            }
        }
    })(),

    /**
     *  IP
     *  
     */
    clientIPAddress: req => {
        const address = req.headers['x-forwarded-for'] || //  IP
        req.connection.remoteAddress || //  connection   IP
        req.socket.remoteAddress || //  socket   IP
        req.connection.socket.remoteAddress;

        return address.replace(/::ffff:/ig, '');
    },

    clientIPAddress: ctx => {    
        return ctx.ip;
    },
}
 

Note : The above acquisition mode current IP of the requesting client, if you need to make the user's IP limiting, anti-brush restrictions, please do not use the above manner, see Popular Science article: How to get the user real and fake IP? In Egg.js where you can also be obtained by ctx.ip, with reference to pre-proxy mode .

  • Initialize Logger
egg-logger-custom/app.js
 
const Logger = require('egg-logger').Logger;
const ConsoleTransport = require('egg-logger').ConsoleTransport;
const AppTransport = require('./app/logger');

module.exports = (ctx, options) => {
    const logger = new Logger();

    logger.set('file', new AppTransport({
        level: options.fileLoggerLevel || 'INFO',
        file: `/var/logs/${options.appName}/bizLog/${options.appName}.log`,
    }, ctx));

    logger.set('console', new ConsoleTransport({
        level: options.consoleLevel || 'INFO',
    }));

    return logger;
}
 

The above is good for the development of the log customization format. If you have actual business needs, you can encapsulate it as a npm middleware inside the team to use according to the needs of your team.

Project extension

Since the definition of good logs middleware package, we also need to step in the actual operation of the project application, Egg provides the framework for extended functions, including five: Application, Context, Request, Response , Helper, you can customize this extension several For the log, because we need to record the traceId carried in the current request to do a link tracing every time we log, we need to use the Context (the request context of Koa) extension.

New app/extend/context.jsfile

const AppLogger = require('egg-logger-custom'); // 

module.exports = {
    get logger() { //    customLogger
        return AppLogger(this, {
            appName: 'test', // 
            consoleLevel: 'DEBUG', // 
            fileLoggerLevel: 'DEBUG', // 
        });
    }
}
 

Suggestion : For the log level, you can use a configuration center such as Consul to configure. The log level is set to INFO when you go online. When you need to troubleshoot production problems, you can dynamically turn on the DEBUG mode. Consul can focus on my writing before service registration discovery Consul series

Project application

Error log records, directly record the complete stack information of the error log, and output it to the errorLog. In order to ensure that the exception can be traced, it must be ensured that all exceptions thrown are of the Error type, because only the Error type will carry the stack information. Locate the problem.

const Controller = require('egg').Controller;

class ExampleController extends Controller {
    async list() {
        const { ctx } = this;

        ctx.logger.error(new Error(' '));

        ctx.logger.debug(' ');

        ctx.logger.info(' ');
    }
}
 

The final log print format is as follows:

2019/05/30 01:50:21[]d373c38a-344b-4b36-b931-1e8981aef14f[]192.168.1.20[]221.69.245.153[]INFO[] 
 

contextFormatter custom log format

The latest version of Egg-Logger supports customizing the log format through the contextFormatter function, please refer to the previous PR: support contextFormatter #51

The application is also very simple, by configuring the contextFormatter function, the following is a simple application

config.logger = {
    contextFormatter: function(meta) {
        console.log(meta);
        return [
            meta.date,
            meta.message
        ].join('[]')
    },
    ...
};
 

The same in your business where you need to print logs, the same as before

ctx.logger.info(' ');
 

The output is as follows:

2019-06-04 12:20:10,421[] 
 

Log cutting

The framework provides egg-logrotator middleware, the default cutting is cut by day, and other methods can be configured by referring to the official website.

  • Framework default log path

egg-logger module lib/egg/config/config.default.js

config.logger = {
    dir: path.join(appInfo.root, 'logs', appInfo.name),
    ...
};
 
  • Custom log directory

It is very simple to redefine the dir path of the logger in the project configuration file according to our needs

config.logger = {
    dir: /var/logs/test/bizLog/
}
 

Is this all right? According to our custom log file name format ( ${projectName}-yyyyMMdd.log) above, it seems that it is not working. The default file name format during the log segmentation process is .log.YYYY-MM-DD, refer to the source code

github.com/eggjs/egg-l...

 _setFile(srcPath, files) {
    //don't rotate logPath in filesRotateBySize
    if (this.filesRotateBySize.indexOf(srcPath) > -1) {
      return;
    }

    //don't rotate logPath in filesRotateByHour
    if (this.filesRotateByHour.indexOf(srcPath) > -1) {
      return;
    }

    if (!files.has(srcPath)) {
      //allow 2 minutes deviation
      const targetPath = srcPath + moment()
        .subtract(23, 'hours')
        .subtract(58, 'minutes')
        .format('.YYYY-MM-DD'); // 
      debug('set file %s => %s', srcPath, targetPath);
      files.set(srcPath, { srcPath, targetPath });
    }
 }
 
  • Log split extension

The middleware egg-logrotator reserves an extended interface. For custom log file names, you can use the app.LogRotator provided by the framework to make a customization.

app/schedule/custom.js

const moment = require('moment');

module.exports = app => {
    const rotator = getRotator(app);

    return {
        schedule: {
            type: 'worker', //only one worker run this task
            cron: '1 0 0 * * *', //run every day at 00:00
        },
        async task() {
            await rotator.rotate();
        }
    };
};

function getRotator(app) {
    class CustomRotator extends app.LogRotator {
        async getRotateFiles() {
            const files = new Map();
            const srcPath = `/var/logs/test/bizLog/test.log`;
            const targetPath = `/var/logs/test/bizLog/test-${moment().subtract(1, 'days').format('YYYY-MM-DD')}.log`;
            files.set(srcPath, { srcPath, targetPath });
            return files;
        }
    }

    return new CustomRotator({ app });
}
 

After splitting, the file is displayed as follows:

$ ls -lh/var/logs/test/bizLog/
total 188K
-rw-r--r-- 1 root root 135K Jun  1 11:00 test-2019-06-01.log
-rw-r--r-- 1 root root  912 Jun  2 09:44 test-2019-06-02.log
-rw-r--r-- 1 root root  40K Jun  3 11:49 test.log
 

Extension : Based on the above log format, ELK can be used for log collection, analysis, and retrieval.

Author: May Jun
Links: www.imooc.com/article/287...
Source: Mu class network

Read recommendations

  • Focus on the Nodejs server technology stack: www.nodejs.red
  • Public number: Nodejs technology stack