https://apify.com/ logo
Join Discord
Powered by
How can I override the default logs of Crawlee?
# crawlee-js
w
Hello I wonder how to override the default logs of crawler, this is how it looks: This logs came from basic-crawle library: (https://github.com/apify/crawlee/blob/3ffcf56d744ac527ed8d883be3b1a62356a5930c/packages/basic-crawler/src/internals/basic-crawler.ts#L891) I am using Playwright, and thats how I mange to override default logs with my custom like that:
Copy code
typescript
//playwright-winston-proxy-logger.ts
import { Log } from 'crawlee'

import type { Logger } from 'winston'

type AdditionalData = Record<string, unknown> | null

export class WinstonLoggerProxy extends Log {
  private logger: Logger

  constructor(logger: Logger) {
    super()
    this.logger = logger
  }

  debug(message: string, data?: AdditionalData): void {
    if (data) {
      this.logger.debug(message, data)
    } else {
      this.logger.debug(message)
    }
  }

  info(message: string, data?: AdditionalData): void {
    if (data) {
      this.logger.info(message, data)
    } else {
      this.logger.info(message)
    }
  }

  warning(message: string, data?: AdditionalData): void {
    if (data) {
      this.logger.warn(message, data)
    } else {
      this.logger.warn(message)
    }
  }

  error(message: string, data?: AdditionalData): void {
    if (data) {
      this.logger.error(message, data)
    } else {
      this.logger.error(message)
    }
  }

  exception(exception: Error, message: string, data?: AdditionalData): void {
    if (data) {
      this.logger.error(message, { exception, ...data })
    } else {
      this.logger.error(message, { exception })
    }
  }
}
and thats how I use them:
Copy code
typescript

...
  private createCrawler = (): PlaywrightCrawler => {
    const loggerCrawler = new WinstonLoggerProxy(
      createLogger({ module: 'PLAYWRIGHT' })
    )

    return new PlaywrightCrawler({
      log: loggerCrawler, // Provide the custom logger proxy
...
https://cdn.discordapp.com/attachments/1278413559398924288/1278413559662903306/image.png?ex=66d0b6e8&is=66cf6568&hm=f78b61adda4c474009b528824e713f837f1009d3a5bd2640d8f6aede5d908ff8&
h
message has been deleted
w
Solution: The solution is to add 
internal
method to your proxy class, here is full code for proxy 
winston
-> 
crawlee
log
Copy code
typescript
import { Log, LogLevel } from 'crawlee'

import type { Logger } from 'winston'

type AdditionalData = Record<string, unknown> | null

export class WinstonLoggerProxy extends Log {
  private logger: Logger

  constructor(logger: Logger) {
    super()
    this.logger = logger
  }

  debug(message: string, data?: AdditionalData): void {
    if (data) {
      this.logger.debug(message, data)
    } else {
      this.logger.debug(message)
    }
  }

  info(message: string, data?: AdditionalData): void {
    if (data) {
      this.logger.info(message, data)
    } else {
      this.logger.info(message)
    }
  }

  internal(
    level: LogLevel,
    message: string,
    data?: any,
    exception?: any
  ): void {
    switch (level) {
      case LogLevel.DEBUG:
        if (data) {
          this.logger.debug(message, { data, exception })
        } else {
          this.logger.debug(message, { exception })
        }
        break
      case LogLevel.INFO:
        if (data) {
          this.logger.info(message, { data, exception })
        } else {
          this.logger.info(message, { exception })
        }
        break
    }
  }

  warning(message: string, data?: AdditionalData): void {
    if (data) {
      this.logger.warn(message, data)
    } else {
      this.logger.warn(message)
    }
  }

  error(message: string, data?: AdditionalData): void {
    if (data) {
      this.logger.error(message, data)
    } else {
      this.logger.error(message)
    }
  }

  exception(exception: Error, message: string, data?: AdditionalData): void {
    if (data) {
      this.logger.error(message, { exception, ...data })
    } else {
      this.logger.error(message, { exception })
    }
  }
}
I manage to override some logs, but still struggling to make statistic work as well
I manage to do this by overriding 
child
method and passing 
data
object to my 
Winston
Logger, but what concern me are those 3 repeating logs
Copy code
[PLAYWRIGHT] [ #1 ] [2024-08-29 13:50:00] INFO : Attempting to run a task.
[PLAYWRIGHT] [ #1 ] [2024-08-29 13:50:00] INFO : Checking for ready tasks. 
[PLAYWRIGHT] [ #1 ] [2024-08-29 13:50:00] INFO : Task will not run. No tasks are ready.
I don't have anything like that in my code, its from library: `Attempting to run a task.`: https://github.com/apify/crawlee/blob/c69a34a616feda0824c88f9ec18871bff0b212c0/packages/core/src/autoscaling/autoscaled_pool.ts#L473 `Checking for ready tasks.`: https://github.com/apify/crawlee/blob/c69a34a616feda0824c88f9ec18871bff0b212c0/packages/core/src/autoscaling/autoscaled_pool.ts#L504 
Task will not run. No tasks are ready.
: https://github.com/apify/crawlee/blob/c69a34a616feda0824c88f9ec18871bff0b212c0/packages/core/src/autoscaling/autoscaled_pool.ts#L519 @Marco hi I see you are a developer at Apify, maybe you know answer to my question
okey fixed, small issue with switch and handling 
PERF
log level, I was treating them as INFO, and I should treat them as DEBUG or mock logic to ignore those strings:
Copy code
typescript
import { Log, LogLevel } from 'crawlee'

import type { LoggerOptions } from 'crawlee'
import type { Logger } from 'winston'

type AdditionalData = Record<string, unknown> | null

export class WinstonLoggerProxy extends Log {
  private logger: Logger

  constructor(logger: Logger, options?: Partial<LoggerOptions>) {
    super(options)
    this.logger = logger
  }

  getLevel(): number {
    return super.getLevel()
  }

  setLevel(level: LogLevel): void {
    super.setLevel(level)
  }

  getOptions(): Required<LoggerOptions> {
    return super.getOptions()
  }

  setOptions(options: Partial<LoggerOptions>): void {
    super.setOptions(options)
  }

  child(options: Partial<LoggerOptions>): WinstonLoggerProxy {
    const childLogger = this.logger.child(options)

    return new WinstonLoggerProxy(childLogger, {
      ...this.getOptions(),
      ...options,
    })
  }

  internal(
    level: LogLevel,
    message: string,
    data?: any,
    exception?: any
  ): void {
    const logMethod = this.getLogMethod(level)
    if (data || exception) {
      this.logger[logMethod](message, { data, exception })
    } else {
      this.logger[logMethod](message)
    }
  }
...

  private getLogMethod(level: LogLevel): keyof Logger {
    switch (level) {
      case LogLevel.DEBUG:
      case LogLevel.PERF: //this ifxed
        return 'debug'
      case LogLevel.INFO:
        return 'info'
      case LogLevel.WARNING:
      case LogLevel.SOFT_FAIL:
        return 'warn'
      case LogLevel.ERROR:
        return 'error'
      default:
        return 'info'
    }
  }
}
a
@Wojciech just advanced to level 3! Thanks for your contributions! 🎉
Next
Powered by