Manipulando salidas repetidas del Shell

Alguna vez nos ha pasado que buscamos un error en un log que se repite mas de una vez pero a la vez es variable su contenido, por ejemplo si es correo el error se repite para un mismo correo o mas.

Si deseamos saber las veces totales unicas que el error a aparecido en el servidor es un poco dificil tratar de contarlos, generalmente puede solucionarse con sort y luego con vim eliminar los repetidos de esa salida. Sin embargo es muy rustico.

El comando uniq viene a solventarnos el problema.

Un ejemplo seria lo siguiente:

# grep kernel /var/log/syslog | head -20
Mar 12 08:09:50 myhostname kernel: [345422.277505] hp_wmi: Unknown event_id - 8 - 0x2
Mar 12 10:52:22 myhostname kernel: [355159.298700] CPU2: Core temperature above threshold, cpu clock throttled (total events = 3401)
Mar 12 10:52:22 myhostname kernel: [355159.298705] CPU3: Core temperature above threshold, cpu clock throttled (total events = 3401)
Mar 12 10:52:22 myhostname kernel: [355159.298710] CPU0: Package temperature above threshold, cpu clock throttled (total events = 3464)
Mar 12 10:52:22 myhostname kernel: [355159.298715] CPU1: Package temperature above threshold, cpu clock throttled (total events = 3464)
Mar 12 10:52:22 myhostname kernel: [355159.298720] CPU3: Package temperature above threshold, cpu clock throttled (total events = 3464)
Mar 12 10:52:22 myhostname kernel: [355159.298725] CPU2: Package temperature above threshold, cpu clock throttled (total events = 3464)
Mar 12 10:52:22 myhostname kernel: [355159.299719] CPU0: Package temperature/speed normal
Mar 12 10:52:22 myhostname kernel: [355159.299721] CPU1: Package temperature/speed normal
Mar 12 10:52:22 myhostname kernel: [355159.299725] CPU3: Core temperature/speed normal
Mar 12 10:52:22 myhostname kernel: [355159.299728] CPU2: Core temperature/speed normal
Mar 12 10:52:22 myhostname kernel: [355159.299731] CPU3: Package temperature/speed normal
Mar 12 10:52:22 myhostname kernel: [355159.299734] CPU2: Package temperature/speed normal
Mar 12 10:54:14 myhostname kernel: [355271.844634] [Hardware Error]: Machine check events logged
Mar 12 10:57:43 myhostname kernel: [355480.139160] CPU3: Core temperature above threshold, cpu clock throttled (total events = 3481)
Mar 12 10:57:43 myhostname kernel: [355480.139165] CPU2: Core temperature above threshold, cpu clock throttled (total events = 3481)
Mar 12 10:57:43 myhostname kernel: [355480.139170] CPU1: Package temperature above threshold, cpu clock throttled (total events = 3544)
Mar 12 10:57:43 myhostname kernel: [355480.139176] CPU0: Package temperature above threshold, cpu clock throttled (total events = 3544)
Mar 12 10:57:43 myhostname kernel: [355480.139180] CPU2: Package temperature above threshold, cpu clock throttled (total events = 3544)
Mar 12 10:57:43 myhostname kernel: [355480.139185] CPU3: Package temperature above threshold, cpu clock throttled (total events = 3544)

El problema es que los 6 primeros campos nos son inutiles por lo tanto tendremos que eliminarlos, transformando la consulta a algo mas util:

# grep kernel /var/log/syslog | awk -F"] " '{print $2}'| head -20
hp_wmi: Unknown event_id - 8 - 0x2
CPU2: Core temperature above threshold, cpu clock throttled (total events = 3401)
CPU3: Core temperature above threshold, cpu clock throttled (total events = 3401)
CPU0: Package temperature above threshold, cpu clock throttled (total events = 3464)
CPU1: Package temperature above threshold, cpu clock throttled (total events = 3464)
CPU3: Package temperature above threshold, cpu clock throttled (total events = 3464)
CPU2: Package temperature above threshold, cpu clock throttled (total events = 3464)
CPU0: Package temperature/speed normal
CPU1: Package temperature/speed normal
CPU3: Core temperature/speed normal
CPU2: Core temperature/speed normal
CPU3: Package temperature/speed normal
CPU2: Package temperature/speed normal
[Hardware Error]: Machine check events logged
CPU3: Core temperature above threshold, cpu clock throttled (total events = 3481)
CPU2: Core temperature above threshold, cpu clock throttled (total events = 3481)
CPU1: Package temperature above threshold, cpu clock throttled (total events = 3544)
CPU0: Package temperature above threshold, cpu clock throttled (total events = 3544)
CPU2: Package temperature above threshold, cpu clock throttled (total events = 3544)
CPU3: Package temperature above threshold, cpu clock throttled (total events = 3544)

Agregando el comando sort ordenamos todas las salidas y como veremos es mas sencillo encontrar unificados los problemas y nuestra tarea seria descartar los repetidos:

# grep kernel /var/log/syslog | awk -F"] " '{print $2}'| sort | head -40
CPU0: Package temperature above threshold, cpu clock throttled (total events = 12688)
CPU0: Package temperature above threshold, cpu clock throttled (total events = 12689)
CPU0: Package temperature above threshold, cpu clock throttled (total events = 29267)
CPU0: Package temperature above threshold, cpu clock throttled (total events = 3464)
CPU0: Package temperature above threshold, cpu clock throttled (total events = 3544)
CPU0: Package temperature above threshold, cpu clock throttled (total events = 4977)
CPU0: Package temperature above threshold, cpu clock throttled (total events = 6992)
CPU0: Package temperature above threshold, cpu clock throttled (total events = 8079)
CPU0: Package temperature/speed normal
CPU0: Package temperature/speed normal
CPU0: Package temperature/speed normal
CPU0: Package temperature/speed normal
CPU0: Package temperature/speed normal
CPU0: Package temperature/speed normal
CPU0: Package temperature/speed normal
CPU0: Package temperature/speed normal
CPU0: Package temperature/speed normal
CPU1: Package temperature above threshold, cpu clock throttled (total events = 12688)
CPU1: Package temperature above threshold, cpu clock throttled (total events = 12689)
CPU1: Package temperature above threshold, cpu clock throttled (total events = 29267)
CPU1: Package temperature above threshold, cpu clock throttled (total events = 3464)
CPU1: Package temperature above threshold, cpu clock throttled (total events = 3544)
CPU1: Package temperature above threshold, cpu clock throttled (total events = 4977)
CPU1: Package temperature above threshold, cpu clock throttled (total events = 6992)
CPU1: Package temperature above threshold, cpu clock throttled (total events = 8079)
CPU1: Package temperature/speed normal
CPU1: Package temperature/speed normal
CPU1: Package temperature/speed normal
CPU1: Package temperature/speed normal
CPU1: Package temperature/speed normal
CPU1: Package temperature/speed normal
CPU1: Package temperature/speed normal
CPU1: Package temperature/speed normal
CPU1: Package temperature/speed normal
CPU2: Core temperature above threshold, cpu clock throttled (total events = 12160)
CPU2: Core temperature above threshold, cpu clock throttled (total events = 12161)
CPU2: Core temperature above threshold, cpu clock throttled (total events = 26206)
CPU2: Core temperature above threshold, cpu clock throttled (total events = 3401)
CPU2: Core temperature above threshold, cpu clock throttled (total events = 3481)
CPU2: Core temperature above threshold, cpu clock throttled (total events = 4571)


Para facilitar la tarea usaremos el comando uniq a la salida:

# grep kernel /var/log/syslog | awk -F"] " '{print $2}'| sort | head -40 | uniq -c
      1 CPU0: Package temperature above threshold, cpu clock throttled (total events = 12688)
      1 CPU0: Package temperature above threshold, cpu clock throttled (total events = 12689)
      1 CPU0: Package temperature above threshold, cpu clock throttled (total events = 29267)
      1 CPU0: Package temperature above threshold, cpu clock throttled (total events = 3464)
      1 CPU0: Package temperature above threshold, cpu clock throttled (total events = 3544)
      1 CPU0: Package temperature above threshold, cpu clock throttled (total events = 4977)
      1 CPU0: Package temperature above threshold, cpu clock throttled (total events = 6992)
      1 CPU0: Package temperature above threshold, cpu clock throttled (total events = 8079)
      9 CPU0: Package temperature/speed normal
      1 CPU1: Package temperature above threshold, cpu clock throttled (total events = 12688)
      1 CPU1: Package temperature above threshold, cpu clock throttled (total events = 12689)
      1 CPU1: Package temperature above threshold, cpu clock throttled (total events = 29267)
      1 CPU1: Package temperature above threshold, cpu clock throttled (total events = 3464)
      1 CPU1: Package temperature above threshold, cpu clock throttled (total events = 3544)
      1 CPU1: Package temperature above threshold, cpu clock throttled (total events = 4977)
      1 CPU1: Package temperature above threshold, cpu clock throttled (total events = 6992)
      1 CPU1: Package temperature above threshold, cpu clock throttled (total events = 8079)
      9 CPU1: Package temperature/speed normal
      1 CPU2: Core temperature above threshold, cpu clock throttled (total events = 12160)
      1 CPU2: Core temperature above threshold, cpu clock throttled (total events = 12161)
      1 CPU2: Core temperature above threshold, cpu clock throttled (total events = 26206)
      1 CPU2: Core temperature above threshold, cpu clock throttled (total events = 3401)
      1 CPU2: Core temperature above threshold, cpu clock throttled (total events = 3481)
      1 CPU2: Core temperature above threshold, cpu clock throttled (total events = 4571)

Como vemos ahora es mas sencillo observar los errores, para nuestro ejemplo unifique la salida de los primeros 40, pero tambien podemos unificar la salida de todo el registro y luego mostrar los primeros 40 resultados, veremos que la salida varia un poco y es mas agradable:

# grep kernel /var/log/syslog | awk -F"] " '{print $2}'| sort | uniq -c | head -40
      1 CPU0: Package temperature above threshold, cpu clock throttled (total events = 12688)
      1 CPU0: Package temperature above threshold, cpu clock throttled (total events = 12689)
      1 CPU0: Package temperature above threshold, cpu clock throttled (total events = 29267)
      1 CPU0: Package temperature above threshold, cpu clock throttled (total events = 3464)
      1 CPU0: Package temperature above threshold, cpu clock throttled (total events = 3544)
      1 CPU0: Package temperature above threshold, cpu clock throttled (total events = 4977)
      1 CPU0: Package temperature above threshold, cpu clock throttled (total events = 6992)
      1 CPU0: Package temperature above threshold, cpu clock throttled (total events = 8079)
      9 CPU0: Package temperature/speed normal
      1 CPU1: Package temperature above threshold, cpu clock throttled (total events = 12688)
      1 CPU1: Package temperature above threshold, cpu clock throttled (total events = 12689)
      1 CPU1: Package temperature above threshold, cpu clock throttled (total events = 29267)
      1 CPU1: Package temperature above threshold, cpu clock throttled (total events = 3464)
      1 CPU1: Package temperature above threshold, cpu clock throttled (total events = 3544)
      1 CPU1: Package temperature above threshold, cpu clock throttled (total events = 4977)
      1 CPU1: Package temperature above threshold, cpu clock throttled (total events = 6992)
      1 CPU1: Package temperature above threshold, cpu clock throttled (total events = 8079)
      9 CPU1: Package temperature/speed normal
      1 CPU2: Core temperature above threshold, cpu clock throttled (total events = 12160)
      1 CPU2: Core temperature above threshold, cpu clock throttled (total events = 12161)
      1 CPU2: Core temperature above threshold, cpu clock throttled (total events = 26206)
      1 CPU2: Core temperature above threshold, cpu clock throttled (total events = 3401)
      1 CPU2: Core temperature above threshold, cpu clock throttled (total events = 3481)
      1 CPU2: Core temperature above threshold, cpu clock throttled (total events = 4571)
      1 CPU2: Core temperature above threshold, cpu clock throttled (total events = 6563)
      1 CPU2: Core temperature above threshold, cpu clock throttled (total events = 7636)
      9 CPU2: Core temperature/speed normal
      1 CPU2: Package temperature above threshold, cpu clock throttled (total events = 12688)
      1 CPU2: Package temperature above threshold, cpu clock throttled (total events = 12689)
      1 CPU2: Package temperature above threshold, cpu clock throttled (total events = 29267)
      1 CPU2: Package temperature above threshold, cpu clock throttled (total events = 3464)
      1 CPU2: Package temperature above threshold, cpu clock throttled (total events = 3544)
      1 CPU2: Package temperature above threshold, cpu clock throttled (total events = 4977)
      1 CPU2: Package temperature above threshold, cpu clock throttled (total events = 6992)
      1 CPU2: Package temperature above threshold, cpu clock throttled (total events = 8079)
      9 CPU2: Package temperature/speed normal
      1 CPU3: Core temperature above threshold, cpu clock throttled (total events = 12160)
      1 CPU3: Core temperature above threshold, cpu clock throttled (total events = 12161)
      1 CPU3: Core temperature above threshold, cpu clock throttled (total events = 26206)
      1 CPU3: Core temperature above threshold, cpu clock throttled (total events = 3401)