Bash scripts to diagnose network issues

These are 2 simple scripts I wrote that run various types of tests to help diagnose network issues between a source and target. They can be safely run as a non-root user as long as the commands exist.

Script 1 - Various network tests

This script runs 3 separate tests and generates 3 logs.

Logs Generated

1. Check port availability (test_network_checkport.log)

This simply uses the nc command to see if the port is available or not. No packets are sent or received, but if you notice high response times occasionally, then something is fishy.

# TIMESTAMP | SOURCE HOSTNAME | TARGET IP | TARGET PORT | RESPONSE TIME (SECS)
2021-12-17 21:14:14.731991069|soadev|192.168.1.31|5901|0.01
2021-12-17 21:14:15.284783737|soadev|192.168.1.31|5901|0.01
2021-12-17 21:14:15.837768193|soadev|192.168.1.31|5901|0.01

2. Check error and dropped packets (test_network_packets.log)

This uses the ip command to count the number of dropped and error packets during transmission and receive.

# TIMESTAMP | SOURCE  HOSTNAME | TRANSMISSION PACKETS ERRORS | TRANSMISSION PACKETS DROPS | RECEIVE PACKET ERRORS | RECEIVE ERROR DROPS
2021-12-17 21:14:13.626336229|soadev|0|0|0|0
2021-12-17 21:14:14.179361714|soadev|0|0|0|0
2021-12-17 21:14:14.731991069|soadev|0|0|0|0

3. Check bad and retransmitted packets (test_network_segments.log)

The netstat command is used here to observe the bad and retransmitted segments. Don't look at the actually values returned, but rather if they are growing instead.

# TIMESTAMP | SOURCE HOSTNAME | BAD SEGMENTS | RETRANSMITTED SEGMENTS | RETRANSMITTED SEGMENTS %
2021-12-17 21:14:13.626336229|soadev|297|30846213|6.25439
2021-12-17 21:14:14.179361714|soadev|297|30846213|6.25439
2021-12-17 21:14:14.731991069|soadev|297|30846213|6.25439

Running the Script

ifconfig -a

./test_network.sh 192.168.1.31 443 8 1 eth0

The ifconfig command is used to get the network interface name to be used in the script (as the last parameter).

In this script, the target IP and target port (443) are passed, and the number of loops (8) with a delay (1 second) on an interface (ens3).

Source Code

Source code for the script:

#!/bin/bash

#--------------------------------------------------------------#
# FILENAME:      test_network.sh                               #
# CREATION DATE: 2021-12-06                                    #
# DESCRIPTION:   Various port, packet, segment checks          #
# AUTHOR:        Ahmed Aboulnaga                               #
# LOG:           test_network_checkport.log                    #
#                test_network_packets.log                      #
#                test_network_segments.log                     #
#--------------------------------------------------------------#

#--------------------------------------------------------------#
# Help                                                         |
#--------------------------------------------------------------#
if [ $# -ne 5 ]; then
  echo ""
  echo "Usage:"
  echo "  ./test_network.sh [target_ip] [target_port] [number_of_loops] [delay_in_secs] [interface]"
  echo ""
  echo "Logs:"
  echo ""
  echo "  test_network_checkport.log"
  echo "    # Port availability, no packets transmitted"
  echo "    # TIMESTAMP | SOURCE HOSTNAME | TARGET IP | TARGET PORT | RESPONSE TIME (SECS)"
  echo "    2021-12-17 21:14:14.731991069|soadev|192.168.1.31|5901|0.01"
  echo "    2021-12-17 21:14:15.284783737|soadev|192.168.1.31|5901|0.01"
  echo "    2021-12-17 21:14:15.837768193|soadev|192.168.1.31|5901|0.01"
  echo ""
  echo "  test_network_packets.log"
  echo "    # Packet transmission and receive packet errors and drops"
  echo "    # TIMESTAMP | SOURCE  HOSTNAME | TARGET IP | TRANSMISSION PACKETS ERRORS | TRANSMISSION PACKETS DROPS | RECEIVE PACKET ERRORS | RECEIVE ERROR DROPS"
  echo "    2021-12-17 21:14:13.626336229|soadev|0|0|0|0"
  echo "    2021-12-17 21:14:14.179361714|soadev|0|0|0|0"
  echo "    2021-12-17 21:14:14.731991069|soadev|0|0|0|0"
  echo ""
  echo "  test_network_segments.log"
  echo "    # Bad and retransmitted segments"
  echo "    # TIMESTAMP | SOURCE HOSTNAME | TARGET IP | BAD SEGMENTS | RETRANSMITTED SEGMENTS | RETRANSMITTED SEGMENTS %"
  echo "    2021-12-17 21:14:13.626336229|soadev|297|30846213|6.25439"
  echo "    2021-12-17 21:14:14.179361714|soadev|297|30846213|6.25439"
  echo "    2021-12-17 21:14:14.731991069|soadev|297|30846213|6.25439"
  echo ""
  echo "Example:"
  echo "  ./test_network.sh"
  echo "  ./test_network.sh 192.168.1.31 443 8 1 eth0"
  echo "  nohup ./test_network.sh 192.168.1.31 443 345600 0.5 eth0 &"
  echo ""
  exit 0
fi

#--------------------------------------------------------------#
# Parameters                                                   |
#--------------------------------------------------------------#
V_TARGETIP=${1}
V_TARGETPORT=${2}
V_LOOP=${3}
V_DELAY=${4}
V_INTERFACE=${5}

#--------------------------------------------------------------#
# Loop                                                         #
#--------------------------------------------------------------#
i=1
while [ ${i} -le ${V_LOOP} ]; do
  V_TIMESTAMP=`date +'%Y-%m-%d %H:%M:%S.%N'`
  i=`expr ${i} + 1`
  sleep ${V_DELAY}

  #--------------------------------------------------------------#
  # Check port                                                   |
  #--------------------------------------------------------------#
  # Log: number of seconds for positive response
  if command -v nc &> /dev/null; then
    nc -vz ${V_TARGETIP} ${V_TARGETPORT} > test_network.tmp 2>&1
    echo "${V_TIMESTAMP}|`hostname`|${V_TARGETIP}|${V_TARGETPORT}|`cat test_network.tmp | tail -1 | awk '{print $9}'`" >> test_network_checkport.log
    rm -f test_network.tmp
  fi

  #--------------------------------------------------------------#
  # Packet errors                                                #
  #--------------------------------------------------------------#
  # Log: Transmitted Packet Errors | Transmitted Packet Drops | Received Packet Errors | Received Packet Drops
  if command -v ip &> /dev/null; then
    V_TX_PACKETERRORS=`cat /proc/net/dev | grep ${V_INTERFACE} | awk {'print $12'}`
    V_TX_PACKETDROPS=`cat /proc/net/dev | grep ${V_INTERFACE} | awk {'print $13'}`
    V_RX_PACKETERRORS=`cat /proc/net/dev | grep ${V_INTERFACE} | awk {'print $4'}`
    V_RX_PACKETDROPS=`cat /proc/net/dev | grep ${V_INTERFACE} | awk {'print $5'}`
    echo "${V_TIMESTAMP}|`hostname`|${V_TX_PACKETERRORS}|${V_TX_PACKETDROPS}|${V_RX_PACKETERRORS}|${V_RX_PACKETDROPS}" >> test_network_packets.log
  fi

  #--------------------------------------------------------------#
  # Segment retransmissions & bad                                #
  #--------------------------------------------------------------#
  # Log: Segments Bad Count | Segments Retransmitted Count | Segments Retransmitted %
  if command -v netstat &> /dev/null; then
    V_SEGS_RETRANSMIT=`netstat -s | grep retransmited | awk {'print $1'}`
    V_SEGS_BAD=`netstat -s | grep bad | grep segments | awk {'print $1'}`
    V_SEGS_RETRANSMITPER=`gawk 'BEGIN {OFS=" "} $1 ~ /Tcp:/ && $2 !~ /RtoAlgorithm/ {print ($13/$12*100)}' /proc/net/snmp`
    echo "${V_TIMESTAMP}|`hostname`|${V_SEGS_BAD}|${V_SEGS_RETRANSMIT}|${V_SEGS_RETRANSMITPER}" >> test_network_segments.log
  fi

done

exit 1

Script 2 - Traffic loss

This script generates a single log file.

Log Generated

1. Check traffic loss (test_trafficloss.log)

This script uses the mtr command, which combines the functionality of traceroute and ping into a single network diagnostic tool.

Start: Fri Dec 17 21:10:33 2021
HOST: soadev                      Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 140.91.196.17              0.0%    10    0.1   0.1   0.1   0.1   0.0
  2.|-- 4.16.73.206                0.0%    10    0.5   6.7   0.5  57.1  17.7
  3.|-- ae60.edge5.Washington12.L  0.0%    10   20.5   6.6   0.5  27.6   9.6
  4.|-- 142.250.166.242            0.0%    10    2.9   2.9   2.7   3.2   0.0
  5.|-- 108.170.240.97             0.0%    10    1.9   1.7   1.5   2.0   0.0
  6.|-- 108.170.240.112            0.0%    10    0.7   2.5   0.6  15.4   4.5
  7.|-- 142.251.49.17             70.0%    10    1.1   1.3   1.1   1.5   0.0
  8.|-- 142.251.49.29             50.0%    10    1.5   1.7   1.5   1.9   0.0
  9.|-- 216.239.56.73              0.0%    10    8.2  14.5   8.2  66.7  18.3
 10.|-- 216.239.40.130            20.0%    10   15.0  14.9  14.8  15.2   0.0
 11.|-- 209.85.240.16             60.0%    10   32.1  32.5  31.8  33.6   0.0
 12.|-- 216.239.62.212             0.0%    10   32.2  33.2  32.2  41.3   2.8
 13.|-- 108.170.252.129            0.0%    10   31.5  31.5  31.5  31.5   0.0
 14.|-- 209.85.244.59              0.0%    10   32.6  33.3  32.2  41.5   2.8
 15.|-- atl14s07-in-f142.1e100.ne  0.0%    10   31.7  31.7  31.7  31.7   0.0

If traffic loss is observed in the last hop, this is generally not an issue with the connection to the target.
If traffic loss is observed in the middle, this is possibly due to ICMP rate limiting which is not an issue.
If traffic loss is observed from the middle to the end, there is likely loss in traffic.

Running the Script

./test_trafficloss.sh 192.168.1.31 10

The target IP is passed with a count of number of times you want to loop.

Source Code

Source code for the script:

#!/bin/bash

#--------------------------------------------------------------#
# FILENAME:      test_trafficloss.sh                           #
# CREATION DATE: 2021-12-17                                    #
# DESCRIPTION:   Test traffic loss                             #
# AUTHOR:        Ahmed Aboulnaga                               #
# LOG:           test_trafficloss.log                          #
#--------------------------------------------------------------#

#--------------------------------------------------------------#
# Help                                                         |
#--------------------------------------------------------------#
if [ $# -ne 2 ]; then
  echo ""
  echo "Usage:"
  echo "  ./test_trafficloss.sh [target_ip] [number_of_loops]"
  echo ""
  echo "Description:"
  echo "  If traffic loss is in the last hop, not an issue with the connection to target."
  echo "  If traffic loss is in the middle, possibly due to ICMP rate limiting; not an issue."
  echo "  If traffic loss is in the middle to the end, likely losing some traffic."
  echo ""
  echo "Example:"
  echo "  ./test_trafficloss.sh"
  echo "  ./test_trafficloss.sh 192.168.1.31 10"
  echo "  nohup ./test_trafficloss.sh 192.168.1.31 10 &"
  echo ""
  exit 0
fi

#--------------------------------------------------------------#
# Parameters                                                   |
#--------------------------------------------------------------#
V_TARGETIP=${1}
V_LOOP=${2}

#--------------------------------------------------------------#
# Loop                                                         #
#--------------------------------------------------------------#
i=1
while [ ${i} -le ${V_LOOP} ]; do
  V_TIMESTAMP=`date +'%Y-%m-%d %H:%M:%S.%N'`
  i=`expr ${i} + 1`

  #--------------------------------------------------------------#
  # Traffic loss                                                 |
  #--------------------------------------------------------------#
  if command -v mtr &> /dev/null; then
    mtr --report ${V_TARGETIP} >> test_trafficloss.log
  fi

done

exit 1

Bash scripts to diagnose network issues

Ahmed Aboulnaga

Ahmed Aboulnaga

Script 1 - Various network tests

Logs Generated

Running the Script

Source Code

Script 2 - Traffic loss

Log Generated

Running the Script

Source Code

A Shell Tricks: Session Color Coding

A Shell Tricks: Forwarding X session

A Shell Tricks: Search and Replace

Docker: Logn Errors

Ansible: Dynamic Inventory