If you have created a load balancer and can't reach your backend servers, check a target firewall configuration, especially if you run Ubuntu.    

If you run some workload on non-standard ports and want to expose them for the other components, there are regular steps you do:

  • Add ingress rule to the Security List
  • Add routing rules if your clients are in the different network/subnet
  • Update firewall rules on the backend nodes to allow incoming traffic.

Relatively mundane configuration steps, yet Ubuntu instance was able to surprise me. The first surprise was IPTables as a local firewall. Fine, I can live with IPTables: 

# Append new rule to the rules table
sudo iptables -A INPUT -p tcp --dport 8080 -j ACCEPT
# Make changes permanent
sudo netfilter-persistence save
Append firewall rule to the INPUT chain

My application is up and accepts requests, but the load balancer still reports that the backend pool is not available.  Let's take a closer look at the INPUT chain.

#List all rules 
 iptables --list INPUT
Chain INPUT (policy ACCEPT)
target     prot opt source    destination
ACCEPT     all  --  anywhere  anywhere   state RELATED,ESTABLISHED
ACCEPT     icmp --  anywhere  anywhere
ACCEPT     all  --  anywhere  anywhere
ACCEPT     udp  --  anywhere  anywhere   udp spt:ntp
ACCEPT     tcp  --  anywhere  anywhere   state NEW tcp dpt:ssh
ACCEPT     tcp  --  anywhere  anywhere   tcp dpt:http
ACCEPT     tcp  --  anywhere  anywhere   tcp dpt:https
REJECT     all  --  anywhere  anywhere   reject-with icmp-host-prohibited
ACCEPT     tcp  --  anywhere  anywhere   tcp dpt:8080
List rules in the INPUT chain

As requested, the new rule is the last rule in the chain, which is actually not good, because the second to last rule rejects all requests, which means our rule never gets a chance to fire.  So, we should put our port before the final REJECT.

# Delete Existing Rule
sudo iptables -D INPUT -p tcp --dport 8080 -j ACCEPT
# Insert Rule to the 6th position
sudo iptables -I INPUT 6 -p tcp --dport 8080 -j ACCEPT

# Check new rule position
iptables -L INPUT  --line-numbers
Chain INPUT (policy ACCEPT)
num  target  prot opt source    destination
1    ACCEPT  all  --  anywhere  anywhere     state RELATED,ESTABLISHED
2    ACCEPT  icmp --  anywhere  anywhere
3    ACCEPT  all  --  anywhere  anywhere
4    ACCEPT  udp  --  anywhere  anywhere     udp spt:ntp
5    ACCEPT  tcp  --  anywhere  anywhere     state NEW tcp dpt:ssh
6    ACCEPT  tcp  --  anywhere  anywhere     tcp dpt:5000
7    ACCEPT  tcp  --  anywhere  anywhere     tcp dpt:http
8    ACCEPT  tcp  --  anywhere  anywhere     tcp dpt:https
9    REJECT  all  --  anywhere  anywhere     reject-with icmp-host-prohibited

# Make changes permanent
sudo netfilter-persistence save
Relocate IPTables rule above 'reject all' clause

This time, the load balancer could reach the backend pool and sends traffic to the backend application.