If you have created a load balancer and can't reach your backend servers, check a target firewall configuration, especially if you run Ubuntu.
If you run some workload on non-standard ports and want to expose them for the other components, there are regular steps you do:
Add ingress rule to the Security List
Add routing rules if your clients are in the different network/subnet
Update firewall rules on the backend nodes to allow incoming traffic.
Relatively mundane configuration steps, yet Ubuntu instance was able to surprise me. The first surprise was IPTables as a local firewall. Fine, I can live with IPTables:
My application is up and accepts requests, but the load balancer still reports that the backend pool is not available. Let's take a closer look at the INPUT chain.
As requested, the new rule is the last rule in the chain, which is actually not good, because the second to last rule rejects all requests, which means our rule never gets a chance to fire. So, we should put our port before the final REJECT.
This time, the load balancer could reach the backend pool and sends traffic to the backend application.