Do we need service discovery?

It is hard to provide a general recommendation whether service discovery tools are needed when working inside a Swarm cluster. If we look at the need to find services as the main use case for those tools, the answer is usually no. We don't need external service discovery for that. As long as all services that should communicate with each other are inside the same network, all we need is the name of the destination service. For example, for the go-demo (https://github.com/vfarcic/go-demo) service to find the related database, it only needs to know its DNS go-demo-db. The Chapter 3, Docker Swarm Networking and Reverse Proxy  proved that proper networking usage is enough for most use cases.

However, finding services and load balancing requests among them is not the only reason for service discovery. We might have other uses for service registries or key-value stores. We might need to store some information such that it is distributed and fault tolerant.

An example of the need for a key-value store can be seen inside the Docker Flow Proxy (https://github.com/vfarcic/docker-flow-proxy) project. It is based on HAProxy which is a stateful service. It loads the information from a configuration file into memory. Having stateful services inside a dynamic cluster represents a challenge that needs to be solved. Otherwise, we might lose state when a service is scaled, rescheduled after a failure, and so on.

Before we go into more details and problems related with stateful services, let's see how we could set up Consul as our key-value store of choice and go through its basic features.