This is a follow-up post on the Improve
docker service update command to fit
in CI/CD world I wrote the other
day. Premise is exactly as before but the target is now
docker service create
command. We will try to ensure that CI/CD setup can get feedback from executing
docker service create command.
Description of issues and assumptions taken in the solution should be identical to those described in Improve
docker service updatecommand to fit in CI/CD world, so I will not repeat myself here.
Following is a very simple wrapping bash script that is doing a little magic to help wait for the feedback from an update process of a service in docker cluster:
- Script accepts 4 parameters:
SERVICE_NAME- docker service name to be created
STABILITY_DELAY_S- after all replicas are created script will wait some seconds and then test status of an update again to ensure that no more replicas are getting created (normally the sign that new version is simply unstable and crashes, letting docker swarm to create new instances all the time)
ERROR_SLIP- number of failing containers per replica that is allowed to still consider service created
CREATE_CMD- update command to be evaluated by this script, i.e. exactly what this script is wrapping (e.g.
docker service create <service_name> ...)
After creating service, the script fetches service status by running
service ps <service_name> which gives information about currently running
containers. There should be no old instances of a service registered, otherwise
docker service create will fail saying that service has already been created.
But the number of old instances of a service can suddenly go up in cases when
installed service is not stable and crashes making swarm to re-create it again
The script is also quite talkative and will produce a lot of information which is typically what you want when running it as part of your CI/CD setup to ensure that troubleshooting process is not a nightmare.
./docker-service-create-wait.sh redis-slave 60 1 \ "docker service create --name redis-slave redis:3.2.6-alpine"
It will output something like this:
1 2 3 4 5 6 7 8
Create command: docker service create --name redis-slave redis:3.2.6-alpine Stability delay: 120 sec Error slip: 1 Creating service: 8oz8ldtxoju18v4ds1g6ith6i done Stability test: done SUCCESS
In case of a failure script is producing non-zero result code which is what is typically getting detected by CI/CD setup.
Author Roman Kuznetsov
License Roman Kuznetsov