There will be the IT-Tage 2016 conference. Where I'm going to give two talks.
One will be about Docker being more then a technical revolution.
The other one is about Best Practices for Docker+MySQL/MariaDB \o/

As always: fun first!

New versions of systemd (like in Ubuntu 16.04) are able to configure pid Cgroups (ulimit -u via Cgroups) using TasksMax.

Check the commit implementing TaksMax.

Author: Lennart Poettering <lennart@poettering.net>
Date:   Fri Nov 13 19:28:32 2015 +0100

    core: enable TasksMax= for all services by default, and set it to 512

    Also, enable TasksAccounting= for all services by default, too.



But not we’ve got silly defaults:

$ systemctl show -p TasksMax docker

$ systemctl show -p TasksMax mysql

So if you use Docker or MySQL most likely you are going to have trouble without really putting load on your server.

Just set TasksMax for your service.


Feel free to configure that setting fitting you load :)

Viel Spaß

Erkan \o/

Update 1: There will be an update for Debian/Ubuntu setting TaskMax=infinity as default.

Update 2: With version 230 there is a new default \o/ KillUserProcesses=yes. Check it out yourself and cry :(

In my last I was looking for a way to do performance monitoring and I stumbled upon Prometheus. Prometheus is much more than monitoring a single node service. Anyway let’s get the idea of gathering metrics using MySQL as example.

This how a simple configuration of Prometheus could look like:

  scrape_interval: 1m
  scrape_timeout: 10s
  evaluation_interval: 1m

  - job_name: mysql
    scheme: http
    - targets: 
        - ''
        zone: mysql

Every minute Prometheus accesses (/metrics is a Prometheus convention) and labels the result with zone=mysql. Querying the data you can use the labels.

This is a simple configuration. The fun of Prometheus is to have a lot of targets/jobs.

Let’s have a look at our specific endpoint:

> curl
mysql_global_status_threads_cached 26
mysql_global_status_threads_connected 99
mysql_global_status_threads_created 125
mysql_global_status_threads_running 2

You as a MySQL administrator know what this is all about. The data is provided by an exporter. In our case a container :)

> docker run -d -p 9104:9104 --link=mysql:backend \
  -e DATA_SOURCE_NAME=prometheus:prometheus@secret(backend:3306)/ \ 

This is old school Docker. Obviously the MySQL is running in a container also (mysql) and we are using the deprecated --link :)

The mysqld-exporter has a lot of options:

$ docker run --rm prom/mysqld-exporter --help
Usage of /bin/mysqld_exporter:
      Collect auto_increment columns and max values from information_schema
      Collect the current size of all registered binlog files
      Collect from SHOW GLOBAL STATUS (default true)
      Collect from SHOW GLOBAL VARIABLES (default true)
      Collect current thread state counts from the information_schema.processlist
      Collect metrics from information_schema.tables (default true)
  -collect.info_schema.tables.databases string
      The list of databases to collect table stats for, or '*' for all (default "*")
      If running with userstat=1, set to true to collect table statistics
      If running with userstat=1, set to true to collect user statistics
      Collect metrics from performance_schema.events_statements_summary_by_digest
  -collect.perf_schema.eventsstatements.digest_text_limit int
      Maximum length of the normalized statement text (default 120)
  -collect.perf_schema.eventsstatements.limit int
      Limit the number of events statements digests by response time (default 250)
  -collect.perf_schema.eventsstatements.timelimit int
      Limit how old the 'last_seen' events statements can be, in seconds (default 86400)
      Collect metrics from performance_schema.events_waits_summary_global_by_event_name
      Collect metrics from performance_schema.file_summary_by_event_name
      Collect metrics from performance_schema.table_io_waits_summary_by_index_usage
      Collect metrics from performance_schema.table_io_waits_summary_by_table
      Collect metrics from performance_schema.table_lock_waits_summary_by_table
      Collect from SHOW SLAVE STATUS (default true)
  -config.my-cnf string
      Path to .my.cnf file to read MySQL credentials from. (default "/home/golang/.my.cnf")
  -log.level value
      Only log messages with the given severity or above. Valid levels: [debug, info, warn, error, fatal, panic]. (default info)
      Add a log_slow_filter to avoid exessive MySQL slow logging.  NOTE: Not supported by Oracle MySQL.
  -web.listen-address string
      Address to listen on for web interface and telemetry. (default ":9104")
  -web.telemetry-path string
      Path under which to expose metrics. (default "/metrics")

Prometheus ships with an expression browser. Giving you the opportunity to access and graph the data. It also provides his own query language :) Without graphing the following two queries should be self-explaining:




Brian Brazil mentioned to use another (function)[https://prometheus.io/docs/querying/functions/] thx \o/


I recommend to use Grafana as dashboard. You just need to provide the Prometheus server as source and reuse the Queries you used in the expression browser. There is also PomDash, but afaik Grafana is the way to go.

Prometheus rocks. Having a central point to do the performance analysis of the whole datacenter is awesome. There a lot of exporters you can use. Even writing your own exporter is quite easy.

Viel Spaß



There is a nice presentation I recommend to check it and see the nice graphs Grafana builds :)


Percona is always great in adopting new stuff. Today they announced there Percona Monitoring and Management. Of course it uses also some exporters, Prometheus and Grafana. I’m quite sure it would/could kill other solutions on the market \o/

This blogpost extends last one. In the last blogpost, we had a look into Docker Network and how it makes the communication between the containers (over multiple hosts) easier. Of course we used Galera for that :)

In this blogpost we are going to use Docker Swarm to bootstrap a Galera Cluster.

Why using Docker Swarm?

Docker Swarm is (simplified) a proxy. So we've got one accesspoint to manage multiple hosts. (The swarm manage service will run on We also use Docker Swarm to abstract from the nodes. As we want the cluster to be running but we don't want to define explicitly where to run them. (Think about a 3-node-cluster on Docker Swarm with 100 nodes.)

Let us point the local docker to Docker Swarm:

export DOCKER_HOST=tcp://

We still got the cluster from the last blogpost running:

$ docker ps -f name=galera3 -f name=galera2 -f name=galera1
CONTAINER ID        IMAGE                      NAMES
751f4f071359        erkules/galera:basic       swarm3/galera3
24d4a2dfe3e2        erkules/galera:basic       swarm2/galera2
d3410d308171        erkules/galera:basic       swarm1/galera1

Docker Swarm extends NAMES. So we see the hosts the containers run on also.

Let's get rid from the old cluster:

$ docker rm -f swarm3/galera3 swarm2/galera2 swarm1/galera1 

We are going to deploy a Galera Cluster. For simplicity we are going to reuse the old overlay-network (named galera).

With Docker Swarm we also change to way we run the containers:

  • We don't mention where to run the container
  • Every container gets the label galera=setup1
  • We tell the container not to run on a node with another container running that label (affinity:...). This is to make sure not two Galera instances are on the same host.

Bootstrapcontainer (galera1):

$ docker run -d --name galera1 --net galera  -e affinity:galera!=setup1 \
   --label galera=setup1 erkules/galera:basic \
   --wsrep-cluster-address=gcomm://galera1,galera2,galera3 --wsrep-new-cluster


$ docker run -d --name galera2 --net galera  -e affinity:galera!=setup1 \
   --label galera=setup1 erkules/galera:basic \


$ docker run -d --name galera3 --net galera  -e affinity:galera!=setup1 \
   --label galera=setup1 erkules/galera:basic \

So let's check if everything went fine:

$ docker ps   -f  label=galera=setup1 
CONTAINER ID        IMAGE                  NAMES
582aaf272bb4        erkules/galera:basic   swarm2/galera3
611501de09b6        erkules/galera:basic   swarm1/galera2
1c3f8576cb12        erkules/galera:basic   swarm3/galera1

$ docker exec  swarm1/galera2 \
   mysql -e 'show global status like "wsrep_cluster_size"'
Variable_name Value
wsrep_cluster_size 3


What happens when we start a third Galera container?

    $ docker run -d --name galera4 --net galera  -e affinity:galera!=setup1 \
      --label galera=setup1 erkules/galera:basic \
    docker: Error response from daemon: unable to find a node that satisfies galera!=setup1.
    See 'docker run --help'

It failed! Very good. Having only three Docker nodes there was no machine left to start a forth Galera container.

For Planetmysql: MySQL g

Viel Spaß


Galera und Docker Network

Using Docker to run Galera on multiple nodes is quite a mess as described here. It is possible but no fun at all. As Docker does NATing every bidirectional setup is complicated.

Starting with Docker Network (version 1.9) you can simply span a Docker owned network (multiple of them) over multiple nodes. These networks are separated as the known bridged network Docker uses by default. It also provides a simple node/container discovery using the --name switch. Feels like a simple DNS.

Let's have a look how easy it is to deploy a Galera cluster. It is not for production. I use my own Docker image. It is just to play around.

There are our nodes:

  • Swarm1 IP=
  • Swarm2 IP=
  • Swarm3 IP=

At the start we still a network. We are going to name it galera.

$ DOCKER_HOST= docker network ls --filter "name=galera"
NETWORK ID          NAME                DRIVER

Creating a network is .. easy:

$ DOCKER_HOST= docker network create -d overlay galera 

$ DOCKER_HOST= docker network ls --filter "name=galera"
NETWORK ID          NAME                DRIVER
b0cadfa91420        galera              overlay

Every other node is part of this network too:

$ DOCKER_HOST= docker network ls --filter "name=galera"
NETWORK ID          NAME                DRIVER
b0cadfa91420        galera              overlay             

$ DOCKER_HOST= docker network ls --filter "name=galera"
NETWORK ID          NAME                DRIVER
b0cadfa91420        galera              overlay

So let's bootstrap a Galera Cluster. Remind the --net switch also. As you see we use the hostname to connect the cluster :)

$ DOCKER_HOST= docker run -d --name galera1 \
  --net galera erkules/galera:basic --wsrep-cluster-address=gcomm://

$ DOCKER_HOST= docker run -d --name galera2 \
  --net galera  erkules/galera:basic --wsrep-cluster-address=gcomm://galera1

$ DOCKER_HOST= docker run -d --name galera3 \
 --net galera  erkules/galera:basic --wsrep-cluster-address=gcomm://galera1

Let's see if everything went fine:

$ DOCKER_HOST= docker exec -ti galera3 \
  mysql -e 'show status like "wsrep_cluster_size"' 
| Variable_name      | Value |
| wsrep_cluster_size | 3     |

This is a quite simple and insecure example. Docker Network will change also the way we build simple MySQL replication setups. If you used to use --links. Get rid of it. As they are deprecated already in favor of Docker Network. Next time we are going to use Docker Swarm (on top of Docker Network) to deploy a Galera Cluster.

Viel Spaß :)


Ahoi, I made three proposals for Percona Live. (Even I know being a lonely (sniff) freelancer with no backup). 
At least I'm for sure an early adopter about the three topics I submitted and know for sure what I'm talking about *g*   

Even being Puppet certified. Ansible is my love. Never been that fast in doing automation \o/

This is a tutorial about Docker and Galera/MySQL. 

This talk is going to tell something about the "Docker Batteries" helping you to even bootstrap clusters etc. 

Feel free to vote for \o/

Docker is kinda awesome, as it releases a lot of creativity and let
us rethink infrastructure. Think about upgrading an application(docker container). We just stop the old one and start the new container. Rollback is easy as stopping the new container and starting from the old image.

Let’s have a look at nginx. Within the Docker ecosystem, in a world where the backends come and go. You profit from writing the nginx configuration in a dynamic way. Most likely using confd or consul-template.

After that you stop the container and start it new from the image.

Kinda silly!


Sending nginx a SIGHUP would have told it to simply reread the configuration without stopping it by spawning a new process.

Nginx even has a nice trick to upgrade. Sending a SIGUSR2 nginx spawns a new process with the new binary.

In a standard Docker workflow you don’t use this features.

Regards yours

DockerHipster \o/

Galera for Mesos

| Keine Kommentare | Keine TrackBacks
This time it is nothing like a linkt to another Blog:  Galera on Mesos

Ok as a fact I was kinda involved. Even the work is done by Stefan all alone.
We meat for a day in a coworking space and discussed about Galera and Mesos.

In the end Stefan produced this incredible blogpost and pushed Mesos forward.

Whats the fun about this post?

We already now Galera is already the standard in a lot of architectures. For example OpenStack.
Doing consultant work for Docker also I encourage to use Galera for all this infrastructures Docker runs on. 

Mesos is about to run easy on 1000 nodes. It has a nice abstraction of nodes and framework. Companies like Airbnb, Paypal, eBay, Groupon use Mesos. Having a Galera poc for Mesos is going to make it likely to have MySQL (etc.) being a native part in Mesos installations.

There had been another customer I was allowed to help to deploy Galera on CoreOS \o/.

At least I plan to help deploy Galera on other cluster or multi-node solutions for Docker also :)

Stay tuned as there is a plan to have a little series about Galera@Docker on the coderships website too.

Have Fun
Erkan Yanar

Using master-master for MySQL? To be frankly we need to get rid of that architecture. We are skipping the active-active setup and show why master-master even for failover reasons is the wrong decision.

So why does a DBA thinks master-master is good for in a failover scenario?

  • The recovered node does get his data automatically.
  • You need not to use a backup for recovery.

Please remember: MySQL Replication is async

Again: MySQL Replication is async. Even the so called semi-sync Replication!

So following is quite likely.

See a nice master-master setup:

 activ                 standby
+------+     c        +------+
|      |------------->|      |
|abcd  |              |ab    |
|      |              |      |
|      |<-------------|      |
+------+              +------+

Oh my god the node went down:

 RIP                   activ
+------+              +------+
|      |-----||------>|      |
|abcd  |              |abc   |
|      |              |      |
|      |<----||-------|      |
+------+              +------+

Np, we’ve got master-master. After the takeover the recovering node fetches up. (As a fact it has one transaction more:( )

recovered              activ
+------+              +------+
|      |------------->|      |
|abcd  |              |abce  |
|      |      e       |      |
|      |<-------------|      |
+------+              +------+

Great we got no sync data anymore!

recovered             activ
+------+              +------+
|      |------------->|      |
|abcde |              |abce  |
|      |              |      |
|      |<-------------|      |
+------+              +------+

As a fact there is no need for master-master anyway. We’ve got GTID nowadays. Use a simple replication. In a failover you can use GTID to check if you got extra transactions on the recovering node.

If not then simply create a replication and you get all the missing data.

But if there are extra transactions on the recovering node you got to rebuild the node anyway.

FYI: This works with GTID@MariaDB and GTID@MySQL.

Welcome to the GTID era! \o/

Viel Spaß

Erkan :)