Deploying the Plumber service in a Docker container

Deploying the Plumber service in a Docker container#

This page provides instructions on how to deploy the plumber service in a Docker container. It takes as a starting point a standard Squirro cloud instance, which runs the plumber service directly on the host, and guides you through the steps to deploy the service in a Docker container. All the steps assume that you are logged in with a normal user that has sudo privileges.

Disable the host-level plumber service#

Firstly, verify that the plumber service is running on the host:

# all the Squirro services report OK (plumber included)
squirro_status

# ensure that the host-level service of plumber is up and running
sudo systemctl status sqplumberd

Then, we stop the plumber service:

# stop and disable the plumber service for now
sudo systemctl stop sqplumberd
sudo systemctl disable sqplumberd

# verify
squirro_status
sudo systemctl status sqplumberd

Modify plumber.ini#

We make copies of the existing plumber.ini file:

sudo cp -p /etc/squirro/plumber.ini /etc/squirro/plumber.ini.host
sudo cp -p /etc/squirro/plumber.ini /etc/squirro/plumber.ini.docker

Then, we perform two modifications to the plumber.ini.docker file:

  • Replace any occurrence of localhost with host.docker.internal

  • Make sure that the following config options exist, otherwise, append them:

[redis_cache]
host = host.docker.internal

[redis_bytes_cache]
host = host.docker.internal

[redis_database_migrations]
host = host.docker.internal

[squirro_client]
squirro_cluster = http://host.docker.internal

[services]
user = http://host.docker.internal:81/api/user

The following script can be used to automate the modifications to the plumber.ini.docker file:

docker_plumber_ini='/etc/squirro/plumber.ini.docker'

# Replace 'localhost' with 'host.docker.internal'
sudo sed -i 's/localhost/host.docker.internal/g' "$docker_plumber_ini"

declare -A section_option_map
section_option_map["redis_cache"]="host = host.docker.internal"
section_option_map["redis_bytes_cache"]="host = host.docker.internal"
section_option_map["redis_database_migrations"]="host = host.docker.internal"
section_option_map["squirro_client"]="squirro_cluster = http://host.docker.internal"
section_option_map["services"]="user = http://host.docker.internal:81/api/user"

# Loop through all sections and append if they do not exist
for section in "${!section_option_map[@]}"; do
    echo "Checking section: [$section]"
    if ! sudo grep -q "^\[$section\]" "$docker_plumber_ini"; then
        echo "Adding section: [$section]"
        echo -e "\n[$section]\n${section_option_map[$section]}" | sudo tee -a "$docker_plumber_ini" > /dev/null
    else
        echo "Section exists: [$section]"
    fi
done

Finally, we replace the current plumber.ini with the edited plumber.ini.docker file:

sudo cp -p -f /etc/squirro/plumber.ini.docker /etc/squirro/plumber.ini

Note

podman uses host.containers.internal instead of host.docker.internal

Note

The script above assumes that if the sections of the aforementioned config options already exist in the file, that they also define the listed config options. In most cases, this should be true. If not, please adjust accordingly.

Modify Redis config files#

The plumber service uses the two Redis servers that come with the Squirro deployment. Therefore, we make sure that they are configured in a way to allow communication from the Docker container of the plumber service.

# for backup
sudo cp -p /etc/redis/redis.conf /etc/redis/redis.conf.original
sudo cp -p /etc/redis/cache.conf /etc/redis/cache.conf.original

# append the 172.17.0.1 address to the bind directive
sudo sed -i 's/bind 127\.0\.0\.1 -::1/bind 127\.0\.0\.1 -::1 172.17.0.1/g' /etc/redis/redis.conf
sudo sed -i 's/bind 127\.0\.0\.1 -::1/bind 127\.0\.0\.1 -::1 172.17.0.1/g' /etc/redis/cache.conf

# restart both services to apply changes
sudo systemctl restart redis-server && sudo systemctl restart redis-server-cache

Note

podman uses 10.88.0.1 instead of 172.17.0.1

Optionally, to verify that they have been configured correctly, you can launch a Python interpreter from Squirro’s virtual environment (squirro_activate), and execute the following (adjust Redis passwords):

import redis
redis_server_password, redis_server_cache_password = "pwd", "pwd"
for port, password in ((6379, redis_server_password), (6380, redis_server_cache_password)):
    assert redis.Redis("172.17.0.1", port=port, password=password).ping()

Enable the containerized plumber service#

The last step involves updating the systemd unit file of the plumber service to manage its containerized version.

# copy the docker-enabled systemd unit file and create backup for the host-level one
cd /usr/lib/systemd/system
sudo cp -p sqplumberd.service sqplumberd.service.host
sudo cp /etc/squirro/plumber.d/sqplumberd.service sqplumberd.service.docker
sudo cp -p -f sqplumberd.service.docker sqplumberd.service

# download the image from the internal registry before starting the container
sudo docker pull $(grep 'mirror.squirro.net' /etc/squirro/plumber.d/run.sh | tr -d '"')

# reload the systemd unit files and start the docker-enabled sqplumberd service
sudo systemctl daemon-reload
sudo systemctl start sqplumberd
sudo systemctl enable sqplumberd

Now, the plumber service should be up and running from the Docker container.

You can verify that by checking:

  • the output of systemctl status sqplumberd (the plumber.d/run.sh should be listed in the output)

  • the output of docker ps (the container squirro-service-plumber should be listed in the output)

To debug any issues with getting the plumber container up and running, you can use:

  • journalctl -u sqplumberd -f

Contents