IT Wonder Lab - Automating Cloud Infrastructure

Running Postgres in Kubernetes with a Persistent NFS Volume

Using a Persistent NFS Volume for a Postgres Database running inside Kubernetes

Learn how to launch a Postgres database instance in a Kubernetes cluster and safely store its data in a persistent location.

The Postgres instance will have all its data stored in a remote NFS Server to preserve database data across Kubernetes cluster destroy or Postgres Pod unintended disruptions.

This configuration is suitable only for the development and testing of applications in a local development Kubernetes cluster and it’s a continuation of the tutorial on How to create a Kubernetes cluster for local development using Vagrant, Ansible, and VirtualBox.

External NSF Server providing storage for a PostgreSQL server inside a Kubernetes Cluster
Persistent NFS Volume for a Postgres local development Kubernetes Cluster

Prerequisites:

Persistent Volumes in Kubernetes

Kubernetes containers are mostly used for stateless applications, where each instance is disposable, and does not store data that needs to be persisted across restarts inside the container is needed for client sessions as its storage is ephemeral.

On the contrary, the stateful applications need to store data and have it available between restarts and sessions. Databases like Postgres or MySQL are typical stateful applications.

Kubernetes provides support for many types of volumes depending on the Cloud provider. For our local development Kubernetes Cluster, the most appropriate and easy to configure is an NFS volume.

NFS Server Installation

If you don’t have an existing NFS Server, it is easy to create a local NFS server for our Kubernetes Cluster.

Install the Ubuntu needed packages and create a local directory /mnt/vagrant-kubernetes:

sudo apt install nfs-kernel-server
sudo mkdir -p /mnt/vagrant-kubernetes
sudo mkdir -p /mnt/vagrant-kubernetes/data
sudo chown nobody:nogroup /mnt/vagrant-kubernetes
sudo chmod 777 /mnt/vagrant-kubernetes

Edit the /etc/exports file to add the exported local directory and limit the share to the CIDR used by the Kubernetes Cluster nodes for external access.

If using the example from the Kubernetes Vagrant Cluster tutorial, use CIDR 192.168.50.0/24 which corresponds to the network used by the VirtualBox machines to access the external networks.

/mnt/vagrant-kubernetes 192.168.50.0/24(rw,sync,no_subtree_check,insecure,no_root_squash)

Start the NFS server

# sudo exportfs -a
# sudo systemctl restart nfs-kernel-server    
# sudo exportfs -v
/mnt/vagrant-kubernetes
	192.168.50.0/24(rw,wdelay,insecure,no_root_squash,no_subtree_check,sec=sys,rw,insecure,no_root_squash,no_all_squash)

Kubernetes Deployment File

The Postgres database deployment is composed of the following Kubernetes Objects:

  • ConfigMap: stores common configuration data for the Postgres database server
  • PersistentVolume: creates an NFS client that connects to the server and makes the NFS share available for volume claims.
  • PersistentVolumeClaim: defines the characteristics of the needed volume, that Kubernetes will try to solve using an available PersistentVolume with the requested configuration.
  • Deployment: starts an instance of a Postgres docker container using the supplied ConfigMap to redefine the value of Postgres configuration environment variables and mounts the PersistentVolumeClaim inside the container to replace PostgreSQL Data directory.
  • Service (NodePort):  publishes the Postgres server port outside the Kubernetes Cluster.

(Full deployment file is shown at the end of the tutorial)

Postgres Kubernetes ConfigMap

The config map is a key-value store that is available to all Kubernetes nodes. The data will be used to set some environment variables of the Postgres container:

  • POSTGRES_DB: db (the database to be created at startup if it doesn’t exist)
  • POSTGRES_USER: user (the admin user that will be created)
  • POSTGRES_PASSWORD: pass (the password for the admin user that will be created)
  • PGDATA: /var/lib/postgresql/data/k8s (the location for the data to be used upon initialization, we will be using the subdirectory k8s for this instance)

Definition of ConfigMap in file postgresql.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: psql-itwl-dev-01-cm
data:
  POSTGRES_DB: db
  POSTGRES_USER: user
  POSTGRES_PASSWORD: pass
  PGDATA: /var/lib/postgresql/data/k8s

Postgres Kubernetes PersistentVolume

Defines an NFS PersistentVolume located at:

  • Server: 192.168.50.1 (The IP assigned by VirtualBox to our host)
  • Path: /mnt/vagrant-kubernetes/data (a subdirectory “data” inside the shared folder)

The volume has the following configuration:

  • Labels:
    • app: psql
    • ver: itwl-dev-01-pv
  • Capacity: It is able to store up to 1 Gigabyte.
  • Access Mode: Supports many clients reading and writing at the same time.
  • Retain policy: After usage, the volume is not destroyed

Definition of PersistentVolume in file postgresql.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: psql-itwl-dev-01-pv
  labels: #Labels 
    app: psql
    ver: itwl-dev-01-pv 
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  nfs:
    server: 192.168.50.1
    path: "/mnt/vagrant-kubernetes/data"

Postgres Kubernetes PersistentVolumeClaim

The PersistentVolumeClaim is the object that is assigned to the deployment. The PersistentVolumeClaim defines how the volume needs to be, and Kubernetes tries to find a corresponding PersistentVolume that satisfies all the requirements.

The PersistentVolumeClaim asks for a volume with the following labels:

  • app: psql
  • ver: itwl-dev-01-pv

An access mode of ReadWriteMany and at least 1 Gigabyte of storage.

Definition of PersistentVolumeClaim in file postgresql.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: psql-itwl-dev-01-pvc
spec:
  selector:
    matchLabels:  #Select a volume with this labels
      app: psql
      ver: itwl-dev-01-pv
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 1Gi

Postgres Kubernetes Deployment

A regular application deployment descriptor with the following characteristics:

A single replica will be deployed. Istio is not needed, annotation sidecar.istio.io/inject false prevents the macro to inject the proxy. See Installing Istio in Kubernetes for an explanation about istio.

The container uses Postgres latest image from the public Docker registry (https://hub.docker.com/) and sets:

  • Export container port 5432
  • Set environment variables for the container using the Kubernetes config map psql-itwl-dev-01-cm
  • Replace the directory /var/lib/postgresql/data inside the Postgres container with a volume named pgdatavol
  • Define the pgdatavol volume as an instance of the persistentVolumeClaim psql-itwl-dev-01-pvc defined before

Definition of Deployment in file postgresql.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: psql-itwl-dev-01
  labels: 
    app: psql 
    ver: itwl-dev-01
spec:
  replicas: 1
  selector:
    matchLabels:  #Deploy in a POD that has labels app: color and color: blue
      app: psql
      ver: itwl-dev-01
  template: #For the creation of the pod      
    metadata:
      labels:
        app: psql
        ver: itwl-dev-01
      annotations:
        sidecar.istio.io/inject: "false"        
    spec:
      containers:
        - name: postgres
          image: postgres:latest
          imagePullPolicy: "IfNotPresent"
          ports:
            - containerPort: 5432 
          envFrom:
            - configMapRef:
                name: psql-itwl-dev-01-cm          
          volumeMounts:
            - mountPath: /var/lib/postgresql/data
              name: pgdatavol
      volumes:
        - name: pgdatavol
          persistentVolumeClaim:
            claimName: psql-itwl-dev-01-pvc

Postgres Kubernetes NodePort Service

Since our local Kubernetes Cluster doesn’t have a Cloud provided Load Balancer, we are using the NodePort functionality to access published ports in containers.

The NodePort will publish Postgres in port 30100 of every Kubernetes Master and Nodes:

  • 192.168.50.11:30100
  • 192.168.50.12:30100
  • 192.168.50.13:30100

Definition of Service in file postgresql.yaml:

apiVersion: v1
kind: Service
metadata:
  name: postgres-service-np
spec:
  type: NodePort
  selector:
    app: psql
  ports:
    - name: psql
      port: 5432        # Cluster IP http://10.109.199.234:port (docker exposed port)
      nodePort: 30100   # (EXTERNAL-IP VirtualBox IPs) 192.168.50.11:nodePort 192.168.50.12:nodePort 192.168.50.13:nodePort
      protocol: TCP

Deploy Postgres in Kubernetes

Full file postgresql.yaml with all the resources to be deployed in Kubernetes:

apiVersion: v1
kind: ConfigMap
metadata:
  name: psql-itwl-dev-01-cm
data:
  POSTGRES_DB: db
  POSTGRES_USER: user
  POSTGRES_PASSWORD: pass
  PGDATA: /var/lib/postgresql/data/k8s

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: psql-itwl-dev-01-pv
  labels: #Labels 
    app: psql
    ver: itwl-dev-01-pv 
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  nfs:
    server: 192.168.50.1
    path: "/mnt/vagrant-kubernetes/data"
          
---

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: psql-itwl-dev-01-pvc
spec:
  selector:
    matchLabels:  #Select a volume with this labels
      app: psql
      ver: itwl-dev-01-pv
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 1Gi
---

apiVersion: apps/v1
kind: Deployment
metadata:
  name: psql-itwl-dev-01
  labels: 
    app: psql 
    ver: itwl-dev-01
spec:
  replicas: 1
  selector:
    matchLabels:  #Deploy in a POD that has labels app: color and color: blue
      app: psql
      ver: itwl-dev-01
  template: #For the creation of the pod      
    metadata:
      labels:
        app: psql
        ver: itwl-dev-01
      annotations:
        sidecar.istio.io/inject: "false"        
    spec:
      containers:
        - name: postgres
          image: postgres:latest
          imagePullPolicy: "IfNotPresent"
          ports:
            - containerPort: 5432 
          envFrom:
            - configMapRef:
                name: psql-itwl-dev-01-cm          
          volumeMounts:
            - mountPath: /var/lib/postgresql/data
              name: pgdatavol
      volumes:
        - name: pgdatavol
          persistentVolumeClaim:
            claimName: psql-itwl-dev-01-pvc

---

apiVersion: v1
kind: Service
metadata:
  name: postgres-service-np
spec:
  type: NodePort
  selector:
    app: psql
  ports:
    - name: psql
      port: 5432        # Cluster IP http://10.109.199.234:port (docker exposed port)
      nodePort: 30100   # (EXTERNAL-IP VirtualBox IPs) http://192.168.50.11:nodePort/ http://192.168.50.12:nodePort/ http://192.168.50.13:nodePort/
      protocol: TCP

Create the resources in Kubernetes

$ kubectl apply -f postgresql.yaml 
configmap/psql-itwl-dev-01-cm created
persistentvolume/psql-itwl-dev-01-pv created
persistentvolumeclaim/psql-itwl-dev-01-pvc created
deployment.apps/psql-itwl-dev-01 created
service/postgres-service-np created

Check Kubernetes

Check that all resources have been created:

$ kubectl get all
NAME                                    READY   STATUS    RESTARTS   AGE
pod/psql-itwl-dev-01-594c7468c7-p9k9l   1/1     Running   0          6s

NAME                          TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
service/kubernetes            ClusterIP   10.96.0.1       <none>        443/TCP          4d1h
service/postgres-service-np   NodePort    10.105.135.29   <none>        5432:30100/TCP   6s

NAME                               READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/psql-itwl-dev-01   1/1     1            1           6s

NAME                                          DESIRED   CURRENT   READY   AGE
replicaset.apps/psql-itwl-dev-01-594c7468c7   1         1         1       6s

See the Kubernetes Postgres Pod log file

$ kubectl logs 
$ kubectl logs pod/psql-itwl-dev-01-594c7468c7-p9k9l -f
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /var/lib/postgresql/data/k8s ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok

WARNING: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
syncing data to disk ... ok

Success. You can now start the database server using:

    pg_ctl -D /var/lib/postgresql/data/k8s -l logfile start

waiting for server to start....2019-06-10 17:59:41.009 UTC [42] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2019-06-10 17:59:41.104 UTC [43] LOG:  database system was shut down at 2019-06-10 17:59:40 UTC
2019-06-10 17:59:41.141 UTC [42] LOG:  database system is ready to accept connections
 done
server started
CREATE DATABASE


/usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*

2019-06-10 17:59:48.590 UTC [42] LOG:  received fast shutdown request
waiting for server to shut down...2019-06-10 17:59:48.597 UTC [42] LOG:  aborting any active transactions
.2019-06-10 17:59:48.603 UTC [42] LOG:  background worker "logical replication launcher" (PID 49) exited with exit code 1
2019-06-10 17:59:48.603 UTC [44] LOG:  shutting down
2019-06-10 17:59:48.715 UTC [42] LOG:  database system is shut down
 done
server stopped

PostgreSQL init process complete; ready for start up.

2019-06-10 17:59:48.854 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2019-06-10 17:59:48.854 UTC [1] LOG:  listening on IPv6 address "::", port 5432
2019-06-10 17:59:48.865 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2019-06-10 17:59:48.954 UTC [60] LOG:  database system was shut down at 2019-06-10 17:59:48 UTC
2019-06-10 17:59:48.997 UTC [1] LOG:  database system is ready to accept connections

Test the Kubernetes Postgres database

Use the Postgres client psql (apt-get install -y postgresql-client)

Password is pass

# psql db -h 192.168.50.11 -p 30100 -U user
Password for user user: 
psql (10.8 (Ubuntu 10.8-0ubuntu0.18.10.1), server 11.3 (Debian 11.3-1.pgdg90+1))
WARNING: psql major version 10, server major version 11.
         Some psql features might not work.
Type "help" for help.

db=#

Create a Table and add data

db=# CREATE TABLE COLOR(
db(#    ID SERIAL PRIMARY KEY   NOT NULL,
db(#    NAME           TEXT     NOT NULL UNIQUE,
db(#    RED            SMALLINT NOT NULL,
db(#    GREEN          SMALLINT NOT NULL,
db(#    BLUE           SMALLINT NOT NULL
db(# );
CREATE TABLE
db=# 
db=# INSERT INTO COLOR (NAME,RED,GREEN,BLUE) VALUES('GREEN',0,128,0);
INSERT 0 1
db=# INSERT INTO COLOR (NAME,RED,GREEN,BLUE) VALUES('RED',255,0,0);
INSERT 0 1
db=# INSERT INTO COLOR (NAME,RED,GREEN,BLUE) VALUES('BLUE',0,0,255);
INSERT 0 1
db=# INSERT INTO COLOR (NAME,RED,GREEN,BLUE) VALUES('WHITE',255,255,255);
INSERT 0 1
db=# INSERT INTO COLOR (NAME,RED,GREEN,BLUE) VALUES('YELLOW',255,255,0);
INSERT 0 1
db=# INSERT INTO COLOR (NAME,RED,GREEN,BLUE) VALUES('LIME',0,255,0);
INSERT 0 1
db=# INSERT INTO COLOR (NAME,RED,GREEN,BLUE) VALUES('BLACK',255,255,255);
INSERT 0 1
db=# INSERT INTO COLOR (NAME,RED,GREEN,BLUE) VALUES('GRAY',128,128,128);
INSERT 0 1
db=# 
db=# SELECT * FROM COLOR;
 id |  name  | red | green | blue 
----+--------+-----+-------+------
  1 | GREEN  |   0 |   128 |    0
  2 | RED    | 255 |     0 |    0
  3 | BLUE   |   0 |     0 |  255
  4 | WHITE  | 255 |   255 |  255
  5 | YELLOW | 255 |   255 |    0
  6 | LIME   |   0 |   255 |    0
  7 | BLACK  | 255 |   255 |  255
  8 | GRAY   | 128 |   128 |  128
(8 rows)

db=#

Delete and Recreate the Postgres Instance

You can now destroy the Postgres instance and create it again, the data is preserved.

$ kubectl delete -f postgresql.yaml 
configmap "psql-itwl-dev-01-cm" deleted
persistentvolume "psql-itwl-dev-01-pv" deleted
persistentvolumeclaim "psql-itwl-dev-01-pvc" deleted
deployment.apps "psql-itwl-dev-01" deleted
service "postgres-service-np" deleted

$ kubectl apply -f postgresql.yaml 
configmap/psql-itwl-dev-01-cm created
persistentvolume/psql-itwl-dev-01-pv created
persistentvolumeclaim/psql-itwl-dev-01-pvc created
deployment.apps/psql-itwl-dev-01 created
service/postgres-service-np created

$ kubectl get all
NAME                                    READY   STATUS              RESTARTS   AGE
pod/psql-itwl-dev-01-594c7468c7-7hrzt   0/1     ContainerCreating   0          4s

NAME                          TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)          AGE
service/kubernetes            ClusterIP   10.96.0.1      <none>        443/TCP          4d1h
service/postgres-service-np   NodePort    10.100.67.43   <none>        5432:30100/TCP   3s

NAME                               READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/psql-itwl-dev-01   0/1     1            0           4s

NAME                                          DESIRED   CURRENT   READY   AGE
replicaset.apps/psql-itwl-dev-01-594c7468c7   1         1         0       4s

$ kubectl logs pod/psql-itwl-dev-01-594c7468c7-7hrzt -f
2019-06-10 18:05:51.456 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2019-06-10 18:05:51.457 UTC [1] LOG:  listening on IPv6 address "::", port 5432
2019-06-10 18:05:51.466 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2019-06-10 18:05:51.547 UTC [22] LOG:  database system was interrupted; last known up at 2019-06-10 18:04:54 UTC
2019-06-10 18:05:53.043 UTC [22] LOG:  database system was not properly shut down; automatic recovery in progress
2019-06-10 18:05:53.055 UTC [22] LOG:  redo starts at 0/1676B10
2019-06-10 18:05:53.055 UTC [22] LOG:  invalid record length at 0/1676BB8: wanted 24, got 0
2019-06-10 18:05:53.055 UTC [22] LOG:  redo done at 0/1676B48
2019-06-10 18:05:53.087 UTC [1] LOG:  database system is ready to accept connections

Use the psql client to check that the database tables have been preserved:

# psql db -h 192.168.50.11 -p 30100 -U user
Password for user user: 
psql (10.8 (Ubuntu 10.8-0ubuntu0.18.10.1), server 11.3 (Debian 11.3-1.pgdg90+1))
WARNING: psql major version 10, server major version 11.
         Some psql features might not work.
Type "help" for help.

db=# select * from COLOR;
 id |  name  | red | green | blue 
----+--------+-----+-------+------
  1 | GREEN  |   0 |   128 |    0
  2 | RED    | 255 |     0 |    0
  3 | BLUE   |   0 |     0 |  255
  4 | WHITE  | 255 |   255 |  255
  5 | YELLOW | 255 |   255 |    0
  6 | LIME   |   0 |   255 |    0
  7 | BLACK  | 255 |   255 |  255
  8 | GRAY   | 128 |   128 |  128
(8 rows)

db=#
Running Postgres in Kubernetes with a Persistent NFS Volume

Leave a Reply

Your email address will not be published. Required fields are marked *
Kubernetes Tutorials

Installing Istio on Kubernetes

How to install Istio in a Kubernetes Cluster to use it as a service mesh for a microservices architecture.