Serverless and Knative – Part 2: Knative Serving

In the first part of this series I went through the installation of Knative on CodeReady Containers which is basically Red Hat OpenShift 4.4 running on a notebook.

In this second part I will cover Knative Serving, which is responsible for deploying and running containers, also networking and auto-scaling. Auto-scaling allows scale to zero and is probably the main reason why Knative is referred to as Serverless platform.

https://pbs.twimg.com/profile_images/1022537350562250752/m5EQknfW_400x400.jpg

Before digging into Knative Serving let me share a piece of information from the Knative Runtime Contract which helps to position Knative. It compares Kubernetes workloads (general-purpose containers) with Knative workloads (stateless request-triggered containers):

In contrast to general-purpose containers, stateless request-triggered (i.e. on-demand) autoscaled containers have the following properties:

  • Little or no long-term runtime state (especially in cases where code might be scaled to zero in the absence of request traffic).
  • Logging and monitoring aggregation (telemetry) is important for understanding and debugging the system, as containers might be created or deleted at any time in response to autoscaling.
  • Multitenancy is highly desirable to allow cost sharing for bursty applications on relatively stable underlying hardware resources.

Or in my own words: Knative sees itself better suited for short running processes. You need to provide central logging and monitoring because the pods come and go. And multi-tenenant hardware can be provided large enough to scale for peaks and at the same time make effective use of the resources.

As a developer, I would expect Knative to make my life easier (Knative claims that it is “abstracting away the complex details and enabling developers to focus on what matters”) but instead when coming from Kubernetes it gets more complicated and confusing at first because Knative uses new terminology for its resources. They are:

  1. Service: Responsible for managing the life cycle of an application/workload. Creates and owns the other Knative objects Route and Configuration.
  2. Route: Maps a network endpoint to one or multiple Revisions. Allows Traffic Management.
  3. Configuration: Desired state of the workload. Creates and maintains Revisions.
  4. Revision: A specific version of a code deployment. Revisions are immutable. Revisions can be scaled up and down. Rules can be applied to the Route to direct traffic to specific Revisions.
(c) knative.dev

Did I already mention that this is confusing? We now need to distinguish between Kubernetes services and Knative services. And on OpenShift, between OpenShift Routes and Knative Routes.

Enough complained, here starts the interesting part:

Creating a sample application

I am following this example from the Knative web site which is a simple Hello World type of application written in Node.js. The sample is also available in Java, Go, PHP, Python, Ruby, and some other languages.

Instead of using the Docker build explained in the example I am using an OpenShift Binary build which builds the Container image on OpenShift and stores it as an Image stream in the OpenShift Image Repository. Of course, the Container image could also be on Docker Hub or Quay.io or any other repository that you can access. If you follow the Knative example step by step, you create the Node.js application, a Dockerfile, and some more files. On OpenShift, for the Binary build, we need the application code and the Dockerfile and then create an OpenShift project and the Container image with these commands:

$ oc new-project knativetutorial
$ oc new-build --name helloworld --binary --strategy docker
$ oc start-build helloworld --from-dir=.

Deploying an app as Knative Service

Next I continue with the Knative example. This is the service.yaml file required to deploy the ‘helloworld’ example as a Knative Service:

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: helloworld-nodejs
spec:
  template:
    metadata:
      name: helloworld-nodejs-v1
    spec:
      containers:
        - image: image-registry.openshift-image-registry.svc:5000/knativetutorial/helloword:latest
          env:
            - name: TARGET
              value: "Node.js Sample v1"

If you are familiar with Kubernetes, you have to start to pay close attention to the first line, to see that this is the definition of a Knative Service.

All you need for your deployment are the highlighted lines, specifically the first ‘metadata’.’name’ and the ‘containers’.’images’ specification to tell Kubernetes where to find the Container image.

Line 11 specifies the location of the Container image just like every other Kubernetes deployment description. In this example, the ‘helloworld’ image is the Image stream in the OpenShift internal Image Repository in a project called ‘knativetutorial’. It is the result of the previous section “Creating a sample application”.

Lines 12, 13, and 14 are setting an environment variable and are used to “create” different versions. (In the Hello World code, the variable TARGET represents the “World” part.)

Lines 7 and 8, ‘metadata’ and ‘name’, are optional but highly recommended. They are used to provide arbitrary names for the Revisions. If you omit this second name, Knative will use default names for the Revisions (“helloworld-nodejs-xhz5df”) and if you have more than one version/revision this makes it difficult to distinguish between them.

With CRC and Knative correctly set up, I simply deploy the service using oc:

$ oc apply -f service.yaml
service.serving.knative.dev/helloworld-nodejs created

The reply isn’t very spectacular but if you look around (oc get all) you can see that a lot has happened:

  1. A Kubernetes Pod is created, running two containers: user-container and Envoy
  2. Multiple Kubernetes services are created, one is equipped with an OpenShift route
  3. An OpenShift Route is created
  4. A Kubernetes deployment and a replica-set are created
  5. Knative service, configuration, route, and revision objects are created

It would have taken a YAML file with a lot more definitions and specifications to accomplish all that with plain Kubernetes. I would say that the Knative claim of “abstracting away the complex details and enabling developers to focus on what matters” is definitely true!

Take a look at the OpenShift Console, in the Developer, Topology view:

I really like the way the Red Hat OpenShift developers have visualized Knative objects here.

If you click on the link (Location) of the Route, you will see the helloworld-nodejs response in a browser:

If you wait about a minute or so, the Pod will terminate: “All Revisions are autoscaled to 0”. If you click on the Route location (URL) then, a Pod will be spun up again.

Another good view of the Knative service is available through the kn CLI tool:

$ kn service list
NAME                URL                                                         LATEST                 AGE   CONDITIONS   READY   REASON
helloworld-nodejs   http://helloworld-nodejs-knativetutorial.apps-crc.testing   helloworld-nodejs-v1   13m   3 OK / 3     True  

$ kn service describe helloworld-nodejs
Name:       helloworld-nodejs
Namespace:  knativetutorial
Age:        15m
URL:        http://helloworld-nodejs-knativetutorial.apps-crc.testing

Revisions:  
  100%  @latest (helloworld-nodejs-v1) [1] (15m)
        Image:  image-registry.openshift-image-registry.svc:5000/knativetutorial/helloword:latest (at 53b1b4)

Conditions:  
  OK TYPE                   AGE REASON
  ++ Ready                  15m 
  ++ ConfigurationsReady    15m 
  ++ RoutesReady            15m 

Adding a new revision

I will now create a second version of our app and deploy it as a second Revision using a new file, service-v2.yaml:

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: helloworld-nodejs
spec:
  template:
    metadata:
      name: helloworld-nodejs-v2
    spec:
      containers:
        - image: image-registry.openshift-image-registry.svc:5000/knativetutorial/helloword:latest
          env:
            - name: TARGET
              value: "Node.js Sample v2 -- UPDATED"

I have changed the revision number to ‘-v2’ and modified the environment variable TARGET so that we can see which “version” is called. Apply with:

$ oc apply -f service-v2.yaml
service.serving.knative.dev/helloworld-nodejs configured

Checking with the kn CLI we can see that Revision ‘-v2’ is now used:

$ kn service describe helloworld-nodejs
Name:       helloworld-nodejs
Namespace:  knativetutorial
Age:        21m
URL:        http://helloworld-nodejs-knativetutorial.apps-crc.testing

Revisions:  
  100%  @latest (helloworld-nodejs-v2) [2] (23s)
        Image:  image-registry.openshift-image-registry.svc:5000/knativetutorial/helloword:latest (at 53b1b4)

Conditions:  
  OK TYPE                   AGE REASON
  ++ Ready                  18s 
  ++ ConfigurationsReady    18s 
  ++ RoutesReady            18s 

It is visible in the OpenShift Web Console, too:

Revision 2 has now fully replaced Revision 1.

Traffic Management

What if we want to Canary test Revision 2? It is just a simple modification in the YAML:

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: helloworld-nodejs
spec:
  template:
    metadata:
      name: helloworld-nodejs-v2
    spec:
      containers:
        - image: image-registry.openshift-image-registry.svc:5000/knativetutorial/helloword:latest
          env:
            - name: TARGET
              value: "Node.js Sample v2 -- UPDATED"
  traffic:
    - tag: v1
      revisionName: helloworld-nodejs-v1
      percent: 75
    - tag: v2
      revisionName: helloworld-nodejs-v2
      percent: 25

This will create a 75% / 25% distribution between revision 1 and 2. Deploy the change and watch in the OpenShift Web Console:

Have you ever used Istio? To accomplish this with Istio requires configuring the Ingress Gateway plus defining a Destination Rule and a Virtual Service. In Knative it is just adding a few lines of code to the Service description. Have you noticed the “Set Traffic Distribution” button in the screen shot of the OpenShift Web Console? Here you can modify the distribution on the fly:

Auto-Scaling

Scale to zero is an interesting feature but without additional tricks (like pre-started containers or pods which aren’t available in Knative) it can be annoying because users have to wait until a new pod is started and ready to receive requests. Or it can lead to problems like time-outs in a microservices architecture if a scaled-to-zero service is called by another service and has to be started first.

On the other hand, if our application / microservice is hit hard with requests, a single pod may not be sufficient to serve them and we may need to scale up. And preferably scale up and down automatically.

Auto-scaling is accomplished by simply adding a few annotation statements to the Knative Service description:

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: helloworld-nodejs
spec:
  template:
    metadata:
      name: helloworld-nodejs-v3
      annotations:
        # the minimum number of pods to scale down to
        autoscaling.knative.dev/minScale: "1"
        # the maximum number of pods to scale up to
        autoscaling.knative.dev/maxScale: "5"
        # Target in-flight-requests per pod.
        autoscaling.knative.dev/target: "1"
    spec:
      containers:
        - image: image-registry.openshift-image-registry.svc:5000/knativetutorial/helloword:latest
          env:
            - name: TARGET
              value: "Node.js Sample v3 -- Scaling"

minScale: “1” prevents scale to zero, there will always be at least 1 pod active.
maxScale: “5” will allow to start a maximum of 5 pods.
target: “1” limits every started pod to 1 concurrent request at a time, this is just to make it easier to demo.

All auto-scale parameters are listed and described here.

Here I deployed the auto-scale example and run a load test using the hey command against it:

$ hey -z 30s -c 50 http://helloworld-nodejs-knativetutorial.apps-crc.testing/

Summary:
  Total:	30.0584 secs
  Slowest:	1.0555 secs
  Fastest:	0.0032 secs
  Average:	0.1047 secs
  Requests/sec:	477.1042
  
  Total data:	501935 bytes
  Size/request:	35 bytes

Response time histogram:
  0.003 [1]	    |
  0.108 [9563]	|■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
  0.214 [3308]	|■■■■■■■■■■■■■■
  0.319 [899]	|■■■■
  0.424 [367]	|■■
  0.529 [128]	|■
  0.635 [42]	|
  0.740 [15]	|
  0.845 [10]	|
  0.950 [5]	    |
  1.056 [3]	    |


Latency distribution:
  10% in 0.0249 secs
  25% in 0.0450 secs
  50% in 0.0776 secs
  75% in 0.1311 secs
  90% in 0.2157 secs
  95% in 0.2936 secs
  99% in 0.4587 secs

Details (average, fastest, slowest):
  DNS+dialup:	0.0001 secs, 0.0032 secs, 1.0555 secs
  DNS-lookup:	0.0001 secs, 0.0000 secs, 0.0197 secs
  req write:	0.0000 secs, 0.0000 secs, 0.0079 secs
  resp wait:	0.1043 secs, 0.0031 secs, 1.0550 secs
  resp read:	0.0002 secs, 0.0000 secs, 0.3235 secs

Status code distribution:
  [200]	14341 responses


$ oc get pod
NAME                                               READY   STATUS    RESTARTS   AGE
helloworld-nodejs-v3-deployment-66d7447b76-4dhql   2/2     Running   0          28s
helloworld-nodejs-v3-deployment-66d7447b76-pvxqg   2/2     Running   0          29s
helloworld-nodejs-v3-deployment-66d7447b76-qxkbc   2/2     Running   0          28s
helloworld-nodejs-v3-deployment-66d7447b76-vhc69   2/2     Running   0          28s
helloworld-nodejs-v3-deployment-66d7447b76-wphwm   2/2     Running   0          2m35s

In the end of the output we see 5 pods are started, one of them for a longer time (2m 35s) than the rest. That is the minScale: “1” pre-started pod.

Jakarta EE Example from Cloud Native Starter

I wanted to see how easy it is to deploy any form of application using Knative Serving.

I used the authors-java-jee microservice that is part of our Cloud Native Starter project and that we use in an exercise of an OpenShift workshop. A Container image of this service is stored on Dockerhub in my colleague Niklas Heidloffs registry as nheidloff/authors:v1

This is the Knative service.yaml:

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: authors-jee
spec:
  template:
    metadata:
      name: authors-jee-v1
    spec:
      containers:
      - image: docker.io/nheidloff/authors:v1

When I deployed this I noticed that it never starts (you need to scroll the following view to the right to see the problem):

$ kn service list
NAME          URL                                                   LATEST   AGE   CONDITIONS   READY     REASON
authors-jee   http://authors-jee-knativetutorial.apps-crc.testing            33s   0 OK / 3     Unknown   RevisionMissing : Configuration "authors-jee" is waiting for a Revision to become ready.

$ oc get pod
NAME                                         READY   STATUS    RESTARTS   AGE
authors-jee-v1-deployment-7dd4b989cf-v9sv9   1/2     Running   0          42s

The user-container in the pod never starts and the Revision never becomes ready. Why is that?

To understand this problem you have to know that there are two versions of the authors service: The first version is written in Node.js and listens on port 3000. The second version is the JEE version we try to deploy here. To make it a drop-in replacement for the Node.js version it is configured to listen on port 3000, too. Very unusual for JEE and something Knative obviously does not pick up from the Docker metadata in the image.

The Knative Runtime Contract has some information about Inbound Network Connectivity, Protocols and Ports:

“The developer MAY specify this port at deployment; if the developer does not specify a port, the platform provider MUST provide a default. Only one inbound containerPort SHALL be specified in the core.v1.Container specification. The hostPort parameter SHOULD NOT be set by the developer or the platform provider, as it can interfere with ingress autoscaling. Regardless of its source, the selected port will be made available in the PORT environment variable.”

I modified the Knative service yaml with ports.containerPort info:

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: authors-jee
spec:
  template:
    metadata:
      name: authors-jee-v2
    spec:
      containers:
      - image: docker.io/nheidloff/authors:v1
        ports:
        - containerPort: 3000

Note the Revision ‘-v2’! Check after deployment:

$ kn service list
NAME          URL                                                   LATEST           AGE   CONDITIONS   READY   REASON
authors-jee   http://authors-jee-knativetutorial.apps-crc.testing   authors-jee-v2   11m   3 OK / 3     True    

$ oc get pod
NAME                                        READY   STATUS    RESTARTS   AGE
authors-jee-v2-deployment-997d44565-mhn7w   2/2     Running   0          51s

The authors-java-jee microservice is using Eclipse Microprofile and has implemented specific health checks. They can be used as Kubernetes readiness and liveness probes, the YAML file then looks like this, syntax is exactly the standard Kubernetes syntax:

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: authors-jee
spec:
  template:
    metadata:
      name: authors-jee-v2
    spec:
      containers:
      - image: docker.io/nheidloff/authors:v1
        ports:
        - containerPort: 3000
        livenessProbe:
          exec:
            command: ["sh", "-c", "curl -s http://localhost:3000/"]
          initialDelaySeconds: 20
        readinessProbe:
          exec:
            command: ["sh", "-c", "curl -s http://localhost:3000/health | grep -q authors"]
          initialDelaySeconds: 40

Microservices Architectures and Knative private services

So far the examples I tested where all exposed on public URLs using the Kourier Ingress Gateway. This is useful for testing and also for externally accessible microservices, e.g. backend-for-frontend services that serve a browser-based web front end or a REST API for other external applications. The multitude of microservices in a cloud native application will only and should only be called cluster local and not be exposed with an external URL.

The Knative documentation has information on how to label a service cluster-local. You can either add a label to the Knative service or the Knative route. The steps described in the documentation are to 1. deploy the service and then 2. convert it to cluster-local via the label.

You can easily add the label to the YAML file and immediately deploy a cluster-local Knative service. This is the modified Jakarta EE example of the previous section:

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: authors-jee
  labels:
    serving.knative.dev/visibility: cluster-local
spec:
  template:
    metadata:
      name: authors-jee-v2
    spec:
      containers:
      - image: docker.io/nheidloff/authors:v1
        ports:
        - containerPort: 3000

When this is deployed to OpenShift, the correct URL shows up in the Route:

Of course you can no longer open the URL in your browser, this address is only available from within the Kubernetes cluster.

Debugging Tips

There are new places to look for information as to why a Knative service doesn’t work. Here is a list of helpful commands and examples:

  1. Display the Knative service:
$ kn service list
NAME          URL                                                   LATEST   AGE    CONDITIONS   READY   REASON
authors-jee   http://authors-jee-knativetutorial.apps-crc.testing            3m7s   0 OK / 3     False   RevisionMissing : Configuration "authors-jee" does not have any ready Revision.

It is normal and to be expected that the revision is not available for some time immediately after the deployment because the application container needs to start first. But in this example the revision isn’t available after over 3 minutes and that is not normal.

You can also display Knative service info using oc instead of kn by using ‘kservice’:

$ oc get kservice
NAME          URL                                                   LATESTCREATED    LATESTREADY   READY   REASON
authors-jee   http://authors-jee-knativetutorial.apps-crc.testing   authors-jee-v2                 False   RevisionMissing

2. Check the pod:

$ oc get pod
No resources found in knativetutorial namespace.

That is bad: no pod means no logs to look at.

3. Get information about the revision:

$ oc get revision
NAME             CONFIG NAME   K8S SERVICE NAME   GENERATION   READY   REASON
authors-jee-v2   authors-jee                      1            False   ContainerMissing

$ oc get revision authors-jee-v2 -o yaml
apiVersion: serving.knative.dev/v1
kind: Revision
[...]
status:
  conditions:
  - lastTransitionTime: "2020-06-03T08:12:49Z"
    message: 'Unable to fetch image "docker.io/nheidloff/authors:1": failed to resolve
      image to digest: failed to fetch image information: GET https://index.docker.io/v2/nheidloff/authors/manifests/1:
      MANIFEST_UNKNOWN: manifest unknown; map[Tag:1]'
    reason: ContainerMissing
    status: "False"
    type: ContainerHealthy
  - lastTransitionTime: "2020-06-03T08:12:49Z"
    message: 'Unable to fetch image "docker.io/nheidloff/authors:1": failed to resolve
      image to digest: failed to fetch image information: GET https://index.docker.io/v2/nheidloff/authors/manifests/1:
      MANIFEST_UNKNOWN: manifest unknown; map[Tag:1]'
    reason: ContainerMissing
    status: "False"
    type: Ready
  - lastTransitionTime: "2020-06-03T08:12:47Z"
    status: Unknown
    type: ResourcesAvailable
[...]

The conditions under the status topic show that I have (on purpose as a demo) mistyped the Container image tag.

This is a real example:

$ oc get revision helloworld-nodejs-v1 -o yaml
[...]
status:
  conditions:
  - lastTransitionTime: "2020-05-28T06:42:14Z"
     message: The target could not be activated.
     reason: TimedOut
     severity: Info
     status: "False"
     type: Active
  - lastTransitionTime: "2020-05-28T06:40:04Z"
     status: Unknown
     type: ContainerHealthy
  - lastTransitionTime: "2020-05-28T06:40:05Z"
     message: '0/1 nodes are available: 1 Insufficient cpu.'
     reason: Unschedulable
     status: "False"
     type: Ready
  - lastTransitionTime: "2020-05-28T06:40:05Z"
     message: '0/1 nodes are available: 1 Insufficient cpu.'
     reason: Unschedulable
       status: "False"
       type: ResourcesAvailable

These conditions clearly show that the cluster is under CPU pressure and unable to schedule a new pod. This was on my first CRC configuration that used only 6 vCPUs.

In my next blog article in this series I will talk about Knative Eventing, stay tuned.

Serverless and Knative – Part 1: Installing Knative on CodeReady Containers

I have worked with Kubernetes for quite some time now, also with Istio Service Mesh. Recently I decided that I want to explore Knative and its possibilities.

https://pbs.twimg.com/profile_images/1022537350562250752/m5EQknfW_400x400.jpg

So what is Knative? The Knative web site describes it as “components build on top of Kubernetes, abstracting away the complex details and enabling developers to focus on what matters.” It has two distinct components, originally it were three:

  1. Knative Build. It is no longer part of Knative, it is now a project of its own: “Tekton
  2. Knative Serving, responsible for deploying and running containers, also networking and auto-scaling. Auto-scaling allows scale to zero and is the main reason why Knative is referred to as Serverless platform.
  3. Knative Eventing, connecting Knative services (deployed by Knative Serving) with events or streams of events.

This will be a series of blogs about installing Knative, Knative Serving, and Knative Eventing.

In order to explore Knative you need to have access to an instance, of course, and that may require installing it yourself. The Knative documentation (for v0.12) has instructions on how to install it on many different Kubernetes platforms, including Minikube. Perfect, Knative on my notebook.

Installation

I followed the instructions for Minikube and installed it, and started a tutorial. At some point, I finished for the day, and stopped Minikube. The next morning it wouldn’t start again. I tried to find out what went wrong and in the end deleted the Minikube profile, recreated it, and reinstalled Knative again. Just out of curiosity I restarted Minikube and ran into the very same problem. This time I was a little more successful with my investigation and found this issue: https://github.com/knative/eventing/issues/2544. I thought about moving to Knative 0.14 shortly but then decided to test it on OpenShift. If you read some of my previous blogs you may know that I am a fan of CodeReady Containers (CRC).

Knative on Red Hat OpenShift is called OpenShift Serverless. It has been a preview (“beta”) for quite some time but since end of April 2020 it is GA, generally available, no longer preview only. According to the Red Hat OpenShift documentation OpenShift Serverless v1.7.0 is based on Knative 0.13.2 (as of May 1st, 2020) and it is tested on OpenShift 4.3 and 4.4. The CRC version I am currently using (v1.10) is built on top of OpenShift 4.4. So it should work.

The hardware or cluster size requirements for OpenShift Serverless are steep: minimum 10 CPUs and 40 GB of RAM. I only have 8 vCPUs (4 cores) and 32 GB of RAM in my notebook and I do need to run an Operating System besides CRC but I thought I give it a try. I started Knative installation on a CRC config using 6 vCPUs and 20 GB of RAM and so far it seems to work. I have tried it on smaller configurations and got unschedulable pods (Memory and/or CPU pressure).

Installation is accomplished via an OpenShift Serverless Operator and it took me probably less then 20 minutes to have both Knative Serving and Eventing installed by just following the instructions:

  1. Install the OpenShift Serverless operator
  2. Create a namespace for Knative Serving
  3. Create Knative Serving via the Serverless operators API. This also installs Kourier as “an open-source lightweight Knative Ingress based on Envoy.” Kourier is a lightweight replacement for Istio.
  4. Create a namespace for Knative Eventing
  5. Create Knative Eventing via the Serverless operators API.

I have started and stopped CRC many times now and it doesn’t have the issues that Minikube had.

As a future exercise I will test the Knative Add-on for the IBM Cloud Kubernetes Service. This installs Knative 0.14 together with Istio on top of Kubernetes and requires a minimum of 3 worker nodes with 4 CPUs and 16 GB om memory (b3c.4×16 is the machine specification).

In the next blog article I will cover Knative Serving with an example from the Knative documentation.

Two great additions to ‘kubectl’

I started to learn Kubernetes in its vanilla form. Almost a year ago I made my first steps on Red Hat OpenShift. From then on, going back to vanilla Kubernetes made me miss the easy way you switch namespaces (aka projects) in OpenShift. With ‘oc project’ it is like switching directories on your notebook. You can do that with ‘kubectl’ somehow but it is not as simple.

Recently I found 2 power tools for kubectl: ‘kubectx’ and ‘kubens’. Ahmet Alp Balkan, a Google Software Engineer, created them and open sourced them (https://github.com/ahmetb/kubectx).

The Github repo has installation instructions for macOS and diferent flavours of Linux. When you install them, also make sure to install ‘fzf’ (“A command-line fuzzy finder”, https://github.com/junegunn/fzf), it is a cool addition.

kubens

‘kubens’ allows you to quickly switch namespaces in Kubernetes. Normally you work in ‘default’ and whenever you need to check something or do something in another namespace you need to add the ‘-n namespace’ parameter to your command.

‘kubens istio-system’ will make ‘istio-system’ your new home and a subsequent ‘kubectl get pod’ or ‘kubectl get svc’ will show the pods and services in istio-system. Thats not all.

‘kubens’ without a parameter will list all namespaces and with ‘fzf’ installed too you have a selectable list:

I think that is even better than ‘oc projects’!

kubectx

‘kubectx’ is really helpful when you work with multiple Kubernetes clusters. I typically work with a Kubernetes cluster on the IBM Cloud (IKS) and then very often start CRC (CodeReady Containers) to try something out on OpenShift. When I log into OpenShift, my connection to the IKS cluster drops. It actually doesn’t drop but the kube context is switched to CRC. With ‘kubectx’ you can switch between them.

In this example I have two contexts, one is CRC, the other IKS (Kubernetes on IBM Cloud):

Not exactly easy to know which one is which, isn’t it? But you can set aliases for the entries like this:

$ kubectx CRC=default/api-crc-testing:6443/kube:admin
$ kubectx IKS=knative/br1td2of0j1q10rc8aj0

And then you get a list with recognizable names:

You can now switch via the list. In addition, with ‘kubectx -‘ you can switch to the previous context.

When you constantly create new kube contexts, e.g. create new CRC or Minikube instances, this list may grow and get unmanageable. But with ‘kubectx -d <NAME>’ you can delete entries from the list. (They will still be in the kube context, though.)

Deploy your Quarkus applications on Kubernetes. Almost automatically!

You want to code Java, not Kubernetes deployment YAML files? And you use Quarkus? You may have seen the announcement blog for Quarkus 1.3.0. Under “much much more” is a feature that is very interesting to everyone using Kubernetes or OpenShift and with a dislike for the required YAML files:

Easy deployment to Kubernetes or OpenShift

The Kubernetes extension has been overhauled and now gives users the ability to deploy their Quarkus applications to Kubernetes or OpenShift with almost no effort. Essentially the extension now also takes care of generating a container image and applying the generated Kubernetes manifests to a target cluster, after the container image has been generated.

Image © quarkus.io

There are two Quarkus extensions required.

  1. Kubernetes Extension
    This extension generates the Kubernetes and OpenShift YAML (or JSON) files and also manages the automatic deployment using these files.
  2. Container Images
    There are actually 3 extensions that can handle automatic build using:
    – Jib
    – Docker
    – OpenShift Source-to-image (s2i)

Both extensions use parameters that are placed into the application.properties file. The parameters are listed in the respective guides of the extensions. Note that I use the term “listed”. Some of these parameters are really just listed without any further explanation.

You can find the list of parameters for the Kubernetes extension here, those for the Container Image extension are here.

I tested the functionality in 4 different scenarios: Minikube, IBM Cloud Kubernetes Service, and Red Hat OpenShift in the form of CodeReady Containers (CRC) and Red Hat OpenShift on IBM Cloud. I will describe all of them here.

Demo Project

I use the simple example from the Quarkus Getting Started Guide as my demo application. The current Quarkus 1.3.1 uses Java 11 and requires Apache Maven 3.6.2+. My notebook runs on Fedora 30 so I had to manually install Maven 3.6.3 because the version provided in the Fedora 30 repositories is too old.

The following command creates the Quarkus Quickstart Demo:

$ mvn io.quarkus:quarkus-maven-plugin:1.3.1.Final:create \
    -DprojectGroupId=org.acme \
    -DprojectArtifactId=config-quickstart \
    -DclassName="org.acme.config.GreetingResource" \
    -Dpath="/greeting"
$ cd config-quickstart

You can run the application locally:

$ ./mvnw compile quarkus:dev

Then test it:

$ curl -w "\n" http://localhost:8080/hello
hello

Now add the Kubernetes and Docker Image extensions:

$ ./mvnw quarkus:add-extension -Dextensions="kubernetes, container-image-docker"

Edit application.properties

The Kubernetes extension will create 3 Kubernetes objects:

  1. Service Account
  2. Service
  3. Deployment

The configuration and naming of these is based on some basic parameters that have to be added in application.properties:

quarkus.kubernetes.part-of=todo-appOne of the Kubernetes “recommended” labels (recommended, not required)
quarkus.container-image.registry=
quarkus.container-image.group=
quarkus.container-image.name=getting-started
quarkus.container-image.tag=1.0
Specifies the container image in the K8s deployment. Result is ‘image: getting-started:1.0’. Make sure there are no excess or trailing spaces! I specify empty registry and group parameters to obtain predictable results.
quarkus.kubernetes.service-type=NodePortCreates a service of type NodePort, default would be ClusterIP (doesn’t really work with Minikube)

Now do a test compile with

$ ./mvnw clean package

This should result in BUILD SUCCESS. Look at the kubernetes.yml file in the target/kubernetes directory.

Every object (ServiceAccount, Service, Deployment) has a set of annotations and labels. The annotations are picked up automatically when the source directory is under version control (e.g. git) and from the last compile time. The labels are picked up from the parameters specified in the table above. You can specify additional parameters but the Kubernetes extensions uses specific defaults:

  • app.kubernetes.io/name and name in the YAML are set to quarkus.container-image.name.
  • app.kubernetes.io/version in the YAML is set to the container-image.tag parameter.

The definition of the port (http, 8080) is picked up by Quarkus from the source code during compile.

Deploy to

With Minikube, we will create the Container (Docker) Image in the Docker installation that is part of the Minikube VM. So after starting Minikube (minikube start) you need to point your local docker command to the Minikube environment:

$ eval $(minikube docker-env)

The Kubernetes extension specifies imagePullPolicy: Always as the default for a container image. This is a problem when using the Minikube Docker environment, it should be never instead. Your application.properites should therefore look like this:

quarkus.kubernetes.part-of=todo-app
quarkus.container-image.registry=
quarkus.container-image.group=
quarkus.container-image.name=getting-started
quarkus.container-image.tag=1.0
quarkus.kubernetes.image-pull-policy=never
quarkus.kubernetes.service-type=NodePort

Now try a test build & deploy in the getting-started directory:

$ ./mvnw clean package -Dquarkus.kubernetes.deploy=true

Check that everything is started with:

$ kubectl get pod 
$ kubectl get deploy
$ kubectl get svc

Note that in the result of the last command you can see the NodePort of the getting-started service, e.g. 31304 or something in that range. Get the IP address of your Minikube cluster:

$ minikube ip

And then test the service, in my example with:

$ curl 192.168.39.131:31304/hello
hello

The result of this execise:

Installing 2 Quarkus extensions and adding 7 statements to the application.properties file (of which 1 is optional) allows you to compile your Java code, build a container image, and deploy it into Kubernetes with a single command. I think this is cool!

What I just described for Minikube also works for the IBM Cloud. IBM Cloud Kubernetes Service (or IKS) does not have an internal Container Image Registry, instead this is a separate service and you may have guessed its name: IBM Cloud Container Registry (ICR). This example works on free IKS clusters, too. A free IKS cluster is free of charge and you can use for 30 days.

For our example to work, you need to create a “Namespace” in an ICR location which is different from a Kubernetes namespace. For example, my test Kubernetes cluster (with the name: mycluster) is located in Houston, so I create a namespace called ‘harald-uebele’ in the registry location Dallas (because it is close to Houston).

Now I need to login and setup the connection using the ibmcloud CLI:

$ ibmcloud login
$ ibmcloud ks cluster config --cluster mycluster
$ ibmcloud cr login
$ ibmcloud cr region-set us-south

The last command will set the registry region to us-south which is Dallas and has the URL ‘us.icr.io’.

application.properties needs a few changes:

  • registry now holds the ICR URL (us.icr.io)
  • group is the registry namespace mentioned above
  • image-pull-policy is changed to always for ICR
  • service-account needs to be ‘default’, the service account created by the Kubernetes extension (‘getting-started’) is not allowed to pull images from the ICR image registry
quarkus.kubernetes.part-of=todo-app
quarkus.container-image.registry=us.icr.io
quarkus.container-image.group=harald-uebele
quarkus.container-image.name=getting-started
quarkus.container-image.tag=1.0
quarkus.kubernetes.image-pull-policy=always
quarkus.kubernetes.service-type=NodePort
quarkus.kubernetes.service-account=default

Compile & build as before:

$ ./mvnw clean package -Dquarkus.kubernetes.deploy=true

Check if the image has been built:

$ ibmcloud cr images

You should see the newly created image, correctly tagged, and hopefully with a ‘security status’ of ‘No issues’. That is the result of a Vulnerability Advisor scan that is automatically performed on every image.

Now check the status of your deployment:

$ kubectl get deploy
$ kubectl get pod
$ kubectl get svc

With kubectl get svc you will see the number of the NodePort of the service, in my example it is 30850. You can obtain the public IP address of an IKS worker node with:

$ ibmcloud ks worker ls --cluster mycluster

If you have multiple worker nodes, any of the public IP addresses will do. Test your service with:

$ curl <externalIP>:<nodePort>/hello

The result should be ‘hello’.

All this also works on

Red Hat OpenShift

I have tested this with CodeReady Containers (CRC) and on Red Hat OpenShift on IBM Cloud. CRC was a bit flaky, sometimes it would build the image, create the deployment config but wouldn’t start the pod.

On OpenShift, the container image is built using Source-to-Image (s2i) and this requires a different Maven extension:

$ ./mvnw quarkus:add-extension -Dextensions="container-image-s2i"

It seems like you can have only container-image extensions in your project. If you installed the container-image-docker extension before, you’ll need to remove it from the dependency section of the pom.xml file, otherwise the build may fail, later.

There is an OpenShift specific section of parameters / options is the documentation of the extension.

Start with log in to OpenShift and creating a new project (quarkus):

$ oc login ...
$ oc new-project quarkus

This is the application.properties file I used:

quarkus.kubernetes.deployment-target=openshift
quarkus.container-image.registry=image-registry.openshift-image-registry.svc:5000
quarkus.container-image.group=quarkus
quarkus.container-image.name=getting-started
quarkus.container-image.tag=1.0
quarkus.openshift.part-of=todo-app
quarkus.openshift.service-account=default
quarkus.openshift.expose=true
quarkus.kubernetes-client.trust-certs=true

Line 1: Create an OpenShift deployment
Line 2: This is the (OpenShift internal) image repository URL for OpenShift 4
Line 3: The OpenShift project name
Line 4: The image name will also be used for all other OpenShift objects
Line 5: Image tag, will also be the application version in OpenShift
Line 6: Name of the OpenShift application
Line 7: Use the ‘default’ service account
Line 8: Expose the service with a route (URL)
Line 9: Needed for CRC because of self-signed certificates, don’t use with OpenShift on IBM Cloud

With these options in place, start a compile & build:

$ ./mvnw clean package -Dquarkus.kubernetes.deploy=true

It will take a while but in the end you should see a “BUILD SUCCESS” and in the OpenShift console you should see an application called “todo-app” with a Deployment Config, Pod, Build, Service, and Route:

Additional and missing options

Namespaces (Kubernetes) and Projects (OpenShift) cannot be specified with an option in application.properties. With OpenShift thats not really an issue because you can specify which project (namespace) to work in with the oc CLI before starting the mvn package. But it would be nice if there were a namespace and/or project option.

The Kubernetes extension is picking up which Port your app is using during build. But if you need to specify an additional port this is how you do it:

quarkus.kubernetes.ports.https.container-port=8443

This will add an https port on 8443 to the service and an https containerPort on 8443 to the containers spec in the deployment.

The number of replicas is supposed to be defined with:

quarkus.kubernetes.replicas=4

This results in WARN io.qua.config Unrecognized configuration key “quarkus.kubernetes.replicas” was provided; it will be ignored and the replicas count remains 1 in the deployment. Instead use the deprecated configuration option without quarkus. (I am sure this will be fixed):

kubernetes.replicas=4

Adding a key value pair environment variables to the deployment:

quarkus.kubernetes.env-vars.DB.value=local

will result in this YAML:

    spec:
      containers:
      - env:
        - name: "DB"
          value: "local"

There are many more options, for readiness and liveness probes, mounts and volumes, secrets, config maps, etc. Have a look at the documentation.

Cloud Native and Reactive Microservices on Red Hat OpenShift 4

My colleague Niklas Heidloff has started to create another version of our Cloud Native Starter using a reactive programming model, and he has also written an extensive series of blogs about it starting here. He uses Minikube to deploy the reactive example and I have created documentation and scripts to deploy it on CloudReady Containers (CRC) which is running Red Hat OpenShift 4.

The reactive version of Cloud Native Starter is based on Quarkus (“Supersonic Subatomic Java”), uses Apache Kafka for messaging, and PostgreSQL for data storage of the articles service. Postgres is accessed via the reactive SQL client. Niklas has blogged about all of the details.

Cloud Native Starter Reactive: High Level Architecture

The deployment on OpenShift is very similar to the deployment of the original Cloud Native Starter which I have written about in my last blog.

The services (web-app, web-api, authors, articles) are build locally in Docker, then tagged with an image path suitable for the OpenShift image repository, then pushed with Docker into the internal repository.

Two things are different, though:

  1. The reactive example currently doesn’t require Istio, no need to install it, then.
  2. Kafka and Postgres weren’t used before.

I install Kafka using the Strimzi operator, and Postgres with the Dev4Devs operator.

In the OpenShift OperatorHub catalog, the Strimzi operator is version 0.14.0, we need version 0.15.0. That’s why I use a script to install the Strimzi Kafka operator and then deploy a Kafka cluster into a kafka namespace/project.

The Dev4Devs Postgres operator is installed through the OperatorHub catalog in the OpenShift web console into its own namespace (postgres).

An example Postgres “cluster” with a single pod is deployed via the operator into the same namespace/project.

Using operators makes it so easy to install components into your architecture. The way they are created in this example is not really applicable to production environments but to create test environments for developers its perfect.

Cloud Native Starter on Red Hat OpenShift 4

I have written about Cloud Native Starter many times in this blog. It is a project created by Niklas Heidloff, Thomas Südbröcker, and myself that demonstrates how to get started with cloud-native applications and microservice based architectures. We have started it on Minikube, and ported it to IBM Cloud Kubernetes Service and to Red Hat OpenShift in the form of Minishift and Red Hat OpenShift on IBM Cloud, the last two based on OpenShift version 3.

Cloud Native Starter Vue.js frontend

OpenShift 4 on the IBM Cloud is imminent and Minishift has a successor based on version 4 called CodeReady Containers or CRC. Time to move Cloud Native Starter to OpenShift 4. Here is a summary of my experience.

© Red Hat Inc.

Install CRC

I have blogged about CRC before and back then in September 2019, CRC was version 1.0.0-beta3 and based on OpenShift 4.1.11. Today CRC is version 1.4, and based on OpenShift 4.2.13. It has matured quite a bit. The installation hasn’t changed: CRC is still free of charge, but you need a Red Hat ID (also free) to obtain the pull secrets to install/start it. If you want to use Istio (of course you do!), the minimum requirement of 8 GB memory will not suffice, in my opinion 16 GB of memory are a requirement in this case. Other than that, setting up CRC is done by entering two commands: ‘crc setup’ which checks the prerequisites and does some setup for virtualization and networking, and ‘crc start’ which does the rest. First start takes around 15 minutes. In the end, it will tell you that the cluster is started (hopefully), issue a warning (“The cluster might report a degraded or error state. This is expected since several operators have been disabled to lower the resource usage.”) and give you the credentials to log into OpenShift as kubeadmin and as developer.

Install OpenShift Service Mesh aka Istio

There is a simple way to install Istio — which is called OpenShift Service Mesh by Red Hat — into OpenShift 4. It uses Operators and I already described it in another blog. Service Mesh requires 4 Operators for Elasticsearch, Jaeger, Kiali, and Service Mesh itself. The official documentation still states that you have to install all 4 of them in this sequence. Actually, last time I tried I simply installed the Service Mesh Operator and this pulled the other three without intervention.

While Service Mesh is Istio under the covers, Red Hat has added some features. You can have more than one Istio Control Plane in an OpenShift cluster, and they can have different configurations (demo and production for example). A ‘Member Roll’ then describes which OpenShift projects (namespaces) are a member of a specific Istio Control Plane. With vanilla upstream Istio, a namespace can be tagged to enable ‘automatic sidecar injection’. When a deployment is made to a tagged namespace, an envoy sidecar is then automatically injected into each pod. This is very convenient in Kubernetes but not helpful in OpenShift. Consider constantly doing binary builds: this automatic sidecar injection would inject an envoy into every build pod where it has zero function because this pod will terminate once the build is complete and it doesn’t communicate. Red Hat decided to trigger sidecar injection by adding an annotation to the deployment yaml file:

“Vanilla” Kubernetes/Istio ignores this annotation, there is no problem to have it in yaml files that are used on vanilla Kubernetes/Istio, too.

The telemetry tools and my favorite, Kiali, are integrated into the OpenShift authentication and accessible via a simple OpenShift route (https://kiali-istio-system.apps-crc.testing):

Kiali as part of OpenShift Service Mesh

Access the OpenShift Internal Image Repository

For the CRC/OpenShift 4 port of Cloud Native Starter, I decided to do the container image builds on the local Docker daemon, then tag the resulting image, and push it into the internal image repository of OpenShift. How do you access the internal image repository? You need to login to OpenShift first, the do a ‘docker login’ to the repository:

$ oc login --token=<APITOKEN> --server=https://api.crc.testing:6443
$ docker login -u developer -p $(oc whoami -t) default-route-openshift-image-registry.apps-crc.testing

Problem is that the Docker CLI uses TLS and doesn’t “know” the internal repository. The ‘docker login’ will terminate with a x509 error.

Error response from daemon: Get https://default-route-openshift-image-registry.apps-crc.testing/v2/: x509: certificate signed by unknown authority

CRC uses self signed certificates that Docker doesn’t know about. But you can extract the required certificate and pass it to docker though, I have described the process here.

With the certificate in place for Docker, ‘docker login’ to the OpenShift repository is possible. ‘docker build’ in our scripts is local, the image is then tagged on the local Docker, and in the end push to OpenShift, e.g. for authors-nodejs service:

$ docker build -f Dockerfile -t  authors:1 .
$ docker tag authors:1 default-route-openshift-image-registry.apps-crc.testing/cloud-native-starter/authors:1
$ docker push default-route-openshift-image-registry.apps-crc.testing/cloud-native-starter/authors:1

After that the deployment is standard Kubernetes business with the notable exception that the image name in the deployment YAML file must reflect the location of the image within the OpenShift repository. Of course, our deployment scripts take care of that.

OpenShift and Container Permissions

In Cloud Native Starter there is one service, web-app-vuejs, that provides a Vue.js application, the frontend, to the browser of the user. To do that, Nginx is used as web server. The docker build has two stages: stage 1 builds the Vue.js application with yarn, stage 2 puts the resulting Vue.js into directory /usr/share/nginx/html in the image and Nginx serves this directory at default port 80. Works with vanilla Kubernetes (e.g. Minikube).

The pod reports “CrashLoopBackoff” and never starts in OpenShift. When you look at the logs you’ll notice messages about non-root users and permissions. This image will never run on OpenShift unless you lower security constraints on the project — they were implemented for a reason.

Information on how to solve this problem can be found in the blog Deploy VueJS applications on OpenShift written by Joel Lord:

  1. Start Nginx on a port number above 1024, default port 80 and anything up to 1024 requires root
  2. Move all temporary files (PID, cache, logs, CGI, etc.) to the /tmp directory (can be accessed by everyone
  3. Use /code as base directory

Look here for my modified nginx.conf and Dockerfile.

Result

This is the Cloud Native Starter project in the OpenShift 4 Console:

Project Overview in OpenShift 4

W-JAX 2019 Impressions

I attended the W-JAX 2019 conference beginning of November in Munich. It is a big conference for software developers and had somewhere between 1300 and 1500 participants. Here are some impressions and pictures.

This is the team preparing our booth: unpacking, setting up the background

front: Miriam and Thomas, back: Niklas and myself

Jason McGee, IBM Fellow, Vice President and CTO of IBM Cloud Platform gave a keynote on “The 20 Year Platform – bringing together Kubernetes, 12-Factor and Functions“.

Full house for Jason McGee

Emily Jiang, our MicroProfile hero from the IBM Hursley lab, did On Stage Hacking: “Building a 12-Factor Microservice“:

l2r: Thomas, Miriam, Niklas, Emily, Harald

Grace Jansen, Developer Advocate and also from the IBM Hursley lab, presented “Reacting to the Future of Application Architecture” :

The next IBM session at W-JAX was Niklas Heidloff and myself explaining “How to develop your first cloud-native app in Java“. I never had such an attentive audience asking so many clever questions … maybe giving out swag (T-Shirts with the cool IBM rebus logo) for asking good questions does help 🙂

And finally Jeremias Werner, Senior Software Developer at the IBM Böblingen lab, presented “A peek behind the scenes and how Knative is changing the serverless landscape” (unfortunately the link to his agenda topic doesn’t work: HTTP 404):

Between the talks we were quite busy at the booth:

Some people work at a new career path as movie star 🙂

Thomas is interviewed by the W-JAX team

And of course Blue Cloud Mirror, our Open Source game project, was an attraction at the booth:

The team on day 1:

l2r: Andrea, Niklas, Miguel, Harald, Miriam, Jason, Thomas

Thomas posted a cool video on Twitter:

Feierabendbier at day 2 … and we do the coolest boomerangs!