Creating a your own hosted registry for the docker

1. Download the docker repository
wget https://download.docker.com/linux/centos/docker-ce.repo -O /etc/yum.repos.d/docker.repo

2. Install the docker-ce on the system as
yum install docker-ce -y

3. Create a directory as
mkdir /root/certs

4. Go to the website
sslforfree.com and generate the keys for your domain by manually verifying your domain and copy in the /root/certs directory

5. unzip the certs downloaded from sslforfree.zip
unzip sslforfree.zip
ls -ltr

-rw-r--r--. 1 centos centos 5599 Feb 19 11:11 sslforfree.zip
-rw-r--r--. 1 root   root   1703 Feb 19  2019 private.key
-rw-r--r--. 1 root   root   1922 Feb 19  2019 certificate.crt
-rw-r--r--. 1 root   root   1646 Feb 19  2019 ca_bundle.crt

6. Create the 2 directories as
[root@ip-10-240-43-119 certs]# mkdir -p /opt/registry/data
[root@ip-10-240-43-119 certs]# mkdir -p /var/lib/registry

7. Start and enable the docker service as
[root@ip-10-240-43-119 certs]# systemctl start docker
[root@ip-10-240-43-119 certs]# systemctl enable docker
Created symlink from /etc/systemd/system/multi-user.target.wants/docker.service to /usr/lib/systemd/system/docker.service.
[root@ip-10-240-43-119 certs]#

6. Run your private repsository as
docker run -d -p 443:443 -v /root/certs:/certs -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/certificate.crt -e REGISTRY_HTTP_TLS_KEY=/certs/private.key -e REGISTRY_HTTP_ADDR=0.0.0.0:443 -v /opt/registry/data:/var/lib/registry --name registry registry:2

[root@ip-10-240-43-119 certs]# docker run -d -p 443:443 -v /root/certs:/certs -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/certificate.crt -e REGISTRY_HTTP_TLS_KEY=/certs/private.key -e REGISTRY_HTTP_ADDR=0.0.0.0:443 -v /opt/registry/data:/var/lib/registry --name registry registry:2
Unable to find image 'registry:2' locally
2: Pulling from library/registry
169185f82c45: Pull complete
046e2d030894: Pull complete
188836fddeeb: Pull complete
832744537747: Pull complete
7ceea07e80be: Pull complete
Digest: sha256:870474507964d8e7d8c3b53bcfa738e3356d2747a42adad26d0d81ef4479eb1b
Status: Downloaded newer image for registry:2
2f5bf3270abefe9e2bbdca51ae93b5dd5cc281837b62f24f0bc976a6801e2e41


7. Add the DNS record pointing to your server as
registry.test.unixcloudfusion.in IN A 52.39.129.41

8. We can test access to the registry using curl. The response should provide headers, for example Docker-Distribution-API-Version, indicating the request was processed by the Registry server.

[root@ip-10-240-43-119 certs]# curl -iv https://registry.unixcloudfusion.in/v2/
* About to connect() to registry.unixcloudfusion.in port 443 (#0)
*   Trying 52.39.129.41...
* Connected to registry.unixcloudfusion.in (52.39.129.41) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: ca_bundle.crt
  CApath: none
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
* subject: CN=*.unixcloudfusion.in
* start date: Feb 19 09:18:56 2019 GMT
* expire date: May 20 09:18:56 2019 GMT
* common name: *.unixcloudfusion.in
* issuer: CN=Let's Encrypt Authority X3,O=Let's Encrypt,C=US
> GET /v2/ HTTP/1.1
> User-Agent: curl/7.29.0
> Host: registry.unixcloudfusion.in
> Accept: */*
>
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Content-Length: 2
Content-Length: 2
< Content-Type: application/json; charset=utf-8
Content-Type: application/json; charset=utf-8
< Docker-Distribution-Api-Version: registry/2.0
Docker-Distribution-Api-Version: registry/2.0
< X-Content-Type-Options: nosniff
X-Content-Type-Options: nosniff
< Date: Tue, 19 Feb 2019 16:31:33 GMT
Date: Tue, 19 Feb 2019 16:31:33 GMT

9. Download the image from the dockerhub, add the tags to identify it belongs to your repository
[root@ip-10-240-43-119 certs]# docker pull alpine:latest;docker tag alpine:latest registry.unixcloudfusion.in/alpine:alpinelocalv1
latest: Pulling from library/alpine
6c40cc604d8e: Pull complete
Digest: sha256:b3dbf31b77fd99d9c08f780ce6f5282aba076d70a513a8be859d8d3a4d0c92b8
Status: Downloaded newer image for alpine:latest

10. Verify the docker image as
[root@ip-10-240-43-119 certs]# docker images
REPOSITORY                           TAG                 IMAGE ID            CREATED             SIZE
registry                             2                   d0eed8dad114        2 weeks ago         25.8MB
alpine                               latest              caf27325b298        2 weeks ago         5.53MB
registry.unixcloudfusion.in/alpine   alpinelocalv1       caf27325b298        2 weeks ago         5.53MB

11. Push the image to your own repository
[root@ip-10-240-43-119 certs]# docker push registry.unixcloudfusion.in/alpine:alpinelocalv1
The push refers to repository [registry.unixcloudfusion.in/alpine]
503e53e365f3: Pushed
alpinelocalv1: digest: sha256:25b4d910f4b76a63a3b45d0f69a57c34157500faf6087236581eca221c62d214 size: 528

[Solved] x509: certificate signed by unknown authority

This error can occur if docker is not able to verify your certificate provider which might be due to the issue of bundle certificates used to verify the Certificate authority in absence of which you might be getting this error.

There is a workaround for this in which case it will ignore the certificate validation.

Create a file as /etc/docker/daemon.json
touch /etc/docker/daemon.json

Enter the following content in the daemon.json file replacing the endpoint for your repository as
[root@ip-10-240-43-119 certs]# cat /etc/docker/daemon.json
{
    "insecure-registries" : [ "registry.unixcloudfusion.in" ]
}

Go ahead and restart your docker service as
systemctl restart docker

Than try to push again to the repository this time you shouldn't get an error message.

[Solved] error: unable to upgrade connection: Forbidden (user=kubernetes, verb=create, resource=nodes, subresource=proxy)

I got this error while running

kubectl exec busybox-744d79879-q4bvl -- /bin/sh

which resulted in

error: unable to upgrade connection: Forbidden (user=kubernetes, verb=create, resource=nodes, subresource=proxy)


Cause/Resolution:-
Your kubernetes apiserver is using a client certificate with CN=kubernetes to connect to the kubelets and that user is not currently authorized to use the kubelet API.

By default system:kubelet-api-admin cluster role defines the permissions required to access that API. You can grant that permission to your apiserver kubelet client user with

kubectl create clusterrolebinding apiserver-kubelet-api-admin --clusterrole system:kubelet-api-admin --user kubernetes

Prometheus Monitoring for Microservices

1. Coming to the age of the microservices the older monitoring systems are not much dependable especially when you have a dynamic environment where containers keep coming up and down.

2. Prometheus is an open-source monitoring and alerting system built at soundcloud in 2012 and now managed by Cloud native computing foundation in 2016 as the second hosted project after Kubernetes.

3. Prometheus main featues include a multi-dimensional data model with time series data identified by metric name and key/value pairs which helps in understand overall performance of the sytem graphically.

4. Prometheus support PromoQL, a flexible query language to leverage this dimensionality.

5. It's not reliant on distributed storage like zookeeper rather single server nodes are autonomous.

6. Time series collection happens via pull model over http and pushing is supported via an intermediary gateway.

7. Targets for the monitoring are discovered via service discovery or static configuration which allows you to dynamically configure monitoring in a dynamic environment.

8. The main components of the prometheus is prometheus server which scrapes and stores time series data, client libraries for instrumenting application code, push gateway for supporting short-lived jobs, exporters like HAProxy, StatsD, Graphite etc, an alertmanager to handle alerts and various support tools.

9. Most of the prometheus components are written in Go programming language, making them easy to build and deploy as static libraries.

10. Prometheus works well with the purely numberic timer series metric. It fits both the machine centric monitoring as well as monitoring of highly dynamic service-oriented architectures. From microservices point of view it supports multi-dimensional data collection and querying is a particular strength.

In our future posts we are going to compare the prometheus with other monitoring tools.

Creating Docker Private Registry from scratch nonproduction only

Consider the following diagram to understand how the container calls the images in the dockerhub initially and how we can replace the dockerhub with our own local registry to store our docker images which will only be available in our own network , thus making it more secure

For a detailed walkthrough on how you can create your own private docker registry, go through the following video in which we have demonstrated how you create your own private docker registry in the nonproduction environment.

Understanding AWS S3 Objects Crossaccount Permissions Architecture

3. Understanding servicemesh event details

Servicemesh is a networking model ?

What is Service Mesh ?

As the introduction of the distributed microservices architecture for creating web/mobile based applications has increased and the orchestration tools such as kubernetes, public clouds has increased and made it more convenient to facilitate these microservice based architecture so the next demand is towards the deployment of the service mesh.

The term service mesh is used to describe the network of microservices that make up the applications running in an environment and how they are interacting amongst themselves. As the environment grows so the is the size of the services and there complexity to communicate both synchronously and asynchronously due to which it becomes harder and challenging to understand and manage such environments.

Than the requirements such as service discovery, load balancing, failure recovery, metrices and continuous monitoring often combines the requirement for more complex operational requirements like A/B testing, canary releases, rate limiting, access control and end-to-end authentication for the various api's and services.

The service mesh provides behavioural insights and operational control over the service mesh as a whole by offering a complete solution to satisfy the diverse requirements for managing the microservice applications.

Some of the leading service mesh provider include Istio developed in collaboration between Lyft, IBM , Google, Vmware and RedHat. Alternatives to Istion include Linkerd, the first service mesh to be ever developed created by Bouyant which open source service mesh written in scale and can be deployed on multiple types of clusters. Than there is Consul developed by Hashicorp which runs on agent-based model i.e. Consul client and finally than there is AWS App Mesh which is specifically developed for the AWS Public cloud.

We will be covering them in more detail in our future posts.

[Solved] S3 Bucket action doesn't apply to any resources

This error occurred when i tried implementing the s3 bucket policy.

this is due to the following policy which i was implementing

            "Action": [
                "s3:GetBucketLocation",
                "s3:ListBucket",
                "s3:GetObject",
                "s3:PutObject"
            ],
            "Resource": [
                "arn:aws:s3:::bucketname"
            ]

The issue here is , I was trying to implement it on the bucket only when the action has to applied in the form of regex to all the objects under the bucket so i replaced it with

            "Action": [
                "s3:GetBucketLocation",
                "s3:ListBucket",
                "s3:GetObject",
                "s3:PutObject"
            ],
            "Resource": [
                "arn:aws:s3:::bucketname",
                "arn:aws:s3:::bucketname/*"
            ]

That resolved my issue.

[Solved] Unable to create a new revision of Task Definition prod-not******:2 Docker label key owner contains invalid characters, does not match pattern ^[_\-a-zA-Z0-9.]+$

If you are getting the below error while updating the AWS ECS service

Unable to create a new revision of Task Definition prod-not****:2
Docker label key owner contains invalid characters, does not match pattern ^[_\-a-zA-Z0-9.]+$

Solution:-
In my case although the key value for the docker label appears to be correct there was an extra space in the key towards the end due to which i was not able to update key value and since it was not matching the regex which AWS has implemented on its end to verify the content , the ecs service was not allowing to update the configuration.

So check you don't have extra spaces and your labels are matching the regex which AWS Ecs service allows.