[Solved] Restructuring the CDN Logs

 Problem:- Cloudfront logs are stored in the following format

 distributionid-year-month-date-hour.gz  

So if you are looking to analyse these logs you need something similar to the Athena which can directly run your queries over the s3 bucket which is storing these logs.

But Athena requires partition data which simply means storing data in a format of (e.g. a folder structure). This allows you to restrict the athena to the limited data which you want to analyze other by default it will take the entire data and cost you more while reading GBs of data which you dont want.

By default Athena tries to "read all" the data. But if you have partitioned it like year/month/day than you can register it like

  year=2021/month=02/day=25 -- s3://logs/2021/02/25  

This allows your to simply use the where clause and with partition indices to restrict the athena to read the data you are interested in

  SELECT uri, count(1)   
   FROM cloudfront_logs  
   WHERE status = 404   
    AND (year || month || day || hour) > ‘20200225’  

Understanding Docker compose with diagram

[Solved] cannot load such file -- puppetserver/ca/cli (LoadError)

Issue:- When try to list the CA certificates in the puppet getting below error 

 [root@puppetmaster bin]# /opt/puppetlabs/bin/puppetserver ca list 

Ignoring executable-hooks-1.3.2 because its extensions are not built. Try: gem pristine executable-hooks --version 1.3.2 

Ignoring gem-wrappers-1.3.2 because its extensions are not built. Try: gem pristine gem-wrappers --version 1.3.2 

Traceback (most recent call last): 

 2: from /opt/puppetlabs/server/apps/puppetserver/cli/apps/ca:3:in `

' 

 1: from /opt/puppetlabs/puppet/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:in `require' 

/opt/puppetlabs/puppet/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:in `require': cannot load such file -- puppetserver/ca/cli (LoadError) 

Resolution:- The above error comes when you are not using sudo command. Even if you are running via root than also you need to use the sudo command to resolve this issue 

 [root@puppetmaster bin]# sudo /opt/puppetlabs/bin/puppetserver ca list 

Requested Certificates: puppetclient.unixcloudfusion.in (SHA256) B3:51:2A:59:CE:68:29:B0:68:9B:C1:53:59:28:97:45:AE:B6:61:97:64:DE:AE:64:40:D7:BE:93:78:65:42:1D