Experience: is what you get soon after you need it.

Experience: is what you get soon after you need it.

****************Got the opportunity to pen a Book on Database Cloud Services.......More details to follow.***************

Title : Database Cloud Revealed
Publisher : Apress
Release Date : Jan-2020

**********************************************************************************


Rasul Allah (sal Allahu alaihi wa sallam) said: "Restore the trusts of those who trust you, and deal not falsely with him who deals falsely with you." [Abu Dawud, Tirmidhi]


My Cloud Certifications:

AWS Certified Solutions Architect Associate

Azure Certified Administrator

Oracle Cloud Infrastructure 2018 Certified Architect Associate.

Oracle Cloud Infrastructure Classic 2018 Certified Architect Associate.

Oracle Database Cloud Administrator Certified Professional.

Oracle Database Cloud Service Operations Certified Associate.

Search This Blog

Sunday, June 2, 2019

AWS Take block volume snapshots at regular intervals


Now using Data Lifecycle manager you can take block volume snapshots at regular intervals without the need of custom scripts and lambda functions.




















Under block volumes -> Life Cycle Manager --> Create Policy





Give a policy Name
You can choose either instance level snapshots (which includes all the EBS volumes attached to the instance) or individual block level volumes.

Pick a schedule:
You can  pick a window anywhere between 2hrs and 12hrs 

Retention:
How many snapshots you want to retain, AFAIK limit is 1000 snapshots and it will overwrite the oldest snapshot.

















Tags:
Pick any tags that you want to have for the snapshots.






































IAM Role:

Pick a role or create a new role with the below policy:

{
  "Version": "2018-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ec2:CreateSnapshot",
        "ec2:DeleteSnapshot",
        "ec2:DescribeVolumes",
        "ec2:DescribeSnapshots"
      ],
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "ec2:CreateTags"
      ],
      "Resource": "arn:aws:ec2:*::snapshot/*"
    }
  ]
}


Enable the policy
























AWS Create Images from EBS Snapshots


You can take block volume snapshots for various reasons like for backup/restore, create golden images, create a custom image which will serve for future auto scaling launch templates and may be for disaster recovery purposes (We will see in a later post on how to take the incremental snapshots on regular intervals using Life cycle manager)

In this post we will cover how to take block volume snapshots and create a custom image from this snapshot.

Step-1:
Go to EBS Volumes and create snapshot:


It may take some time for the initial snapshot to complete based on the amount of size of storage volume






Step-2:

From the snapshots plane, go to actions and select "create Image", this will create a custom AMI based on the snapshot that we just took.














Step-3:

Now go to Images  and you should see the image you just created in step-2




Step-4:

You can now launch a EC2 instance using the private image you just created.















ssh: connect to host XX.XX.XX.XX port 22: Connection refused while connecting to Guest OS on Virtual Box

Issue:
While connecting from Windows - 10 to Guest OS - Oracle Linux -7.1  on Oracle Virtual Box - 5.2, you may get the below error.
ssh: connect to host XX.XX.XX.XX port 22: Connection refused


shaikprod@shaikprod ~
$ ping 192.168.56.1

Pinging 192.168.56.1 with 32 bytes of data:
Reply from 192.168.56.1: bytes=32 time<1ms TTL=128
Reply from 192.168.56.1: bytes=32 time<1ms TTL=128
Reply from 192.168.56.1: bytes=32 time<1ms TTL=128
Reply from 192.168.56.1: bytes=32 time<1ms TTL=128

Ping statistics for 192.168.56.1:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 0ms, Maximum = 0ms, Average = 0ms

shaikprod@shaikprod ~
$ ssh 192.168.56.1
ssh: connect to host 192.168.56.1 port 22: Connection refused

shaikprod@shaikprod ~
$ ssh 192.168.56.1
ssh: connect to host 192.168.56.1 port 22: Connection refused



Fix:
Enable port forwarding on the NAT -Network from the Oracle Virtual Box



AWS Deploy custom monitors

So how do you deploy custom monitors in AWS and push them to CloudWatch:

By default CloudWatch monitors the below four metrics and if you want to monitor beyond these four metrics then you have to deploy custom monitors, below we will go though the depoyment of custom metric and push the metrics to CloudWatch monitoring dashboard.

CloudWatch default metrics for EC2 Instances:

1. CPU
2. Disk IO
3. Network bytes in/out
4. Status check


Download the latest version of monitoring scripts from the below URL:
http://aws-cloudwatch.s3.amazonaws.com/downloads/CloudWatchMonitoringScripts-1.2.2.zip


Above script provides following metrics:
  • Memory Utilization – Memory allocated by applications and the operating system, exclusive of caches and buffers, in percentages.
  • Memory Used – Memory allocated by applications and the operating system, exclusive of caches and buffers, in megabytes.
  • Memory Available – System memory available for applications and the operating system, in megabytes.
  • Disk Space Utilization – Disk space usage as percentages.
  • Disk Space Used – Disk space usage in gigabytes.
  • Disk Space Available – Available disk space in gigabytes.
  • Swap Space Utilization – Swap space usage as a percentage.
  • Swap Space Used – Swap space usage in megabytes.

Install the required packages

[root@shaikprod ~]# sudo yum install -y perl-Switch perl-DateTime perl-Sys-Syslog perl-LWP-Protocol-https perl-Digest-SHA.x86_64 -y


Complete!
[root@shaikprod ~]#

Download  the monitoring scripts:

[root@shaikprod ~]# curl 
https://aws-cloudwatch.s3.amazonaws.com/downloads/CloudWatchMonitoringScripts-1.2.2.zip -O


  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 24225  100 24225    0     0   241k      0 --:--:-- --:--:-- --:--:--  241k


Install the monitoring scripts:


unzip CloudWatchMonitoringScripts-1.2.2.zip && \
rm CloudWatchMonitoringScripts-1.2.2.zip && \
cd aws-scripts-mon


[root@shaikprod ~]# unzip CloudWatchMonitoringScripts-1.2.2.zip && \
> rm CloudWatchMonitoringScripts-1.2.2.zip && \
> cd aws-scripts-mon
Archive:  CloudWatchMonitoringScripts-1.2.2.zip
 extracting: aws-scripts-mon/awscreds.template
  inflating: aws-scripts-mon/AwsSignatureV4.pm
  inflating: aws-scripts-mon/CloudWatchClient.pm
  inflating: aws-scripts-mon/LICENSE.txt
  inflating: aws-scripts-mon/mon-get-instance-stats.pl
  inflating: aws-scripts-mon/mon-put-instance-data.pl
  inflating: aws-scripts-mon/NOTICE.txt
rm: remove regular file ‘CloudWatchMonitoringScripts-1.2.2.zip’? y

Below files will be deployed by the above install:

[root@shaikprod aws-scripts-mon]# ls -lrt
total 96
-rw-r--r-- 1 root root   138 Mar 26  2018 NOTICE.txt
-rwxr-xr-x 1 root root 18144 Mar 26  2018 mon-put-instance-data.pl
-rwxr-xr-x 1 root root  9739 Mar 26  2018 mon-get-instance-stats.pl
-rw-r--r-- 1 root root  9124 Mar 26  2018 LICENSE.txt
-r--r--r-- 1 root root 22519 Mar 26  2018 CloudWatchClient.pm
-r--r--r-- 1 root root 17021 Mar 26  2018 AwsSignatureV4.pm
-rw-r--r-- 1 root root    30 Mar 26  2018 awscreds.template


If you view the perl scripts,  it mentions the usage and provides examples on how to push the metrics to CloudWatch

[root@shaikprod aws-scripts-mon]# grep -A5 "Usage:" *.pl
mon-get-instance-stats.pl:Usage: mon-get-instance-stats.pl [options]
mon-get-instance-stats.pl-
mon-get-instance-stats.pl-  Queries Amazon CloudWatch for statistics on CPU, memory, swap, and
mon-get-instance-stats.pl-  disk space utilization within a given time interval. This data is
mon-get-instance-stats.pl-  provided for the Amazon EC2 instance on which this script is executed.
mon-get-instance-stats.pl-
--
mon-put-instance-data.pl:Usage: mon-put-instance-data.pl [options]
mon-put-instance-data.pl-
mon-put-instance-data.pl-  Collects memory, swap, and disk space utilization on an Amazon EC2
mon-put-instance-data.pl-  instance and sends this data as custom metrics to Amazon CloudWatch.
mon-put-instance-data.pl-

 Verify the script execution:

[root@shaikprod aws-scripts-mon]# ./mon-put-instance-data.pl --mem-util --verify --verbose

MemoryUtilization: 9.47831427948723 (Percent)

Using IAM role <admin-role>
Endpoint: https://monitoring.us-east-2.amazonaws.com
Payload: {"MetricData":[{"Timestamp":1559456122,"Dimensions":[{"Value":"i-00e1a0eaccb314f6f","Name":"InstanceId"}],"Value":9.47831427948723,"Unit":"Percent","MetricName":"MemoryUtilization"}],"Namespace":"System/Linux","__type":"com.amazonaws.cloudwatch.v2010_08_01#PutMetricDataInput"}

Verification completed successfully. No actual metrics sent to CloudWatch.


Now publish the metrics to CloudWatch:


[root@shaikprod aws-scripts-mon]# ./mon-put-instance-data.pl --mem-util --mem-used --mem-avail --aggregated=only

Successfully reported metrics to CloudWatch. Reference Id: 1bb50b87-84fe-11e9-89db-9fb70138859a


You can add the above metric execution to crontab so that the metrics are continuously published to CludWatch.





Saturday, June 1, 2019

AWS Auto Stop Start EC2 Instances

In the below post we will create a simple Lambda function to Auto STOP/START EC2 instances.

Step-1
Before we create a lambda function we need to have a role that has access to our EC2 instances.

Go ahead and create a IAM role with appropriate permissions to stop/start the EC2 instances, you can either use the built-in policies or create a custom policy for this role.

 


Step-2
Now create a lambda function, choose option author from scratch as we will use custom code to stop/start the instances.











copy/paste the below code into the lmbda function, remember to replace the region & instance ids.


import boto3
region = 'us-east-2'
instances = ['i-00XXXXXX14f6f']

def lambda_handler(event, context):
    ec2 = boto3.client('ec2', region_name=region)
    ec2.start_instances(InstanceIds=instances)
    print 'started your instances: ' + str(instances)


Under roles select the IAM role you created for this function and change the timeout to at least 20 seconds.










Save and test the function.

You should see a successful message.























Now create a stop function to stop the EC2 instances. Follow step-2 except the in the code you will replace start command with stop command


import boto3
region = 'us-east-2'
instances = ['i-00XXXXXX14f6f']

def lambda_handler(event, context):
    ec2 = boto3.client('ec2', region_name=region)
    ec2.stop_instances(InstanceIds=instances)
    print 'stopped your instances: ' + str(instances)


Step-3:
Now  create Cloud Watch events that will trigger the lambda functions that were created in Step2.






















Create a new rule for the stop function :

















Here I have selected cron expression, you can select which ever is convenient for you.











More about schedule expressions can be found here:
https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/ScheduledEvents.html


****Remember scheduled events in CloudWatch uses default timezone as UTC




Repeat step-4 for start function so that the ec2 instances are started at a desired interval:

Validation:

Instance is stopped at the specified time:














Tuesday, March 12, 2019

Create new service in Oracle Cloud DBCS system





shaikprod@shaikprod ~/.ssh
$ ssh opc@129.213.136.114
The authenticity of host '129.213.136.114 (129.213.136.114)' can't be establishe                              d.
RSA key fingerprint is SHA256:M1nYMnHul6e0Mpb3llWcJovCwyJBVD+iNOLQz6q1niI.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '129.213.136.114' (RSA) to the list of known hosts.

[opc@ocidb ~]$ sudo su - oracle

[oracle@ocidb ~]$ ps -ef| grep smon
grid     72807     1  0 16:39 ?        00:00:00 asm_smon_+ASM1
oracle   82804 82133  0 17:31 pts/0    00:00:00 grep smon
oracle   86482     1  0 17:01 ?        00:00:00 ora_smon_ocidb

[oracle@ocidb ~]$



SQL> show parameter service

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
service_names                        string      ocidb_iad1sp.sub05111533480.vcn
                                                 0511153348.oraclevcn.com
SQL> select service_id,name,pdb,con_id from cdb_services order by con_id;

SQL> set linesize 1000
SQL> select service_id,name,pdb,con_id from cdb_services order by con_id;

SERVICE_ID NAME                                                             PDB                                                    CON_ID
---------- ---------------------------------------------------------------- ----------------------------------                     ---------------------------------------------------------------------------------------------- ----------
         1 SYS$BACKGROUND                                                   CDB$ROOT                                                    1
         6 ocidb_iad1sp.sub05111533480.vcn0511153348.oraclevcn.com           CDB$ROOT                                                    1
         5 ocidbXDB                                                          CDB$ROOT                                                    1
         2 SYS$USERS                                                        CDB$ROOT                                                    1
         7 ocidb_pdb1.sub05111533480.vcn0511153348.oraclevcn.com             ocidb_PDB1                                                   3




SQL>  col name for a20
SQL> select con_id,name,open_mode from v$containers;

    CON_ID NAME                 OPEN_MODE
---------- -------------------- ----------
         3 ocidb_PDB1            READ WRITE



SQL> alter session set container=ocidb_PDB1;

Session altered.


SQL> col pdb for a20
SQL> select service_id,name,pdb,con_id from cdb_services order by con_id;

SERVICE_ID NAME                                                                   PDB                      CON_ID
---------- ---------------------------------------------------------------------- -------------------- ----------
         7 ocidb_pdb1.sub05111533480.vcn0511153348.oraclevcn.com                   ocidb_PDB1                     3



SQL> exec dbms_service.CREATE_SERVICE('shaikpdb','shaik_pdb.sub05111533480.vcn0511153348.oraclevcn.com');

PL/SQL procedure successfully completed.



SQL> select service_id,name,pdb,con_id from cdb_services order by con_id;

SERVICE_ID NAME                                                                   PDB                      CON_ID
---------- ---------------------------------------------------------------------- -------------------- ----------
         7 ocidb_pdb1.sub05111533480.vcn0511153348.oraclevcn.com                   ocidb_PDB1                     3
         1 shaikpdb                                                               ocidb_PDB1                     3



SQL> exec dbms_service.start_service('shaikpdb');

PL/SQL procedure successfully completed.

SQL> alter system register;

System altered.



SQL> ALTER PROFILE "DEFAULT" LIMIT PASSWORD_VERIFY_FUNCTION NULL;

Profile altered.

SQL> create user user99 identified by WelCome99;

User created.


SQL> grant create session to user99;

Grant succeeded.



SQL> alter pluggable database save state;

Pluggable database altered.




SQL> !lsnrctl status

LSNRCTL for Linux: Version 12.2.0.1.0 - Production on 12-MAR-2019 17:43:42

Copyright (c) 1991, 2016, Oracle.  All rights reserved.

Connecting to (ADDRESS=(PROTOCOL=tcp)(HOST=)(PORT=1521))
STATUS of the LISTENER
------------------------
Alias                     LISTENER
Version                   TNSLSNR for Linux: Version 12.2.0.1.0 - Production
Start Date                12-MAR-2019 16:40:06
Uptime                    0 days 1 hr. 3 min. 36 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /u01/app/12.2.0.1/grid/network/admin/listener.ora
Listener Log File         /u01/app/grid/diag/tnslsnr/ocidb/listener/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=10.0.0.39)(PORT=1521)))
Services Summary...
Service "+APX" has 1 instance(s).
  Instance "+APX1", status READY, has 1 handler(s) for this service...
Service "+ASM" has 1 instance(s).
  Instance "+ASM1", status READY, has 1 handler(s) for this service...
Service "+ASM_DATA" has 1 instance(s).
  Instance "+ASM1", status READY, has 1 handler(s) for this service...
Service "+ASM_RECO" has 1 instance(s).
  Instance "+ASM1", status READY, has 1 handler(s) for this service...
Service "83e996f4d58506f6e0532700000a2d76.sub05111533480.vcn0511153348.oraclevcn.com" has 1 instance(s).
  Instance "ocidb", status READY, has 2 handler(s) for this service...
Service "ocidbXDB.sub05111533480.vcn0511153348.oraclevcn.com" has 1 instance(s).
  Instance "ocidb", status READY, has 1 handler(s) for this service...
Service "ocidb_iad1sp.sub05111533480.vcn0511153348.oraclevcn.com" has 1 instance(s).
  Instance "ocidb", status READY, has 2 handler(s) for this service...
Service "ocidb_pdb1.sub05111533480.vcn0511153348.oraclevcn.com" has 1 instance(s).
  Instance "ocidb", status READY, has 2 handler(s) for this service...
Service "shaik_pdb.sub05111533480.vcn0511153348.oraclevcn.com" has 1 instance(s).
  Instance "ocidb", status READY, has 2 handler(s) for this service...
The command completed successfully

SQL>


SQL> !
[oracle@ocidb ~]$  sqlplus user99/WelCome99@ocidb:1521/shaik_pdb.sub05111533480.vcn0511153348.oraclevcn.com

SQL*Plus: Release 12.2.0.1.0 Production on Tue Mar 12 17:48:15 2019

Copyright (c) 1982, 2016, Oracle.  All rights reserved.


Connected to:
Oracle Database 12c Standard Edition Release 12.2.0.1.0 - 64bit Production

SQL> show user
USER is "USER99"






[oracle@ocidb ~]$ vi $ORACLE_HOME/network/admin/tnsnames.ora

[oracle@ocidb ~]$ cat $ORACLE_HOME/network/admin/tnsnames.ora
# tnsnames.ora Network Configuration File: /u01/app/oracle/product/12.2.0.1/dbhome_1/network/admin/tnsnames.ora
# Generated by Oracle configuration tools.

LISTENER_ocidb =
  (ADDRESS = (PROTOCOL = TCP)(HOST = ocidb.sub05111533480.vcn0511153348.oraclevcn.com)(PORT = 1521))


ocidb_IAD1SP =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = ocidb.sub05111533480.vcn0511153348.oraclevcn.com)(PORT = 1521))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = ocidb_iad1sp.sub05111533480.vcn0511153348.oraclevcn.com)
    )
  )

shaikpdb =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = ocidb.sub05111533480.vcn0511153348.oraclevcn.com)(PORT = 1521))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = shaik_pdb.sub05111533480.vcn0511153348.oraclevcn.com)
    )
  )



[oracle@ocidb ~]$ tnsping shaikpdb

TNS Ping Utility for Linux: Version 12.2.0.1.0 - Production on 12-MAR-2019 17:50:06

Copyright (c) 1997, 2016, Oracle.  All rights reserved.

Used parameter files:
/u01/app/oracle/product/12.2.0.1/dbhome_1/network/admin/sqlnet.ora


Used TNSNAMES adapter to resolve the alias
Attempting to contact (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = ocidb.sub05111533480.vcn0511153348.oraclevcn.com)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = shaik_pdb.sub05111533480.vcn0511153348.oraclevcn.com)))
OK (0 msec)


[oracle@ocidb ~]$ sqlplus user99@shaikpdb

SQL*Plus: Release 12.2.0.1.0 Production on Tue Mar 12 17:50:16 2019

Copyright (c) 1982, 2016, Oracle.  All rights reserved.

Enter password:
Last Successful login time: Tue Mar 12 2019 17:48:15 +00:00

Connected to:
Oracle Database 12c Standard Edition Release 12.2.0.1.0 - 64bit Production


SQL> show user
USER is "USER99"
SQL>