Whitepaper on inter node communications between different networks (VCNs) in Oracle Cloud Infrastructure.
Contents:
Summary:
Part-1: Configure instances to access
Internet. we will create compartments, VCNs and Subnets.
Part-2: Configure Instances to access
Internet in Primary & Secondary compartment. We will create Internet Gateway,
Instances and configure Security Lists & Route tables.
Part-3: Setup VCN peering between instances.
We will create Local Peering Gateways (LPG), establish connection
between LPGs and Configure Route Tables.
Part-4: Configure password less SSH
connection between instances. Generate SSH keys, Transfer SSH keys between
Instances and Configure Route tables.
Purpose of this document
In the current cloud computing world
there is always a need to setup communication between instances residing in
different VCNs to configure high availability for an application or due to 3
tier architecture models or for various other reasons. This document guides you
through the necessary steps needed to establish communication between instances
residing in different VCNs, separate compartments and in separate Availability Domains.
This document also walks you through on how
to create the following
cloud resources and other important setups.
·
Compartments,
Virtual Cloud Networks, Subnets, Setup Route Tables, Configure security rules.
· Internet Gateways
· Access Internet from Oracle Cloud Infrastructure Instances.
· Local Peering Gateways.
· Setup Inter Node communication between Instances.
· Setup Password less SSH connection between Instances.
Scope & Assumptions
Scope of this
document is to provide the reader with quick steps on how to setup VCN local
peering between Instances in two different VCNs. This document should not be
used as a complete reference guide for any production deployments.
Instructions/steps provided in this document are for informational and testing
purposes only. The author is not responsible for any mistakes/damages/security
flaws caused by the following steps. Configuring Peering between regions is
beyond the scope of this document.
This document
assumes that the reader understands Oracle Cloud infrastructure architecture
and has sound networking knowledge. The reader should also have working cloud
account with privileges to create network components like VCN, IGW, INSTANCES.
Virtual Cloud Network (VCN)
A VCN is
a software defined virtual network in Oracle Cloud Infrastructure. A VCN
offers you complete control over your cloud network environment. VCN offers
CIDR ranges from /16 to /30, and you can assign your own private IP addresses,
create subnets, create route tables and firewalls. You can have multiple VCNs
to provide grouping and isolation between related resources. The scope of VCN
is limited to all Availability Domains(ADs) in a region.
VCN Peering
Peering
is a process of connecting multiple VCNs so that traffic can be routed between
them privately. By default, instances from one VCN cannot communicate to
instances in another VCN. You can configure peering between VCNs using LOCAL
Peering Gateways so that instances from one VCN can communicate with instance
in another VCN as if they are in the same network. VCN peering is currently
allowed between same tenants and within a single region.
Core components of a VCN
- Subnet: A subnet is a range of IP addresses within a VCN
providing logical isolation for resource groups.
- Security List: A set of stateful virtual firewall rules
associated with a subnet. Security Lists consists of Ingress (Inbound) and
Egress (Outbound) firewall rules. Security lists are attached to an
instance at the time of instance launch.
- Route Table: A set of virtual route rules, viewed in
table format. Route table provides
mapping for the network traffic from subnets to destination address via
gateways.
- DHCP Options: IP address of a custom DNS server provided
during instance launch.
- Internet Gateway: A software-defined virtual router providing
a path for network traffic from your VCN to the public internet.
- Local Peering Gateway: is a software-defined virtual
router providing a path for network traffic from one VCN to another VCN.
It is used for local VNC peering.
The Figure 1 outlines the VCN peering
architecture and how its core components interact with each other.
![]() |
Figure 1 VNC Local Peering Architecture |
Billing
for VCN
There is
no charge for creating VCNs and using them. There are no data transfer charges
for any communication among resources within a VCN.
Cloud Resources Assignments
Figure 2 shows cloud resource
names, CIDR block ranges and IP addresses that have been used throughout this
document.
Compartment – Root
|
Compartment -
Secondary
|
|
VCN
|
Primary
|
SecondaryVCN2
|
Subnet
|
PrimarySubnet1
|
SecondarySubnet1
|
InternetGateway
|
PrimaryIGW1
|
SecondaryIGW1
|
Local Peering Gateway
|
PrimaryLPG1
|
SecondaryLPG1
|
VCN CIDR
|
10.0.0.0/16
|
20.0.0.0/16
|
Subnet CIDR
|
10.0.0.0/24
|
20.0.0.0/24
|
Private IP
|
10.0.0.2
|
20.0.0.2
|
Public IP
|
129.213.45.153
|
129.213.40.193
|
Instance Name
|
PrimaryVM1
|
SecondaryVM1
|
Instance Shape
|
VMStandard2.1
|
VMStandard1.2
|
Figure 2 Cloud resources name assignments
As mentioned earlier, the goal of
this document is to establish communication between two instances residing in
two different Compartments, two different Virtual Cloud Networks and two
different Availability Domains(ADs) using VCN Local Peering gateways. For this purpose,
this document has been divided into the following four parts:
Part-1:
Configure instances to access Internet. we will create a compartment, VCNs and
Subnets
Part-2:
Configure Instances to access Internet in Primary & Secondary compartment.
We will create Internet
Gateway, Instances and configure
Security Lists & Route tables.
Part-3: Setup
VCN peering between instances. We will create Local Peering Gateways (LPG), establish
connection between LPGs and
Configure Route Tables.
Part-4:
Configure password less SSH connection between instances. Generate SSH keys,
Transfer SSH keys
between Instances and Configure
Route tables.
PART-1
Step – 1: Create compartment
We will be using two compartments for
our setup; root compartment and secondary compartment. Root compartment will be
used as Primary compartment. Figure 3 highlights the steps for root compartment. Primary cloud resources will be deployed in
root compartment and all the secondary cloud resources will be deployed in
Secondary compartment. Figure 4 specifys the steps for creating secondary
compartment.
![]() |
Figure 3 Create Compartment |
![]() |
Figure 4 Create Secondary compartment
Step – 2: Create Virtual Cloud Network
To create a virtual cloud network,
under Primary compartment:
1.
click on Networking Tab
2.
click on create Virtual Cloud Network.
Figure 5 is for reference.
|
![]() |
Figure 6 VCN details
Step – 3: Create Subnet in Primary VCN
Create subnet with values shown in Figure 7, 8 and 9:
1.
Compartment
à root
2.
SubnetName
à Primarysubnet1
3.
CIDR
Range à 10.0.0.0/24
4.
Select
à Default DHCP options and Default
security lists.
5.
Click
Ã
Create
|
![]() |
![]() |
Figure 9 Security lists
Wait until you see that
Subnet is created and turns the status of the subnet to Available. Figure 10 is
for reference.
|
![]() |
Figure 10 Subnet Creation Confirmation
PART-2
Step - 1: Create Internet Gateway
Select Primary (root)
compartment à Internet Gateway Ã
Create Internet Gateway
Figure 11 and 12 are for
reference:
Figure 11 Create Internet Gateway
Provide the name as PrimaryIGW1 and
click Create Internet Gateway. Make sure that PrimaryIGW1 is in Available state
as shown in the Figure 13.
Figure 12 Name Internet Gateway
Figure 13 Internet Gateway creation confirmation
Step - 2: Configure Route Tables
Now we need to configure Route tables to allow network
traffic to passthrough Internet Gateway. In our example we are using default
Route tables and we will modify the default route table in our Primary VCN to
allow internet traffic. Figure 14 is for reference.
Figure 14 Default Route Table for Primary VCN
Select the Default
Route Table for Primary VCN and edit Route Rules as shown in Figure 15 below.
Figure 15 Edit Route Rules for Default Route Table in Primary
VCN
Click on “+ Another Route
Rule” as shown in Figure 16
Figure 16 Add Route Rule for Default Route Table in Primary VCN
Figure 17 Add Route rule to allow internet traffic via
Internet Gateway
Figure 18 Route rule addition confirmation.
Step – 3: Launch Instance
Go to Primary (root)
compartment and select Compute à Instances. Click on Launch Instance as shown in Figure
19.
Figure 19 Launch Instance in Primary Compartment.
Choose the following options for Instance Launch in Primary
Compartment. Figure 20 is for reference
Instance Name à PrimaryVM1
Availability Domain
à AD1
IMAGE SOURCE
Ã
ORACLE PROVIDED IMAGE
IMAGE O.S Ã Oracle Linux 7.4
SHAPE TYPE Ã VIRTUAL MACHINE
SHAPE TYPE Ã VM.Standard2.1
VIRTUAL CLOUD NETWROK Ã
Primary
SUBNET Ã PrimarySubnet1
UPLOAD SSH KEYS Ã USER
Launch Instance.
Figure 20 Options for Instance creation
Step – 4: Create
VCN in Secondary compartment.
Now we will mimic and create cloud resources in Secondary
compartment as we did in Primary Compartment. First, we will start by creating
VCN - SecondaryVCN2 as shown in Figure 21 & Figure 22
Figure 21 Create VCN in Secondary Compartment
Change compartment to Secondary in
the bottom left and select Virtual Cloud Networks. Click on Create Virtual
Cloud Network.
Fill in the details as shown in Figure 22
1.
Compartment
à Secondary
2.
VCN
name Ã
SecondaryVCN2
3.
Select
Virtual Cloud Network Only.
4.
CIDR
Range à 20.0.0.0/16
5.
Click
Ã
Create Virtual Cloud Network
Figure 22 Secondary VCN creation
Figure 23 Create Subnet in Secondary VCN2
In
SecondaryVCN2 select Subnets and click on Create Subnet with the below values.
Create
subnet with values shown in Figure 24.
· Compartment à Secondary
· SubnetName à Secondarysubnet1 · CIDR Range à 20.0.0.0/24 · Select à Default DHCP options and Default security lists. · Select à Default Security List for SecondaryVCN2 · Click à Create
Figure 24 Create Subnet Secondary subnet1
Step – 6: Create Internet Gateway in Secondary compartment
Select Secondary compartment à Internet Gateway à Create Internet Gateway
Figure 25 Create Internet Gateway
Figure 26 Create Internet Gateway in Secondary Compartment.
Provide the
name as SecondaryIGW1 and click Create Internet Gateway. Make sure that
SecondaryIGW1 is in Available state.
Step – 7: Configure Route Tables in Secondary
Compartment
Now we need to configure Route tables to allow
network traffic to passthrough Internet Gateway. In our example we are using
default Route tables and we will modify the default route table in SecondaryVCN2
to allow internet traffic.
Figure 27 Configure Route Table in Secondary Compartment
Figure 28 Add Route rule to allow internet traffic via
Secondary Internet Gateway.
Step – 8: Launch Instance in Secondary Compartment
Choose the Below options for Instance Launch in Secondary Compartment,
Figure 29.
Instance Name - SecondaryVM1
Availability Domain - AD1
IMAGE SOURCE - ORACLE PROVIDED IMAGE
IMAGE O.S - Oracle
Linux 7.4
SHAPE TYPE - VIRTUAL
MACHINE
SHAPE TYPE - VM.Standard2.1
VIRTUAL CLOUD NETWROK - Secondary
SUBNET - SecondarySubnet1
UPLOAD SSH KEYS - USER
Figure 29 Launch Instance in Secondary compartment.
Step – 9: Test Internet Traffic
Login into instances using their Public IPs as shown below in
Figure 30 and ping any public URL. Here we will ping www.google.com and we should see the successful
packet transfers.
Figure 30 Login into Instances and Test internet traffic
PART-3
In this part, we will create Local Peering
Gateways (LPG) in Primary & Secondary compartments, establish private connection
between LPGs and Configure Route Tables to allow network traffic to pass
through LPGs. The steps for doing so are listed below.
Step – 1: Creating Local Peering Gateway (LPG)
Select Primary (root) compartment à Networking à Virtual Cloud Networks à Primary VCN à Local Peering
Gateway
Click Create Local
Peering Gateway as shown in Figure 31
Figure 31 Create Local Peering Gateway
Figure 32 Create PrimaryLPG1 – Local Peering Gateway in Primary
compartment
Now switch to
Secondary compartment and create Local peering gateway as follows:
Select Secondary
compartment -> Networking -> Virtual Cloud Networks -> SecondaryVCN2 ->
Local Peering Gateway.
Click Create Local
Peering Gateway as shown in Figure 33
Figure 33 Create Local Peering Gateway
Figure 34 Create SecondaryLPG1 – Local Peering Gateway in
Secondary compartment
Confirm that the SecondaryLPG1 is created
successfully and status is Available as shown in Figure35
Figure 35 Secondary Local Peering Gateway creation confirmation
Step – 2: Establish Connection Between LPGs
Switch to Primary (root) compartment à Networking à Virtual Cloud
Networks à Primary VCN à Local Peering Gateway à Click Establish Connection. (Figure 36 and 37
are for reference)
Figure 36 Establish Connection between Local Peering Gateways
Figure 37 Setup Connection with Secondary Local Peering Gateway
You need to
establish connection only once between two LPGs and it doesn’t really matter
which LPG you pick to establish the connection from.
Figure 38 Perring confirmation between Local Peering Gateways
Step – 3: Configure Route Tables
So far, we have
created LPGs and bridged a connection between LPGs but the instances in the two
VCNs cannot communicate yet due to the Firewall restrictions. In this section
we will configure Route tables and modify Security Lists to ease Firewall restrictions
and allow network traffic from the instances to passthrough Local Peering
Gateway. In our example we are using default Route tables and we will modify
the default route table in our Primary & Secondary VCN to allow destination
LPG CIDR traffic to pass through.
Select Secondary
compartment à Networking à Virtual Cloud Networks à SecondaryVCN2 à Route Tables à Default Route
Table
Add new Route Rule to route network traffic from
SecondaryLPG1 to destination CIDR block 10.0.0.0/24 as shown in Figure 39.
Figure 39 Add Route rule to route Secondary LPG traffic
Figure 40 Route rule addition confirmation for SecondaryLPG1
traffic
Figure 41 Add Route rule to route Primary LPG traffic
Select Primary
(root) compartment à Networking à Virtual Cloud Networks à Primary VCN à Route Tables à Default Route
Table.
Add new Route Rule
to route network traffic from PrimaryLPG1 to destination CIDR block 20.0.0.0/24
as shown in Figure 42.
Figure 42 Add Route rule to allow PrimaryLPG1 traffic
Step – 4: Configure Security Lists
Now edit Default security list in SecondaryVCN2
to allow all protocols from CIDR block range 10.0.0.0/24 into SecondaryVCN2.
Figure 43 Security List modification in SecondaryVCN2
Select Secondary
compartment Networking Virtual Cloud Networks SecondaryVCN2 à Default Security List à Add Rule.
Add rule to allow all protocols from CIDR block range 10.0.0.0/24 into
SecondaryVCN2 as shown in Figure 44.
Figure 44 Add security Rule in SecondaryVCN2
Now edit Default
security list in Primary VCN to allow all protocols from CIDR block range
20.0.0.0/24 into primary VCN.
Figure 45 Modify Security List in Primary VCN
Select Primary
(root) compartment à Networking à Virtual Cloud Networks à Primary VCN à Default Security List à Add Rule.
Add rule to allow
all protocols from CIDR block range 20.0.0.0/24 into Primary VCN as shown in Figure
46.
Figure 46 Add security Rule in Primary VCN
Step –
5: Test VNC Local peering
Note down the
public and Private IP addresses of instances in PrimaryVCN and SecondaryVCN.
Login into
PrimaryVM1 as opc user and ping the private IP address of the SecondaryVM1 you
should be able to see the packet response from SecondaryVM1.
Figure 47 PrimaryVM1 IP addresses
[opc@primaryvm1 ~]$ ifconfig -a
ens3: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000
inet 10.0.0.2 netmask 255.255.255.0 broadcast 10.0.0.255
ether
02:00:17:01:9e:6d txqueuelen 1000 (Ethernet)
RX packets 56766 bytes 328814777 (313.5 MiB)
RX errors 0 dropped 0
overruns 0 frame 0
TX packets 48827 bytes 124483559 (118.7 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0
collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
loop txqueuelen 0
(Local Loopback)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0
overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0
collisions 0
[opc@primaryvm1 ~]$ ping 20.0.0.2
PING 20.0.0.2 (20.0.0.2) 56(84) bytes of data.
64 bytes from 20.0.0.2: icmp_seq=1 ttl=64 time=0.212 ms
64 bytes from 20.0.0.2: icmp_seq=2 ttl=64 time=0.175 ms
64 bytes from 20.0.0.2: icmp_seq=6 ttl=64 time=0.177 ms
64 bytes from 20.0.0.2: icmp_seq=7 ttl=64 time=0.140 ms
64 bytes from 20.0.0.2: icmp_seq=8 ttl=64 time=0.145 ms
^C
--- 20.0.0.2 ping statistics ---
8 packets transmitted, 8 received, 0% packet loss, time 6999ms
rtt min/avg/max/mdev = 0.140/0.173/0.212/0.025 ms
Figure 48 SecondaryVM1 IP addresses
[opc@secondaryvm1
~]$ ifconfig -a
ens3:
flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000
inet 20.0.0.2 netmask 255.255.255.0 broadcast 20.0.0.255
ether 02:00:17:01:53:5a txqueuelen 1000 (Ethernet)
RX packets 57725 bytes 328343608 (313.1 MiB)
RX errors 0 dropped 0
overruns 0 frame 0
TX packets 54826 bytes 181204328 (172.8 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0
collisions 0
lo:
flags=73<UP,LOOPBACK,RUNNING> mtu
65536
inet 127.0.0.1 netmask 255.0.0.0
loop
txqueuelen 0 (Local Loopback)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0
overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0
collisions 0
[opc@secondaryvm1
~]$ ping 10.0.0.2
PING 10.0.0.2
(10.0.0.2) 56(84) bytes of data.
64 bytes from
10.0.0.2: icmp_seq=1 ttl=64 time=0.222 ms
64 bytes from
10.0.0.2: icmp_seq=2 ttl=64 time=0.172 ms
64 bytes from
10.0.0.2: icmp_seq=3 ttl=64 time=0.166 ms
64 bytes from
10.0.0.2: icmp_seq=4 ttl=64 time=0.205 ms
64 bytes from
10.0.0.2: icmp_seq=5 ttl=64 time=0.157 ms
64 bytes from
10.0.0.2: icmp_seq=13 ttl=64 time=0.171 ms
^C
--- 10.0.0.2 ping
statistics ---
13 packets
transmitted, 13 received, 0% packet loss, time 11999ms
rtt
min/avg/max/mdev = 0.137/0.172/0.222/0.027 ms
Figure 49 Inter instance
communication via LPGs
PART - 4
Step – 1: Setup User equivalency or password-less
ssh login between OCI instances
We will generate ssh rsa keys without any passphrase for simplicity
and then add the public key id_rsa.pub into PrimaryVM1’s ~/.ssh/authorized_keys
file to enable password less login for opc user.
Step – 2: Generate SSH RSA Keys
[opc@secondaryvm1
~]$ pwd
/home/opc
[opc@secondaryvm1
~]$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key
(/home/opc/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in
/home/opc/.ssh/id_rsa.
Your public key has been saved in
/home/opc/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:iOniHDTl09z/2kc9X5cw8tia9RmfmrcmW9jPZq0GT+w
opc@secondaryvm1
The key's randomart image is:
+---[RSA 2048]----+
| |
| |
|
. |
| o =
o . o
|
| o = +
S =.o..|
| . o .
. ..+*++|
| o
. . +B.o@|
| o o
= .EBB|
|
o ..o*B=+|
+----[SHA256]-----+
[opc@secondaryvm1
~]$ cat .ssh/id_rsa.pub
ssh-rsa
AAAAB3NzaC1yc2EAAAADAQABAAABAQCap+PEKhwVbvP05Gikv6toUq6POzUmuwgh8FrnILBkCmKfWLQRChmcUNlCvEG7Wqc2SUW6lefHhNpcIkrT/Q/TDwkhVN5HMupgU0oJZEP1fUH/xj365deQJrZPWCa1ZzPGEozj76kcqYOqPMVSbtc0O4znuyXvFeZBcwmFiwzp0LcAOeJlfvTKBXe0y3+7CHYgJzjMgfPZgPbMUu40LdBlHS3kEVkRwWlOSJFxjGrZyl0FbX74jHFQ1RDy/LYjXeZvZdyqjYdpwNw6t0LEFHjYAy50gOkrw9SDwsDR2QJOxXiZ7ccz0WoBSI2Hp9FQXmlnk7stWS0t9xNe2XevRdsF
opc@secondaryvm1
Either
scp public key or copy and paste into authorized_keys file in primaryVM1 host.
[opc@primaryvm1
~]$ vi .ssh/authorized_keys
[opc@secondaryvm1 ~]$ ssh 10.0.0.2
The
authenticity of host '10.0.0.2 (10.0.0.2)' can't be established.
Are
you sure you want to continue connecting (yes/no)? yes
Warning:
Permanently added '10.0.0.2' (ECDSA) to the list of known hosts.
Last login: Fri Feb 16 21:38:30 2018
from ool-18b8eaa0.dyn.optonline.net
Generate
SSH RSA keys on PrimaryVM1 and add the public key id_rsa.pub into
SecondaryVM1’s ~/.ssh/authorized_keys file to enable password less login for
opc user.
[opc@primaryvm1 ~]$ ssh-keygen -t rsa
Generating
public/private rsa key pair.
Enter
file in which to save the key (/home/opc/.ssh/id_rsa):
Enter
passphrase (empty for no passphrase):
Enter
same passphrase again:
Your
identification has been saved in /home/opc/.ssh/id_rsa.
Your
public key has been saved in /home/opc/.ssh/id_rsa.pub.
The
key fingerprint is:
SHA256:SNeDgqryhoIYc1w6983KFrU59ajye5RAEVnGqemYFzo
opc@primaryvm1
The
key's randomart image is:
+---[RSA
2048]----+
| o*o.
|
| .
o+o |
| . o.ooo
|
|
..o .*S* + |
|o.=
. E * + . |
|==
o . * + |
|*.. .+ + .
|
|... .o+oo
|
+----[SHA256]-----+
[opc@primaryvm1 ~]$ cat ~/.ssh/id_rsa.pub
ssh-rsa
AAAAB3NzaC1yc2EAAAADAQABAAABAQDBeF90LFhO/QF/u0uEVnxKEX6z3Q09XZIjsPpN1F+RVZrJ47+9qNytPKFqg8OErUuBb5iUK9VyFG2fmLJQgGgARGZXy4SlEoW8GBq/CIQbHF5JLcDcWVrUuNTu2QbKZKD3bJq7+zG+P28YdLs8jDprVHhkKiNK1PPQpgnVLAGRYTbDhsAX9zO944FCCrbsKDVSVVdR1ySRryvAveU4j8r1HsjqgYvPzLb8Dmlhgnz9b336JGOicJ1Wy+AunHSdUAJgVg4gG6qJWA2M6Vi3ePUn5ImipAhJ0WGDRhbP/7Vz+51chiUPyceG2VnnFFSHuLQiuFrPm7iBPRvfuTysV2ir
opc@primaryvm1
[opc@primaryvm1 ~]$ vi .ssh/authorized_keys
[opc@primaryvm1 ~]$ ssh 20.0.0.2
The
authenticity of host '20.0.0.2 (20.0.0.2)' can't be established.
Are
you sure you want to continue connecting (yes/no)? yes
Warning:
Permanently added '20.0.0.2' (ECDSA) to the list of known hosts.
Last login: Fri Feb 16 21:38:32 2018
from ool-18b8eaa0.dyn.optonline.net
|