AWS by harsha VPC and EC2
Class 1:(Introduction to aws)
Iaas vs Paas vs Saas:
Iaas:
* Entire infrastructure should be managed by us.It provides Ec2 instances, we need o install app servers on top of EC2 instances and database and web servers as well
* we will have more control
* If we face down time or any issues we need to contact aws support.Since infrastructure is managed by aws
Pass:
* Vendors will provide the sevices. we don't have complete control on that applications
* One good example was Dynamo Databases, we just need to use them and we didn't install them on any bare machine
* If there is an issue, we need to contact specific vendor, Eg: Postgres DB,oracle db etc
Saas:
* Every thing is taken care by applications owners
* We just need to connect and use it ..Thats it
* We don't have any control on backgroud activity
Eg: Office 365,Sales force,Google workspace
Class 2:(Datacenter architecture)
The definition and documentation of AWS Region is stated in the above comments. In summary, AWS Region is a separate geographic area. AWS Region has Availability Zones which are isolated data centers. Availability Zones is used for high availability. There are 2 or more Availability Zones for each region.
Which factors are important before selecting region in AWS?
There are several factors to consider.
- Latency - The faster your data center, the better your performance. This link can display the latency between ec2 instances. https://www.cloudping.co/
- Cost - Different region has different cost. So far, North Virginia is the cheapest.
- AWS Services to use - Not all AWS Services are available in all regions. This link can display the supported services per region. https://aws.amazon.com/about-aws/global-infrastructure/regional-product-services/
Class 3:(Ip addressing and subnetting)
- One route(i.e first route will be created automatically by aws)(Destination-10.1.0.0/16 and Traget- local), which means all the traffic from hosts within same vpc(i.e all host present in all subnets attached to current vpc) should use of local route table
- Second route i.e (Destination 0.0.0.0/0 and target Internet Gateway) -specifies that if host in current vpc wants to communicate to another vpcs inside aws or internet then they should go to internet gateway.This can be seen from the above picture
- Create internet gateway and attach it to the vpc.
- When you create IG.It will be in ddetached mode.It should be attached to any vpc
- When hosts in a vpc wants to communicate with internet or other vpc, hen traffic should go through Internet gateway
- Creating routes is very very important. It will decide how data should be transfered
- One default route will be created by aws with target local, which is responsible for establishing communication among hosts within same vpc
- Since our traffic should go to outside world, create a route in entry with desination '0.0.0.0/0' and in target select internet gateway created in above step
- The reason behind creating that is, if hosts wihin vpc want to communicate with hosts vpcs or internet they should go through IGw and within vpc the route will be handled by local.
- Create a test security group, which allows all traffic in in bound rule.More discussion about it will be found in later classes
- While creating ec2 instance, select the vpc,subnet and securiy group created in above steps
- You can generate key-pair and download pem file(pem file is used while sshing from linux or mac machines..if you want to connect from windows using putty then convert pem file to ppk file using putty gen and connect )
- key-pair is created accross availabilty zonee only.If we want to have same key-pair across different regions ..Import the public key(Not pem file because it is private key..While converting pem to ppk in putty gen you can find public key or If you have ec2 insance running in the region from which you wan to import ..You can get it from the machine)
Destination
- when we create a vpc, a default rouble table will be created
- when we create subnets in that vpc, all subnets will be associated with that default route
- If we create a Internet gateway and attach it to vpc and edit the above created route table to connect to IGW with destination as '0.0.0.0/0' and target as 'Igw created'.Then all subnets will be connected to internet(vvv IMP)
- (vvimp) Routes are associated with subnets. So for a subnet we can create only one route
- NAT gateway should be created in public subnet(If we have more than 1 public subnet then select any one)
- We should assign elastic ip(which is fixed public ip) to NAT gateway
- we need to add route in private subnet to point to this NAT gateway by selecting destination as '0.0.0.0.0/0' and target as NAT gateway
Note:If we add route to a subnet to internet gateway then that subnet is called public subnet and similary if we ad route to a subnet to point to NAT gateway.We called it as private subnet
Lab practice steps:
1.create a vpc (right click and enable dns host name(This is optional))
2.create 2 subnets (1 was public and 1 was private..just name it we will do it later)
For public subnet, select auto assign public ips bt right clicking on the public subnet,edit subnet settings and enabling respective check box
3.Create internet gateway and attach it to the vpc created in step1
4.By default 1 route was created(Tag name of it as main) and 2 subnets will be added to that route(Aws does this)
5.You need to create another route (name it NAT route) and associate private subnet to this route(Imp note:When we do this step, default route will be disassociated from private subnet as you added it to new NAT route table(since each subnet is associated with single routable only) )
6.Add route to main route table.So that it points to Internet gateway
- 1 route rule with target as local will created by default by aws
(which is responsible for sending traffic across different subnets within vpc) - Create route rule, to allow traffic to outside world Internet or other vpc's by pointing to Internet gateway created in step3
- destination should be '0.0.0.0/0' and target is 'IGW'
7.Create a NAT gateway in public subnet, by using elastic ip as well as shown above
8. Add route in NAT route table created in step 5 to redirect traffic from private subnet to exernal worlds
- destination should be '0.0.0.0/0' and target is 'NAT gateway'
After adding routes, we can call this architecture has 1 public and 1 private subnet.
Test this by creating insances in public an private subnets and pinging them
Note:
Internet Gateway (IGW) allows instances with public IPs to access the internet.
NAT Gateway (NGW) allows instances with no public IPs to access the internet.
A subnet is deemed to be a Public Subnet if it has a Route Table that directs traffic to the Internet Gateway.
The instances in the public subnet can send outbound traffic directly to the internet, whereas the instances in the private subnet can't. Instead, the instances in the private subnet can access the internet by using a network address translation (NAT) gateway that resides in the public subnet. The database servers can connect to the internet for software updates using the NAT gateway, but the internet cannot establish connections to the database servers.
Let’s assume you created a NAT gateway and you have an EC2 instance routing to the internet through the NAT gateway. Your EC2 instance behind the NAT gateway sends a 1 GB file to one of your Amazon Simple Storage Service (Amazon S3) buckets. The EC2 instance, NAT gateway, and S3 Bucket are in the same region of the US East (Ohio), and the NAT gateway and EC2 instance are in the same Availability Zone. We calculate your cost as follows:
- NAT Gateway Hourly Charge: NAT Gateway is charged on an hourly basis. For this region, the rate is $0.045 per hour.
- NAT Gateway Data Processing Charge: 1 GB data went through the NAT gateway. The Data Processing charge will result in a charge of $0.045.
- Data Transfer Charge: This is the standard EC2 Data Transfer charge. 1 GB data was transferred from the EC2 instance to S3 via the NAT gateway. There was no charge for the data transfer from the EC2 instance to S3, as it is Data Transfer Out to Amazon EC2 to S3 in the same region. There was also no charge for the data transfer between the NAT gateway and the EC2 instance since the traffic stays in the same Availability Zone using private IP addresses. There will be data transfer charges between your NAT gateway and EC2 instance if they are in a different Availability Zone.
- In simple words, For NAT Gateway we need to pay hourly charges and data transfer charge
- Created active directory in windows 2016 server(Which inturn creates dns )
- When we create a vpc, default DHCP options will be created which has its own dns server with name(AmazonProvidedDNS)
- You can give our own dns server details here
- So when we create new ec2 instance in this vpc, they will become members of specified dns
- If we want to add newly created systems to domain from workgroup, we need to do so using some scripts
user and groups:
Managed policies – Standalone identity-based policies that you can attach to multiple users, groups, and roles in your AWS account. There are two types of managed policies:
AWS managed policies – Managed policies that are created and managed by AWS.
Customer managed policies – Managed policies that you create and manage in your AWS account. Customer managed policies provide more precise control over your policies than AWS managed policies.
Inline policies – Policies that you add directly to a single user, group, or role. Inline policies maintain a strict one-to-one relationship between a policy and an identity. They are deleted when you delete the identity.
Authentication in AWS is done via (IAM users, groups and roles) whereas Authorization is done by Policies.More info: here
IAM identities provide access to resources under specific conditions within an AWS account. The purpose of an identity is not only to provide access to resources, but to ensure that users are authenticated and authorized so that your digital resources can be properly managed and remain secure. Identities come in three varieties: users, groups, and roles.
Roles are designed so that a set of permissions can easily be delegated to users on an individual basis. For example, instead of assigning an individual all their necessary permissions one at a time, they can be assigned a specific role that contains all the necessary permissions in a single step.
Once a role is created, it can be assigned to as many individuals as needed. This makes roles particularly useful when assigning permissions to new users or changing permissions to users who have shifted jobs within their organization.
When it comes to AWS roles vs. policies, a good rule of thumb is to remember that policies are applied to roles, which can then be assigned to individual users via roles.
Example of an AWS IAM Role
An organization undergoes major expansion to undertake a new project: new employees are coming in, and current employees are shifting positions laterally within the organization. The current employees no longer need access to some of their old permissions but need access to new permissions. Additionally, the new employees need access to a wide variety of permissions to do their jobs. In order to accommodate this rapid new growth, IT administrators need a way to quickly and easily control access to their cloud resources while keeping their infrastructure secure.
Enter AWS IAM roles. Administrators create roles that are tied to specific policies that are appropriate for the new project. Employees—both current and new employees—can then be easily assigned to their new roles so that they only have access to what their new positions require.
The Difference Between AWS IAM Users, Groups, and Roles
As previously mentioned, users, groups, and roles are all specific types of AWS identities. Identities can be authenticated and authorized to perform actions using AWS resources under certain conditions.
IAM User vs. IAM Role
An IAM user is a single person or service entity/application that interacts with AWS resources through service requests and modifications. AWS users consist of a name, password, and a pair of unique API access keys that grant them permissions according to policy condition criteria established by an administrator.
The difference between an IAM role and a user is that a role can be temporarily or permanently applied to a user to give the user bulk permissions for a task. Unlike a user, a role does not have associated passwords or credentials and can be easily applied to multiple users to grant access to a set of permissions at once.
AWS Groups vs. Roles
An AWS group is simply a collection of multiple users. When changes are made to the permissions of the group, the changes affect each individual user within that group. Policies can be attached directly to groups, so there is no need to assign permissions on an individual basis if they are applicable to the entire group. Moving users between groups can attach appropriate permissions when necessary, instead of editing permissions for a single user.
I am role:
An IAM role is an IAM identity that you can create in your account that has specific permissions. An IAM role is similar to an IAM user, in that it is an AWS identity with permission policies that determine what the identity can and cannot do in AWS. However, instead of being uniquely associated with one person, a role is intended to be assumable by anyone who needs it. Also, a role does not have standard long-term credentials such as a password or access keys associated with it. Instead, when you assume a role, it provides you with temporary security credentials for your role session.
You can use roles to delegate access to users, applications, or services that don't normally have access to your AWS resources. For example, you might want to grant users in your AWS account access to resources they don't usually have, or grant users in one AWS account access to resources in another account. Or you might want to allow a mobile app to use AWS resources, but not want to embed AWS keys within the app (where they can be difficult to rotate and where users can potentially extract them). Sometimes you want to give AWS access to users who already have identities defined outside of AWS, such as in your corporate directory. Or, you might want to grant access to your account to third parties so that they can perform an audit on your resources.
For these scenarios, you can delegate access to AWS resources using an IAM role. This section introduces roles and the different ways you can use them, when and how to choose among approaches, and how to create, manage, switch to (or assume), and delete roles.
Overview of users,groups,roles and policies: nice video
usecases of roles: video
Vey very Imp: IAM JSON policy elements reference here
Sample policies for practicing : here
very very Imp, how my company aws account was managed: here
2.Gateway Endpoints
Interface endpoints also let VPC resources connect to supported AWS Marketplace partner services in addition to endpoint services, which are hosted by AWS customers or partners in their own VPCs.
Traffic from VPC resources to the endpoint network interface is controlled by security group rules. AWS PrivateLink then enables the endpoint to connect the traffic to other services without going over the internet.
AWS charges usage and data processing rates for PrivateLink, so there are additional costs involved with creating and using an interface endpoint.
A gateway endpoint is a gateway that is a target for a route in your route table used for traffic destined to either Amazon S3 or DynamoDB.
- Created a vpc with 2 subnets 1 public and 1 private
- As we know for public subnet we create and add Internet Gateway
- And for private subnet we add route to NAT gateway(by creating NAT Gateway)
- Deployed 2 vms in public and private subnet respectively
- SShed into private subnet vm from vm in public subnet and tried to access data in s3 bucket(prior to this we created s3 bucket and uploaded a sample file)
- First and Imp limitation was security, It is unnecessarily leaving aws network and reaching internet for accessing aws inetrnal resources
- Second thing was cost, If we transfer data uisng NAT Gateway then hourly charges were leived on data transfer and since we are also blocking elastic ip for creation of NAT gateway it could also result in higher costs
- we created vpc endpoint to connect to s3 bucket
- We can create gateway endpoints for s3 and RDS services only.(As of now, when Iam writing this blog in Mar 2022)
- When we create gateway end points, routes will be added in the routing table(which we will be selected while creating gateway), in our case route table of private subn
- All the vpc and 2 sunbet creation steps were same
- we created a interface endpoint to access aws system manager(Used to open vms shell(session manger))
- We created a role for accessing SSMS service and assigned that role to vms which we want to manage using session manger
- In our case, we created 1 public server in public subnet( without assigning public IP ) and 1 private server in private subnet
- Before performing creating end points we verified wheather those vms are shod in session manager window by navigating to that page.we didn't see any vms
- Since session manager is aws service,If we want to connect to that service from our vpc we need to create end points
- We need roles if vms needs to access session manager,so we created a role
- Added the role for those 2 vms
- Created a Interface endpoint, there we need to select the vpc and subnets(In case of Gateway endpoint we select vpc and route table)
- 1 interface end point will be created in each subnet i.e(public and private), since we selected both(we have vms running in both subnets)
- For session manager we need to create 3 end points (as shown here).So 6 Network Interfaces(3 in each subnet) will be created
- Those end points are chargable on hourly bases(Delete them once testing is done)
- Now we can manage 2 vms using session manger
- blog: here
- Security groups only allows rules(There is no concept of denying)
- We can assign security groups to security groups(Best use case: here)
- At max we can use 5 security groups for an EC2 instance or any other resource
- Site to site vpn, connect your office(On prem) devices to aws vpc by establishing a tunnel.
- You can communicate with machines in aws vpc using their private ips(without public ips) by establishing vpn coonection
- All the steps involved in establishing connection is documented here
- In realtime, we use site to site vpn only. All the computers in office network connect to aws vpc using corporate vpn
- In very rare cases, If you want your remote employees to coonect to aws vpc or particular subnet.you can use openvpn. details here
- If we want to establish connection between vpc's then vpc peering is one way
- If no of vps increases then veering connections also increases.
- For connecting n vpcs we require n*n-1/2 connections
- We can use transit gateway for connecting any number of vpc only one attachment is required for each vpc
- Architecture is so simple
- No charge for vpc peering connections, only charge for data transferred using peering
- But in transit gateway, there will be charge for gateway attachments as well and data transfer cost is common
- Create a transit gateway
- Create gateway attachment for 3 vpcs
- Add routes, Lets say in 1st vpc route table(public subnet or private subnet) add routes for 2 other vpcs(Destination as vpc CIDR) and targert as respective gateway attachments created in step2
- Perform same steps for other 2 vpcs
- Now test with ping connectivity(While creating machines if all ports are allowed using security group), then these 3 machines can be pinged
- We can also connected vpc's in other region as well
- Similarly we can connect transit gateway with vpn which is in on prem(When employees in office wanted to connected to multiple vpcs with their private ips)
- check in video if you want lab for above 2
- Hybernation behaviour: here
- Enable cloud watch detailed monitoring: By default ec2 instances activity is moinitored by cloud watch every 5 minutes.I we enable this check box then monitoring will be performed for every 1 min
- Tenancy: Shared(99.9% uses shared) .. dedicated and decicated hosts were very rarely used which are very costly
- We can even add second network interface(which is not recomended)
- We can even add secondary ip to the same network interface(default : etho)(Even though only 1 will be seen using ipconfig cmd)
- We can run custom commands after boot straping instance in userdata section
- we can even create a template will details of ec2 instances and launch itvery fastly withot specifying these details again and again
- Created EBS volumes(In the same AZ where ec2 instance is created)
- Attached volume to ec2 instance, by right clicking on volume and attach
- Created partions and mounted file system and saved files in mount point
- Deleted volume(First unmount using umount(deletion of partion using fdisk is not required)and got to volumes and click detach volume and then delete volume) (More details on deleting patitions here)
- All above steps here
- gp2 ssd's comes with baseline IOPs of 3, 100 Iops is minumum for any least size
eg:IF we tak 1 GP EBS gp2 size, we will get 100 IOPs
30 GB -> 30*3 =90<100 = 100 IOPS
40 GB -> 40*3 >100 = 120 IOPS - There is no option to customize.
- For gp2, Io2 and Io3 we can customize IOPS, exact range can be found on documentation
- IOPs are very costly when compared to gp's
- Rest all (apart from gp2,gp3,io2,io3) are HDD's, so there is no concept of IOPS, It deals with spindles.There are very cheap
The main differences are the following:gp3 can scale up to 16.000 IOPS whereas io1 can scale up to 64.000 IOPS
io1 volumes support EBS Multi-Attach and gp3 doesn't
If you need EBS Multi-Attach or more than 16k IOPS, you'll probably want to go with io1 volumes, otherwise gp3 will usually be fine.
- Snapshots and how incremental snapshots work: here(see pic)
- snapshots deletion, here
- state 3 in above pic(deletion) is very very imp,
In state 3, the volume has not changed since State 2, but Snapshot A has been deleted. The 6 GiB of data stored in Snapshot A that were referenced by Snapshot B have now been moved to Snapshot B, as shown by the heavy arrow. As a result, you are still charged for storing 10 GiB of data; 6 GiB of unchanged data preserved from Snap A and 4 GiB of changed data from Snap B. - Summary of above point, since backups are incremental, if a snapshot is being deleted which is referred by other snapshots, then data will be copied to other snapshots before deletion.So we always have fixed size of snapshot as volume size(Even though snapshots are incremental)
- snaphots can be copied to other regions, by right cliking on snapshot.
- XFS filesystem has UUID which are unique identifiers of the filesystem.
- Two file systems with same UUID can not be mounted on the same system.
- If the backing storage of filesystem is cloned or replicated, the metadata that contains UUID information for filesystem will be identical on both filesystems.
mount -o nouuid /dev/xdfv1 /mnt- Buy a domain in godaddy
- Create hosted zone with domain bought in step 1(SOA and Nameserer records will be created)
- Launch an EC2 instance and install nginix on it
- Navigate to route 53 and create A record to point to ec2 public ip
- ipconfig /displaydns - It displays all dns resolver cache detaisl in local machie
- ipconfig /flushdns - It will removes all dns local cache
- Create a vpc and 3 subnets in 3 availabilty zones
- Add routes to all 3 subnets to IGW(Which means those are publoc subnets)
- Create 3 ec2 instances in 3 subnets
- Install nginx in all the 3 instances and change index.html test to uniquely identify those(For testing)
- Create a target group, select TCP conection over port 80
- Add 3 instances to target group
- create a network load balancer and specify target group created above
- Load balancer has public dns, try accessing public dns of loadbalancer using browser
- You should above the same link points to all 3 sites when refreshed one at a time(browsers store cache so try in new browser or different browser)
- If you want to create route 53 records, create a CNAME record which points to public dns of load balancer
- OR since loadbalancer is aws resource, you can create A record and select loadbalancer and select it
- If you want to access it over https, create a public certifacte in certificate manager, specify domain name as '*.bhuvandevops.xyz' and verify your doamin by creating CNAME record which aws provides you(This will be created by aws itself just click on create records)
- Navigate to load balancers, click on listners tab and create a new listner, now select TLS(port 443 will be created automatically) and specify the certificate and cleick on create
- Delete the prevous listner(TCP:80), Now you should access nginx over https using your domain
- Same as above,create 3 vms in 3 az in a vpc
- The only change and is very very very imps is, while creating target groups select (http 80) but not TCP 80(This is the most common mistake, I fyou do like this then after creating application load balancer, when you try to forward traffic to target groups,No targer group will show up)
- Another very import thing, while defining paths in path based in rules, end with '/' else you will get error,same with the case of health checks in targer groups
- Eg: /path is not correct and /path/ is correct
- The difference between application and network load balancer were, we cannot add paths in network loadbalancer and edit the rules
- There is no concept of cross zone load balancer here, since it is enabled by default in application load balancer
I can't route traffic to an Amazon S3 bucket that's configured for website hosting
When you configure an Amazon S3 bucket for website hosting, you must give the bucket the same name as the record that you want to use to route traffic to the bucket. For example, if you want to route traffic for example.com to an S3 bucket that is configured for website hosting, the name of the bucket must be example.com.
If you want to route traffic to an S3 bucket that is configured for website hosting but the name of the bucket doesn't appear in the Alias Target list in the Amazon Route 53 console, or if you're trying to create an alias record programmatically and you're getting an InvalidInput error from the Route 53 API, one of the AWS SDKs, the AWS CLI, or AWS Tools for Windows PowerShell, check the following:
The name of the bucket exactly matches the name of the record, such as example.com or www.example.com.
The S3 bucket is correctly configured for website hosting. For more information, see Hosting a static website on Amazon S3 in the Amazon Simple Storage Service User Guide.
- We need to use third party tools to upload and retrieve data from glacier.
- we can also send data to glacier using s3 by setting s3 lifecycle to one day
- Cost is very very low but retriewing time is high, We need to customize retrieval time by paying more
- It is archieve storage
- In the lab we used fast glacier client to upload data to the glacier and retrieve data
- Same efs storage can be mounted in different ec2 instances
- This is fully managed by aws, we don't need to specify size while creating it. It's size is automatically adjusted and we will pay for what we have used
- EFS storage is region specific, one file system storage can be accessed by all ec2 instances in the same zone spread across different availability zones
- It will create network interfaces in all the availability zones,(which can be observed by navigating to network instances section in EC2 tab)
- So all the instances in an availability zone makes use of respective network interfaces in the backend they all point to same nfs filesystem
Lab Activity:
- Created a vpc and 2 subnets(all 2 public suubnets, added routes to internet gateway)
- Created 2 instances one each in 2 sunets which are spread across 2 availabilty zones
- Created a efs file system(after creating observed 2 network interfaces were created in both availabilty zones)
- In EC2 instances, no need of creating partitions and creating file system. We can directly mount by creating mount points directory and mount it to efs file system(nfs client is installed by default in amozon linux machine)
- The command to mount will be available in aws UI, when we click on attach in efs tab
- Create 2 mount points and tested if data is replicated when we create it in one machine mount point
- Those mount points will be deleted once computer gets restarted so we need to enter those details in fstab
Comments
Post a Comment