Set up a Hive cluster on AWS

1. Open the AWS Management Console.

2. Search for VPC and click the resulting link:

3. Click the orange Create VPC button:

4. Scroll to the bottom of the page and click the orange Create VPC button:

You will see the progress under the Create VPC workflow heading:

5. Search for EMR and click the resulting link:

6. Click the orange Create cluster button:

7. Under the Name and applications > Amazon EMR release > Application bundle section:
   • Add Tez 0.10.2 to the bundle selection.

8. Under the Networking section:
   • Click the Browse button next to the Create VPC button.

9. Click the Refresh button at the top right:

10. Select the newly created VPC, then click the orange Choose button:

You will see values similar to the following:

11. Expand EC2 security groups (firewall) and set:
    • EMR-managed security group (Primary node): Create ElasticMapReduce-Primary
    • EMR-managed security group (Core and task nodes): Create ElasticMapReduce-Core

12. Scroll down to Security configuration and EC2 key pair, then click the Create key pair button:

13. Give the key a name (e.g., "hive-key"), then click the orange Create key pair button:

The generated key file (e.g., hive-key.pem) will download automatically.

14. Return to the previous window and click the Browse button next to the Create key pair button:

15. Click the Refresh button at the top right, select the generated key pair, and click the orange Choose button:

You will see the name of the selected key pair:

16. Under the Identity and Access Management (IAM) roles section:
    • Select Create a service role. You will see the default values pre-selected:

17. Under the EC2 instance profile for Amazon EMR section:
    • Select Create an instance profile:

18. Finally, at the bottom right, click the orange Create cluster button:

After a while, you will see a success message indicating the cluster has been created:

19. Scroll down to the Network and security section:
    • Expand EC2 security groups (firewall).
      • Click the link below Primary node EMR managed security group:

20. In the newly opened tab, click Edit inbound rules under the Inbound rules section:

21. In the new window, click the Add rule button at the bottom left:

22. Click the first dropdown menu and type (or select) SSH:

23. Click the second dropdown menu and select My IP:

24. Click the orange Save rules button at the bottom right:

25. Return to the EMR window and click the Refresh button at the top right. You should see the cluster's Status change to Waiting:

26. Before connecting to the cluster, ensure the key file has the correct permissions. Locate the hive-key.pem file (usually in your Downloads folder, not your home folder) and set the appropriate permissions:

% chmod 400 hive-key.pem

27. Return to your browser and click the Connect to the Primary node using SSH link under the Primary node public DNS heading. You will see instructions on how to connect to your newly created cluster:

28. Copy the command and paste it into your terminal. Before executing it, ensure the path to your hive-key.pem file is correct:

% ssh -i ~/hive-key.pem hadoop@**************************.amazonaws.com

You will see the fingerprint message:

The authenticity of host '**************************.amazonaws.com (************)' can't be established.
ED25519 key fingerprint is: SHA256:*******************************************
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])? 

29. Type yes and press Enter.

Finally, you will access the SSH shell for your cluster:

** WARNING: connection is not using a post-quantum key exchange algorithm.
** This session may be vulnerable to "store now, decrypt later" attacks.
** The server may need to be upgraded. See https://openssh.com/pq.html
   ,     #_
   ~\_  ####_        Amazon Linux 2023
  ~~  \_#####\
  ~~     \###|
  ~~       \#/ ___   https://aws.amazon.com/linux/amazon-linux-2023
   ~~       V~' '->
    ~~~         /
      ~~._.   _/
         _/ _/
       _/m/'
Last login: ************************

EEEEEEEEEEEEEEEEEEEE MMMMMMMM           MMMMMMMM RRRRRRRRRRRRRRR
E::::::::::::::::::E M:::::::M         M:::::::M R::::::::::::::R
EE:::::EEEEEEEEE:::E M::::::::M       M::::::::M R:::::RRRRRR:::::R
  E::::E       EEEEE M:::::::::M     M:::::::::M RR::::R      R::::R
  E::::E             M::::::M:::M   M:::M::::::M   R:::R      R::::R
  E:::::EEEEEEEEEE   M:::::M M:::M M:::M M:::::M   R:::RRRRRR:::::R
  E::::::::::::::E   M:::::M  M:::M:::M  M:::::M   R:::::::::::RR
  E:::::EEEEEEEEEE   M:::::M   M:::::M   M:::::M   R:::RRRRRR::::R
  E::::E             M:::::M    M:::M    M:::::M   R:::R      R::::R
  E::::E       EEEEE M:::::M     MMM     M:::::M   R:::R      R::::R
EE:::::EEEEEEEE::::E M:::::M             M:::::M   R:::R      R::::R
E::::::::::::::::::E M:::::M             M:::::M RR::::R      R::::R
EEEEEEEEEEEEEEEEEEEE MMMMMMM             MMMMMMM RRRRRRR      RRRRRR

[hadoop@************* ~]$