Installation and Setup

Prerequisites

This guide assumes that you have executed the following prerequisite steps and have the required details ready before getting started with the cloud installation.

AWS account and Initial setup

Create AWS access Keys for IAM user
- This credential will be used to access your AWS account during deployment
- On your AWS account, procure access key ID and secret access key for programmatic access to your AWS resources.
- Prefer to obtain a non root IAM user with administrator access.
- Refer to the AWS documentation to create access key https://aws.amazon.com/premiumsupport/knowledge-center/create-access-key/.
Register a Domain for your reference cloud and configure Route53 hosted zone to deploy applications under. You need to register a new domain within the same AWS account used for the installation.
- reference-Cloud deployment needs a Domain and Route53 hosted zone to support HTTPS endpoints.
- Request domain name through AWS Route53.
- Refer to the AWS documentation guide https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/domain-register.html.
- Under the deployment variable file, set domain.external to false.
- Fill in domain.name with the newly registered domain name. e.g., dev.my-domain.com.
- Follow the installation guide to run the script to set up the infrastructure.
The script times out in a short period of time. If you can’t finish the aforementioned steps within the time limit, simply run the installation script again once the name servers are populated in the parent domain, and the certificate will be able to get verified.

NGC account - MoJ Org

Installation scripts and reference cloud containers are available for distribution under MoJ Org in NGC. Make sure your NGC account has access to the MoJ org.

Note:Installation of MoJ Cloud includes utilization of various AWS managed services such as AWS Private CA, AWS Secret Manager, AWS ElasticCache, and AWS RDS, among others. Please be advised that this may incur AWS infrastructure costs exceeding $100 USD per day, depending on your usage.

Prepare Installation laptop

Follow through the instructions in this section and install the software and tools in your laptop/Desktop from where you are planning to trigger cloud installations.

Setup the OS
- Use any linux based distribution(Ubuntu, Centos… ) with openssl version >= 3.0.0
Install Terraform
- Use the following guidance in the following link and install terraform https://developer.hashicorp.com/terraform/tutorials/aws-get-started/install-cli
- Always use the latest version. Current version tested on: terraform v1.4.6
Install AWS CLI:
- Refer to the official documentation and install AWS cli https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html
- Always use the latest version. Current version tested on: aws-cli/1.19.38 Python/3.8.2 Darwin/19.6.0 botocore/1.20.38
Install jq tool:
- Jq will be used by the cloud script during the creation of PKI assets
- Refer to the official documentation and install jq https://jqlang.github.io/jq/download/
Install curl tool
- curl will be used by the cloud script during the creation of PKI assets to check if the registry credential is correct
- Refer to the official documentation and install curl https://everything.curl.dev/get
Update openssl if needed
- Openssl is needed to create PKI
- Make sure your OS has the latest version of openssl. The minimum versioning required is 3.0.0
- Refer to this documentation. As of today the latest version used for setting up pki is OpenSSL 3.1.3 19 Sep 2023 (Library: OpenSSL 3.1.3 19 Sep 2023)
- https://webhostinggeeks.com/howto/install-update-openssl-centos/

Download & Extract installation Artifacts

Log in to NGC and navigate to moj org, release team
Download the reference CLOUD SCRIPTS
- Navigate to resources on the left panel. Should see “MOJ CLOUD SCRIPTS” as one of the listed items. Select and Download the latest archive file to your laptop.
Untar and extract

unzip files.zip
gunzip moj-cloud-scripts.tar.gz
tar -xvf moj-cloud-scripts.tar

Download installation terraform script

Download Terraform archive file moj-cloud-terraform.tar.gz from NGC moj/nvidia-dev to your laptop. This file should be available under Resources -> “MOJ CLOUD TERRAFORM” resource. Select and download the file from the latest version of the resource.

Untar and extract installation script

gunzip moj-cloud-terraform.tar.gz
tar -xvf moj-cloud-terraform.tar

Pre-Installation steps

Configure AWS CLI

Run the following command to configure your cli. You will be asked to enter the AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and region. Use the credential you downloaded from steps before. The credential has both AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. As for the region, pick the region you want to deploy your reference Cloud at. Leave the default format to None.

Remember the region that you have configured, it will be used throughout this process. For the complete list of available region, check out https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html.

aws configure

This is how they will respectively look on the screen as you run the above command. Make sure you enter the value for each of them as discussed above:

AWS Access Key ID [****************LG5P]:
AWS Secret Access Key [****************xZVi]:
Default region name [us-east-1]:
Default output format [None]:

This will create a default profile in your AWS home config file at ~/.aws/config.

cat  ~/.aws/config

The credential is located at ~/.aws/credentials.

cat ~/.aws/credentials

Configure pki assets

Running the cloud script will create a general-purpose private CA in your AWS account. Be aware of the pricing for AWS private CA before running the script. https://aws.amazon.com/private-ca/pricing/

You should use the CLOUD SCRIPTS you just untar-ed in the previous section

Change directory into

$ cd moj-cloud-scripts/bin/tcpmux/pki

Update config files if needed:

configuration/openssl.cnf: This file contains config value for a Private CA to be created. You should not update this file

configuration/tcpmux-subordinate-ca.json: This file contains configuration value that will be used to create a subordinate CA in AWS. Update the Subject section with your organization information.

configuration/openssl_v3_tcpmux_server.cfg: This file contains configuration value for iotgateway server certificate. Update the req_distinguished_name with your organization information. Very important is the commonName_default. It has to match your domain name you will be using. You should set commonName_default to your-domain-name.com. Make sure you start with a star sign as shown in in *.rosie-cloud-demo.com. That will help support subdomains that will be crated. This is critical otherwise your device will fail the verification process during the provisioning stage. Here is the list of attributes to update
- countryName_default
- stateOrProvinceName_default
- localityName_default
- organizationName_default
- organizationalUnitName_default
- commonName_default
- emailAddress_default
For more information about this, visit https://www.ibm.com/docs/en/i/7.1?topic=concepts-distinguished-name Its purpose is simply to identify your organization.

Execution steps:

In the first stage after executing the script, you will be asked to enter the image registry credential(username and password). That is the credential needed to pull the reference cloud container images form their remote location. If you do not enter the correct credential, you will be asked to retry until the correct values are entered. Instruction to get the credential are documented in the Quick Start Guide. Here is a sample output.
Enter the username for the container image registry : $oauthtoken
{
    "username": "$oauthtoken",
    "password": ""
}
Enter the password for the container image registry : xxxxxxxxxxxxxxxxxxxxx
{
    "username": "$oauthtoken",
    "password": "xxxxxxxxxxxxxxxxxxxxx"
}
In the second stage, You will be given the option to create AWS private CA. Type YES and confirm. This will create the pki assets. You will have another chance to update the subject names for the tcpmux-server-cert. If you already configured configuration/openssl_v3_tcpmux_server.cfg as shown in the previous steps, you can simply confirm the value that will be used and hit Enter. But if you wish to update the value, type in the value and confirm. In the sample below, only the email address is updated while others are left unchanged
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter ., the field will be left blank.
-----
State or Province Name (full name) [CA]:
Locality Name (eg, city) [SANTA CLARA]:
Organization Name (eg, company) [NVIDIA INC]:
Organizational Unit Name (eg, section) [Engineering]:
Domain Name [*.your-domain.com]:
Email Address [it@nvidia.com]:test@nvidia.com

The script can take up to a minute to execute. Remember to respond to prompt when question is asked or confirmation is needed. Run the following command to create the pki assets.

$ chmod u+x ./ca-manager.sh
$ ./ca-manager.sh

Make sure the script complete the creation successfully.

PKI assets used by MOJ cloud are the following files:

assets/moj/tcpmux-server-cert.pem: used for iotgateway mutual auth

assets/moj/tcpmux-server-key.pem: used for iotgateway mutual auth

assets/moj/tcpmux-subordinate-ca.pem: intermediate certificate

assets/moj/tcpmux-root-ca.pem: root certificate

assets/moj/authorizer-pri-key.pem: authorizer private key used to sign JWT token

assets/moj/authorizer-pub-key.pem: authorizer public key used to sign JWT token

assets/moj/tcpmux_subordinate_arn.txt: tcpmux subordinate arn value. Will be used by the provisioning server to issue certificate

assets/moj/image-registry-creds.json: credential to download container form registry

Copy the pki folder from its source moj-cloud-scripts/bin/tcpmux/pki/pki to the moj-cloud-terraform script at location deployment/secretmanager/. Assuming you are in the moj-cloud-scripts/bin/tcpmux/pki folder from where you ran the script, run this copy command:

$ cp -r pki  FULLPATH-OF-moj-cloud-terraform/deployment/secretmanager/

Other files of interest: These should be kept in a safe place and not be shared. Will need it when creating a new subordinate CA assets/private/root_ca_password.txt this is the password for your root CA. assets/private/root_rosie_ca_private_key.pem this is your root private key assets/private/tcpmux-root-ca.pem this does not have to be kept secret but you will need it every time to use the the root ca

You should also securely save a copy of pki.tar.gz somewhere for future use. This will prevent you from re-running the pki cloud script.

NB: The script creates a general-purpose private CA in the region configured in STEP 1.

Configure reference-cloud infrastructure

Following changes are specific to your setup. You are free to configure them however you want. See the list of variable definitions and sample values from the file below:

$ cd deployment
$ vi variables.tf

Some default value are set to simplify your deployment, but there are mandatory field that will need to be filled out depending on your deployment environment. Simply create an override file my-env.tfvars to override default variables and provide values for mandatory variables. For example,

region = "us-east-1"
vpc_cidr_block = "10.0.0.0/16"
public_subnets_cidr = ["10.0.1.0/24", "10.0.2.0/24"]
private_subnets_cidr = ["10.0.3.0/24", "10.0.4.0/24"]
...

Configure Terraform to use your AWS Account

There are few options to configure Terraform to access your AWS account where reference-cloud needs to be set up. Below is one of the simpler options. Export your_aws_access_key_id and your_aws_secret_access_key as an environment variable.

export AWS_ACCESS_KEY_ID=your_aws_access_key
export AWS_SECRET_ACCESS_KEY=your_aws_secret_access_key

Setup for Terraform Deployment State

Execute the following command to Create required DynamoDB table and a S3 bucket to store deployment state and terraform locks. Enter the same region value as in previous steps. The terraform deployment state and lock keep track of the infrastructure, and are used as a reference for update and deprovision in the future. Use the same deployment state and lock for the lifetime of your infrastructure and avoid deleting the resources from AWS.

    $ cd deployment_state
$ vi s3.tf   # Choose a unique name for the s3 bucket to store the deployment state
    resource "aws_s3_bucket" "rcb-terraform-state" {
        bucket = "xxxxxxxx"
    }
    $ terraform init
    $ terraform plan
    $ terraform apply

terraform init is for creating the state of your current infrastructure to be deployed. terraform plan is a testing process to see if your configuration is correct. You may be asked to provide the region. If so, do enter the region and hit Enter terraform apply is for actually deploying the infrastructure. You will be asked to confirm the deployment. Do type yes then hit Enter or Return depending on your keyboard.

Adjust the region and s3 bucket name in the terraform script to use the newly created deployment state.

$ cd deployment
$ vi main.tf

backend "s3" {
   region         = "xxxxx"  # Enter the same region value as in the previous step.
   bucket         = "xxxxx"  # Enter the same s3 bucket name as in the previous step.
}

Cloud Installation

Create Infrastructure and Deploy reference-cloud services:

$ cd deployment
$ terraform init
$ terraform plan --var-file ./my-env.tfvars
$ terraform apply --var-file ./my-env.tfvars

It will take a while for the above step to complete. Ensure above steps are executed to its completeness. During deployment, the script might error out due to failure during certificate provision. This is caused by delay in DNS progation in AWS infrastructure, simply re-run the terraform apply step should solve the problem.

Post Installation validation

Once the deployment is finished, you can view all the service status under ECS in AWS console.

As shown in the above sample, status of all the MoJ cloud microservices should be Active and healthy.

If everything looks healthy, you can access your endpoints. All endpoints URL are printed to the console once your installation finishes.

https://cloud-gateway.<domain-name>
https://cloud-proxy.<domain-name>
cloud_gateway_endpoint = "https://cloud-gateway.<domain-name>"
cloud_proxy_endpoint = "https://cloud-proxy.<domain-name>"
cognito_jwk_endpoint = "https://cognito-idp.us-east-1.amazonaws.com/<region>_<user-pool-id>/.well-known/jwks.json"
cognito_login_endpoint = "https://cognito.<domain-name>/login?redirect_uri=<redirect uri>&client_id=<client id>&response_type=code"
cognito_token_endpoint = "https://cognito.<domain-name>/oauth2/token?redirect_uri=<redirect uri>&client_id=<client id>&grant_type=authorization_code&code=xxxxxxxxxx"

Check to see if all the DNS records are resolvable before moving onto the device onboarding.

nslookup cloud-gateway.<domain-name>
nslookup cloud-proxy.<domain-name>
nslookup cognito.<domain-name>

Please be noted that the DNS record can take up to 48 hours (usually happen within a couple hours) to get propagate globally after a successful deployment, this is because DNS resolvers across the internet typically request the name servers only once every two days and cache the answer. You might need to wait for the propagation to finish if the endpoints above do not resolve.

With cloud installation successful above a continue with device onboarding, claim ownership of the device in cloud and connect to the device via REST APIs after successful authentication and authorization. You may re-run terraform apply after a successful installation, all the endpoints will be printed again.

Uninstallation

To destroy the infrastructure. run the following command

$ cd deployment
$ terraform destroy

Remove the private CA from your account

Deleting resources in your infrastructure will not remove the AWS Private CA. You will need to manually delete it to prevent incurring cost for the following month. To delete it using aws cli run the following command. You will need to copy the content of assets/moj/tcpmux_subordinate_arn.txt and replace in XXX-CERT-ARN.

$ aws acm-pca delete-certificate-authority --certificate-authority-arn XXX-CERT-ARN --permanent-deletion-time-in-days 7

for more information about the deletion process, follow the official documentation at https://docs.aws.amazon.com/privateca/latest/userguide/PCADeleteCA.html