Onboarding Manual

This document serves as a step-by-step guide to set up and onboard the Securacloud application for development and deployment.

System Requirements

Ensure your system meets the following prerequisites before proceeding with the setup:

Development Tools

· VS Code (or equivalent IDE)

o Recommended Extensions: ESLint, Prettier, Docker, Git Lens, Python, MongoDB, and Nx Console.

· Node.js

o Version: v20.11.1

· npm

o Version: 10.2.4

· Python

o Version: install the latest Python version.

· Lens

o Lens is used for monitoring logs and scaling of backend services running on Kubernetes.

· Docker

o To run containerized services locally.

· Git

o Version control system for repo connection.

Repo Connection

Clone the required repository from the ADO (Azure DevOps) on local machine VS Code. Make sure to use Git Credentials if you’re using HTTPS URL to clone.

API Gateway:

https://Royal-Cyber@dev.azure.com/Royal-Cyber/SecuraCloud/_git/SecuraCloud-API

Backend Services:

https://Royal-Cyber@dev.azure.com/Royal-Cyber/SecuraCloud/_git/SecuraCloud-Backend

AI Module:

https://Royal-Cyber@dev.azure.com/Royal-Cyber/SecuraCloud/_git/SecuraCloud-AI

Securacloud Frontend:

https://Royal-Cyber@dev.azure.com/Royal-Cyber/SecuraCloud/_git/SecuraCloud-Frontend

Setup

· Run npm install in each cloned repository except SecuraCloud-AI to install dependencies. For SecuraCloud-AI you will need to install python packages with this command pip install -r requirements.txt

· Ensure .env and other configuration files like GCP service account file securacloud-backend-dev.json are correctly populated for each project and placed in the root folder. These can be found in the project-specific documentation or contact your lead for providing you with necessary configuration and .env file.

GKE Connection

Securacloud uses Google Kubernetes Engine (GKE) for deploying services. Follow these steps to connect to the GKE cluster:

Install Google Cloud CLI:

o Download and install Google Cloud CLI.

Authenticate:

gcloud auth login

gcloud config set project <your-project-id>

Get Kubernetes Credentials:

gcloud container clusters get-credentials <cluster-name> --region <region>

Verify connection using:

kubectl get nodes

Projects

Securacloud consists of four key projects.

API Gateway

Handles authentication, routing, and acts as a bridge between the front-end and back-end services. And ensure to read instructions provided in the Setup section of this document for .env and other configurations files.

· Location: SecuraCloud-API

· Install dependencies: npm install

· Run Locally:

npm run develop

Backend Services

The backend services are managed and run using Nx, a powerful mono-repo management tool. And ensure to read instructions provided in the Setup section of this document for .env and other configurations files.

· Location: SecuraCloud-Backend

· Install dependencies: npm install

· Start the Backend:

npm run start # Runs 'nx serve'

· Service-Specific Commands:

o Plugin Executor:

npm run plugin # Runs 'nx serve plugin-executor'

o Resource Executor:

npm run resource # Runs 'nx serve resource-extractor'

o Account Verification:

npm run account-verify # Runs 'nx serve account-verification'

o PDF Report Generator:

npm run pdf # Runs 'nx serve pdf-report'

AI Module

The AI module, developed in Python, serves as an AI Assistant for the Securacloud application, offering intelligent responses to user prompts.

Virtual Setup

If you want to work in a virtual environment, follow these steps in VS Code.

Open the Command Palette (Ctrl+Shift+P), start typing the Python: Create Environment.

Follow the steps to select the interpreter you’ve just installed.

Then choose the virtual environment (venv) and then choose requirements.txt file in your root folder to install the packages.

You’re ready to work in a virtual environment.

In your virtual environment you can ensure that GCP service account file is part of your env by running this command:

$env:GOOGLE_APPLICATION_CREDENTIALS="C:\Users\ABC\Desktop\...\SecuraCloud-AI\securacloud-backend-dev.json"

The above step is important if the virtual environment is not picking up GCP credentials.

If you need to run in the same virtual environment again later just use this command:

myenv\Scripts\activate
where myenv is the name of your virtual environment, it could be different in your case.

Run the AI Module Locally

Run the following command for executing AI Module Locally.

python .\main.py

Securacloud Frontend

The securacloud-frontend is the user interface of the Securacloud application, built using React.js with Typescript.

· Location: SecuraCloud-Frontend

· Install dependencies: npm install

· Run application on three different environments

o Npm run start:dev

o Npm run start:uat

o Npm run start:prod

Local Connectivity

To run all services locally:

· API Gateway: Port 1337

· Backend Services: Port 4000

· AI Module: Port 5000

· Frontend: 3000

Ensure the .env file in each project contains the correct configuration for local development.

Lens Connectivity

Lens is used for scaling the backend services and monitor the logs of the servers. Following are the steps through which you will be able to connect Lens with Securacloud applications.

· Start with installing lens on your desktop

· Acquire a kube config file from your GCP project pertaining to your environment

· After acquiring kube.config file from GCP console, upload it to lens

Database Connection

Securacloud employs multiple databases across environments to handle structured and unstructured data efficiently.

· Local Environment

o Database Type: SQLite

o File Used (only for API-Gateway): data.db

o This file acts as the local Strapi database, reflecting the structure of the dev environment database.

· Remote Environment (Dev/UAT/Prod)

o MongoDB

§ Used for storing configurations, logs, and semi-structured data.

§ Dev Environment Connection:

MONGO_URL= <Connection String>

MONGO_DB=dev

o MySQL

§ Used for structured data like user accounts, billing, and compliance reports.

§ Dev Environment Configuration:

DB=mysql

DATABASE_HOST=----------

DATABASE_PORT=----------

DATABASE_NAME=---------

DATABASE_USERNAME=------

DATABASE_PASSWORD=------

DATABASE_SSL_SELF=-----

· Backend Database Configuration

o The backend services share the same database configurations as the environment they operate in (e.g., Dev/UAT/Prod).

Deployment

Securacloud's deployment process ensures a seamless update and rollout for both frontend and backend services across multiple environments.

Hosting Platforms

Frontend:

o Hosted on Firebase

Backend Services:

o Deployed on GCP Kubernetes

Deployment Tools

· Deployment pipelines are managed using Azure DevOps, with distinct pipelines for each environment (Dev, UAT, Prod).

Multi-Environment Deployment

The pipeline is designed to work seamlessly for all environments:

· Dev: For development and testing new features.

· UAT: For user acceptance testing.

· Prod: For live, production-ready builds.

Each environment has a dedicated pipeline configuration and uses environment-specific project credentials.

In the following sections, we will describe Securacloud application’s deployment for frontend and backend. Because the deployment process for all backend projects are the same.

Frontend Deployment Process

The Azure DevOps pipeline automates the building and deployment of the Securacloud frontend application for all environments: dev, UAT, and prod. The frontend is hosted on Firebase, and each environment's pipeline ensures the deployment process is efficient and consistent.

Pipeline Stages

Trigger

o Automatically triggers the pipeline whenever changes are pushed to the corresponding branch for an environment (e.g., master for prod).

Build Stage

o Node.js Installation:

Installs Node.js version 20.x to serve as the runtime environment for the build.

o Dependency Installation:

Installs project dependencies using npm install --legacy-peer-deps to prevent conflicts.

o Production Build:

Executes npm run build:prod to create optimized artifacts for production.

o Artifact Publishing:

Packages the build artifacts and make them available for deployment in the next stage.

Deployment Stage

o Download Artifacts:

Retrieves the build artifacts from the build stage.

o Firebase Tools Setup:

Installs firebase-tools globally and configures deployment credentials using environment variables (firebase-token, firebase-project).

o Deploy to Firebase Hosting:

Uses firebase deploy --only hosting to deploy the artifacts to the appropriate Firebase Hosting environment.

o Completion Notification:

Provides a confirmation message upon successful deployment.

Environment Variables

The deployment process uses the following variables for Firebase hosting:

· firebase-token: Authentication token for the Firebase CLI.

· firebase-project: Project ID representing the target Firebase Hosting environment.

Locating the Deployment Configuration

The YAML file defining the Azure DevOps pipeline for the frontend application is included in the project repository. Developers can find it in the project they have cloned from DevOps.

Backend Deployment Process

The backend services of the Securacloud project, including SecuraCloud-Backend, SecuraCloud-API, and SecuraCloud-AI, are deployed to GCP Kubernetes. The process of deployment for all three is same that’s why I will be describing only one which is SecuraCloud-API The deployment process consists of two stages:

Pipeline Execution:

The pipeline must be triggered manually for all environments (dev, UAT, and prod) whenever changes are pushed to the repository.

Release Execution:

Once the pipeline is successfully completed, the release process is executed manually to deploy the updated service.

Pipeline Stages

Build Stage

o Update Deployment Configuration:

Dynamically modifies the deployment.yaml file to reference the current Docker image version.

o Artifact Publishing:

Publishes the modified deployment.yaml as a build artifact for the release stage.

Docker Build & Push

o Login to Google Artifact Registry (GCR):

Authenticates with GCR to store and manage Docker images.

o Docker Image Build and Push:

Builds a Docker image using the Dockerfile and pushes it to GCR with two tags:

o Build ID (e.g., http://gcr.io/ <project-name>/<service-name>:1234)

o Latest (e.g., http://gcr.io/ <project-name>/<service-name>:latest)

Automated Deployment Flow

Automatic Pipeline Trigger:

The pipeline is automatically triggered whenever changes are pushed to the branch corresponding to a specific environment (dev, UAT, or prod).

The pipeline builds and publishes updated artifacts and Docker images to the Google Artifact Registry.

Automatic Release Execution:

Upon successful completion of the pipeline, the release is triggered automatically.

The release uses the published deployment.yaml artifact to deploy the updated Docker image to the GCP Kubernetes cluster for the respective environment.

Environment Variables

The pipeline utilizes environment-specific variables for deployment:

DockerImageName: GCR path for the Docker image (e.g., http://gcr.io/ <env>/service-name).
DOCKERFILE: Path to the service's Dockerfile.
TAG: Unique tag for the Docker image (e.g., Build ID).
gcr: Google Artifact Registry connection.

Locating Deployment Configurations

Developers can locate the YAML pipeline files in the repository for each service.

SecuraCloud-KB

Onboarding Manual

System Requirements

Development Tools

Repo Connection

Setup

GKE Connection

Projects

API Gateway

Backend Services

AI Module

Virtual Setup

Run the AI Module Locally

Securacloud Frontend

Local Connectivity

Lens Connectivity

Database Connection

Deployment

Frontend Deployment Process

Backend Deployment Process

Related pages