Development Process

Development Environment

There is a project in a CloudCIX region with multiple Ubuntu 20.04 VMs configured as virtual desktops.

Installed Packages:

• Python3

• Pip3

• Git

• PyCharm

• Sublime

• PgAdmin4

• Docker

Each virtual desktop has access to

• CloudCIX GitLab instance on ports 22, 443, 5005

• Dev PODs on port 443

Required: Source IP Range to allow access for remote Developers to their Virtual Desktop.

From the Virtual Desktops, developers can access GitLab via port 443, git commands are executed over port 22 and docker containers for testing can be pulled over port 5005.

The GitLab Runners access GitLab on port 22 and deploys to the Dev Pods on port 22. Containers exposed on port 443 in the Dev POD allow connections from the virtual desktops.

Introductory Training

An Application is a CloudCIX API system. Each Application represents a set of functionality powered by a REST API, Django REST Framework. An example of an Application is Membership. Each Application consists of services. A service is a group of methods defined to interact with a model. An example of a service in Membership is User. CRUDL methods are defined to handle an end user request with a service. The end user requests will be validated against the controller, authorized via the permission and the data returned in the response defined in the serializer.

Each of the applications are built within the same base framework. The CloudCIX Developers Manual, published to GitHub, includes the layout of each of the application. It includes layout and style guidelines for

1. Python3 projects

2. TypeScript projects

3. CloudCIX Applications

CloudCIX Architecture

We design and develop CloudCIX software in our self managed GitLab instance. We publish CloudCIX source code to GitHub, containers to Docker Hub and SDK to PyPi.

The network, volumes and containers required to build each pod is defined in docker compose files published to pod_yaml repository in GitHub.

Development Tools

Django REST Framework

CloudCIX APIs are developed based on the Django REST Framework.

The CloudCIX framework is the base Djago Rest Framework for Python3 CloudCIX APIs. It contains functions and settings that are common and available to all CloudCIX APIs.

It is extended by

1. Copying the app repository into the framework e.g. framework/membership

2. Moving the settings_local.py file for the app to framework/system_conf/settings_local.py

3. Moving the urls_local.py file for the app to framework/system_conf/urls_local.py

4. Moving the errors folder file for the app to framework/errors

Django

CloudCIX Apps which consume the CloudCIX APIs are developed based on Django Python web framework.

GitLab Ultimate

• Epics are a GitLab tool we use to manage and group issues across multiple projects when adding or extending a feature. Via Epics we can create child epics to group issues by their stage in the development process. An issue is assigned to one or more members of the team to complete instructions outlined.

• Inbuilt Security Scanning

  • Static Application Security Tests (SAST): Scans the source code for known vulnerabilities and sorts by priority.

  • Dependency Scanning: finds security vulnerabilities in the software dependencies (third party libraties).

  • Container Scanning: Scans docker containers after they are built for known vulnerabilities that may arise from the base Docker image used.

  • Secret Detection: Secrets include keys, passwords, API token and other sensitive informauion. Secret Detection, scans the source code to ensure none are accidentally commited to the master branch.

• GitLab Container Registry: Build docker containers and publish to the GitLab registry for testing before deploying the CloudCIX Docker Hub account

The network, volumes and containers required to build each pod is defined in docker compose files published to GitHub.

Introduction to CloudCIX Membership

Membership is our core application as it is utiliized by all our other applications to define what method can be executed on a service based on member, address or user.

There are two types of Members, Self Managed and Non Self Managed.

A Self Managed Member can only have it’s details modified by an Administrator in the members account.

A Non Self Managed Member is created by another Self Managed Member. The creating Member users can modify the details of the Non Self Managed Member. Users in the Non Self Managed Member can read all details associated with their Member. An Administrator in the Non Self Managed Member account can make the Member Self Managed. Once this is done, Users in the creating Member can no longer modify details.

A CloudCIX Member can have one or more Addresses.

A User in CloudCIX is associated with an Address and Member. Users can have multiple roles: * Administrator - In membership can create Addresses and Users in their Member account * Global - A global user has the option the ability to see all data related to all addresses in their Member. * Global Active - Only a global user can be global active. When global active, they can see data related to all addresses in their Member. If the global user only wants to see the data for their current address, this flag will be False. A global active user can also move their User to another Address in their Member.

Connections in CloudCIX are between Members and Addresses. These are referred to as Member Links and Address Links. There will be two link records in each direction to complete the connection. The Link service is utilized to define the business relationship an Address has with the Contra Address.

• Financial Roles

  • Customer:

  • Supplier:

• Support Roles

  • Client

  • Service Centre

  • Warrantor

• Cloud Roles

  • Cloud Builder

  • Cloud Distributor

  • Cloud Seller

  • Cloud Customer

Process Overview

This diagram outlines the key stages in the CloudCIX development process.

POC Concept

A Proof of Concept (POC) is initially developed by a subject matter expert (SME) in the relevant technology to consider the feasability of the proposed feature and to investigate alternative strategies for implementation. If the POC is sucessfull, then a developer shadows the SME to capture the sequence of events and commands executed to build in preparation for automating this process.

POC Envelope Test

The POC Envelope Testing looks to find the hard limits of the feature and determines how scaleable it is. The sequence of events and commands utilized to manually build the feature in the POC Concept stage are converted to scripts to assist in the POC Envelope Testing Process.

ERD / CRUDL / Service Definition

Form the previous POC stages, we now examine the services Entity Relationship Diagram (ERD) to see if

1. The feature is a model we already have or need to add.

2. What fields are required for the user GUI and what fields Robot requires to build the feature.

Next the CRUDL (Create, Read, Update, Delete, List) API methods are designed based on the updated ERD. This involves validating data sent by the user in a create/update request, defining the permissions on who can make a CRUDL request and what should be returned in the response to the requesting user.

GUI Develpment

After the API is developed, it is deployed to a staging environment where the GUI is designed, developed and tested to support the new feature. Typically this involves adding a capability to one of the CloudCIX Apps.

Robot Development

The scripts from the POC Envelope Test stage are translated into Python code and Jinja templates and Robot is updated.

ISOfication

New featues may require changes to the installation ISO. This ISOfication step involves updating the ISO and/or updating Docker images for deployment to production. The ISO is designed to build Pod Appliances and PodNets with the default requirements, libraries installed, IP Assignments and the relevant docker-compose files.

Deploy to DevTest / Test

DevTest is a region specifically built to test candidate versions of CloudCIX. If there are changes to the ISO, then the DevTest Region will be rebuilt using the latest ISO to ensure it installs as intended. Any bugs that arise are reported back, correct and depending on the corrections made, the ISO and/or the Docker images are updated and testing recommenses.

The Services API, GUI and Robot are all deployed and tested together in DevTest. Requests sent via the GUI are processed and accepted by the API. Robot reads the API, retrieves the request and executes as required. Any issues arising via Robot are reported back and examined in the Robot Development

Deploy to Production

When testing is complete, software is deployed to the relevant repositories as required (i.e. ISO download, PyPi, GitHub and DockerHub). The new feature is rolled out to production Pods.