PCM Development/Build Infrastructure/Infrastructure Operation

Aus SDQ-Wiki
Warning
This page was moved to the GitHub wiki

Overview on CI/CD Infrastructure

The infrastructure providing the means to do continuous integration (CI) and continuous deployment (CD) consists of the core components shown in the overview in the following figure.

Palladio-BuildDocumentation-Infrastructure.png

  • All repositories from the Palladio-Simulator organization are built using our self-hosted runners.
  • All repositories from the MDSD-Tools organization are built using GitHub-hosted runners.

Self-Hosted GitHub Runner

  • Hosted on bwCloud
  • 4 VMs running the GitHub Actions Service
  • Run a minimal graphical user interface (required for tycho-surefire UI tests)
  • Run Maven 3.8.7 and Java 17

Set up

  1. Create VM in bwCloud
    • Ubuntu 22.04
    • M1.large
    • 60 gb Disk (hard coupled with the vm)
  2. Run script to install all necessary dependencies
  3. Install Maven 3.8.7
  4. Install Java 17
  5. Start docker
    • sudo usermod -aG docker $USER
    • newgrp docker
  6. Restart VM
  7. In the GitHub Organization:
    1. Go to 'Settings' -> 'Actions' -> 'Runner'
    2. Click 'New runner'
    3. Follow directions
  8. To start the runner as a service, run:
    1. sudo ./svc.sh install
    2. sudo ./svc.sh start

Overview on CI/CD Infrastructure (Deprecated)

The infrastructure providing the means to do continuous integration (CI) and continuous deployment (CD) consists of the core components shown in the overview in the following figure. Public access means that the component is accessible to everyone. Internal access means that access is limited to certain IP addresses.

Palladio-BuildDocumentation-InfrastructureComponentsOverview.svg

Github

We store all source code in Github repositories. Github notifies the Jenkins build server about every change in the registered Github organizations.

Jenkins

We use Jenkins as build server software. The build server receives notifications from Github and frequently polls changes from the Github organization (e.g. the addition of a new repository). Jenkins creates a build job for every repository containing a Jenkinsfile. If the code of a repository changed, a build is triggered. Jenkins distributes the actual work to worker nodes.

Worker

The worker nodes execute the tasks of a build job. Jobs usually are executed within Docker containers. The workers pull the necessary images from a selfhosted docker registry or a public registry.

Docker Registry

To save time during the build, we run a docker registry containing preconfigured docker images.

Updatesite

We run a SFTP server, through which the workers can publish Eclipse artifacts onto an updatesite. The updatesite itself is just a webserver serving the contents of the SFTP server.

Sonatype

Sonatype operates an open source repository for maven artifacts. Workers can deploy built maven artifacts to Sonatype in order to release them to maven central.

TSM-Backup

All infrastructure components, which contain important data, backup this data to a backup system operated by KIT. We backup all nodes except for worker nodes because they do not persist any important data and are easy to setup. The configuration is covered by the setup protocols references below.

Breakdown of Network Services (Deprecated)

The services described above are realized by various applications allocated on various hosts. The following image gives an overview on the involved hosts and applications. We describe the URLs to access the services in the following. Credentials are available in the internal wiki.

Palladio-BuildDocumentation-NetworkComponents.svg

Docker Registry
  • accessible via [1] without credentials
  • write access via [2] requires credentials
  • setup protocol and credentials: Docker Registry (internal)
Jenkins Palladio
Jenkins MDSD.tools
Updatesite Palladio
Updatesite MDSD.tools
SFTP Server Palladio
  • only accessible from the institute's subnet via i43vm-docker.ipd.kit.edu:22001
  • login only possible by an authorized SSH key
  • setup protocol: SFTP Server (internal)
SFTP Server MDSD.tools
  • only accessible from the institute's subnet via i43vm-docker.ipd.kit.edu:22002
  • login only possible by an authorized SSH key
  • setup protocol: SFTP Server (internal)
Jenkins Workers Palladio
Jenkins Workers MDSD.tools
Backup Jenkins
Backup Updatesite / Docker Registry
Sonatype
  • Registered groups: org.palladiosimulator, tools.mdsd
  • User account: kitsdq

Maintenance Tasks (Deprecated)

  • Responsible: Build Server Managers
  • Virtual machines: regularly install updates
  • Docker images: regularly build up to date images
  • Docker containers: pull and use most recent images
  • Overview on managed applications and VMs (internal)
Abgerufen von „https://sdq.kastel.kit.edu/mediawiki-sdq-extern/index.php?title=PCM_Development/Build_Infrastructure/Infrastructure_Operation&oldid=35845“