CI4TOSCA - Evaluation

This repository contains all materials used to evaluate the CI4TOSCA prototype, including runner configurations, scripts, workflows, raw results, and visualizations.

Overview

The CI4TOSCA framework extends the TOSCA standard with a declarative build stage and enables continuous integration workflows across various artifact types and CI/CD tools. This repository documents the complete evaluation process.

Repository Structure

• 1_runner_files: Runner configuration files
• 2_scripts_and_workflows: Python scripts and n8n workflows for evaluation
• 3_pipeline_files: Pipeline definitions for various CI/CD tools and artifact types
• 4_results: Raw evaluation results in JSON format
• 5_plots: Visualizations generated from the results

How to Reproduce the Evaluation

To reproduce the evaluation results, follow these steps:

1. Set up the CI/CD Runners
   Prepare six runners across three platforms:

   • CI4TOSCA Runners (3x):
     Use the CI4TOSCA Prototype and the webhook script to deploy runners for GitHub Actions, GitLab CI, and Jenkins.
   • GitHub Runner:
     Run in "permanent ephemeral" mode to ensure isolated builds.
   • GitLab Runner:
     Set up in Docker mode.
   • Jenkins Runner:
     Standard Jenkins setup connected to your pipeline environment.

2. Replace Placeholders
   Execute the replacement script and update all placeholders with actual values:

   python 2_scripts_and_workflows/0_replace_placeholders.py

3. Configure the n8n Workflow
   Load and configure the evaluation automation via:

   • 1_n8n_evaluation_workflow.json

4. Upload external pipelines to Jenkins Upload the external pipelines to your Jenkins server. Name them: binary, docker, maven & npm.

   Important: Execute each pipeline once.

5. Run the Evaluation
   Execute the main evaluation script:

   python 2_scripts_and_workflows/2_evaluate.py

6. Simplify the Results
   Post-process the raw results into a cleaner structure:

   python 2_scripts_and_workflows/3_simplify_results.py

7. Generate Visualizations
   Create charts and plots from the processed data:

   python 2_scripts_and_workflows/4_visualize_results.py

8. Calculate Switching Costs
   Compute the costs of transitioning between CI/CD tools and pipelines:

   python 2_scripts_and_workflows/5_calculate_switching_costs.py

9. Visualize Switching Costs
   Generate comparative charts to illustrate switching costs:

   python 2_scripts_and_workflows/6_visualize_switching_costs.py

License

This evaluation is licensed under the MIT License. See the LICENSE file for details.