Run Python Script In Gitlab Ci

Stackoverflow.com

Instead of having duplicate scripts inside your.gitlab-ci.yml, we created a Python script inside our repo. To make it more readable, we modified the Python script with Click!, turning it into a CLI. This page contains links to a variety of examples that can help you understand how to implement GitLab CI/CD for your specific use case. Examples are available in several forms. As a collection of:.gitlab-ci.yml template files maintained in GitLab, for many common frameworks and programming languages. Repositories with example projects for. In this Python tutorial, you'll learn the core concepts behind Continuous Integration (CI) and why they are essential for modern software engineering teams. Find out how to how set up Continuous Integration for your Python project to automatically create environments, install dependencies, and run tests.

Project pipelines¶

A pipeline is a group of jobs executed by GitLab CI.

Reference¶

  • v4 API:

    • gitlab.v4.objects.Project.pipelines

  • GitLab API: https://docs.gitlab.com/ce/api/pipelines.html

Examples¶

List pipelines for a project:

Get a pipeline for a project:

Get variables of a pipeline:

Create a pipeline for a particular reference with custom variables:

Retry the failed builds for a pipeline:

Cancel builds in a pipeline:

Delete a pipeline:

Triggers¶

Triggers provide a way to interact with the GitLab CI. Using a trigger a useror an application can run a new build/job for a specific commit.

Reference¶

  • v4 API:

    • gitlab.v4.objects.Project.triggers

  • GitLab API: https://docs.gitlab.com/ce/api/pipeline_triggers.html

Examples¶

List triggers:

Get a trigger:

Create a trigger:

Remove a trigger:

Full example with wait for finish:

You can trigger a pipeline using token authentication instead of userauthentication. To do so create an anonymous Gitlab instance and use lazyobjects to get the associated project:

Reference: https://docs.gitlab.com/ee/ci/triggers/#trigger-token

Pipeline schedule¶

You can schedule pipeline runs using a cron-like syntax. Variables can beassociated with the scheduled pipelines.

Reference¶

  • v4 API

    • gitlab.v4.objects.Project.pipelineschedules

    • gitlab.v4.objects.Project.pipelineschedules

  • GitLab API: https://docs.gitlab.com/ce/api/pipeline_schedules.html

Gitlab Ci Cd Example

Examples¶

List pipeline schedules:

Get a single schedule:

Create a new schedule:

Update a schedule:

Trigger a pipeline schedule immediately:

Delete a schedule:

List schedule variables:

Create a schedule variable:

Edit a schedule variable:

Delete a schedule variable:

Jobs¶

Jobs are associated to projects, pipelines and commits. They provideinformation on the jobs that have been run, and methods to manipulatethem.

Reference¶

  • v4 API

    • gitlab.v4.objects.Project.jobs

  • GitLab API: https://docs.gitlab.com/ce/api/jobs.html

Examples¶

Jobs are usually automatically triggered, but you can explicitly trigger a newjob:

List jobs for the project:

Get a single job:

List the jobs of a pipeline:

Note

Job methods (play, cancel, and so on) are not available onProjectPipelineJob objects. To use these methods create a ProjectJobobject:

Get the artifacts of a job:

Get the artifacts of a job by its name from the latest successful pipeline ofa branch or tag:

project.artifacts(ref_name=’master’, job=’build’)

Warning

Artifacts are entirely stored in memory in this example.

You can download artifacts as a stream. Provide a callable to handle thestream:

You can also directly stream the output into a file, and unzip it afterwards:

Get a single artifact file:

Get a single artifact file by branch and job:

Mark a job artifact as kept when expiration is set:

Delete the artifacts of a job:

Get a job trace:

Warning

Traces are entirely stored in memory unless you use the streaming feature.See the artifacts example.

Gitlab Api Python

Cancel/retry a job:

Play (trigger) a job:

Erase a job (artifacts and trace):

Pipeline bridges¶

Get a list of bridge jobs (including child pipelines) for a pipeline.

Reference¶

  • v4 API

    • gitlab.v4.objects.ProjectPipeline.bridges

  • GitLab API: https://docs.gitlab.com/ee/api/jobs.html#list-pipeline-bridges

Examples¶

List bridges for the pipeline:

Pipeline test report¶

Get a pipeline’s complete test report.

Reference¶

  • v4 API

    • gitlab.v4.objects.ProjectPipeline.test_report

  • GitLab API: https://docs.gitlab.com/ee/api/pipelines.html#get-a-pipelines-test-report

Examples¶

Get the test report for a pipeline:

I joined iib in January and since the beginning it was clear that I’d have a lot of fun automating a few manual process. The development team is formed by three persons: the CTO, another developer and me. The team is small but step by step we’re going towards the technical excellence we want. :)

One of the things that needed automation was the release of an internal library. The process to install it was basically git pull + python setup.py install. Since we have many other things to do, this problem was waiting a little longer to be solved. But the things became ugly when I had to setup a test pipeline for Airflow (must say that now I understand why the most famous blog post on this topic is called Data’s Inferno: 7 Circles of Data Testing Hell with Airflow - but this discussion I’ll leave for another day).

At iib we use a self-hosted Gitlab instance, recently updated to the version 13. Gitlab offers a private Package Registry out of the box. The package registry was there, we had our project using poetry and a test pipeline. Publishing the package would be a piece of cake, right?

Where is this Package Registry after all?

I’m going to start by the most obvious thing that I took a few minutes to figure: you have to enable the Package Registry in your project in order to use it. Go to General > Visibility, project features, permissions > Repository > Packages.

Forbidden

Python Gitlab Module

As you can imagine, you’ll need the right permissions to upload the package to your own pypi. This can be done using CI_JOB_TOKEN, an variable provided by Gitlab during a pipeline run. Below you can see how we did it:

The username for the token `CI_JOB_TOKEN is gitlab-ci-token. Another important variable provided during the pipeline execution is CI_PROJECT_ID (you can see the number in the repository page too).

Automation FTW

The thing that got me really excited was the automation flow for our releases. With the job shown previously, we only needed to create a new tag and then a new version would be released. I added BUMP_RULE as an environment variable because we can control the rule used by just changing the repository environment variable. Poetry provides different options (following the SEMVER style).

Poetry publish #fail

Note that we used Twine instead of Poetry. Unfortunately it wasn’t possible to use poetry publish due to a different errors:

  • RuntimeError Repository gitlab is not defined (when having it configured in pyproject.toml)
  • UploadError HTTP Error 404: Not Found (when configuring the repository via CLI)

There are open issues for it, so I hope it will be possible in the future. By now, Twine is doing the trick just fine.

pip install

Since the package is published in the private package registry, we’ll need to adapt the command a little bit.

A trick here is: you’ll need the index URL and the extra index URL for the dependencies that are coming from Pypi. In the end the command will look like this:

Without the --extra-index-url you’ll get errors like ERROR: Could not find a version that satisfies the requirement.

Unfortunately, Gitlab’s documentation is missing some details but you can find their step by step here.

Hope it helps! See ya next time.

Please enable JavaScript to view the comments powered by Disqus.comments powered by Disqus
Comments are closed.