GitLab, a cloud-based, open-source Git repository used by over 38,000 companies around the globe, plays a pivotal role in modern software development. For the organizations that rely on it, it’s a vital collaboration tool and innovation enabler, allowing developers to swiftly test, monitor, and deploy code and housing the full CI/CD pipeline.
For the attackers and internal bad actors who prey on it, GitLab represents something else: a rich source of organizational value filled with intellectual property. So, understanding the risks for potential attacks and misuse is important for GitLab users. As cloud-driven organizations rely more and more on GitLab's offerings, safeguarding their security is crucial. Recent CVEs such as CVE-2023-7028, CVE-2023-5356, CVE-2023-4812, CVE-2023-6955, and CVE-2023-2030, emphasize the essential requirement for strong GitLab security measures in place.
In this blog post, we will help you understand the potential attacks and misuse by describing GitLab’s uses as a SaaS (Software as a Service) platform, uncovering its risks, sharing how attackers could potentially gain access without proper authorization, and explaining ways GitLab’s risks can be investigated and mitigated.
In this blog we'll answer questions including:
- What is GitLab?
- How can attackers gain access to GitLab?
- What GitLab forensic data can be investigated?
- What are caveats of the managed SaaS GitLab logs?
- What detections can be written with managed SaaS GitLab logs?
- How do you prevent and mitigate GitLab threats?
To do so, we’ll elaborate on the available forensic data and provide effective methods for identifying security threats. We will also show you how to perform threat hunting for suspicious activities with the forensic logs, focusing on managed SaaS GitLab environments, and delve into hunting potential threats, by analyzing forensic logs, specifically focusing on managed SaaS GitLab environments.
What is GitLab?
GitLab is DevOps lifecycle tool that offers a Git repository manager for source code management (SCM), continuous integration, and more. It provides a versatile platform for software development, project collaboration, source code tracking and managing software development and deployment processes.
Within GitLab's unified interface, users access an array of essential features: version control, issue tracking, code review, continuous integration and delivery (CI/CD), and a container registry, all seamlessly integrated. One of GitLab's key strengths is its all-in-one approach, which means developers can manage their entire workflow within a single platform, from writing code to deploying applications.
GitLab supports both private and public repositories, enabling teams and organizations to work on proprietary projects while also fostering open-source collaboration.
The platform can be either self-hosted on-premises or utilized as a cloud-based service, providing organizations with solutions for their infrastructure needs.
With the SaaS implementation of GitLab, organizations have seamless accessibility to the platform, rapid deployment and automatic updates, freeing teams from maintenance and infrastructure management. The cloud-based version often includes scalability options and backup solutions.
What approaches might attackers take when targeting GitLab?
Attackers can manipulate GitLab in many ways to achieve their goals. Some methods may target the platform itself to achieve more aggressive intentions including compromising the integrity of victims’ code, product, or intellectual property.
Gaining control over projects with misconfigurations - Privilege Escalation
Attackers may use a compromised account (or alternative Initial Access tactics) to infiltrate private projects (repositories in GitLab). Through the exploitation of misconfigured GitLab user roles and group permissions, they can elevate privileges.
By attaining administrative privileges, attackers gain extensive control over projects and various configurations within the system.
Data theft and disruption - Defense evasion and execution
If an attacker gains access to a GitLab CI/CD pipeline configuration file, they could modify it to execute unauthorized commands or introduce malicious code during the build process. Another approach an attacker might take is tampering with the repository code, inserting malicious code directly into the repository code itself. For instance, an attacker gaining access to the CI/CD pipeline configuration could embed malicious code designed to exfiltrate sensitive data, such as Personally Identifiable Information (PII) or authentication tokens, redirecting them to an adversary-controlled server. Similarly, tampering with repository code might involve inserting malware that compromises system integrity or introduces backdoors for unauthorized access.
Malicious code or abuse of the pipeline could lead to data theft, code disruption, unauthorized access, and supply chain attacks.
Exfiltrating data - Collection and exfiltration
Once adversaries gain access and elevated privileges within GitLab repositories (projects), they can extract critical and sensitive data. This could encompass source code, proprietary intellectual property, or confidential customer information. They transfer this information externally, aligning with methods to both selectively gather specific data for their aims and move it out of the organization's network. Additionally, discovery of credentials or secrets can enable attackers to move laterally to other platforms, cloud environments and systems. Its vital to have visibility of attacker activities when abusing identified secrets or credentials within the source code, including the code itself, old pull requests, branches, or repository history. Organizations can often monitor users and current credentials, but older, untracked secrets are a challenge – especially in older and larger repositories, complicating the tracking process as they expand in size and age.
In addition to the mentioned methods, attackers may seek to exploit GitLab Pages for malware distribution, misuse GitLab Runners, and attempt lateral movement to other systems using information obtained from GitLab.
How can attackers gain access to GitLab?
There a several ways attackers might access GitLab, which include tokens or keys—some are just account compromises.
Compromised account
Adversaries that get access to an existing GitLab user and compromised their account, can abuse the compromised account to sign-in to GitLab. Attackers might compromise a user account through various methods, including phishing attacks, credential stuffing, session hijacking, or leveraging weak passwords to gain unauthorized access.
Tokens
Within GitLab, several token types facilitate API access without requiring explicit authentication (or passing MFA requirements), presenting an opportunity for exploitation by attackers. When an attacker obtains a token through various means like compromised sources or public repositories, they can exploit it to gain access to GitLab via its API. So, let's dig in further to these tokens.
Project/Group Access Tokens:
Project Access Tokens are authentication credentials used to authenticate with the GitLab API or to authenticate with Git using HTTP Basic Authentication on a specific Project/Group.
Each token features an expiration period (in GitLab 16.0, tokens default to a 365-day expiration unless specified otherwise within that time), a user-assigned role, and an access level scope. Roles dictate project (repository) and group (organizational unit) permissions, while access levels define actions permissible upon authentication. The combination of role and access level determines overall capabilities. If a role lacks sufficient project permissions, even with a higher access level, it will not supersede the role's constraints.
Personal Access Tokens:
Like Project or Group Tokens, Personal Access Tokens are used by users or systems to authenticate with GitLab's API or Git using HTTP Basic Authentication. Serving as an alternative to OAuth2, these tokens supply access without the user's password and are tied to a specific user, incorporating that user's relevant role and an assigned expiration date. Unlike Project or Group Access Tokens, Personal Access Tokens solely require the specification of an access level. These tokens are required when using two-factor authentication.
Deploy Tokens:
Deploy Tokens work similarly to personal access tokens for HTTP authentication. However, they are specifically intended for the GitLab package registry public API and git commands, focusing solely on repository-related scopes for CI/CD build servers and processes.
Deploy Tokens can either be for a specific Project of all projects in a Group.
By default, Deploy Tokens lack an expiration date, though one can be set as needed.
Maintaining oversight of these tokens is critical as they supply entry to GitLab and should be regularly rotated, especially personal access tokens.
Project/Public Deploy Keys
Deploy Keys in GitLab serve as SSH keys for repository, particularly from external hosts like CI servers. These keys provide read-only or read-write permissions and are shareable across multiple projects. They are primarily used for accessing repositories and are generated as public SSH keys on external hosts. Unlike Deploy Tokens, which offer broader access within GitLab, Deploy Keys are specifically linked to repositories and cannot be used for Git operations if external authorization is enabled. Maintainers or Owners can set an expiration date for these tokens, Deploy Tokens defaults to never expire and Project access tokens default to 30 days.
What GitLab forensic data can be investigated?
When delving into GitLab's forensic data, we can concentrate on distinct log types that fall into two primary categories: logs accessible through the REST API and logs present locally within the system.
Internal logs
These logs are stored within files on the GitLab instance itself and are not accessible through REST API requests. In the context of a SaaS GitLab instance, external vendors aiming to access these logs encounter limited visibility. Notably, these logs contain critical data like API calls and authentication records, essential components of comprehensive logging.
Let us take an example involving authentication logs. When utilizing the SaaS GitLab, our access allows us to observe our user's authentication context but limited to the perspective of the logged-in user. Even as Group Owners, within the Web UI, our visibility is confined solely to our individual authentication details found under “User Setting.”
REST API logs
These logs are available for access via the REST API to GitLab and can be collected programmatically. The two REST API logs available are “events” and “audit events.”
Events logs:
The Contribution Events logs record Git operations within a Project, also known as "contribution events." These actions include push events, branch creation and deletion, merge requests, merges, comments, and issue-related activities.
Audit Events logs:
The Audit Events logs monitor actions within a Project or Group. These include tasks like adding members, downloading the Project, generating tokens (within Projects or Groups), adjusting visibility (public or private), creating protected branches, initiating Projects, and more.
What are caveats of the managed SaaS GitLab logs?
As discussed earlier, in a managed SaaS GitLab deployment, there are two REST API logs. However, these logs pose certain limitations that affect the visibility of external security vendors and investigators.
Authentication visibility gap
As previously highlighted, user authentication logs are stored locally and tied to specific user contexts. This setup presents a challenge for external security vendors or products aiming to create detections, metrics, or investigate incidents within GitLab. To overcome these limitations, they might need to resort to alternative solutions, such as leveraging external IDP logs used for authentication, to gain a complete picture of the events.
GET and local operations visibility gap
In our research, we observed that GET operations and local actions remain unlogged, which can make sense. This implies that actions such as local branch creations, pull operations, and even individual file downloads (except for full repository downloads) are not recorded. This poses a challenge for security vendors aiming to create modules that handle numerous pull operations or atypical single file downloads.
MFA configuration visibility gap
While implementing additional MFA within our test environment and examining customer data, we found that the logging for adding or removing MFA was absent. This lack of logging poses a challenge for security vendors aiming to detect MFA removal actions tied to an identity.
Token lifespan visibility gap
When a Project/Group token is created, the event doesn't include the expiration date in the Audit Events logs. This absence of token lifespan data limits the documentation of any anomalies in token durations or investigations centered around verifying the token's legitimacy based on its lifespan.
What detections can be written with managed SaaS GitLab logs?
At Mitiga, we've designed multiple IOAs (Indicators of Attack) for GitLab's managed SaaS logs to pinpoint events worth investigating. Here are concise outlines of several IOAs.
Creation of a permissive access token
Creating a permissive access token allows attackers to gain broad control in GitLab, potentially enabling unauthorized actions, data theft, or even persistence within the system, making it easier to move laterally or perform malicious activities undetected.
The code below looks for specific Access Token scopes with write or high privileges. The function gets an audit events log dataframe, and filters for a Personal Access Token resource creation, and for project access tokens with the preset scopes.
Changing a GitLab Project or Group to Public
Changing a GitLab project or group to public can help attackers to access sensitive code and project details, leading to potential data breaches or unauthorized access. Detecting and preventing these changes protect sensitive information, preserving confidentiality, and mitigating unauthorized access or data exposure.
The code below looks for changes in a Project or Group visibility or visibility_level. The function gets an audit events log dataframe, and filters for a change message fitting the conditions mentioned.
Anomaly based IOAs (Indicators of Attack):
Some of our IOAs use intricate, proprietary logic. While we can discuss the overall logic and concepts behind these detections, we won't delve into specific implementation details.
Unusual GitLab Project download
Unusual GitLab project downloads may indicate attackers attempting to obtain sensitive data or source code. This behavior can suggest potential threats of data theft or unauthorized access. Detecting these anomalies proactively strengthens security measures, helping identification of unauthorized access and investigate data protection.
This IOA identifies unusual GitLab Project (full repository) Downloads. The anomaly allows reducing False Positives of general downloads by utilizing factors such as:
- Identity Download metrics
- Repository Download metrics
- Group and Project metrics
- Additional enrichment metrics
Unusual spike in GitLab bot users' activity
To understand this IOA, it's important to get a handle on GitLab Bot users. According to GitLab's documentation:
“Bot users for projects are GitLab-created service accounts. Each time you create a project access token, a bot user is created and added to the project....
The bot users for projects have permissions that correspond with the selected role and scope of the project access token...
The bot users for projects have permissions that correspond with the selected role and scope of the project access token.”
With this understanding, the goal of this detection is to flag any sudden surge in Bot user activity. This is significant because Project Access Tokens are usually used for routine operations and automated tasks. An unexpected increase in Bot User activity can signal a potential compromise of a Project Access token.
Some factors that can be considered are:
- Bot user metrics
- Group and Project metrics
- Additional enrichment metrics
Prevention and mitigation for GitLab threats
Rotate access tokens
Regularly rotate access tokens and credentials, including personal access tokens, deploy tokens, or API keys. Automate token rotation where possible and promptly revoke any unused or compromised tokens.
Implement least privilege
Adhere to the principle of least privilege, granting users only the minimum permissions required to perform their tasks. Review and refine access permissions regularly to ensure users have the necessary access and nothing more.
Identity provider (IDP) monitoring
Monitor activities within your Identity Provider (IDP) or Single Sign-On (SSO) system. Watch for unusual login patterns, access attempts, or any suspicious activities associated with users.
Outbound request filtering
Use GitLab's outbound request filtering to contain potential data exposure risks. Configure the networking setting relevant to your organization. For more information (https://docs.gitlab.com/ee/security/webhooks.html
Enforce two-factor authentication
Enforce multiple layers of authentication to prevent attackers from accessing via username and password.
Git services can be a prized goal for adversaries, due to the sensitive nature of the information they hold, such as source code, intellectual property, and confidential data. Ensuring their security is vital for preventing possible data breaches, potential compromises, or operational and financial damage.
Mitiga offers its customers a wide range of security offerings for Git services—from continuous and strategic hunting to incident readiness and response—ensuring you’re ready for GitLab threats and prepared to swiftly detect and respond to any attack.