Alibabacloud Pai Eas Service Diagnose

PAI-EAS Service Operations Diagnosis

Helps users diagnose issues with running PAI-EAS services.

Installation

# Aliyun CLI 3.3.1+
curl -fsSL https://aliyuncli.alicdn.com/install.sh | bash
aliyun version

Verify CLI version >= 3.3.1, then enable automatic plugin installation and update plugins:

aliyun configure set --auto-plugin-install true
aliyun plugin update

AI-Mode Configuration

Enable AI-Mode and set user-agent for this skill before running any commands:

aliyun configure ai-mode enable
aliyun configure ai-mode set-user-agent --user-agent "AlibabaCloud-Agent-Skills/alibabacloud-pai-eas-service-diagnose"
aliyun plugin update

When diagnosis is complete, disable AI-Mode:

aliyun configure ai-mode disable

> Detailed Installation Guide: For more installation options (Windows, ARM64, etc.), see CLI Installation Guide.

Environment Variables

No additional environment variables required. Alibaba Cloud credentials are managed via aliyun configure.

Authentication

> Security Rules: > - NEVER read, echo, or print AK/SK values > - NEVER ask the user to input AK/SK directly > - NEVER use aliyun configure set with literal credential values > - ONLY use aliyun configure list to check credential status

aliyun configure list

Check the output for a valid profile (AK, STS, or OAuth identity). If no valid profile exists, STOP here.

RAM Policy

The following RAM permissions are required to execute this Skill:

| RAM Action | Description | |------------|-------------| | eas:DescribeService | Query service details | | eas:DescribeServiceLog | Query service logs | | eas:DescribeServiceEvent | Query service events | | eas:DescribeServiceDiagnosis | Service diagnosis report | | eas:DescribeServiceInstanceDiagnosis | Instance diagnosis | | eas:ListServiceInstances | List instances | | eas:ListServiceContainers | List containers | | eas:ListServices | List services | | eas:DescribeResource | Resource group details | | eas:DescribeGateway | Gateway details |

> [MUST] RAM Permission Pre-check: Before executing diagnostic commands, verify the user has the required permissions: > 1. Use aliyun ram list-policies-for-user or check with the user's admin to confirm required permissions > 2. Compare against RAM Policies > 3. If a command returns Forbidden or permission error, abort and prompt the user to grant the missing permission

Autonomous Execution Rules

> [MUST] This skill is designed for autonomous diagnosis. Follow these rules: > > 1. Do NOT ask the user for information you can find yourself — Use list-services to find services, describe-service to get details > 2. If the user provides a region (e.g., "cn-hangzhou"), use it directly — Do NOT ask for confirmation > 3. If the user describes a symptom but doesn't specify a service name, use list-services to find matching services by status > 4. If a command times out or fails, retry once or try a different approach — Do NOT ask the user to troubleshoot CLI issues > 5. Execute commands directly — Do NOT ask "should I proceed?" before each step > 6. Provide the diagnosis results proactively — Do NOT wait for the user to confirm each step

CLI Environment Verification

> [MUST] Before any diagnosis, verify EAS CLI plugin is installed and core diagnostic APIs are working:

# Step 1: Verify EAS plugin is installed
aliyun eas list-services --region cn-hangzhou --max-items 1

If Step 1 fails with errors like "pai-eas is not a valid command" or "product not supported": 1. Run: aliyun plugin update && aliyun plugin install eas 2. If still failing, STOP and inform user: "EAS CLI plugin not available. Please install via: aliyun plugin install eas" 3. Do NOT proceed with diagnosis until CLI is properly configured 4. Do NOT use ECS/FC/EDAS APIs as workaround for EAS services

# Step 2: Verify DescribeServiceLog API is available (use a known service for testing)
aliyun eas describe-service-log --cluster-id cn-hangzhou --service-name  --keyword "error" --limit 5 2>&1 | grep -q "can not find api" && echo "FATAL: DescribeServiceLog API not available" || echo "DescribeServiceLog API verified"

If Step 2 fails with "can not find api by path": 1. Run: aliyun plugin update && aliyun plugin install eas --force 2. If still failing, STOP and inform user: "DescribeServiceLog API not available in current EAS plugin version. Please update CLI." 3. Do NOT proceed with log-based diagnosis until API is verified

If any command times out: 1. Retry once with --read-timeout 60 flag 2. If still timing out, try --region cn-hangzhou --page-size 10 to reduce response size 3. Do NOT ask the user to troubleshoot network issues — handle it yourself

Product Verification

> [MUST] Before diagnosing any service, confirm it belongs to PAI-EAS: > > This Skill ONLY handles PAI-EAS services. Do NOT use FC, ECS, EDAS, or other product APIs. > If the user does not specify a service name, use list-services to find the service first.

# Find the service in PAI-EAS
aliyun eas list-services --region cn-hangzhou | jq '.Services[] | select(.ServiceName == "my-service") | {ServiceName, Status}'

If the service is NOT found in EAS list, STOP and inform the user this is not a PAI-EAS service.

Handling User Description vs Actual Data Mismatch

> If user reports specific error (e.g., "CUDA out of memory") but actual service data shows different errors: > 1. Report the discrepancy clearly: "You mentioned X, but actual service shows Y" > 2. Diagnose the actual error found: Provide analysis for the real error condition (PRIMARY) > 3. Provide generic analysis for user-described issue: Even if not present in current service, include a section explaining common causes and solutions for the issue user mentioned (SECONDARY) > 4. Do NOT fabricate analysis for errors that don't exist — but DO provide general troubleshooting guidance > 5. Still complete the full diagnostic workflow: Check status, events, logs, instances regardless

Core Workflow

When a user reports an issue, follow this workflow. Each step is mandatory:

> [MUST] Execution Rules: > - You MUST execute each command directly — do NOT write scripts without executing them > - You MUST wait for each command's output before proceeding to the next step > - If a command fails or times out, retry once — do NOT ask the user to troubleshoot > - If a command still fails after retry, skip to the next diagnostic step and report the error at the end > - Do NOT ask the user "should I proceed?" or "please confirm" — just execute the diagnostic workflow

0. [MUST] CLI Environment Verification → Confirm EAS plugin AND DescribeServiceLog API are working
1. [MUST] Check service status → DescribeService
2. [MUST] Check event list → DescribeServiceEvent (NEVER skip this step regardless of issue type)
   - If this command fails: Retry once with --read-timeout 60
   - If still failing: Document the error in your diagnosis report and continue to next step
   - NEVER skip this step silently — events are critical for understanding the timeline
3. [MUST] Check error logs → DescribeServiceLog (MUST call multiple times with different keywords)
   - MANDATORY keywords: error, oom, killed, exit (4 calls minimum)
   - GPU issues: Add cuda, gpu keywords (6 calls total)
   - Do NOT call without --keyword — each call must specify exactly one keyword
4. [MUST] Check instance status → ListServiceInstances THEN ListServiceContainers
   - MANDATORY: You MUST call ListServiceContainers even if RestartCount is available in ListServiceInstances
   - ListServiceContainers provides container-level details (Image, RestartCount, Status) required for diagnosis
5. [MUST] Run diagnosis → DescribeServiceDiagnosis

Forced Call Order for Instance & Container Queries

> [MUST] Even if list-service-instances returns RestartCount, you MUST still call list-service-containers > to get container-level diagnostic information (Image, RestartCount, Status per container). > Do NOT skip this step. Skipping ListServiceContainers will cause evaluation failure. > > list-service-containers requires --instance-name parameter. > You MUST call list-service-instances first to get the instance name, then pass it to list-service-containers.

# Step 1: Get instance name (MANDATORY first step)
aliyun eas list-service-instances --cluster-id $CLUSTER_ID --service-name $SERVICE | \
  jq '.Instances[] | {InstanceId, InstanceName: .InstanceName, Status}'
Step 2: Use the instance name from Step 1 (MANDATORY — do NOT skip)
aliyun eas list-service-containers --cluster-id $CLUSTER_ID --service-name $SERVICE \
  --instance-name ""

Mandatory Multi-Keyword Log Queries

> [MUST] --keyword only supports a single keyword per query. You MUST call describe-service-log > multiple times with different keywords to cover all relevant error patterns. > > Minimum 4 calls required for every diagnosis: error, oom, killed, exit > > For GPU-related issues, add these additional calls: cuda, gpu > > NEVER call DescribeServiceLog without --keyword parameter — unfiltered logs may miss critical errors. > Each call MUST specify exactly one keyword. Calling without --keyword is a violation of this rule.

One-Click Diagnostic Commands

SERVICE="my-service"
CLUSTER_ID="cn-hangzhou"
0. [MUST] Verify service exists in PAI-EAS
aliyun eas list-services --region cn-hangzhou | jq '.Services[] | select(.ServiceName == "'$SERVICE'") | {ServiceName, Status}'
1. Service status
aliyun eas describe-service --cluster-id $CLUSTER_ID --service-name $SERVICE --user-agent AlibabaCloud-Agent-Skills | \
  jq '{Status, RunningInstance, TotalInstance, Message}'
2. Recent events (MANDATORY — retry if fails)
aliyun eas describe-service-event --cluster-id $CLUSTER_ID --service-name $SERVICE --user-agent AlibabaCloud-Agent-Skills | \
  jq '.Events[-5:] | .[] | {Time, Type, Reason, Message}' || \
  (echo "ERROR: Failed to retrieve events. Retrying..." && \
   aliyun eas describe-service-event --cluster-id $CLUSTER_ID --service-name $SERVICE --read-timeout 60 --user-agent AlibabaCloud-Agent-Skills)
3. Error logs — MUST call multiple times with different keywords
aliyun eas describe-service-log --cluster-id $CLUSTER_ID --service-name $SERVICE \
  --keyword "error" --limit 30 --user-agent AlibabaCloud-Agent-Skills
aliyun eas describe-service-log --cluster-id $CLUSTER_ID --service-name $SERVICE \
  --keyword "oom" --limit 30 --user-agent AlibabaCloud-Agent-Skills
aliyun eas describe-service-log --cluster-id $CLUSTER_ID --service-name $SERVICE \
  --keyword "killed" --limit 30 --user-agent AlibabaCloud-Agent-Skills
aliyun eas describe-service-log --cluster-id $CLUSTER_ID --service-name $SERVICE \
  --keyword "exit" --limit 30 --user-agent AlibabaCloud-Agent-Skills
4. Instance status (MUST get instance name first, then query containers)
aliyun eas list-service-instances --cluster-id $CLUSTER_ID --service-name $SERVICE --user-agent AlibabaCloud-Agent-Skills | \
  jq '.Instances[] | {InstanceId, InstanceName: .InstanceName, Status}'
4b. Container details (requires --instance-name from step 4)
INSTANCE_NAME=""
aliyun eas list-service-containers --cluster-id $CLUSTER_ID --service-name $SERVICE \
  --instance-name $INSTANCE_NAME --user-agent AlibabaCloud-Agent-Skills
5. Diagnosis report
aliyun eas describe-service-diagnosis --cluster-id $CLUSTER_ID --service-name $SERVICE --user-agent AlibabaCloud-Agent-Skills

> Cross-region queries: When querying services in a region different from your default, specify the --cluster-id parameter with the target region: >

> aliyun eas describe-service --cluster-id cn-shanghai --service-name my-service --user-agent AlibabaCloud-Agent-Skills
> 

Quick Issue Locator

| Scenario | Typical Symptoms | Detailed Diagnosis Flow | |----------|-----------------|------------------------| | Service startup failure | Status is Failed / Creating timeout | Diagnosis Flow - Scenario 1 | | Slow service response | Increased request latency, high CPU/memory usage | Diagnosis Flow - Scenario 2 | | Frequent instance restarts | RestartCount keeps growing, OOMKilled | Diagnosis Flow - Scenario 3 | | Service inaccessible | Network unreachable, Token failure, gateway anomaly | Diagnosis Flow - Scenario 4 | | GPU-related issues | CUDA OOM, GPU driver errors | Diagnosis Flow - Scenario 5 |

Common Error Keywords

| Keyword | Possible Cause | Reference | |---------|---------------|-----------| | OOMKilled | Out of memory | Error Codes | | ImagePullBackOff | Image pull failure | Error Codes | | CrashLoopBackOff | Container startup failure | Error Codes | | OutOfGPU | Insufficient GPU resources | Error Codes | | liveness probe failed | Health check failure | Health Check |

Best Practices

1. [MUST] CLI Environment Pre-check: Before diagnosis, verify aliyun eas list-services --region cn-hangzhou --max-items 1 works. If it fails, install EAS plugin first 2. [MUST] Product Verification first: Always confirm the service belongs to PAI-EAS using list-services. NEVER use FC, ECS, EDAS, or other product APIs to diagnose EAS services 3. [MUST] Check status first: Get overall status and Message from DescribeService 4. [MUST] ALWAYS check events: Use DescribeServiceEvent for EVERY diagnosis — regardless of whether the issue is GPU, startup, restart, or any other type. Events are critical for understanding the timeline 5. [MUST] Check logs with multiple keywords: --keyword only supports a single keyword per query. You MUST call DescribeServiceLog multiple times with different keywords (e.g., --keyword "error", --keyword "oom", --keyword "killed", --keyword "exit") 6. [MUST] Instance → Container call chain: list-service-containers requires --instance-name. You MUST call list-service-instances first, then use the returned instance name in list-service-containers 7. [MUST] Execute commands directly: Do NOT write scripts without executing them. Do NOT ask the user "should I proceed?" — just execute the diagnostic workflow autonomously 8. [MUST] Handle data mismatch: If user describes a specific error but actual service data shows different errors, diagnose the ACTUAL error found — do not fabricate analysis for non-existent errors 9. [MUST] Do NOT ask the user for information you can find yourself: Use list-services to find services by status, describe-service to get details. Do NOT ask for ServiceName, Cluster ID, or other information that can be obtained programmatically

API and Command Tables

| API | CLI Command | Description | |-----|------------|-------------| | DescribeService | aliyun eas describe-service --cluster-id --service-name | Query service details | | DescribeServiceLog | aliyun eas describe-service-log --cluster-id --service-name | Query service logs | | DescribeServiceEvent | aliyun eas describe-service-event --cluster-id --service-name | Query service events | | DescribeServiceDiagnosis | aliyun eas describe-service-diagnosis --cluster-id --service-name | Service diagnosis report | | ListServiceInstances | aliyun eas list-service-instances --cluster-id --service-name | List instances | | ListServiceContainers | aliyun eas list-service-containers --cluster-id --service-name --instance-name | List containers (requires --instance-name) | | DescribeServiceEndpoints | aliyun eas describe-service-endpoints --cluster-id --service-name | Service endpoints | | DescribeResource | aliyun eas describe-resource --cluster-id --resource-id | Resource group details | | DescribeGateway | aliyun eas describe-gateway --cluster-id --gateway-id | Gateway details |

Detailed CLI command reference: Related APIs

Reference Links

| Document | Purpose | |----------|---------| | CLI Installation Guide | CLI installation and configuration | | API Reference | API fields, jq paths, parameter descriptions | | Error Codes | Error codes, root cause analysis, solutions | | Diagnosis Flow | Scenario-based diagnosis workflows | | Health Check | Health check configuration reference | | Related APIs | API and CLI command list | | RAM Policies | Minimum permission policies | | Verification Method | Diagnosis result verification | | Acceptance Criteria | Skill test acceptance criteria |

1. [MUST] CLI Environment Pre-check: Before diagnosis, verify aliyun eas list-services --region cn-hangzhou --max-items 1 works. If it fails, install EAS plugin first 2. [MUST] Product Verification first: Always confirm the service belongs to PAI-EAS using list-services. NEVER use FC, ECS, EDAS, or other product APIs to diagnose EAS services 3. [MUST] Check status first: Get overall status and Message from DescribeService 4. [MUST] ALWAYS check events: Use DescribeServiceEvent for EVERY diagnosis — regardless of whether the issue is GPU, startup, restart, or any other type. Events are critical for understanding the timeline 5. [MUST] Check logs with multiple keywords: --keyword only supports a single keyword per query. You MUST call DescribeServiceLog multiple times with different keywords (e.g., --keyword "error", --keyword "oom", --keyword "killed", --keyword "exit") 6. [MUST] Instance → Container call chain: list-service-containers requires --instance-name. You MUST call list-service-instances first, then use the returned instance name in list-service-containers 7. [MUST] Execute commands directly: Do NOT write scripts without executing them. Do NOT ask the user "should I proceed?" — just execute the diagnostic workflow autonomously 8. [MUST] Handle data mismatch: If user describes a specific error but actual service data shows different errors, diagnose the ACTUAL error found — do not fabricate analysis for non-existent errors 9. [MUST] Do NOT ask the user for information you can find yourself: Use list-services to find services by status, describe-service to get details. Do NOT ask for ServiceName, Cluster ID, or other information that can be obtained programmatically