Alibabacloud Bailian Videoanalysis

Bailian Video Analysis

This skill provides video analysis functionality based on Alibaba Cloud Bailian Video Analysis Light Application. It uses the Bailian (QuanMiaoLightApp) API for intelligent video comprehension, including shot analysis, ASR transcription, title generation, caption extraction, and mind mapping.

Architecture: CLI (Credential Chain) + OSS (File Storage) + Bailian Workspace + Quanmiao Video Analysis Service + Python SDK Scripts

Output Language Policy

Detection Priority: User input language → Environment language → Default: Chinese (中文)

Apply to: All user-facing messages, summaries, error explanations, and interactive prompts.

Keep original language: Code snippets, API responses, file paths, technical identifiers, HTTP status codes.

Installation

1. Python Dependencies

python -m pip install -r scripts/requirements.txt
On some systems, use pyhon3 instead of python

2. Alibaba Cloud CLI

> Pre-check: Aliyun CLI >= 3.3.1 required > Run aliyun version to verify >= 3.3.1. If not installed or version too low, > see references/cli-installation-guide.md for installation instructions. > Then [MUST] run aliyun configure set --auto-plugin-install true to enable automatic plugin installation.

> AI-Mode Configuration for aliyun CLI > > When using the aliyun CLI in this skill, you MUST configure AI-Mode settings: > >

> # Enable AI-Mode (required before using CLI commands)
> aliyun configure ai-mode enable
>
> # Set User-Agent for tracking (required)
> aliyun configure ai-mode set-user-agent AlibabaCloud-Agent-Skills/alibabacloud-bailian-videoanalysis
>
> # Update plugins to latest versions (recommended)
> aliyun plugin update
> 

> aliyun configure ai-mode disable
> 

3. Important Prerequisite: Activate Bailian and Video Analysis Service

Before using this skill, you must first activate the Video Analysis service in the Alibaba Cloud Bailian console!

1. Visit Bailian Video_Analysis page 2. Click the "Paid activation" button 3. Confirm activation and wait for the service to take effect (usually 1-2 minutes)

4. Important Prerequisite: Guide Users to Grant Permissions in Alibaba Cloud Access Control and Bailian

Before using this skill, you must grant the following permissions to the previously configured AK, otherwise calls will fail and may cause 403 errors

#### 1) RAM Permission Grant See references/ram-policies.md for the full list of required RAM permissions and authorization instructions.

> [MUST] Permission Failure Handling: When any command or API call fails due to permission errors at any point during execution, follow this process: > 1. Read references/ram-policies.md to get the full list of permissions required by this SKILL > 2. Use ram-permission-diagnose skill to guide the user through requesting the necessary permissions > 3. Pause and wait until the user confirms that the required permissions have been granted

#### 2) Bailian Workspace Permission Grant 1. Visit Alibaba Cloud Bailian Permission Management 2. If the RAM user corresponding to the AK does not exist, click "Add User" in the upper right corner of the page, select the corresponding RAM user and click confirm to add. 3. There is a 30s effective time after configuration, please wait patiently for a while.

Authentication

> Pre-check: Alibaba Cloud Credentials Required > > Security Rules: > - NEVER read, echo, or print AK/SK values (e.g., echo $ALIBABA_CLOUD_ACCESS_KEY_ID is FORBIDDEN) > - NEVER ask the user to input AK/SK directly in the conversation or command line > - NEVER use aliyun configure set with literal credential values > - ONLY use aliyun configure list to check credential status > >

> aliyun configure list
> 

Parameter Confirmation

> IMPORTANT: Parameter Confirmation — Before executing any command or API call, > confirm user-provided or customizable parameters (video source, OSS bucket, oss object key). > System-auto-resolved parameters (workspace_id, default OSS bucket) do NOT > require explicit confirmation unless the user wants to override them.

| Parameter | Type | Description | Default / Resolution | |------------------|---------------|-------------------------------------------|------------------------------------------------------------------------------------------| | video_source | Required | Local file path OR downloadable video URL | N/A (user must provide) | | workspace_id | Auto-resolved | Bailian workspace ID | Auto-detected(user may override) | | ossBucket | Optional | OSS bucket name for file upload | Auto-detect from first available bucket; user may specify (e.g. --ossBucket my-bucket) | | ossObjectKey | Optional | OSS object key for the uploaded file | /temp/quanmiao/YYYYMMDD/filename | | expireSeconds | Optional | Temporary URL expiration time (seconds) | 14400 (4 hours) |

Confirmation Workflow: 1. Auto-detection first: The skill will auto-detect workspace_id and ossBucket when possible 2. User override: If user wants to specify custom values, confirm before using 3. Local vs URL: Confirm whether user is providing a local file path or a public URL

Core Workflow

> ⚠️ CRITICAL: Cloud API Mandatory — This skill MUST use Bailian (QuanMiaoLightApp) API for video analysis. Local tools (ffmpeg, whisper, OpenCV, ffprobe, mediainfo, etc.) are FORBIDDEN. If API calls fail due to credentials or permissions, follow Permission Failure Handling process — DO NOT fall back to local analysis.

Step 1: Environment Check

Run python scripts/check_env.py to verify:

If check_env.py fails or returns {"ready": false}:

Expected output: {"ready": true} indicates environment is properly configured.

Step 2: Get Workspace ID

Do not ask the user for workspace_id upfront. Always auto-fetch available workspaces first:

aliyun modelstudio list-workspaces --user-agent AlibabaCloud-Agent-Skills/alibabacloud-bailian-videoanalysis

Workspace selection logic:

Step 3: Upload File(video_source) to OSS

Based on the input resource type from Input Resource Validation:

Case A: User provided a downloadable URL → Verify URL accessibility: Test if the URL is downloadable using appropriate method for your OS → Skip this step. Use the video_source as file_url in Step 4.

Case B: User provided a local file path → Auto-detects OSS bucket、Upload local file to OSS and get a temporary URL(file_url) for Step 4:

aliyun ossutil ls --user-agent AlibabaCloud-Agent-Skills/alibabacloud-bailian-videoanalysis

IMPORTANT - Upload Path Restriction:

aliyun ossutil cp  oss://{oss_bucket}/{oss_object_key} --user-agent AlibabaCloud-Agent-Skills/alibabacloud-bailian-videoanalysis --region {oss_region}

aliyun ossutil sign oss://{oss_bucket}/{oss_object_key} --expires-duration {expire_seconds} --user-agent AlibabaCloud-Agent-Skills/alibabacloud-bailian-videoanalysis --region {oss_region}

Recommended validation URL downloadable methods:

Validation criteria:

Step 4: Submit Video Analysis Task

> ⚠️ MANDATORY API CALL — You MUST call SubmitVideoAnalysisTask on QuanMiaoLightApp product (version 2024-08-01). Do NOT use videorecog, Mts, or any other product. Do NOT attempt local analysis.

> API Selection Checklist — Before calling, verify: > - ✅ Product: QuanMiaoLightApp (NOT videorecog, NOT Mts) > - ✅ Version: 2024-08-01 > - ✅ Action: SubmitVideoAnalysisTask > - ✅ Parameters: workspace_id, file_url

python scripts/quanmiao_submit_videoAnalysis_task.py --workspace_id  --file_url 

Parameters requiring confirmation:

Error Handling:

Returns task_id for polling.

Step 5: Poll for Task Result

> ⚠️ MANDATORY API CALL — You MUST poll GetVideoAnalysisTask on QuanMiaoLightApp product (version 2024-08-01) until status is SUCCESSED. Do NOT generate summary from local tools or filename inference.

Video analysis is asynchronous. Poll until completion:

Task Status: PENDING → RUNNING → SUCCESSED | FAILED | CANCELED

Variables:

Polling Loop: 1. Wait 10 seconds after submission 2. Run: python scripts/quanmiao_get_videoAnalysis_task_result.py --workspace_id --task_id --save_path 3. Check the returned status field: - SUCCESSED → Script auto-saves JSON to result_json_path, append entry to index_file, display saved locations, then proceed to Step 6 - FAILED or CANCELED → check error message, inform user, stop - PENDING or RUNNING → display any partial results available, wait 10s, repeat from step 2 4. Max 180 retries (approximately 30 minutes)

When taskStatus = SUCCESSED:

1. Append to index file (index_file):

    {"task_id": "", "video_source": "", "workspace_id": "", "result_file": "", "timestamp": ""}
    

2. Display saved locations:

✅ Files saved successfully:
Raw JSON result: 
Index updated: 

Parameters requiring confirmation:

Step 6: Summarize Video Content

CRITICAL: Use the results from Step 5 directly. Do NOT call the API again. Do NOT re-execute any analysis.

Extract data from the SUCCESSED response obtained in Step 5 and summarize according to user requirements.

Case A: If the user has a specific analysis request (e.g., "analyze the speaker's body language", "extract key business insights", "compare two people in the video"), base your answer primarily on:

Case B: If no specific request, use the standard output format: Title → Outline → Summary → Captions → Shot Analysis → Timeline → Token Usage

Important Constraints

Success Verification

See references/verification-method.md for step-by-step verification commands and expected outcomes.

Cleanup

To clean up resources created by this skill:

Delete uploaded OSS objects:

aliyun ossutil rm oss://{oss_bucket}/{oss_object_key} --user-agent AlibabaCloud-Agent-Skills/alibabacloud-bailian-videoanalysis

Cleanup best practices:

Best Practices

1. Always verify environment first — run check_env.py before any other operation to catch missing dependencies or credentials early. 2. Auto-detect workspace_id — always fetch workspaces via list-workspaces; default to the first result, but present a selection list when the user explicitly asks to choose. 3. Use default OSS settings — unless the user specifies a particular bucket, let the script auto-detect the bucket and generate the oss object key. 4. Display partial results during polling — when task status is RUNNING, show available results (title, captions) to give the user real-time feedback. 5. Save complete result for summary — when status becomes SUCCESSED, use the full result payload directly for Step 6 without re-calling the API. 6. Respect URL expiration — temporary OSS URLs expire after expireSeconds (default 14400s); ensure the task is submitted before the URL expires. 7. Handle permission errors gracefully — follow the Permission Failure Handling process in the RAM Policy section; never improvise credential fixes.

Command Tables

See references/related-commands.md for the full list of available scripts and their parameters.

Reference Links

| Reference | Purpose | |------------------------------------------|----------------------------------------------------------| | references/cli-installation-guide.md | Installing and upgrading Aliyun CLI | | references/ram-policies.md | RAM permission checklist and authorization guide | | references/acceptance-criteria.md | Acceptance criteria and correct/incorrect usage patterns | | references/related-commands.md | Available scripts and CLI command reference | | references/verification-method.md | Step-by-step success verification commands |

Troubleshooting

Common scenarios:

1. Always verify environment first — run check_env.py before any other operation to catch missing dependencies or credentials early. 2. Auto-detect workspace_id — always fetch workspaces via list-workspaces; default to the first result, but present a selection list when the user explicitly asks to choose. 3. Use default OSS settings — unless the user specifies a particular bucket, let the script auto-detect the bucket and generate the oss object key. 4. Display partial results during polling — when task status is RUNNING, show available results (title, captions) to give the user real-time feedback. 5. Save complete result for summary — when status becomes SUCCESSED, use the full result payload directly for Step 6 without re-calling the API. 6. Respect URL expiration — temporary OSS URLs expire after expireSeconds (default 14400s); ensure the task is submitted before the URL expires. 7. Handle permission errors gracefully — follow the Permission Failure Handling process in the RAM Policy section; never improvise credential fixes.