10 Commits

Author SHA1 Message Date
ABAPPLO
f2ac751c4c feat: 接入企业微信通知(临期催办 + 群机器人汇总)
新增 lib/wecom-client.ts 与 lib/wecom-notifier-service.ts,支持
群机器人 webhook 与应用消息双通道;scheduler 加入临期 N 天催办
与每日管理员汇总;action 路由补 bind_wecom_external_id 与
send_test_wecom 两个管理端动作,配套测试。
2026-08-18 17:05:16 +08:00
巫凤萍
cac6c5e83b fix: 修复回填更新与截图刷新 2026-08-16 22:22:50 +08:00
巫凤萍
74671a9b9f docs: 更新 main 私有化部署指南 2026-08-15 03:57:50 +08:00
巫凤萍
f37d05dd88 feat: 完善视频任务与 KOC 资源库 2026-08-15 03:53:09 +08:00
巫凤萍
ad3dbdcc86 feat: collapse mobile note after first backfill 2026-08-12 20:44:57 +08:00
巫凤萍
ee6caaf9e5 fix: type resource import enrichment fallback 2026-08-12 19:59:38 +08:00
巫凤萍
e05041e037 fix: preserve KOC portal path in delegation links 2026-08-12 19:53:19 +08:00
巫凤萍
51934b0638 feat: simplify KOC resource imports 2026-08-12 17:51:57 +08:00
巫凤萍
7ef150e08b feat: add distribution release and recovery controls 2026-08-12 11:12:23 +08:00
巫凤萍
ddad4b7659 feat: ship self-hosted KOC LOOP workflows 2026-08-11 23:07:26 +08:00
132 changed files with 13761 additions and 9487 deletions

12
.dockerignore Normal file
View File

@@ -0,0 +1,12 @@
.git
.next
dist
node_modules
koc-portal/.next
koc-portal/out
koc-portal/dist
koc-portal/node_modules
.data
.env*
!.env.self-hosted.example
npm-debug.log*

39
.env.self-hosted.example Normal file
View File

@@ -0,0 +1,39 @@
# 复制为 .env.self-hosted 后填写。不要把真实密钥提交到 Git。
MYSQL_ROOT_PASSWORD=replace-with-strong-root-password
MYSQL_DATABASE=koc_loop
MYSQL_USER=koc
MYSQL_PASSWORD=replace-with-strong-app-password
# 对外地址。部署域名确定后替换。
APP_ORIGIN=https://koc.example.com
KOC_PORTAL_URL=https://koc.example.com/koc/
# 后台首次启动会创建唯一超级管理员;账号创建后建议移除明文密码并重启。
SUPER_ADMIN_USERNAME=admin
SUPER_ADMIN_PASSWORD=replace-with-strong-admin-password
ADMIN_INTERNAL_TOKEN=replace-with-random-internal-token
# KOC LOOP 自身 MCP 的访问密钥。
KOC_MCP_API_KEY=replace-with-random-mcp-key
# 飞书内容表读取。
FEISHU_APP_ID=
FEISHU_APP_SECRET=
# 小红书公开数据采集 MCP。
AI_TOOL_CENTER_MCP_URL=
AI_TOOL_CENTER_MCP_KEY=
# 企业微信通知(临期催办 + 管理员汇总)。群机器人只需 webhookKOC 侧催办还需 corp/agent/secret
# 并在后台 partners 编辑里把 wecom_external_user_id 填好。
WECOM_ROBOT_WEBHOOK=
WECOM_CORP_ID=
WECOM_AGENT_ID=
WECOM_SECRET=
WECOM_NOTIFY_DUE_DAYS=3
WECOM_NOTIFY_ENABLED=true
# 每天北京时间 09:00 自动执行采集计划。
ENABLE_SCHEDULER=true
SEED_DEMO_DATA=false
HTTP_PORT=80

2
.gitignore vendored
View File

@@ -31,6 +31,7 @@ yarn-error.log*
# env files (can opt-in for committing if needed)
.env*
!.env.self-hosted.example
.dev.vars
# vercel
@@ -38,6 +39,7 @@ yarn-error.log*
# typescript
next-env.d.ts
*.tsbuildinfo
/dist/
/.wrangler/
/outputs/

31
Dockerfile Normal file
View File

@@ -0,0 +1,31 @@
FROM node:22.18.0-bookworm-slim AS dependencies
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm ci
FROM node:22.18.0-bookworm-slim AS builder
WORKDIR /app
ENV NEXT_TELEMETRY_DISABLED=1
COPY --from=dependencies /app/node_modules ./node_modules
COPY . .
RUN npm run build
FROM node:22.18.0-bookworm-slim AS runner
WORKDIR /app
ENV NODE_ENV=production \
NEXT_TELEMETRY_DISABLED=1 \
PORT=3000 \
HOSTNAME=0.0.0.0 \
UPLOAD_DIR=/data/koc/uploads
RUN groupadd --system --gid 1001 nodejs \
&& useradd --system --uid 1001 --gid nodejs nextjs \
&& mkdir -p /data/koc/uploads \
&& chown -R nextjs:nodejs /data/koc
COPY --from=builder --chown=nextjs:nodejs /app/.next/standalone ./
COPY --from=builder --chown=nextjs:nodejs /app/.next/static ./.next/static
COPY --from=builder --chown=nextjs:nodejs /app/public ./public
COPY --from=builder --chown=nextjs:nodejs /app/mysql ./mysql
COPY --from=builder --chown=nextjs:nodejs /app/scripts ./scripts
USER nextjs
EXPOSE 3000
CMD ["sh", "-c", "node scripts/migrate-mysql.mjs && node server.js"]

View File

@@ -1,20 +1,22 @@
# KOC LOOP
KOC 内容分发与数据回收闭环,运行于 vinext、Cloudflare D1 和 R2
KOC 内容分发与数据回收闭环`main` 分支运行于标准 Next.js Node.js、MySQL 8、Nginx 和本地持久化文件存储
## Prerequisites
- Node.js `>=22.13.0`
- MySQL `>=8.0`(本地完整运行)
## Quick Start
```bash
npm install
npm run db:migrate
npm run dev
npm run build
```
复制 `.dev.vars.example``.dev.vars` 并配置运行时变量。飞书动态导入需要:
复制 `.env.self-hosted.example``.env.self-hosted`,设置 `DATABASE_URL``MYSQL_*` 连接信息。飞书动态导入需要:
- `FEISHU_APP_ID`
- `FEISHU_APP_SECRET`
@@ -31,16 +33,7 @@ npm run build
系统首次登录时创建唯一的超级管理员。后续管理员和普通用户均由“用户管理”页面创建,普通用户不能访问 KOC 资源库。
This starter does not use `wrangler.jsonc`.
## Included Shape
- edit site code under `app/`
- `.openai/hosting.json` declares optional Sites D1 and R2 bindings
- `vite.config.ts` simulates declared bindings for local development
- `db/schema.ts` starts intentionally empty
- `examples/d1/` contains an optional D1 example surface
- `drizzle.config.ts` supports local migration generation when needed
完整私有化部署请看 [KOC LOOP 私有化部署指南](docs/KOC%20LOOP%20私有化部署指南.md)。Docker Compose 会启动 Nginx、KOC 服务和 MySQL外部领取页与后台使用同一域名下的 `/koc/` 路径。
## 后台账号与角色
@@ -50,6 +43,24 @@ This starter does not use `wrangler.jsonc`.
后台不提供注册、找回密码和普通用户个人改密。密码重置统一由管理员在后台完成。
## KOC 资源导入
超级管理员和管理员可在“KOC资源”页面下载标准 Excel 模板,批量导入已有资源。模板只需填写账号主页;账号名称、账号 ID、IP 属地、粉丝数、性别、简介、标签和合作来源均可选填。多个标签使用逗号分隔,每个账号最多 5 个标签。
- 单次最多导入 10,000 个账号,支持 `.xlsx``.csv`,文件不超过 20MB。
- 当前自动解析支持小红书账号主页;上传后先展示新增、更新和异常数据。异常行会跳过,其余有效账号可以正常导入。
- 按“平台 + 账号主页”去重;解析出相同小红书号时也会更新已有账号。
- 重复账号更新公开资料和合作来源,不产生两份资源。
- 大批量导入会先写入资源库,再在后台逐步补全缺失的公开资料。
- KOC 使用手机号或微信号领取任务后,系统会把该值写入“当前联系人”;原“合作来源”继续保留渠道信息。
- 导入的标签、当前联系人和合作来源会进入资源搜索或导出结果。
## KOC 批量回填 Excel
KOC 领取端支持导出和上传批量回填表。视频任务只生成“序号、标题、笔记内容、视频、发布链接、笔记截图、数据分析截图”列,不生成“图片”列。视频链接通过当前公网域名生成,下载接口返回可播放的 `.mp4` 附件。
反向代理部署必须正确传递 `Host``X-Forwarded-Host``X-Forwarded-Proto`,并把 `APP_ORIGIN` 配置为实际公网地址;不要填写 `localhost` 或容器内部地址。
## Agent MCP
生产地址:
@@ -110,11 +121,15 @@ Authorization: Bearer <KOC_MCP_API_KEY>
## Useful Commands
- `npm run dev`: start local development
- `npm run build`: verify the vinext build output
- `npm test`: build the starter and verify its rendered loading skeleton
- `npm run build`: 验证标准 Next.js Node.js 生产构建
- `npm test`: 构建并执行业务与私有化架构测试
- `npm run db:generate`: generate Drizzle migrations after schema changes
- `npm run db:migrate`: 应用 MySQL 增量迁移
- `npm run db:check`: 检查 MySQL 连接
- `npm run db:import-json -- <file>`: 导入 D1 JSON 数据
- `npm run storage:import -- <dir>`: 导入 R2 对象目录
## Learn More
- [vinext Documentation](https://github.com/cloudflare/vinext)
- [Drizzle D1 Guide](https://orm.drizzle.team/docs/get-started/d1-new)
- [Next.js Self-Hosting](https://nextjs.org/docs/app/guides/self-hosting)
- [Drizzle MySQL Guide](https://orm.drizzle.team/docs/get-started-mysql)

File diff suppressed because it is too large Load Diff

View File

@@ -1,6 +1,11 @@
import { env } from "cloudflare:workers";
import { getRequestExecutionContext } from "vinext/shims/request-context";
import { backfillAccountProfiles } from "../../../lib/account-enrichment-service";
import { getRuntimeEnv } from "../../../lib/runtime-env";
import { runInBackground } from "../../../lib/background";
const env = getRuntimeEnv();
import {
backfillAccountProfiles,
enrichDistributionAccount,
} from "../../../lib/account-enrichment-service";
import {
ensureSchema,
getDashboardData,
@@ -19,19 +24,28 @@ import {
type CollectionMcpBindings,
} from "../../../lib/mcp-collection-client";
import { adminForbidden, isAdminRequest } from "../../../lib/admin-auth";
import { isManagerRequest } from "../../../lib/user-auth";
import {
FeishuSourceError,
readFeishuSource,
type FeishuBindings,
} from "../../../lib/feishu-client";
import { createDistributionTask } from "../../../lib/task-service";
import {
createDistributionTask,
createScreenshotTask,
} from "../../../lib/task-service";
import {
DistributionReleaseError,
releaseUnfinishedDistribution,
} from "../../../lib/distribution-release-service";
export const runtime = "edge";
import {
resolveWecomConfig,
sendWecomAppMessage,
sendWecomRobotMessage,
WecomClientError,
type WecomBindings,
} from "../../../lib/wecom-client";
import { isManagerRequest } from "../../../lib/user-auth";
import { extractPublishUrl } from "../../../lib/publish-url";
type ActionBody = {
action?: string;
@@ -60,6 +74,14 @@ export async function POST(request: Request) {
sheetName: source.sheetName,
syncedAt: source.syncedAt,
rowCount: source.rows.length,
imageCount: source.rows.reduce(
(total, row) => total + row.images.length,
0,
),
videoCount: source.rows.reduce(
(total, row) => total + row.videos.length,
0,
),
columns: source.columns,
preview: source.rows.slice(0, 3),
});
@@ -69,6 +91,8 @@ export async function POST(request: Request) {
const name = String(body.name ?? "").trim();
const brand = String(body.brand ?? "").trim();
const dueAt = String(body.dueAt ?? "").trim();
const platform = body.platform === "抖音" ? "抖音" : "小红书";
const contentFormat = body.contentFormat === "video" ? "video" : "image_text";
if (!name || !brand || !dueAt) {
return Response.json(
{ error: "请补全任务名称、品牌和截止日期" },
@@ -81,9 +105,21 @@ export async function POST(request: Request) {
name,
brand,
dueAt,
platform,
contentFormat,
},
env as unknown as FeishuBindings,
);
} else if (body.action === "create_screenshot_task") {
await createScreenshotTask({
name: String(body.name ?? "").trim(),
brand: String(body.brand ?? "").trim(),
dueAt: String(body.dueAt ?? "").trim(),
keyword: String(body.keyword ?? "").trim(),
instructions: String(body.instructions ?? "").trim(),
quantity: numberValue(body.quantity),
exampleImageKey: String(body.exampleImageKey ?? "").trim(),
});
} else if (body.action === "claim") {
const partnerId = String(body.partnerId ?? "");
const taskId = String(body.taskId ?? "");
@@ -93,9 +129,9 @@ export async function POST(request: Request) {
`SELECT id FROM contents
WHERE task_id = ? AND status = 'available'
ORDER BY created_at, id
LIMIT ?`,
LIMIT ${quantity}`,
)
.bind(taskId, quantity)
.bind(taskId)
.all<{ id: string }>();
if (available.results.length === 0) {
return Response.json({ error: "当前没有可领取内容" }, { status: 409 });
@@ -133,6 +169,159 @@ export async function POST(request: Request) {
db,
String(body.distributionId ?? "").trim(),
);
} else if (body.action === "update_distribution_publish_url") {
if (!(await isManagerRequest(request))) return adminForbidden();
const distributionId = String(body.distributionId ?? "").trim();
if (!distributionId) {
return Response.json(
{ error: "作品记录不存在" },
{ status: 400 },
);
}
const current = await db
.prepare(
`SELECT d.id, d.task_id, d.partner_id, d.publish_url,
t.task_type, t.platform, t.collection_start_date, t.collection_days,
COALESCE(a.nickname, '待识别账号') AS account_nickname
FROM distributions d
JOIN tasks t ON t.id = d.task_id
LEFT JOIN accounts a ON a.id = d.account_id
WHERE d.id = ?`,
)
.bind(distributionId)
.first<{
id: string;
task_id: string;
partner_id: string;
publish_url: string | null;
task_type?: string | null;
platform: string;
collection_start_date: string | null;
collection_days: string;
account_nickname: string;
}>();
if (!current) {
return Response.json({ error: "作品记录不存在" }, { status: 404 });
}
if (current.task_type === "screenshot_collect") {
return Response.json(
{ error: "截图回收任务不需要填写发布链接" },
{ status: 400 },
);
}
const platform = current.platform === "抖音" ? "抖音" : "小红书";
const publishUrl = extractPublishUrl(
String(body.publishUrl ?? "").trim(),
platform,
);
if (!publishUrl) {
return Response.json(
{ error: `请填写包含${platform}作品链接的发布内容` },
{ status: 400 },
);
}
if (current.publish_url === publishUrl) {
return Response.json(await getDashboardData());
}
let collectionDays: number[] = [];
try {
const parsed = JSON.parse(current.collection_days || "[]");
if (Array.isArray(parsed)) {
collectionDays = [...new Set(parsed.map(Number))]
.filter(
(day) =>
Number.isInteger(day) && day >= 1 && day <= 7,
)
.sort((a, b) => a - b);
}
} catch {
collectionDays = [];
}
const isScheduled = Boolean(
current.collection_start_date && collectionDays.length > 0,
);
const statements = [
db
.prepare(
`UPDATE distributions SET
publish_url = ?,
publish_time = CURRENT_TIMESTAMP,
status = 'published',
d2_likes = NULL,
d2_comments = NULL,
d2_collects = NULL,
d5_likes = NULL,
d5_comments = NULL,
d5_collects = NULL,
d7_likes = NULL,
d7_comments = NULL,
d7_collects = NULL,
latest_likes = NULL,
latest_comments = NULL,
latest_collects = NULL,
latest_shares = NULL,
collection_status = ?,
collection_status_description = ?,
collection_updated_at = NULL,
last_collection_day = NULL,
updated_at = CURRENT_TIMESTAMP
WHERE id = ?`,
)
.bind(
publishUrl,
isScheduled ? "scheduled" : "pending",
isScheduled
? `管理员已更新链接,等待${collectionDays.length}个采集日`
: "管理员已更新链接,等待设置采集计划",
distributionId,
),
db
.prepare("DELETE FROM collection_runs WHERE distribution_id = ?")
.bind(distributionId),
];
if (!current.publish_url) {
statements.push(
db
.prepare(
`UPDATE partners SET completed_total = completed_total + 1
WHERE id = ?`,
)
.bind(current.partner_id),
);
}
await db.batch(statements);
if (isScheduled && current.collection_start_date) {
await createCollectionRunTasks(
db,
current.task_id,
current.collection_start_date,
collectionDays,
);
runInBackground(
runDueScheduledCollections(
db,
Date.now(),
resolveCollectionMcpConfig(
env as unknown as CollectionMcpBindings,
),
"catchup",
current.task_id,
).catch(() => undefined),
"collection catchup after publish URL update",
);
}
runInBackground(
enrichDistributionAccount(
db,
distributionId,
publishUrl,
current.account_nickname,
resolveCollectionMcpConfig(
env as unknown as CollectionMcpBindings,
),
).catch(() => undefined),
"account enrichment after publish URL update",
);
} else if (body.action === "save_collection_schedule") {
const taskId = String(body.taskId ?? "").trim();
const startDate = String(body.startDate ?? "").trim();
@@ -157,12 +346,18 @@ export async function POST(request: Request) {
);
}
const task = await db
.prepare("SELECT id FROM tasks WHERE id = ?")
.prepare("SELECT id, task_type FROM tasks WHERE id = ?")
.bind(taskId)
.first<{ id: string }>();
.first<{ id: string; task_type?: string | null }>();
if (!task) {
return Response.json({ error: "任务不存在" }, { status: 404 });
}
if (task.task_type === "screenshot_collect") {
return Response.json(
{ error: "截图回收任务不需要设置数据采集计划" },
{ status: 400 },
);
}
await db.batch([
db
.prepare(
@@ -204,12 +399,7 @@ export async function POST(request: Request) {
"catchup",
taskId,
).catch(() => undefined);
const executionContext = getRequestExecutionContext();
if (executionContext) {
executionContext.waitUntil(catchup);
} else {
await catchup;
}
runInBackground(catchup, "collection catchup");
} else if (body.action === "run_due_collections") {
await runDueScheduledCollections(
db,
@@ -235,6 +425,24 @@ export async function POST(request: Request) {
if (body.action === "collect" && day === null) {
return Response.json({ error: "采集周期无效" }, { status: 400 });
}
const distribution = await db
.prepare(
`SELECT t.task_type
FROM distributions d
JOIN tasks t ON t.id = d.task_id
WHERE d.id = ?`,
)
.bind(distributionId)
.first<{ task_type?: string | null }>();
if (!distribution) {
return Response.json({ error: "笔记记录不存在" }, { status: 404 });
}
if (distribution.task_type === "screenshot_collect") {
return Response.json(
{ error: "截图回收任务不支持公开数据采集" },
{ status: 400 },
);
}
await collectDistributionMetrics(
db,
distributionId,
@@ -250,6 +458,19 @@ export async function POST(request: Request) {
if (!taskId) {
return Response.json({ error: "请选择需要补采的任务" }, { status: 400 });
}
const task = await db
.prepare("SELECT task_type FROM tasks WHERE id = ?")
.bind(taskId)
.first<{ task_type?: string | null }>();
if (!task) {
return Response.json({ error: "任务不存在" }, { status: 404 });
}
if (task.task_type === "screenshot_collect") {
return Response.json(
{ error: "截图回收任务没有需要补采的公开数据" },
{ status: 400 },
);
}
await retryFailedCollections(
db,
taskId,
@@ -265,12 +486,7 @@ export async function POST(request: Request) {
),
Math.max(1, Math.min(25, numberValue(body.limit, 10))),
).catch(() => undefined);
const executionContext = getRequestExecutionContext();
if (executionContext) {
executionContext.waitUntil(backfill);
} else {
await backfill;
}
runInBackground(backfill, "account profile backfill");
} else if (body.action === "set_public_account_ids") {
const items = Array.isArray(body.items) ? body.items.slice(0, 50) : [];
const normalized = items
@@ -325,6 +541,66 @@ export async function POST(request: Request) {
)
.bind(exposure, views, distributionId)
.run();
} else if (body.action === "bind_wecom_external_id") {
const partnerId = String(body.partnerId ?? "").trim().slice(0, 80);
const externalId = String(body.wecomExternalUserId ?? "")
.trim()
.slice(0, 128);
if (!partnerId) {
return Response.json(
{ error: "缺少 partnerId" },
{ status: 400 },
);
}
await db
.prepare(
"UPDATE partners SET wecom_external_user_id = ? WHERE id = ?",
)
.bind(externalId || null, partnerId)
.run();
return Response.json({
partnerId,
wecomExternalUserId: externalId || null,
});
} else if (body.action === "send_test_wecom") {
const wecomConfig = resolveWecomConfig(
env as unknown as WecomBindings,
);
const partnerId = String(body.partnerId ?? "").trim();
let partnerExternalId: string | null = null;
if (partnerId) {
const row = await db
.prepare(
"SELECT wecom_external_user_id FROM partners WHERE id = ?",
)
.bind(partnerId)
.first<{ wecom_external_user_id: string | null }>();
partnerExternalId = row?.wecom_external_user_id ?? null;
}
const testContent = `[KOC LOOP 测试] 群机器人连通性正常,时间 ${new Date().toISOString()}`;
let robotStatus: "ok" | "skipped" = "skipped";
if (wecomConfig.robotWebhook) {
await sendWecomRobotMessage(testContent, wecomConfig);
robotStatus = "ok";
}
let appStatus: "ok" | "skipped" | "failed" = "skipped";
if (
partnerExternalId &&
wecomConfig.corpId &&
wecomConfig.agentId &&
wecomConfig.secret
) {
const result = await sendWecomAppMessage(
[partnerExternalId],
testContent,
wecomConfig,
);
appStatus = result.failed > 0 ? "failed" : "ok";
}
return Response.json({
robot: robotStatus,
app: appStatus,
});
} else {
return Response.json({ error: "不支持的操作" }, { status: 400 });
}
@@ -336,7 +612,8 @@ export async function POST(request: Request) {
{
status:
error instanceof FeishuSourceError ||
error instanceof DistributionReleaseError
error instanceof DistributionReleaseError ||
error instanceof WecomClientError
? error.status
: 500,
},

View File

@@ -3,12 +3,11 @@ import {
createSessionCookie,
ensureInitialSuperAdmin,
normalizeUsername,
requestUsesHttps,
verifyPassword,
} from "../../../../lib/user-auth";
import { getRawDb } from "../../../../lib/mvp-db";
export const runtime = "edge";
export async function POST(request: Request) {
try {
await ensureInitialSuperAdmin();
@@ -33,7 +32,7 @@ export async function POST(request: Request) {
return Response.json({ error: "账号或密码错误" }, { status: 401 });
}
const token = await createSession(user.id);
const secure = new URL(request.url).protocol === "https:";
const secure = requestUsesHttps(request);
return Response.json(
{ user: { id: user.id, username: user.username, role: user.role } },
{ headers: { "Set-Cookie": createSessionCookie(token, secure) } },

View File

@@ -1,15 +1,14 @@
import {
clearSessionCookie,
deleteSession,
requestUsesHttps,
sessionCookieFromHeader,
} from "../../../../lib/user-auth";
export const runtime = "edge";
export async function POST(request: Request) {
const token = sessionCookieFromHeader(request.headers.get("cookie"));
await deleteSession(token);
const secure = new URL(request.url).protocol === "https:";
const secure = requestUsesHttps(request);
return Response.json(
{ loggedOut: true },
{ headers: { "Set-Cookie": clearSessionCookie(secure) } },

View File

@@ -1,5 +1,7 @@
import { env } from "cloudflare:workers";
import { getRequestExecutionContext } from "vinext/shims/request-context";
import { getRuntimeEnv } from "../../../lib/runtime-env";
import { runInBackground } from "../../../lib/background";
const env = getRuntimeEnv();
import { backfillAccountProfiles } from "../../../lib/account-enrichment-service";
import { runDueScheduledCollections } from "../../../lib/collection-service";
import {
@@ -14,8 +16,6 @@ import {
} from "../../../lib/mvp-db";
import { authForbidden, getRequestPrincipal } from "../../../lib/user-auth";
export const runtime = "edge";
export async function GET(request: Request) {
const principal = await getRequestPrincipal(request);
if (!principal) return authForbidden();
@@ -55,12 +55,7 @@ export async function GET(request: Request) {
collectionCatchup,
accountBackfill,
]);
const executionContext = getRequestExecutionContext();
if (executionContext) {
executionContext.waitUntil(catchup);
} else {
await catchup;
}
runInBackground(catchup, "bootstrap catchup");
const dashboard = await getDashboardData();
return Response.json(
principal.kind === "user" && principal.user.role === "user"

View File

@@ -2,8 +2,6 @@ import feishuSnapshot from "../../../lib/feishu-source-snapshot.json";
import { adminForbidden, isAdminRequest } from "../../../lib/admin-auth";
import { ensureSchema, getUploadBucket } from "../../../lib/mvp-db";
export const runtime = "edge";
export async function POST(request: Request) {
if (!(await isAdminRequest(request))) return adminForbidden();
try {

View File

@@ -5,8 +5,6 @@ import {
} from "../../../lib/mvp-db";
import { adminForbidden, isAdminRequest } from "../../../lib/admin-auth";
export const runtime = "edge";
export async function GET(request: Request) {
if (!(await isAdminRequest(request))) return adminForbidden();
try {
@@ -43,7 +41,7 @@ export async function GET(request: Request) {
if (!headers.get("Content-Type")) {
headers.set("Content-Type", "image/jpeg");
}
return new Response(object.body, { headers });
return new Response(await object.arrayBuffer(), { headers });
} catch (error) {
return Response.json(
{ error: error instanceof Error ? error.message : "截图读取失败" },

24
app/api/health/route.ts Normal file
View File

@@ -0,0 +1,24 @@
import { checkDatabaseConnection } from "../../../lib/database";
import { getObjectStore } from "../../../lib/object-store";
export async function GET() {
try {
const database = await checkDatabaseConnection();
return Response.json({
status: database ? "ok" : "degraded",
database,
storage: getObjectStore().root,
scheduler: process.env.ENABLE_SCHEDULER !== "false",
timestamp: new Date().toISOString(),
}, { status: database ? 200 : 503 });
} catch (error) {
return Response.json(
{
status: "error",
database: false,
error: error instanceof Error ? error.message : "health check failed",
},
{ status: 503 },
);
}
}

View File

@@ -1,4 +1,4 @@
import { env } from "cloudflare:workers";
import { getRuntimeEnv } from "../../../lib/runtime-env";
import {
createMcpHandler,
McpServer,
@@ -16,10 +16,11 @@ import {
import { registerMcpOperationTools } from "../../../lib/mcp-tools";
import type { CollectionMcpBindings } from "../../../lib/mcp-collection-client";
export const runtime = "edge";
const env = getRuntimeEnv();
type McpBindings = FeishuBindings & CollectionMcpBindings & {
KOC_MCP_API_KEY?: string;
KOC_LOOP_MCP_API_KEY?: string;
KOC_PORTAL_URL?: string;
};
@@ -31,6 +32,8 @@ const toolOutputSchema = z.object({
due_date: z.string(),
sheet_name: z.string(),
note_count: z.number().int().nonnegative(),
platform: z.enum(["小红书", "抖音"]),
content_format: z.enum(["image_text", "video"]),
claim_url: z.string().url(),
});
@@ -56,7 +59,7 @@ function createServer(context: McpRequestContext) {
{
title: "创建 KOC 分发任务",
description:
"读取飞书电子表格中的标题、正文和配图,在 KOC LOOP 创建分发任务,并返回可直接发给 KOC 的领取链接。飞书表格若有多个工作表,链接必须包含目标 sheet 参数。",
"读取飞书电子表格中的标题、正文及图片或视频,在 KOC LOOP 创建小红书/抖音分发任务,并返回可直接发给 KOC 的领取链接。飞书表格若有多个工作表,链接必须包含目标 sheet 参数。",
inputSchema: z.object({
feishu_url: z
.string()
@@ -73,6 +76,14 @@ function createServer(context: McpRequestContext) {
.max(100)
.optional()
.describe("品牌或项目名称;未提供时系统记录为“未设置项目”"),
platform: z
.enum(["小红书", "抖音"])
.optional()
.describe("发布平台,默认小红书"),
content_format: z
.enum(["image_text", "video"])
.optional()
.describe("内容形式image_text 图文video 视频;默认图文"),
}),
outputSchema: toolOutputSchema,
annotations: {
@@ -82,7 +93,7 @@ function createServer(context: McpRequestContext) {
openWorldHint: true,
},
},
async ({ feishu_url, task_name, due_date, brand_project }) => {
async ({ feishu_url, task_name, due_date, brand_project, platform, content_format }) => {
try {
const bindings = getBindings();
const portalUrl = String(bindings.KOC_PORTAL_URL ?? "").trim();
@@ -95,6 +106,8 @@ function createServer(context: McpRequestContext) {
name: task_name,
brand: brand_project?.trim() || "未设置项目",
dueAt: due_date,
platform: platform ?? "小红书",
contentFormat: content_format ?? "image_text",
},
bindings,
{ deduplicate: true },
@@ -107,6 +120,8 @@ function createServer(context: McpRequestContext) {
due_date: result.dueAt,
sheet_name: result.sheetName,
note_count: result.noteCount,
platform: result.platform,
content_format: result.contentFormat,
claim_url: buildClaimUrl(portalUrl, result.shareToken),
};
const actionText = result.created ? "已创建" : "已找到相同任务";
@@ -114,7 +129,7 @@ function createServer(context: McpRequestContext) {
content: [
{
type: "text",
text: `${actionText}${result.name}”,共 ${result.noteCount}笔记。领取链接:${output.claim_url}`,
text: `${actionText}${result.name}”,平台:${result.platform},内容形式:${result.contentFormat === "video" ? "视频" : "图文"}${result.noteCount} 篇。领取链接:${output.claim_url}`,
},
],
structuredContent: output,
@@ -190,7 +205,10 @@ function originRejected(request: Request) {
}
async function authorize(request: Request) {
const expected = String(getBindings().KOC_MCP_API_KEY ?? "").trim();
const bindings = getBindings();
const expected = String(
bindings.KOC_MCP_API_KEY ?? bindings.KOC_LOOP_MCP_API_KEY ?? "",
).trim();
const authorization = request.headers.get("Authorization") ?? "";
const match = authorization.match(/^Bearer\s+(.+)$/i);
if (!expected || !match || !(await secretsMatch(match[1].trim(), expected))) {

View File

@@ -0,0 +1,750 @@
import { getRuntimeEnv } from "../../../lib/runtime-env";
import { runInBackground } from "../../../lib/background";
import type { DatabaseStatement } from "../../../lib/database";
import {
downloadFeishuMedia,
type FeishuBindings,
} from "../../../lib/feishu-client";
import {
ensureSchema,
getRawDb,
getUploadBucket,
hashText,
uid,
} from "../../../lib/mvp-db";
import {
PARTNER_BATCH_MAX_BYTES,
buildPartnerBatchWorkbookColumns,
parsePartnerBatchWorkbook,
resolvePartnerWorkbookOrigin,
} from "../../../lib/partner-batch-workbook";
import {
accountFromPublishLink,
} from "../../../lib/partner-utils";
import { extractPublishUrl } from "../../../lib/publish-url";
import {
partnerOptions,
withPartnerCors,
} from "../../../lib/partner-cors";
import {
buildRecoveryWorkbook,
type RecoveryWorkbookImage,
type RecoveryWorkbookRow,
} from "../../../lib/recovery-workbook";
import {
resolveCollectionMcpConfig,
type CollectionMcpBindings,
} from "../../../lib/mcp-collection-client";
import { enrichDistributionAccount } from "../../../lib/account-enrichment-service";
import {
createCollectionRunTasks,
runDueScheduledCollections,
} from "../../../lib/collection-service";
import { normalizeWorkbookImage } from "../../../lib/workbook-image";
const env = getRuntimeEnv();
type StoredAsset = {
index: number;
key: string;
fileToken?: string;
width?: number | null;
height?: number | null;
};
type BatchRow = {
distribution_id: string;
title: string;
body: string;
source_row: number | null;
image_assets: string;
video_assets: string;
publish_url: string | null;
publish_screenshot_key: string | null;
screenshot_key: string | null;
partner_id: string;
account_id: string | null;
claimant_name: string;
};
type TaskRow = {
id: string;
name: string;
task_type: string;
collection_start_date: string | null;
collection_days: string;
platform: "小红书" | "抖音";
content_format: "image_text" | "video";
};
type BatchAccess = {
task: TaskRow;
rows: BatchRow[];
};
function textValue(value: string | null, maxLength = 100) {
return String(value ?? "").trim().slice(0, maxLength);
}
function safeFileName(value: string) {
return value.replace(/[\\/:*?"<>|]/g, " ").trim().slice(0, 60) || "领取笔记";
}
function exactArrayBuffer(bytes: Uint8Array) {
const copy = new Uint8Array(bytes.byteLength);
copy.set(bytes);
return copy.buffer;
}
function contentTypeFromObject(object: { writeHttpMetadata(headers: Headers): void }) {
const headers = new Headers();
object.writeHttpMetadata(headers);
return headers.get("Content-Type") || "application/octet-stream";
}
function parseAssets(
value: string,
prefixes = ["content-assets/", "task-assets/"],
) {
try {
const assets = JSON.parse(value || "[]") as StoredAsset[];
return Array.isArray(assets)
? assets
.filter(
(asset) =>
Number.isInteger(Number(asset.index)) &&
Number(asset.index) > 0 &&
typeof asset.key === "string" &&
prefixes.some((prefix) => asset.key.startsWith(prefix)),
)
.map((asset) => ({ ...asset, index: Number(asset.index) }))
.sort((left, right) => left.index - right.index)
: [];
} catch {
return [];
}
}
async function loadImage(
key: string,
description: string,
fileToken?: string,
width?: number | null,
height?: number | null,
compactSource = false,
) {
const bucket = getUploadBucket();
let object = await bucket.get(key);
if (!object && fileToken) {
const media = await downloadFeishuMedia(
fileToken,
env as unknown as FeishuBindings,
);
await bucket.put(key, media.bytes, {
httpMetadata: { contentType: media.contentType },
customMetadata: { source: "feishu-api" },
});
object = await bucket.get(key);
}
if (!object) return null;
return normalizeWorkbookImage(
{
bytes: new Uint8Array(await object.arrayBuffer()),
contentType: contentTypeFromObject(object),
description,
width,
height,
} satisfies RecoveryWorkbookImage,
compactSource
? { maxDimension: 1600, outputFormat: "jpeg", jpegQuality: 82 }
: undefined,
);
}
async function findAccess(
taskToken: string,
claimToken: string,
delegationToken: string,
): Promise<BatchAccess | null> {
const db = getRawDb();
const task = delegationToken
? await db
.prepare(
`SELECT t.id, t.name, t.task_type, t.collection_start_date, t.collection_days,
t.platform, t.content_format
FROM tasks t
JOIN delegation_bundles b ON b.task_id = t.id
WHERE b.share_token = ? AND b.status = 'active'`,
)
.bind(delegationToken)
.first<TaskRow>()
: await db
.prepare(
`SELECT t.id, t.name, t.task_type, t.collection_start_date, t.collection_days,
t.platform, t.content_format
FROM tasks t
JOIN claims cl ON cl.task_id = t.id
WHERE t.share_token = ? AND cl.claim_token = ?`,
)
.bind(taskToken, claimToken)
.first<TaskRow>();
if (!task) return null;
const rows = delegationToken
? await db
.prepare(
`SELECT d.id AS distribution_id, d.partner_id, d.account_id,
d.publish_url, d.publish_screenshot_key, d.screenshot_key,
c.title, c.body, c.source_row, c.image_assets, c.video_assets,
cl.claimant_name
FROM distributions d
JOIN contents c ON c.id = d.content_id
JOIN delegation_bundles b ON b.id = d.delegation_bundle_id
JOIN claims cl ON cl.id = d.claim_id
WHERE b.share_token = ? AND b.task_id = ? AND b.status = 'active'
ORDER BY d.claimed_at, d.id`,
)
.bind(delegationToken, task.id)
.all<BatchRow>()
: await db
.prepare(
`SELECT d.id AS distribution_id, d.partner_id, d.account_id,
d.publish_url, d.publish_screenshot_key, d.screenshot_key,
c.title, c.body, c.source_row, c.image_assets, c.video_assets,
cl.claimant_name
FROM distributions d
JOIN contents c ON c.id = d.content_id
JOIN claims cl ON cl.id = d.claim_id
WHERE cl.claim_token = ? AND cl.task_id = ?
ORDER BY d.claimed_at, d.id`,
)
.bind(claimToken, task.id)
.all<BatchRow>();
return { task, rows: rows.results };
}
async function handleGet(request: Request) {
try {
await ensureSchema();
const url = new URL(request.url);
const taskToken = textValue(url.searchParams.get("task"));
const claimToken = textValue(url.searchParams.get("claim"));
const delegationToken = textValue(url.searchParams.get("share"));
if (!delegationToken && (!taskToken || !claimToken)) {
return Response.json({ error: "领取凭证不完整" }, { status: 400 });
}
const access = await findAccess(taskToken, claimToken, delegationToken);
if (!access) return Response.json({ error: "领取凭证无效" }, { status: 403 });
if (access.task.task_type !== "content_publish") {
return Response.json({ error: "当前任务不支持笔记批量回填" }, { status: 400 });
}
const maxSourceImages = Math.max(
0,
...access.rows.map((row) => parseAssets(row.image_assets).length),
);
const maxSourceVideos = Math.max(
0,
...access.rows.map(
(row) => parseAssets(row.video_assets, ["content-videos/"]).length,
),
);
const columns = buildPartnerBatchWorkbookColumns({
contentFormat: access.task.content_format,
maxSourceImages,
maxSourceVideos,
});
const downloadOrigin = resolvePartnerWorkbookOrigin(
request,
env.APP_ORIGIN,
);
const workbookRows: RecoveryWorkbookRow[] = [];
for (let rowIndex = 0; rowIndex < access.rows.length; rowIndex += 1) {
const row = access.rows[rowIndex];
const images: RecoveryWorkbookRow["images"] = [];
const hyperlinks: NonNullable<RecoveryWorkbookRow["hyperlinks"]> = [];
const assets =
columns.sourceImageCount > 0 ? parseAssets(row.image_assets) : [];
for (let index = 0; index < assets.length; index += 1) {
const asset = assets[index];
const image = await loadImage(
asset.key,
`${row.title} 原图${asset.index}`,
asset.fileToken,
asset.width,
asset.height,
true,
);
if (image) {
images.push({
column: columns.sourceImageStartColumn + index,
image,
maxWidth: 160,
maxHeight: 118,
});
}
}
if (row.publish_screenshot_key) {
const image = await loadImage(
row.publish_screenshot_key,
`${row.title} 笔记截图`,
);
if (image) {
images.push({ column: columns.publishScreenshotColumn, image });
}
}
if (row.screenshot_key) {
const image = await loadImage(
row.screenshot_key,
`${row.title} 数据分析截图`,
);
if (image) {
images.push({ column: columns.creatorScreenshotColumn, image });
}
}
const videoAssets = parseAssets(row.video_assets, ["content-videos/"]);
for (let index = 0; index < videoAssets.length; index += 1) {
const params = new URLSearchParams({
distribution: row.distribution_id,
index: String(videoAssets[index].index),
kind: "video",
download: "1",
});
if (delegationToken) params.set("share", delegationToken);
else {
params.set("task", taskToken);
params.set("claim", claimToken);
}
hyperlinks.push({
column: columns.sourceVideoStartColumn + index,
url: `${downloadOrigin}/api/partner-image?${params}`,
});
}
workbookRows.push({
cells: [
rowIndex + 1,
row.title,
row.body,
...Array.from({ length: columns.sourceImageCount }, () => ""),
...Array.from(
{ length: columns.sourceVideoCount },
(_, index) => (index < videoAssets.length ? `下载视频${index + 1}` : ""),
),
row.publish_url || "",
"",
"",
row.distribution_id,
row.publish_screenshot_key || "",
row.screenshot_key || "",
],
images,
hyperlinks: [
...hyperlinks,
...(row.publish_url
? [{ column: columns.publishUrlColumn, url: row.publish_url }]
: []),
],
});
}
const workbook = buildRecoveryWorkbook({
sheetName: "批量回填",
headers: columns.headers,
columnWidths: columns.columnWidths,
rows: workbookRows,
hiddenColumns: [
columns.systemColumn,
columns.systemColumn + 1,
columns.systemColumn + 2,
],
});
const fileName = `${safeFileName(access.task.name)}-批量回填.xlsx`;
return new Response(exactArrayBuffer(workbook), {
headers: {
"Cache-Control": "private, no-store",
"Content-Type":
"application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
"Content-Disposition": `attachment; filename="koc-batch.xlsx"; filename*=UTF-8''${encodeURIComponent(fileName)}`,
"X-Content-Type-Options": "nosniff",
},
});
} catch (error) {
return Response.json(
{ error: error instanceof Error ? error.message : "导出失败" },
{ status: 500 },
);
}
}
function extensionForImage(image: { fileName: string; contentType: string }) {
const fromName = image.fileName.split(".").at(-1)?.replace(/[^a-zA-Z0-9]/g, "");
if (fromName) return fromName.toLowerCase() === "jpeg" ? "jpg" : fromName;
return image.contentType.includes("png")
? "png"
: image.contentType.includes("webp")
? "webp"
: image.contentType.includes("gif")
? "gif"
: "jpg";
}
async function storeImportedImage(
kind: "publish" | "creator",
distributionId: string,
image: { bytes: Uint8Array; contentType: string; fileName: string },
) {
const prefix = kind === "publish" ? "publish-evidence" : "creator-center";
const key = `${prefix}/${distributionId}/${uid("sheet")}.${extensionForImage(image)}`;
await getUploadBucket().put(key, image.bytes, {
httpMetadata: { contentType: image.contentType },
customMetadata: { source: "partner-batch-workbook" },
});
return key;
}
async function isDifferentFromStoredImage(
existingKey: string | null,
image: { bytes: Uint8Array },
) {
if (!existingKey) return true;
const stored = await getUploadBucket().get(existingKey);
if (!stored) return true;
const storedBytes = new Uint8Array(await stored.arrayBuffer());
if (storedBytes.byteLength !== image.bytes.byteLength) return true;
for (let index = 0; index < storedBytes.byteLength; index += 1) {
if (storedBytes[index] !== image.bytes[index]) return true;
}
return false;
}
async function handlePost(request: Request) {
try {
await ensureSchema();
const form = await request.formData();
const file = form.get("file");
if (!(file instanceof File) || file.size === 0) {
return Response.json({ error: "请选择填写完成的Excel表" }, { status: 400 });
}
if (file.size > PARTNER_BATCH_MAX_BYTES) {
return Response.json({ error: "批量回填表不能超过80MB" }, { status: 400 });
}
if (!/\.xlsx$/i.test(file.name)) {
return Response.json({ error: "仅支持系统导出的 .xlsx 表格" }, { status: 400 });
}
const taskToken = textValue(String(form.get("taskToken") ?? ""));
const claimToken = textValue(String(form.get("claimToken") ?? ""));
const delegationToken = textValue(String(form.get("delegationToken") ?? ""));
const access = await findAccess(taskToken, claimToken, delegationToken);
if (!access) return Response.json({ error: "领取凭证无效" }, { status: 403 });
if (access.task.task_type !== "content_publish") {
return Response.json({ error: "当前任务不支持笔记批量回填" }, { status: 400 });
}
const importedRows = parsePartnerBatchWorkbook(await file.arrayBuffer());
const assignmentById = new Map(
access.rows.map((row) => [row.distribution_id, row]),
);
const seen = new Set<string>();
const errors: string[] = [];
const prepared = importedRows.map((row) => {
const assignment = assignmentById.get(row.distributionId);
if (!assignment) {
errors.push(`${row.spreadsheetRow}行不属于当前领取批次,请重新导出表格`);
} else if (seen.has(row.distributionId)) {
errors.push(`${row.spreadsheetRow}行笔记重复`);
} else if (row.title && row.title !== assignment.title) {
errors.push(`${row.spreadsheetRow}行标题已被修改,请重新导出表格`);
}
seen.add(row.distributionId);
const publishUrl = row.publishUrl
? extractPublishUrl(row.publishUrl, access.task.platform)
: assignment?.publish_url || "";
if (row.publishUrl && !publishUrl) {
errors.push(`${row.spreadsheetRow}行发布链接不是有效的${access.task.platform}作品链接`);
}
const hasPublishScreenshot = Boolean(
assignment?.publish_screenshot_key || row.publishScreenshot,
);
if (publishUrl && !hasPublishScreenshot) {
errors.push(`${row.spreadsheetRow}行填写了发布链接,请同时插入笔记截图`);
}
if (row.publishScreenshot && !publishUrl) {
errors.push(`${row.spreadsheetRow}行插入了笔记截图,请同时填写发布链接`);
}
if (row.creatorScreenshot && !publishUrl) {
errors.push(`${row.spreadsheetRow}行需先回填发布链接,再补数据分析截图`);
}
return {
imported: row,
assignment,
publishUrl,
};
});
if (errors.length > 0) {
return Response.json(
{ error: errors.slice(0, 8).join(""), errors },
{ status: 400 },
);
}
const db = getRawDb();
let updatedRows = 0;
let publishedCount = 0;
let analysisScreenshotCount = 0;
let noteScreenshotCount = 0;
let publishUrlChangedCount = 0;
const enrichments: Array<{ id: string; url: string; nickname: string }> = [];
for (const item of prepared) {
const assignment = item.assignment!;
let publishScreenshotKey = assignment.publish_screenshot_key;
let creatorScreenshotKey = assignment.screenshot_key;
const hasNewPublishScreenshot = item.imported.publishScreenshot
? await isDifferentFromStoredImage(
assignment.publish_screenshot_key,
item.imported.publishScreenshot,
)
: false;
const hasNewCreatorScreenshot = item.imported.creatorScreenshot
? await isDifferentFromStoredImage(
assignment.screenshot_key,
item.imported.creatorScreenshot,
)
: false;
if (hasNewPublishScreenshot && item.imported.publishScreenshot) {
publishScreenshotKey = await storeImportedImage(
"publish",
assignment.distribution_id,
item.imported.publishScreenshot,
);
noteScreenshotCount += 1;
}
if (hasNewCreatorScreenshot && item.imported.creatorScreenshot) {
creatorScreenshotKey = await storeImportedImage(
"creator",
assignment.distribution_id,
item.imported.creatorScreenshot,
);
analysisScreenshotCount += 1;
}
const statements: DatabaseStatement[] = [];
if (publishScreenshotKey !== assignment.publish_screenshot_key) {
statements.push(
db
.prepare(
`UPDATE distributions SET publish_screenshot_key = ?, updated_at = CURRENT_TIMESTAMP WHERE id = ?`,
)
.bind(publishScreenshotKey, assignment.distribution_id),
);
}
if (creatorScreenshotKey !== assignment.screenshot_key) {
statements.push(
db
.prepare(
`UPDATE distributions SET screenshot_key = ?,
ocr_status = CASE WHEN exposure IS NOT NULL AND views IS NOT NULL THEN ocr_status ELSE 'uploaded' END,
updated_at = CURRENT_TIMESTAMP WHERE id = ?`,
)
.bind(creatorScreenshotKey, assignment.distribution_id),
);
}
if (item.publishUrl && item.publishUrl !== assignment.publish_url) {
const isReplacement = Boolean(assignment.publish_url);
const account = accountFromPublishLink(
item.publishUrl,
access.task.platform,
);
if (!account) {
return Response.json(
{ error: `${item.imported.spreadsheetRow}行发布链接格式不正确` },
{ status: 400 },
);
}
const matchedAccount = await db
.prepare(
`SELECT id FROM accounts
WHERE platform = ? AND platform_uid = ?
LIMIT 1`,
)
.bind(account.platform, account.platformUid)
.first<{ id: string }>();
const accountId =
matchedAccount?.id ||
`account-${hashText(`${account.platform}:${account.platformUid}`)}`;
statements.push(
...(isReplacement
? [
db
.prepare(
`UPDATE distributions SET
d2_likes = NULL, d2_comments = NULL, d2_collects = NULL,
d5_likes = NULL, d5_comments = NULL, d5_collects = NULL,
d7_likes = NULL, d7_comments = NULL, d7_collects = NULL,
latest_likes = NULL, latest_comments = NULL,
latest_collects = NULL, latest_shares = NULL,
collection_status = 'pending',
collection_status_description = '批量回填已更新链接,等待重新采集',
collection_updated_at = NULL, last_collection_day = NULL,
updated_at = CURRENT_TIMESTAMP
WHERE id = ?`,
)
.bind(assignment.distribution_id),
db
.prepare("DELETE FROM collection_runs WHERE distribution_id = ?")
.bind(assignment.distribution_id),
]
: []),
db
.prepare(
`INSERT INTO accounts
(id, platform, platform_uid, nickname, profile_url,
current_contact, post_count)
VALUES (?, ?, ?, ?, ?, ?, 1)
ON CONFLICT(platform, platform_uid) DO UPDATE SET
nickname = excluded.nickname,
profile_url = excluded.profile_url,
current_contact = CASE
WHEN excluded.current_contact != ''
THEN excluded.current_contact
ELSE accounts.current_contact
END,
last_seen_at = CURRENT_TIMESTAMP`,
)
.bind(
accountId,
account.platform,
account.platformUid,
account.nickname,
account.profileUrl,
assignment.claimant_name,
),
db
.prepare(
`UPDATE distributions SET account_id = ?, publish_url = ?,
publish_time = COALESCE(publish_time, CURRENT_TIMESTAMP),
status = 'published', updated_at = CURRENT_TIMESTAMP
WHERE id = ?`,
)
.bind(accountId, item.publishUrl, assignment.distribution_id),
);
statements.push(
db
.prepare(
`UPDATE accounts SET post_count = (
SELECT COUNT(*) FROM distributions WHERE account_id = ?
) WHERE id = ?`,
)
.bind(accountId, accountId),
);
if (assignment.account_id && assignment.account_id !== accountId) {
statements.push(
db
.prepare(
`UPDATE accounts SET post_count = (
SELECT COUNT(*) FROM distributions WHERE account_id = ?
) WHERE id = ?`,
)
.bind(assignment.account_id, assignment.account_id),
);
}
if (isReplacement) publishUrlChangedCount += 1;
if (!assignment.publish_url) {
statements.push(
db
.prepare(
`UPDATE partners SET completed_total = completed_total + 1 WHERE id = ?`,
)
.bind(assignment.partner_id),
);
publishedCount += 1;
}
enrichments.push({
id: assignment.distribution_id,
url: item.publishUrl,
nickname: account.nickname,
});
}
if (statements.length > 0) {
await db.batch(statements);
updatedRows += 1;
}
}
if (
(publishedCount > 0 || publishUrlChangedCount > 0) &&
access.task.collection_start_date &&
access.task.collection_days !== "[]"
) {
let collectionDays: number[] = [];
try {
const parsed = JSON.parse(access.task.collection_days);
if (Array.isArray(parsed)) collectionDays = parsed.map(Number);
} catch {
collectionDays = [];
}
if (collectionDays.length > 0) {
await db
.prepare(
`UPDATE distributions SET collection_status = 'scheduled',
collection_status_description = ?
WHERE task_id = ? AND publish_url IS NOT NULL AND publish_url != ''`,
)
.bind(
`已安排${collectionDays.length}个采集日每日09:00执行`,
access.task.id,
)
.run();
await createCollectionRunTasks(
db,
access.task.id,
access.task.collection_start_date,
collectionDays,
);
runInBackground(
runDueScheduledCollections(
db,
Date.now(),
resolveCollectionMcpConfig(
env as unknown as CollectionMcpBindings,
),
"catchup",
access.task.id,
).catch(() => undefined),
"collection catchup after batch publish update",
);
}
}
for (const enrichment of enrichments) {
runInBackground(
enrichDistributionAccount(
db,
enrichment.id,
enrichment.url,
enrichment.nickname,
resolveCollectionMcpConfig(env as unknown as CollectionMcpBindings),
).catch(() => undefined),
"batch distribution account enrichment",
);
}
return Response.json({
imported: true,
updatedRows,
publishedCount,
publishUrlChangedCount,
noteScreenshotCount,
analysisScreenshotCount,
skippedRows: importedRows.length - updatedRows,
});
} catch (error) {
return Response.json(
{ error: error instanceof Error ? error.message : "批量回填失败" },
{ status: 500 },
);
}
}
export async function GET(request: Request) {
return withPartnerCors(request, await handleGet(request));
}
export async function POST(request: Request) {
return withPartnerCors(request, await handlePost(request));
}
export async function OPTIONS(request: Request) {
return partnerOptions(request);
}

View File

@@ -1,4 +1,4 @@
import { env } from "cloudflare:workers";
import { getRuntimeEnv } from "../../../lib/runtime-env";
import {
ensureSchema,
getRawDb,
@@ -12,8 +12,10 @@ import {
partnerOptions,
withPartnerCors,
} from "../../../lib/partner-cors";
import { parseResultScreenshotKeys } from "../../../lib/result-screenshots";
import { hasMp4FileSignature } from "../../../lib/video-file";
export const runtime = "edge";
const env = getRuntimeEnv();
type StoredAsset = {
index: number;
@@ -25,7 +27,11 @@ function textValue(value: string | null, maxLength = 100) {
return String(value ?? "").trim().slice(0, maxLength);
}
function findAsset(value: string, imageIndex: number) {
function findAsset(
value: string,
imageIndex: number,
prefixes = ["content-assets/", "task-assets/"],
) {
try {
const assets = JSON.parse(value) as StoredAsset[];
return Array.isArray(assets)
@@ -33,7 +39,7 @@ function findAsset(value: string, imageIndex: number) {
(asset) =>
asset.index === imageIndex &&
typeof asset.key === "string" &&
asset.key.startsWith("content-assets/") &&
prefixes.some((prefix) => asset.key.startsWith(prefix)) &&
(asset.fileToken === undefined ||
typeof asset.fileToken === "string"),
)
@@ -52,18 +58,24 @@ async function handleGet(request: Request) {
const delegationToken = textValue(url.searchParams.get("share"));
const distributionId = textValue(url.searchParams.get("distribution"));
const imageIndex = Number(url.searchParams.get("index"));
const imageKind = textValue(url.searchParams.get("kind"), 20);
const downloadRequested = url.searchParams.get("download") === "1";
if (
(!delegationToken && (!taskToken || !claimToken)) ||
!distributionId ||
!Number.isInteger(imageIndex) ||
imageIndex < 1
) {
return Response.json({ error: "图片链接不完整" }, { status: 400 });
return Response.json({ error: "素材链接不完整" }, { status: 400 });
}
const row = delegationToken
? await getRawDb()
.prepare(
`SELECT c.image_assets
`SELECT c.image_assets,
c.video_assets,
d.result_screenshot_key,
d.publish_screenshot_key,
d.screenshot_key
FROM distributions d
JOIN contents c ON c.id = d.content_id
JOIN delegation_bundles b ON b.id = d.delegation_bundle_id
@@ -72,10 +84,20 @@ async function handleGet(request: Request) {
AND b.status = 'active'`,
)
.bind(distributionId, delegationToken)
.first<{ image_assets: string }>()
.first<{
image_assets: string;
video_assets: string;
result_screenshot_key: string | null;
publish_screenshot_key: string | null;
screenshot_key: string | null;
}>()
: await getRawDb()
.prepare(
`SELECT c.image_assets
`SELECT c.image_assets,
c.video_assets,
d.result_screenshot_key,
d.publish_screenshot_key,
d.screenshot_key
FROM distributions d
JOIN contents c ON c.id = d.content_id
JOIN claims cl ON cl.id = d.claim_id
@@ -86,35 +108,107 @@ async function handleGet(request: Request) {
AND cl.task_id = t.id`,
)
.bind(distributionId, claimToken, taskToken)
.first<{ image_assets: string }>();
const asset = row ? findAsset(row.image_assets, imageIndex) : undefined;
if (!asset) {
return Response.json({ error: "没有找到这张笔记图片" }, { status: 404 });
.first<{
image_assets: string;
video_assets: string;
result_screenshot_key: string | null;
publish_screenshot_key: string | null;
screenshot_key: string | null;
}>();
const asset =
imageKind === "result"
? row
? {
index: imageIndex,
key: parseResultScreenshotKeys(row.result_screenshot_key)[
imageIndex - 1
],
}
: undefined
: imageKind === "publish"
? row?.publish_screenshot_key?.startsWith("publish-evidence/")
? { index: 1, key: row.publish_screenshot_key }
: undefined
: imageKind === "creator"
? row?.screenshot_key?.startsWith("creator-center/")
? { index: 1, key: row.screenshot_key }
: undefined
: imageKind === "video"
? row
? findAsset(row.video_assets, imageIndex, ["content-videos/"])
: undefined
: row
? findAsset(row.image_assets, imageIndex)
: undefined;
if (!asset?.key) {
return Response.json({ error: "没有找到这个素材" }, { status: 404 });
}
const bucket = getUploadBucket();
let object = await bucket.get(asset.key);
if (!object && asset.fileToken) {
let objectBytes = object ? await object.arrayBuffer() : null;
const invalidStoredVideo =
imageKind === "video" &&
objectBytes !== null &&
!hasMp4FileSignature(objectBytes);
if ((!object || invalidStoredVideo) && asset.fileToken) {
const media = await downloadFeishuMedia(
asset.fileToken,
env as unknown as FeishuBindings,
fetch,
{
maxBytes: imageKind === "video" ? 500 * 1024 * 1024 : undefined,
label: imageKind === "video" ? "视频" : "图片",
},
);
if (imageKind === "video" && !hasMp4FileSignature(media.bytes)) {
return Response.json(
{ error: "源视频不是可下载的 MP4 文件,请重新上传 MP4 视频" },
{ status: 422 },
);
}
await bucket.put(asset.key, media.bytes, {
httpMetadata: { contentType: media.contentType },
httpMetadata: {
contentType: imageKind === "video" ? "video/mp4" : media.contentType,
},
customMetadata: { source: "feishu-api" },
});
object = await bucket.get(asset.key);
objectBytes = object ? await object.arrayBuffer() : media.bytes;
}
if (!object) {
return Response.json({ error: "图片同步失败,请稍后重试" }, { status: 404 });
if (!object || !objectBytes) {
return Response.json({ error: "素材同步失败,请稍后重试" }, { status: 404 });
}
if (imageKind === "video" && !hasMp4FileSignature(objectBytes)) {
return Response.json(
{ error: "已存储的视频不是有效的 MP4 文件,请联系运营重新同步" },
{ status: 422 },
);
}
const headers = new Headers();
object.writeHttpMetadata(headers);
headers.set("Cache-Control", "private, max-age=3600");
headers.set("Content-Disposition", `inline; filename="note-${imageIndex}"`);
return new Response(object.body, { headers });
const isMutableEvidence =
imageKind === "publish" || imageKind === "creator";
headers.set(
"Cache-Control",
isMutableEvidence ? "private, no-store" : "private, max-age=3600",
);
if (imageKind === "video") {
headers.set("Content-Type", "video/mp4");
}
headers.set("Content-Length", String(objectBytes.byteLength));
headers.set("X-Content-Type-Options", "nosniff");
const fileName =
imageKind === "video"
? `video-${imageIndex}.mp4`
: `image-${imageIndex}`;
headers.set(
"Content-Disposition",
`${downloadRequested ? "attachment" : "inline"}; filename="${fileName}"; filename*=UTF-8''${encodeURIComponent(fileName)}`,
);
return new Response(objectBytes, { headers });
} catch (error) {
return Response.json(
{ error: error instanceof Error ? error.message : "图片读取失败" },
{ error: error instanceof Error ? error.message : "素材读取失败" },
{ status: 500 },
);
}

View File

@@ -8,8 +8,11 @@ import {
partnerOptions,
withPartnerCors,
} from "../../../lib/partner-cors";
export const runtime = "edge";
import {
MAX_RESULT_SCREENSHOTS,
parseResultScreenshotKeys,
serializeResultScreenshotKeys,
} from "../../../lib/result-screenshots";
async function readUpload(request: Request) {
const contentType = request.headers.get("content-type") ?? "";
@@ -64,7 +67,7 @@ async function handlePost(request: Request) {
!upload.distributionId ||
upload.fileBytes.byteLength === 0
) {
return Response.json({ error: "请选择发布截图" }, { status: 400 });
return Response.json({ error: "请选择需要上传的截图" }, { status: 400 });
}
if (
!upload.fileType.startsWith("image/") ||
@@ -76,21 +79,26 @@ async function handlePost(request: Request) {
);
}
const isCreatorCenter = upload.uploadKind === "creator-center";
const isTaskResult = upload.uploadKind === "task-result";
if (!["publish", "creator-center", "task-result"].includes(upload.uploadKind)) {
return Response.json({ error: "不支持的截图类型" }, { status: 400 });
}
const assignment = upload.delegationToken
? await getRawDb()
.prepare(
`SELECT d.id, d.publish_url
`SELECT d.id, d.publish_url, d.result_screenshot_key, t.task_type
FROM distributions d
JOIN delegation_bundles b ON b.id = d.delegation_bundle_id
JOIN tasks t ON t.id = d.task_id
WHERE d.id = ?
AND b.share_token = ?
AND b.status = 'active'`,
)
.bind(upload.distributionId, upload.delegationToken)
.first<{ id: string; publish_url: string | null }>()
.first<{ id: string; publish_url: string | null; result_screenshot_key: string | null; task_type: string }>()
: await getRawDb()
.prepare(
`SELECT d.id, d.publish_url
`SELECT d.id, d.publish_url, d.result_screenshot_key, t.task_type
FROM distributions d
JOIN claims c ON c.id = d.claim_id
JOIN tasks t ON t.id = d.task_id
@@ -100,7 +108,7 @@ async function handlePost(request: Request) {
AND c.task_id = t.id`,
)
.bind(upload.distributionId, upload.claimToken, upload.taskToken)
.first<{ id: string; publish_url: string | null }>();
.first<{ id: string; publish_url: string | null; result_screenshot_key: string | null; task_type: string }>();
if (!assignment) {
return Response.json({ error: "笔记与领取凭证不匹配" }, { status: 403 });
}
@@ -110,15 +118,45 @@ async function handlePost(request: Request) {
{ status: 409 },
);
}
if (isTaskResult && assignment.task_type !== "screenshot_collect") {
return Response.json({ error: "当前任务不支持截图结果回填" }, { status: 400 });
}
if (!isTaskResult && assignment.task_type === "screenshot_collect") {
return Response.json({ error: "截图回收任务请上传任务结果截图" }, { status: 400 });
}
const existingResultKeys = isTaskResult
? parseResultScreenshotKeys(assignment.result_screenshot_key)
: [];
if (isTaskResult && existingResultKeys.length >= MAX_RESULT_SCREENSHOTS) {
return Response.json(
{ error: `每份任务最多上传${MAX_RESULT_SCREENSHOTS}张截图` },
{ status: 409 },
);
}
const extension =
upload.fileName.split(".").at(-1)?.replace(/[^a-zA-Z0-9]/g, "") || "jpg";
const key = isCreatorCenter
const key = isTaskResult
? `task-results/${upload.distributionId}/${uid("shot")}.${extension}`
: isCreatorCenter
? `creator-center/${upload.distributionId}/${uid("shot")}.${extension}`
: `publish-evidence/${upload.distributionId}/${uid("shot")}.${extension}`;
await getUploadBucket().put(key, upload.fileBytes, {
httpMetadata: { contentType: upload.fileType },
});
if (isCreatorCenter) {
if (isTaskResult) {
await getRawDb()
.prepare(
`UPDATE distributions SET
result_screenshot_key = ?,
updated_at = CURRENT_TIMESTAMP
WHERE id = ?`,
)
.bind(
serializeResultScreenshotKeys([...existingResultKeys, key]),
upload.distributionId,
)
.run();
} else if (isCreatorCenter) {
await getRawDb()
.prepare(
`UPDATE distributions SET
@@ -145,7 +183,12 @@ async function handlePost(request: Request) {
}
return Response.json({
uploaded: true,
kind: isCreatorCenter ? "creator-center" : "publish",
screenshotCount: isTaskResult ? existingResultKeys.length + 1 : undefined,
kind: isTaskResult
? "task-result"
: isCreatorCenter
? "creator-center"
: "publish",
});
} catch (error) {
return Response.json(

View File

@@ -1,7 +1,13 @@
import { env } from "cloudflare:workers";
import { getRequestExecutionContext } from "vinext/shims/request-context";
import { getRuntimeEnv } from "../../../lib/runtime-env";
import { runInBackground } from "../../../lib/background";
import type { DatabaseStatement } from "../../../lib/database";
const env = getRuntimeEnv();
import { enrichDistributionAccount } from "../../../lib/account-enrichment-service";
import { createCollectionRunTasks } from "../../../lib/collection-service";
import {
createCollectionRunTasks,
runDueScheduledCollections,
} from "../../../lib/collection-service";
import {
resolveCollectionMcpConfig,
type CollectionMcpBindings,
@@ -14,16 +20,14 @@ import {
} from "../../../lib/mvp-db";
import {
accountFromPublishLink,
extractXhsPublishUrl,
} from "../../../lib/partner-utils";
import { extractPublishUrl } from "../../../lib/publish-url";
import { parseClaimantIdentifier } from "../../../lib/claimant-identifier";
import {
partnerOptions,
withPartnerCors,
} from "../../../lib/partner-cors";
export const runtime = "edge";
type PartnerBody = {
action?: string;
taskToken?: string;
@@ -39,6 +43,7 @@ type PartnerBody = {
publishUrl?: string;
exposure?: number | string;
views?: number | string;
resultScreenshotKey?: string;
};
type ImageAsset = {
@@ -99,22 +104,28 @@ function publicImageAssets(value: unknown): ImageAsset[] {
}
}
type PartnerTask = {
id: string;
name: string;
brand: string;
quantity: number;
claimed_quantity: number;
due_at: string;
status: string;
task_type: string;
platform: string;
content_format: string;
};
async function findTask(taskToken: string) {
return getRawDb()
.prepare(
`SELECT id, name, brand, quantity, claimed_quantity, due_at, status
`SELECT id, name, brand, quantity, claimed_quantity, due_at, status,
task_type, platform, content_format
FROM tasks WHERE share_token = ?`,
)
.bind(taskToken)
.first<{
id: string;
name: string;
brand: string;
quantity: number;
claimed_quantity: number;
due_at: string;
status: string;
}>();
.first<PartnerTask>();
}
async function findDelegationAccess(delegationToken: string) {
@@ -128,6 +139,9 @@ async function findDelegationAccess(delegationToken: string) {
t.claimed_quantity,
t.due_at,
t.status,
t.task_type,
t.platform,
t.content_format,
b.id AS bundle_id,
b.label AS bundle_label,
b.quantity AS bundle_quantity,
@@ -137,14 +151,7 @@ async function findDelegationAccess(delegationToken: string) {
WHERE b.share_token = ? AND b.status = 'active'`,
)
.bind(delegationToken)
.first<{
id: string;
name: string;
brand: string;
quantity: number;
claimed_quantity: number;
due_at: string;
status: string;
.first<PartnerTask & {
bundle_id: string;
bundle_label: string;
bundle_quantity: number;
@@ -165,13 +172,17 @@ async function findAccessibleAssignment(
d.account_id,
d.publish_url,
d.publish_screenshot_key,
d.screenshot_key`;
d.screenshot_key,
d.result_screenshot_key,
d.result_submitted_at,
c.claimant_name`;
if (delegationToken) {
return db
.prepare(
`${select}
FROM distributions d
JOIN delegation_bundles b ON b.id = d.delegation_bundle_id
JOIN claims c ON c.id = d.claim_id
WHERE d.id = ?
AND b.share_token = ?
AND b.task_id = ?
@@ -185,6 +196,9 @@ async function findAccessibleAssignment(
publish_url: string | null;
publish_screenshot_key: string | null;
screenshot_key: string | null;
result_screenshot_key: string | null;
result_submitted_at: string | null;
claimant_name: string;
}>();
}
if (!claimToken) return null;
@@ -203,6 +217,9 @@ async function findAccessibleAssignment(
publish_url: string | null;
publish_screenshot_key: string | null;
screenshot_key: string | null;
result_screenshot_key: string | null;
result_submitted_at: string | null;
claimant_name: string;
}>();
}
@@ -263,6 +280,8 @@ async function handleGet(request: Request) {
d.publish_url,
d.publish_time,
d.publish_screenshot_key,
d.result_screenshot_key,
d.result_submitted_at,
d.screenshot_key AS creator_screenshot_key,
d.ocr_status,
d.exposure,
@@ -271,6 +290,7 @@ async function handleGet(request: Request) {
c.body,
c.source_row,
c.image_assets,
c.video_assets,
a.nickname AS account_nickname,
b.id AS delegation_bundle_id,
b.label AS delegation_label
@@ -295,6 +315,7 @@ async function handleGet(request: Request) {
b.created_at,
b.revoked_at,
SUM(CASE WHEN d.publish_url IS NOT NULL AND d.publish_url != '' THEN 1 ELSE 0 END) AS completed_count,
SUM(CASE WHEN d.result_submitted_at IS NOT NULL THEN 1 ELSE 0 END) AS result_completed_count,
SUM(CASE WHEN d.screenshot_key IS NOT NULL AND d.exposure IS NOT NULL AND d.views IS NOT NULL THEN 1 ELSE 0 END) AS creator_completed_count
FROM delegation_bundles b
LEFT JOIN distributions d ON d.delegation_bundle_id = b.id
@@ -312,7 +333,9 @@ async function handleGet(request: Request) {
assignments: assignments.results.map((assignment) => ({
...assignment,
images: publicImageAssets(assignment.image_assets),
videos: publicImageAssets(assignment.video_assets),
image_assets: undefined,
video_assets: undefined,
})),
delegations: delegations.results,
};
@@ -325,6 +348,8 @@ async function handleGet(request: Request) {
d.publish_url,
d.publish_time,
d.publish_screenshot_key,
d.result_screenshot_key,
d.result_submitted_at,
d.screenshot_key AS creator_screenshot_key,
d.ocr_status,
d.exposure,
@@ -333,6 +358,7 @@ async function handleGet(request: Request) {
c.body,
c.source_row,
c.image_assets,
c.video_assets,
a.nickname AS account_nickname
FROM distributions d
JOIN contents c ON c.id = d.content_id
@@ -344,13 +370,15 @@ async function handleGet(request: Request) {
.all();
delegation = {
id: delegationAccess.bundle_id,
label: "转派发布包",
label: task.task_type === "screenshot_collect" ? "转派截图任务包" : "转派发布包",
quantity: delegationAccess.bundle_quantity,
createdAt: delegationAccess.bundle_created_at,
assignments: assignments.results.map((assignment) => ({
...assignment,
images: publicImageAssets(assignment.image_assets),
videos: publicImageAssets(assignment.video_assets),
image_assets: undefined,
video_assets: undefined,
})),
};
}
@@ -362,6 +390,9 @@ async function handleGet(request: Request) {
brand: task.brand,
dueAt: task.due_at,
status: task.status,
type: task.task_type,
platform: task.platform,
contentFormat: task.content_format,
}
: {
name: task.name,
@@ -370,6 +401,9 @@ async function handleGet(request: Request) {
claimedQuantity: task.claimed_quantity,
dueAt: task.due_at,
status: task.status,
type: task.task_type,
platform: task.platform,
contentFormat: task.content_format,
availableQuantity: available?.count ?? 0,
},
claim,
@@ -402,10 +436,11 @@ async function handlePost(request: Request) {
if (
delegationToken &&
body.action !== "submit" &&
body.action !== "submit_creator_metrics"
body.action !== "submit_creator_metrics" &&
body.action !== "submit_screenshot_result"
) {
return Response.json(
{ error: "分享链接只能用于查看和回填包内笔记" },
{ error: "分享链接只能用于查看和回填包内任务" },
{ status: 403 },
);
}
@@ -438,7 +473,10 @@ async function handlePost(request: Request) {
c.quantity,
c.created_at,
COUNT(d.id) AS note_count,
SUM(CASE WHEN d.publish_url IS NOT NULL AND d.publish_url != '' THEN 1 ELSE 0 END) AS completed_count,
SUM(CASE
WHEN ? = 'screenshot_collect' AND d.result_submitted_at IS NOT NULL THEN 1
WHEN ? != 'screenshot_collect' AND d.publish_url IS NOT NULL AND d.publish_url != '' THEN 1
ELSE 0 END) AS completed_count,
MIN(co.title) AS first_title
FROM claims c
LEFT JOIN distributions d ON d.claim_id = c.id
@@ -448,7 +486,7 @@ async function handlePost(request: Request) {
ORDER BY c.created_at DESC, c.id DESC
LIMIT 20`,
)
.bind(task.id, partnerId, legacyPartnerId)
.bind(task.task_type, task.task_type, task.id, partnerId, legacyPartnerId)
.all<{
claim_token: string;
quantity: number;
@@ -469,7 +507,7 @@ async function handlePost(request: Request) {
quantity: claim.note_count || claim.quantity,
completedCount: claim.completed_count || 0,
createdAt: claim.created_at,
firstTitle: claim.first_title || "领取的笔记",
firstTitle: claim.first_title || (task.task_type === "screenshot_collect" ? "领取的截图任务" : "领取的笔记"),
})),
});
}
@@ -495,9 +533,9 @@ async function handlePost(request: Request) {
`SELECT id FROM contents
WHERE task_id = ? AND status = 'available'
ORDER BY COALESCE(source_row, 999999), created_at, id
LIMIT ?`,
LIMIT ${quantity}`,
)
.bind(task.id, quantity)
.bind(task.id)
.all<{ id: string }>();
if (available.results.length === 0) {
return Response.json({ error: "当前任务已领完" }, { status: 409 });
@@ -508,7 +546,7 @@ async function handlePost(request: Request) {
const claimantName = claimantIdentifier.display;
const claimId = uid("claim");
const claimToken = crypto.randomUUID().replaceAll("-", "");
const statements: D1PreparedStatement[] = [
const statements: DatabaseStatement[] = [
db
.prepare(
`INSERT INTO partners
@@ -584,7 +622,7 @@ async function handlePost(request: Request) {
}
if (distributionIds.length === 0) {
return Response.json(
{ error: "请至少选择一篇待发布笔记" },
{ error: task.task_type === "screenshot_collect" ? "请至少选择一份待提交任务" : "请至少选择一篇待发布笔记" },
{ status: 400 },
);
}
@@ -602,7 +640,7 @@ async function handlePost(request: Request) {
const placeholders = distributionIds.map(() => "?").join(", ");
const selected = await db
.prepare(
`SELECT id, publish_url, delegation_bundle_id
`SELECT id, publish_url, result_submitted_at, delegation_bundle_id
FROM distributions
WHERE claim_id = ? AND id IN (${placeholders})`,
)
@@ -610,6 +648,7 @@ async function handlePost(request: Request) {
.all<{
id: string;
publish_url: string | null;
result_submitted_at: string | null;
delegation_bundle_id: string | null;
}>();
if (selected.results.length !== distributionIds.length) {
@@ -618,9 +657,13 @@ async function handlePost(request: Request) {
{ status: 403 },
);
}
if (selected.results.some((item) => item.publish_url)) {
if (selected.results.some((item) =>
task.task_type === "screenshot_collect"
? item.result_submitted_at
: item.publish_url,
)) {
return Response.json(
{ error: "已发布的笔记不能再次转派" },
{ error: task.task_type === "screenshot_collect" ? "已提交的截图任务不能再次转派" : "已发布的笔记不能再次转派" },
{ status: 409 },
);
}
@@ -632,7 +675,7 @@ async function handlePost(request: Request) {
}
const bundleId = uid("delegate");
const shareToken = crypto.randomUUID().replaceAll("-", "");
const statements: D1PreparedStatement[] = [
const statements: DatabaseStatement[] = [
db
.prepare(
`INSERT INTO delegation_bundles
@@ -658,6 +701,7 @@ async function handlePost(request: Request) {
WHERE id = ?
AND claim_id = ?
AND publish_url IS NULL
AND result_submitted_at IS NULL
AND delegation_bundle_id IS NULL`,
)
.bind(bundleId, distributionId, claimRow.id),
@@ -721,19 +765,18 @@ async function handlePost(request: Request) {
{ status: 404 },
);
}
const published = await db
const completed = await db
.prepare(
`SELECT COUNT(*) AS count
FROM distributions
WHERE delegation_bundle_id = ?
AND publish_url IS NOT NULL
AND publish_url != ''`,
AND (publish_url IS NOT NULL AND publish_url != '' OR result_submitted_at IS NOT NULL)`,
)
.bind(bundle.id)
.first<{ count: number }>();
if ((published?.count ?? 0) > 0) {
if ((completed?.count ?? 0) > 0) {
return Response.json(
{ error: "该分享包已有笔记发布需保留链接继续完成第7天数据回收" },
{ error: task.task_type === "screenshot_collect" ? "该分享包已有截图提交,不能撤销" : "该分享包已有笔记发布需保留链接继续完成第7天数据回收" },
{ status: 409 },
);
}
@@ -759,6 +802,9 @@ async function handlePost(request: Request) {
}
if (body.action === "submit") {
if (task.task_type === "screenshot_collect") {
return Response.json({ error: "截图回收任务无需填写发布链接" }, { status: 400 });
}
const claimToken = textValue(body.claimToken, 80);
const distributionId = textValue(body.distributionId, 80);
const publishInput = textValue(body.publishUrl, 5000);
@@ -768,14 +814,15 @@ async function handlePost(request: Request) {
{ status: 400 },
);
}
const publishUrl = extractXhsPublishUrl(publishInput);
const platform = task.platform === "抖音" ? "抖音" : "小红书";
const publishUrl = extractPublishUrl(publishInput, platform);
if (!publishUrl) {
return Response.json(
{ error: "请粘贴包含小红书长链或短链的分享内容" },
{ error: `请粘贴包含${platform}作品链接的分享内容` },
{ status: 400 },
);
}
const account = accountFromPublishLink(publishUrl);
const account = accountFromPublishLink(publishUrl, platform);
if (!account) {
return Response.json({ error: "发布链接格式不正确" }, { status: 400 });
}
@@ -786,28 +833,69 @@ async function handlePost(request: Request) {
delegationToken,
);
if (!assignment) {
return Response.json({ error: "笔记与领取凭证不匹配" }, { status: 403 });
return Response.json({ error: "作品与领取凭证不匹配" }, { status: 403 });
}
if (!assignment.publish_screenshot_key) {
return Response.json({ error: "请先上传发布截图" }, { status: 400 });
}
const reuseExistingAccount =
assignment.publish_url === publishUrl && assignment.account_id;
const matchedAccount = reuseExistingAccount
? null
: await db
.prepare(
`SELECT id FROM accounts
WHERE platform = ? AND platform_uid = ?
LIMIT 1`,
)
.bind(account.platform, account.platformUid)
.first<{ id: string }>();
const accountId =
reuseExistingAccount ||
matchedAccount?.id ||
`account-${hashText(`${account.platform}:${account.platformUid}`)}`;
const statements: D1PreparedStatement[] = [];
const statements: DatabaseStatement[] = [];
const publishUrlChanged = Boolean(
assignment.publish_url && assignment.publish_url !== publishUrl,
);
if (publishUrlChanged) {
statements.push(
db
.prepare(
`UPDATE distributions SET
d2_likes = NULL, d2_comments = NULL, d2_collects = NULL,
d5_likes = NULL, d5_comments = NULL, d5_collects = NULL,
d7_likes = NULL, d7_comments = NULL, d7_collects = NULL,
latest_likes = NULL, latest_comments = NULL,
latest_collects = NULL, latest_shares = NULL,
collection_status = 'pending',
collection_status_description = '发布链接已更新,等待重新采集',
collection_updated_at = NULL, last_collection_day = NULL,
updated_at = CURRENT_TIMESTAMP
WHERE id = ?`,
)
.bind(assignment.id),
db
.prepare("DELETE FROM collection_runs WHERE distribution_id = ?")
.bind(assignment.id),
);
}
if (!reuseExistingAccount) {
statements.push(
db
.prepare(
`INSERT INTO accounts
(id, platform, platform_uid, nickname, profile_url, post_count)
VALUES (?, ?, ?, ?, ?, 1)
(id, platform, platform_uid, nickname, profile_url,
current_contact, post_count)
VALUES (?, ?, ?, ?, ?, ?, 1)
ON CONFLICT(platform, platform_uid) DO UPDATE SET
nickname = excluded.nickname,
profile_url = excluded.profile_url,
post_count = accounts.post_count + ?,
current_contact = CASE
WHEN excluded.current_contact != ''
THEN excluded.current_contact
ELSE accounts.current_contact
END,
last_seen_at = CURRENT_TIMESTAMP`,
)
.bind(
@@ -816,7 +904,7 @@ async function handlePost(request: Request) {
account.platformUid,
account.nickname,
account.profileUrl,
assignment.publish_url ? 0 : 1,
assignment.claimant_name,
),
);
}
@@ -833,6 +921,26 @@ async function handlePost(request: Request) {
)
.bind(accountId, publishUrl, assignment.id),
);
statements.push(
db
.prepare(
`UPDATE accounts SET post_count = (
SELECT COUNT(*) FROM distributions WHERE account_id = ?
) WHERE id = ?`,
)
.bind(accountId, accountId),
);
if (assignment.account_id && assignment.account_id !== accountId) {
statements.push(
db
.prepare(
`UPDATE accounts SET post_count = (
SELECT COUNT(*) FROM distributions WHERE account_id = ?
) WHERE id = ?`,
)
.bind(assignment.account_id, assignment.account_id),
);
}
if (!assignment.publish_url) {
statements.push(
db
@@ -866,35 +974,53 @@ async function handlePost(request: Request) {
collectionDays = [];
}
if (collectionDays.length > 0) {
await db
.prepare(
`UPDATE distributions SET collection_status = 'scheduled',
collection_status_description = ? WHERE id = ?`,
)
.bind(
`已安排${collectionDays.length}个采集日每日09:00执行`,
assignment.id,
)
.run();
await createCollectionRunTasks(
db,
task.id,
collectionSchedule.collection_start_date,
collectionDays,
);
runInBackground(
runDueScheduledCollections(
db,
Date.now(),
resolveCollectionMcpConfig(
env as unknown as CollectionMcpBindings,
),
"catchup",
task.id,
).catch(() => undefined),
"collection catchup after partner publish update",
);
}
}
if (account.platform === "小红书") {
const enrichment = enrichDistributionAccount(
db,
assignment.id,
publishUrl,
account.nickname,
resolveCollectionMcpConfig(
env as unknown as CollectionMcpBindings,
),
).catch(() => undefined);
const executionContext = getRequestExecutionContext();
if (executionContext) {
executionContext.waitUntil(enrichment);
} else {
await enrichment;
}
}
const enrichment = enrichDistributionAccount(
db,
assignment.id,
publishUrl,
account.nickname,
resolveCollectionMcpConfig(
env as unknown as CollectionMcpBindings,
),
).catch(() => undefined);
runInBackground(enrichment, "distribution account enrichment");
return Response.json({ ok: true });
}
if (body.action === "submit_creator_metrics") {
if (task.task_type === "screenshot_collect") {
return Response.json({ error: "截图回收任务不需要创作者数据" }, { status: 400 });
}
const claimToken = textValue(body.claimToken, 80);
const distributionId = textValue(body.distributionId, 80);
const exposure = creatorMetricValue(body.exposure);
@@ -941,6 +1067,49 @@ async function handlePost(request: Request) {
return Response.json({ submitted: true });
}
if (body.action === "submit_screenshot_result") {
if (task.task_type !== "screenshot_collect") {
return Response.json({ error: "当前任务不支持这种回填方式" }, { status: 400 });
}
const claimToken = textValue(body.claimToken, 80);
const distributionId = textValue(body.distributionId, 80);
const assignment = await findAccessibleAssignment(
task.id,
distributionId,
claimToken,
delegationToken,
);
if (!assignment) {
return Response.json({ error: "任务与领取凭证不匹配" }, { status: 403 });
}
if (!assignment.result_screenshot_key) {
return Response.json({ error: "请先上传任务截图" }, { status: 400 });
}
const statements: DatabaseStatement[] = [
db
.prepare(
`UPDATE distributions SET
result_submitted_at = CURRENT_TIMESTAMP,
status = 'complete',
updated_at = CURRENT_TIMESTAMP
WHERE id = ?`,
)
.bind(assignment.id),
];
if (!assignment.result_submitted_at) {
statements.push(
db
.prepare(
`UPDATE partners SET completed_total = completed_total + 1
WHERE id = ?`,
)
.bind(assignment.partner_id),
);
}
await db.batch(statements);
return Response.json({ submitted: true });
}
return Response.json({ error: "不支持的操作" }, { status: 400 });
} catch (error) {
return Response.json(

View File

@@ -1,4 +1,5 @@
import { env } from "cloudflare:workers";
import { getRuntimeEnv } from "../../../lib/runtime-env";
import type { StoredObjectBody } from "../../../lib/object-store";
import { adminForbidden, isAdminRequest } from "../../../lib/admin-auth";
import { parseStoredDate } from "../../../lib/date-utils";
import {
@@ -16,8 +17,9 @@ import {
type RecoveryWorkbookRow,
} from "../../../lib/recovery-workbook";
import { consumeMcpExportToken } from "../../../lib/mcp-export-token";
import { normalizeWorkbookImage } from "../../../lib/workbook-image";
export const runtime = "edge";
const env = getRuntimeEnv();
type StoredAsset = {
index: number;
@@ -52,6 +54,7 @@ type ExportRow = {
latest_likes: number | null;
latest_comments: number | null;
latest_collects: number | null;
latest_shares: number | null;
collection_status: string | null;
collection_status_description: string | null;
collection_updated_at: string | null;
@@ -120,12 +123,16 @@ function latestMetrics(row: ExportRow) {
const likes = row.latest_likes ?? legacyLikes;
const comments = row.latest_comments ?? legacyComments;
const collects = row.latest_collects ?? legacyCollects;
const shares = row.latest_shares;
return {
likes,
comments,
collects,
shares,
total:
likes === null ? null : likes + (comments ?? 0) + (collects ?? 0),
likes === null
? null
: likes + (comments ?? 0) + (collects ?? 0) + (shares ?? 0),
};
}
@@ -159,7 +166,7 @@ function collectionLabel(row: ExportRow) {
: label;
}
function contentTypeFromObject(object: R2ObjectBody) {
function contentTypeFromObject(object: StoredObjectBody) {
const headers = new Headers();
object.writeHttpMetadata(headers);
return headers.get("Content-Type") || "application/octet-stream";
@@ -180,13 +187,13 @@ async function loadImage(reference: ImageReference) {
object = await bucket.get(reference.key);
}
if (!object) return null;
return {
return normalizeWorkbookImage({
bytes: new Uint8Array(await object.arrayBuffer()),
contentType: contentTypeFromObject(object),
width: reference.width,
height: reference.height,
description: reference.description,
} satisfies RecoveryWorkbookImage;
} satisfies RecoveryWorkbookImage);
}
async function loadImages(references: ImageReference[]) {
@@ -235,9 +242,9 @@ export async function GET(request: Request) {
}
const db = getRawDb();
const task = await db
.prepare("SELECT id, name, brand FROM tasks WHERE id = ?")
.prepare("SELECT id, name, brand, platform FROM tasks WHERE id = ?")
.bind(taskId)
.first<{ id: string; name: string; brand: string }>();
.first<{ id: string; name: string; brand: string; platform: string }>();
if (!task) {
return Response.json({ error: "没有找到这个任务" }, { status: 404 });
}
@@ -268,6 +275,7 @@ export async function GET(request: Request) {
d.latest_likes,
d.latest_comments,
d.latest_collects,
d.latest_shares,
d.collection_status,
d.collection_status_description,
d.collection_updated_at,
@@ -317,18 +325,19 @@ export async function GET(request: Request) {
}
});
const loadedImages = await loadImages(references);
const isDouyin = task.platform === "抖音";
const metricHeaders = isDouyin
? ["点赞", "收藏", "转发", "评论", "总互动"]
: ["点赞", "收藏", "评论", "总互动"];
const headers = [
"序号(不能改)",
"标题",
"笔记内容(正文+话题)",
...Array.from({ length: maxContentImages }, (_, index) => `图片${index + 1}`),
"小红书昵称",
`${task.platform}昵称`,
"发布链接",
"发布时间",
"点赞",
"收藏",
"评论",
"总互动",
...metricHeaders,
"曝光量-实际第7天",
"阅读量-实际第7天",
"数据分析截图(单篇笔记数据分析截图)",
@@ -340,7 +349,7 @@ export async function GET(request: Request) {
];
const originalImageStart = 3;
const accountColumn = originalImageStart + maxContentImages;
const creatorScreenshotColumn = accountColumn + 9;
const creatorScreenshotColumn = accountColumn + (isDouyin ? 10 : 9);
const publishScreenshotColumn = creatorScreenshotColumn + 1;
const workbookRows: RecoveryWorkbookRow[] = rows.map((row, rowIndex) => {
const metrics = latestMetrics(row);
@@ -349,7 +358,7 @@ export async function GET(request: Request) {
{ length: maxContentImages },
(_, index) => {
const asset = contentAssets.find((item) => item.index === index + 1);
return asset && loadedImages.get(asset.key) ? "见图" : asset ? "图片读取失败" : "";
return "";
},
);
const creatorImage = row.screenshot_key
@@ -368,12 +377,13 @@ export async function GET(request: Request) {
formatExportDate(row.publish_time),
metrics.likes,
metrics.collects,
...(isDouyin ? [metrics.shares] : []),
metrics.comments,
metrics.total,
row.exposure,
row.views,
creatorImage ? "见图" : row.screenshot_key ? "截图读取失败" : "",
publishImage ? "见图" : row.publish_screenshot_key ? "截图读取失败" : "",
"",
"",
formatExportDate(row.collection_updated_at || row.updated_at),
row.distribution_id ? collectionLabel(row) : "未领取",
row.partner_name || "",
@@ -402,6 +412,7 @@ export async function GET(request: Request) {
20,
11,
11,
...(isDouyin ? [11] : []),
11,
11,
18,

View File

@@ -7,8 +7,6 @@ import {
} from "../../../lib/recovery-workbook";
import { consumeMcpExportToken } from "../../../lib/mcp-export-token";
export const runtime = "edge";
type AccountRow = {
id: string;
platform: string;
@@ -17,7 +15,12 @@ type AccountRow = {
profile_url: string;
ip_location: string;
followers: number;
gender: string;
bio: string;
tags: string;
post_count: number;
cooperation_source: string;
current_contact: string;
first_seen_at: string;
last_seen_at: string;
};
@@ -25,6 +28,7 @@ type AccountRow = {
type CooperationRow = {
account_id: string;
partner_name: string;
claimant_name: string | null;
delegation_bundle_id: string | null;
};
@@ -75,9 +79,11 @@ async function exportAccounts(accountIds: string[]) {
`SELECT
d.account_id,
p.name AS partner_name,
cl.claimant_name,
d.delegation_bundle_id
FROM distributions d
JOIN partners p ON p.id = d.partner_id
LEFT JOIN claims cl ON cl.id = d.claim_id
WHERE d.account_id IS NOT NULL`,
)
.all<CooperationRow>(),
@@ -110,15 +116,33 @@ async function exportAccounts(accountIds: string[]) {
"账号主页",
"IP地",
"粉丝数",
"性别",
"简介",
"标签",
"合作发布数",
"历史合作来源",
"当前联系人",
"资源归属",
"首次合作时间",
"最近合作时间",
];
const rows: RecoveryWorkbookRow[] = accounts.map((account, index) => {
const cooperation = cooperationByAccount.get(account.id) ?? [];
const sources = [...new Set(cooperation.map((item) => item.partner_name))];
const sources = [
...new Set([
...cooperation
.filter(
(item) =>
!item.claimant_name ||
item.partner_name !== item.claimant_name,
)
.map((item) => item.partner_name),
...(account.cooperation_source || "")
.split(/[、,;|]/)
.map((item) => item.trim())
.filter(Boolean),
]),
];
const partnerManagedOnly =
cooperation.some((item) => item.delegation_bundle_id) &&
cooperation.every((item) => item.delegation_bundle_id);
@@ -131,8 +155,12 @@ async function exportAccounts(accountIds: string[]) {
account.profile_url || "",
account.ip_location || "待识别",
account.followers,
account.gender || "",
account.bio || "",
account.tags || "",
account.post_count,
sources.join("、"),
account.current_contact || "",
partnerManagedOnly ? "合作社资源 · 不可直联" : "可直联",
formatExportDate(account.first_seen_at),
formatExportDate(account.last_seen_at),
@@ -154,9 +182,13 @@ async function exportAccounts(accountIds: string[]) {
44,
14,
14,
10,
36,
32,
14,
32,
22,
22,
21,
21,
],

View File

@@ -0,0 +1,500 @@
import { isManagerRequest, managerForbidden } from "../../../lib/user-auth";
import { runInBackground } from "../../../lib/background";
import { ensureSchema, getRawDb } from "../../../lib/mvp-db";
import {
resolveCollectionMcpConfig,
resolveProfileDetailsFromMcp,
resolveXhsPublicAccountDetails,
type CollectionMcpBindings,
} from "../../../lib/mcp-collection-client";
import { getRuntimeEnv } from "../../../lib/runtime-env";
import {
mergeCooperationSources,
normalizeProfileUrl,
parseResourceImportFile,
RESOURCE_IMPORT_MAX_BYTES,
RESOURCE_IMPORT_MAX_ROWS,
resourcePlatformUid,
resourceImportMissingFields,
type ResourceImportRow,
} from "../../../lib/resource-import";
type AccountRow = {
id: string;
platform: string;
platform_uid: string;
public_account_id: string;
nickname: string;
profile_url: string;
ip_location: string;
followers: number;
gender: string;
bio: string;
tags: string;
cooperation_source: string;
};
type AnalyzedRow = ResourceImportRow & {
action: "create" | "update" | "error";
accountId: string;
platformUid: string;
cooperationSource: string;
};
const RESOURCE_IMPORT_PREVIEW_ROWS = 100;
const RESOURCE_IMPORT_SYNC_ENRICH_ROWS = 100;
const RESOURCE_IMPORT_DB_BATCH_SIZE = 100;
function identityKey(platform: string, value: string) {
return `${platform.trim().toLocaleLowerCase("zh-CN")}|${value
.trim()
.toLocaleLowerCase("zh-CN")}`;
}
async function loadAccounts() {
return getRawDb()
.prepare(
`SELECT id, platform, platform_uid, public_account_id, nickname,
profile_url, ip_location, followers, gender, bio, tags,
cooperation_source
FROM accounts`,
)
.all<AccountRow>();
}
async function mapConcurrent<T, R>(
items: T[],
limit: number,
worker: (item: T) => Promise<R>,
) {
const results = new Array<R>(items.length);
let cursor = 0;
await Promise.all(
Array.from({ length: Math.min(limit, items.length) }, async () => {
while (cursor < items.length) {
const index = cursor;
cursor += 1;
results[index] = await worker(items[index]);
}
}),
);
return results;
}
function mergeExistingFields(rows: ResourceImportRow[], accounts: AccountRow[]) {
const existingByProfile = new Map<string, AccountRow>();
for (const account of accounts) {
const profileUrl = normalizeProfileUrl(account.profile_url || "");
if (profileUrl) {
existingByProfile.set(identityKey(account.platform, profileUrl), account);
}
}
return rows.map((row) => {
if (
row.errors.length > 0 ||
!row.profileUrl ||
!["小红书", "抖音"].includes(row.platform)
) {
return row;
}
const existing = existingByProfile.get(identityKey(row.platform, row.profileUrl));
const existingNickname =
existing?.nickname && existing.nickname !== "待识别账号"
? existing.nickname
: "";
const existingIpLocation =
existing?.ip_location && existing.ip_location !== "待识别"
? existing.ip_location
: "";
const existingGender: ResourceImportRow["gender"] =
existing?.gender === "男" || existing?.gender === "女"
? existing.gender
: "";
return {
...row,
nickname: row.nickname || existingNickname,
publicAccountId: row.publicAccountId || existing?.public_account_id || "",
ipLocation: row.ipLocation || existingIpLocation,
followers: row.followersResolved
? row.followers
: Number(existing?.followers || 0),
followersResolved:
row.followersResolved || Number(existing?.followers || 0) > 0,
gender: row.gender || existingGender,
bio: row.bio || existing?.bio || "",
tags: row.tags.length > 0
? row.tags
: (existing?.tags || "")
.split(/[,,、;|]/)
.map((item) => item.trim())
.filter(Boolean)
.slice(0, 5),
};
});
}
async function enrichRows(rows: ResourceImportRow[], accounts: AccountRow[]) {
const mcpConfig = resolveCollectionMcpConfig(
getRuntimeEnv() as unknown as CollectionMcpBindings,
);
const baselineRows = mergeExistingFields(rows, accounts);
return mapConcurrent(baselineRows, 4, async (baseline) => {
const row = baseline;
if (
row.errors.length > 0 ||
!row.profileUrl ||
!["小红书", "抖音"].includes(row.platform)
) {
return row;
}
if (resourceImportMissingFields(baseline).length === 0) {
return baseline;
}
let details: {
nickname: string | null;
redId: string | null;
followers: number | null;
ipLocation: string | null;
gender: "" | "男" | "女";
bio: string;
recentNoteTitles: string[];
providerTags: string[];
} = await resolveProfileDetailsFromMcp(
row.profileUrl,
row.platform === "抖音" ? "抖音" : "小红书",
mcpConfig,
).catch(() => ({
nickname: null,
redId: null,
followers: null,
ipLocation: null,
gender: "" as const,
bio: "",
recentNoteTitles: [],
providerTags: [],
}));
const mcpResult = {
nickname: baseline.nickname || details.nickname?.trim() || "",
publicAccountId: baseline.publicAccountId || details.redId?.trim() || "",
ipLocation: baseline.ipLocation || details.ipLocation?.trim() || "",
followersResolved: baseline.followersResolved || details.followers !== null,
gender: baseline.gender || details.gender,
bio: baseline.bio || details.bio,
tags: baseline.tags,
};
if (
row.platform === "小红书" &&
resourceImportMissingFields(mcpResult).length > 0
) {
const publicDetails = await resolveXhsPublicAccountDetails(row.profileUrl).catch(
() => ({ nickname: null, redId: null, followers: null, ipLocation: null }),
);
details = {
...details,
nickname: details.nickname || publicDetails.nickname,
redId: details.redId || publicDetails.redId,
followers: details.followers ?? publicDetails.followers,
ipLocation: details.ipLocation || publicDetails.ipLocation,
};
}
return {
...baseline,
nickname: baseline.nickname || details.nickname?.trim() || "待识别账号",
publicAccountId: baseline.publicAccountId || details.redId?.trim() || "",
ipLocation: baseline.ipLocation || details.ipLocation?.trim() || "待识别",
followers: baseline.followersResolved
? baseline.followers
: (details.followers ?? 0),
followersResolved:
baseline.followersResolved || details.followers !== null,
gender: baseline.gender || details.gender,
bio: baseline.bio || details.bio,
tags: baseline.tags,
};
});
}
function analyzeRows(rows: ResourceImportRow[], accountRows: AccountRow[]) {
const profileMap = new Map<string, AccountRow>();
const publicIdMap = new Map<string, AccountRow>();
const platformUidMap = new Map<string, AccountRow>();
for (const account of accountRows) {
const profileUrl = normalizeProfileUrl(account.profile_url || "");
if (profileUrl) profileMap.set(identityKey(account.platform, profileUrl), account);
if (account.public_account_id) {
publicIdMap.set(identityKey(account.platform, account.public_account_id), account);
}
platformUidMap.set(identityKey(account.platform, account.platform_uid), account);
}
return rows.map<AnalyzedRow>((row) => {
const platformUid = resourcePlatformUid(row);
const profileMatch = row.profileUrl
? profileMap.get(identityKey(row.platform, row.profileUrl))
: undefined;
const publicIdMatch = row.publicAccountId
? publicIdMap.get(identityKey(row.platform, row.publicAccountId))
: undefined;
const uidMatch = platformUidMap.get(identityKey(row.platform, platformUid));
const matches = [profileMatch, publicIdMatch, uidMatch].filter(
(account): account is AccountRow => Boolean(account),
);
const matchedIds = [...new Set(matches.map((account) => account.id))];
const errors = [...row.errors];
if (matchedIds.length > 1) {
errors.push("账号主页和账号ID匹配到不同的现有账号请先核对");
}
const existing = matchedIds.length === 1 ? matches[0] : undefined;
const accountId = existing?.id ?? `account-${crypto.randomUUID().slice(0, 12)}`;
const analyzed: AnalyzedRow = {
...row,
errors,
action: errors.length > 0 ? "error" : existing ? "update" : "create",
accountId,
platformUid: existing?.platform_uid ?? platformUid,
cooperationSource: mergeCooperationSources(
existing?.cooperation_source ?? "",
row.cooperationSource,
),
};
if (analyzed.action !== "error") {
const virtual: AccountRow = {
id: accountId,
platform: row.platform,
platform_uid: analyzed.platformUid,
public_account_id: row.publicAccountId || existing?.public_account_id || "",
nickname: row.nickname,
profile_url: row.profileUrl || existing?.profile_url || "",
ip_location: row.ipLocation || existing?.ip_location || "待识别",
followers: row.followers || existing?.followers || 0,
gender: row.gender || existing?.gender || "",
bio: row.bio || existing?.bio || "",
tags: (row.tags.length > 0
? row.tags
: (existing?.tags || "").split(/[,,、;|]/).filter(Boolean)
).slice(0, 5).join(","),
cooperation_source: analyzed.cooperationSource,
};
if (row.profileUrl) profileMap.set(identityKey(row.platform, row.profileUrl), virtual);
if (row.publicAccountId) {
publicIdMap.set(identityKey(row.platform, row.publicAccountId), virtual);
}
platformUidMap.set(identityKey(row.platform, analyzed.platformUid), virtual);
}
return analyzed;
});
}
function summarize(rows: AnalyzedRow[]) {
return {
total: rows.length,
create: rows.filter((row) => row.action === "create").length,
update: rows.filter((row) => row.action === "update").length,
error: rows.filter((row) => row.action === "error").length,
};
}
function previewAnalyzedRows(rows: AnalyzedRow[]) {
const errorRows = rows.filter((row) => row.action === "error");
if (errorRows.length === 0) {
return rows.slice(0, RESOURCE_IMPORT_PREVIEW_ROWS);
}
const importableRows = rows.filter((row) => row.action !== "error");
return [
...errorRows.slice(0, RESOURCE_IMPORT_PREVIEW_ROWS),
...importableRows.slice(
0,
Math.max(0, RESOURCE_IMPORT_PREVIEW_ROWS - errorRows.length),
),
];
}
function statementForAnalyzedRow(
db: ReturnType<typeof getRawDb>,
row: AnalyzedRow,
) {
return row.action === "update"
? db
.prepare(
`UPDATE accounts SET
nickname = ?,
public_account_id = CASE WHEN ? != '' THEN ? ELSE public_account_id END,
profile_url = CASE WHEN ? != '' THEN ? ELSE profile_url END,
ip_location = CASE
WHEN ? != '' AND ? != '待识别' THEN ? ELSE ip_location END,
followers = CASE WHEN ? = 1 THEN ? ELSE followers END,
gender = CASE WHEN ? != '' THEN ? ELSE gender END,
bio = CASE WHEN ? != '' THEN ? ELSE bio END,
tags = CASE WHEN ? != '' THEN ? ELSE tags END,
cooperation_source = ?,
last_seen_at = CURRENT_TIMESTAMP
WHERE id = ?`,
)
.bind(
row.nickname || "待识别账号",
row.publicAccountId,
row.publicAccountId,
row.profileUrl,
row.profileUrl,
row.ipLocation,
row.ipLocation,
row.ipLocation,
row.followersResolved ? 1 : 0,
row.followers,
row.gender,
row.gender,
row.bio,
row.bio,
row.tags.join(","),
row.tags.join(","),
row.cooperationSource,
row.accountId,
)
: db
.prepare(
`INSERT INTO accounts
(id, platform, platform_uid, public_account_id, nickname,
profile_url, ip_location, followers, post_count, avg_views,
gender, bio, tags, cooperation_source)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, 0, 0, ?, ?, ?, ?)`,
)
.bind(
row.accountId,
row.platform,
row.platformUid,
row.publicAccountId,
row.nickname || "待识别账号",
row.profileUrl,
row.ipLocation || "待识别",
row.followers,
row.gender,
row.bio,
row.tags.join(","),
row.cooperationSource,
);
}
async function writeAnalyzedRows(rows: AnalyzedRow[]) {
const db = getRawDb();
const statements = rows
.filter((row) => row.action !== "error")
.map((row) => statementForAnalyzedRow(db, row));
for (let index = 0; index < statements.length; index += RESOURCE_IMPORT_DB_BATCH_SIZE) {
await db.batch(statements.slice(index, index + RESOURCE_IMPORT_DB_BATCH_SIZE));
}
}
function deferredEnrichmentCount(rows: ResourceImportRow[]) {
return rows.filter(
(row) =>
row.errors.length === 0 &&
row.profileUrl &&
resourceImportMissingFields(row).length > 0,
).length;
}
async function enrichImportedRowsInBackground(rows: ResourceImportRow[]) {
const accounts = await loadAccounts();
const baselineRows = mergeExistingFields(rows, accounts.results);
const missingRows = baselineRows.filter(
(row) =>
row.errors.length === 0 &&
row.profileUrl &&
resourceImportMissingFields(row).length > 0,
);
if (missingRows.length === 0) return;
const enriched = await enrichRows(missingRows, accounts.results);
const latestAccounts = await loadAccounts();
const analyzed = analyzeRows(enriched, latestAccounts.results).filter(
(row) => row.action !== "error",
);
await writeAnalyzedRows(analyzed);
}
export async function POST(request: Request) {
if (!(await isManagerRequest(request))) return managerForbidden();
try {
await ensureSchema();
const form = await request.formData();
const file = form.get("file");
const mode = String(form.get("mode") ?? "preview");
if (!(file instanceof File)) {
return Response.json({ error: "请选择需要导入的 Excel 文件" }, { status: 400 });
}
if (file.size <= 0 || file.size > RESOURCE_IMPORT_MAX_BYTES) {
return Response.json({ error: "文件不能为空,且不能超过 20MB" }, { status: 400 });
}
const rows = parseResourceImportFile(file.name, new Uint8Array(await file.arrayBuffer()));
const accounts = await loadAccounts();
const shouldEnrichSynchronously = rows.length <= RESOURCE_IMPORT_SYNC_ENRICH_ROWS;
const preparedRows = shouldEnrichSynchronously
? await enrichRows(rows, accounts.results)
: mergeExistingFields(rows, accounts.results);
const analyzed = analyzeRows(preparedRows, accounts.results);
const summary = summarize(analyzed);
const importableRows = analyzed.filter((row) => row.action !== "error");
const importableRowNumbers = new Set(
importableRows.map((row) => row.rowNumber),
);
const importablePreparedRows = preparedRows.filter((row) =>
importableRowNumbers.has(row.rowNumber),
);
const deferredEnrichment = shouldEnrichSynchronously
? 0
: deferredEnrichmentCount(importablePreparedRows);
if (mode !== "commit") {
return Response.json({
summary,
rows: previewAnalyzedRows(analyzed).map((row) => ({
rowNumber: row.rowNumber,
platform: row.platform,
nickname: row.nickname,
publicAccountId: row.publicAccountId,
profileUrl: row.profileUrl,
ipLocation: row.ipLocation,
followers: row.followers,
gender: row.gender,
bio: row.bio,
tags: row.tags,
cooperationSource: row.cooperationSource,
action: row.action,
errors: row.errors,
})),
truncated: analyzed.length > RESOURCE_IMPORT_PREVIEW_ROWS,
deferredEnrichment,
maxRows: RESOURCE_IMPORT_MAX_ROWS,
});
}
if (importableRows.length === 0) {
return Response.json(
{ error: "没有可导入的有效账号,请修正异常数据后重新上传", summary },
{ status: 400 },
);
}
await writeAnalyzedRows(importableRows);
if (deferredEnrichment > 0) {
const importableSourceRows = rows.filter((row) =>
importableRowNumbers.has(row.rowNumber),
);
runInBackground(
enrichImportedRowsInBackground(importableSourceRows),
"bulk resource profile enrichment",
);
}
return Response.json({
summary,
deferredEnrichment,
message: `已导入 ${summary.create} 个新账号,更新 ${summary.update} 个已有账号${
summary.error > 0 ? `;跳过 ${summary.error} 条异常数据` : ""
}${
deferredEnrichment > 0
? `${deferredEnrichment} 个账号的缺失公开资料将在后台补全`
: ""
}`,
});
} catch (error) {
const message = error instanceof Error ? error.message : "导入失败";
return Response.json({ error: message }, { status: 400 });
}
}

View File

@@ -0,0 +1,102 @@
import { zipSync, strToU8 } from "fflate";
import { adminForbidden, isAdminRequest } from "../../../lib/admin-auth";
import { ensureSchema, getRawDb, getUploadBucket } from "../../../lib/mvp-db";
import { parseResultScreenshotKeys } from "../../../lib/result-screenshots";
function safeName(value: string) {
return value.replace(/[\\/:*?"<>|\r\n]/g, "_").trim().slice(0, 60) || "KOC";
}
function csvCell(value: unknown) {
const text = String(value ?? "");
return `"${text.replaceAll('"', '""')}"`;
}
export async function GET(request: Request) {
if (!(await isAdminRequest(request))) return adminForbidden();
try {
await ensureSchema();
const taskId = new URL(request.url).searchParams.get("task")?.trim();
if (!taskId) {
return Response.json({ error: "缺少任务参数" }, { status: 400 });
}
const task = await getRawDb()
.prepare("SELECT name, task_type FROM tasks WHERE id = ?")
.bind(taskId)
.first<{ name: string; task_type: string }>();
if (!task || task.task_type !== "screenshot_collect") {
return Response.json({ error: "没有找到截图回收任务" }, { status: 404 });
}
const rows = await getRawDb()
.prepare(
`SELECT d.id, d.result_screenshot_key, d.result_submitted_at,
d.claimed_at, c.source_row, c.title, p.name AS partner_name,
cl.claimant_name
FROM distributions d
JOIN contents c ON c.id = d.content_id
JOIN partners p ON p.id = d.partner_id
LEFT JOIN claims cl ON cl.id = d.claim_id
WHERE d.task_id = ?
ORDER BY COALESCE(c.source_row, 999999), d.claimed_at, d.id`,
)
.bind(taskId)
.all<{
id: string;
result_screenshot_key: string | null;
result_submitted_at: string | null;
claimed_at: string;
source_row: number | null;
title: string;
partner_name: string;
claimant_name: string | null;
}>();
const files: Record<string, Uint8Array> = {};
const manifest = [
["序号", "搜索关键词", "领取人", "领取时间", "提交时间", "文件名"],
];
let exported = 0;
for (const [rowIndex, row] of rows.results.entries()) {
const fileNames: string[] = [];
const screenshotKeys = row.result_submitted_at
? parseResultScreenshotKeys(row.result_screenshot_key)
: [];
for (const [imageIndex, screenshotKey] of screenshotKeys.entries()) {
const object = await getUploadBucket().get(screenshotKey);
if (!object) continue;
const extension = screenshotKey.split(".").at(-1) || "jpg";
const fileName = `${String(row.source_row ?? rowIndex + 1).padStart(3, "0")}-${safeName(row.claimant_name || row.partner_name)}-${imageIndex + 1}.${extension}`;
files[fileName] = new Uint8Array(await object.arrayBuffer());
fileNames.push(fileName);
exported += 1;
}
manifest.push([
String(row.source_row ?? ""),
row.title,
row.claimant_name || row.partner_name,
row.claimed_at,
row.result_submitted_at || "",
fileNames.join(""),
]);
}
files["回收清单.csv"] = strToU8(
`\uFEFF${manifest.map((cells) => cells.map(csvCell).join(",")).join("\r\n")}`,
);
const archive = zipSync(files, { level: 0 });
const headers = new Headers({
"Content-Type": "application/zip",
"Content-Disposition": `attachment; filename*=UTF-8''${encodeURIComponent(`${safeName(task.name)}-截图回收.zip`)}`,
"Cache-Control": "private, no-store",
"X-KOC-Exported-Count": String(exported),
});
const body = archive.buffer.slice(
archive.byteOffset,
archive.byteOffset + archive.byteLength,
) as ArrayBuffer;
return new Response(body, { headers });
} catch (error) {
return Response.json(
{ error: error instanceof Error ? error.message : "截图打包失败" },
{ status: 500 },
);
}
}

View File

@@ -0,0 +1,30 @@
import { adminForbidden, isAdminRequest } from "../../../lib/admin-auth";
import { ensureSchema, getUploadBucket, uid } from "../../../lib/mvp-db";
export async function POST(request: Request) {
if (!(await isAdminRequest(request))) return adminForbidden();
try {
await ensureSchema();
const form = await request.formData();
const file = form.get("file");
if (!(file instanceof File) || file.size === 0) {
return Response.json({ error: "请选择示例截图" }, { status: 400 });
}
if (!file.type.startsWith("image/") || file.size > 8_000_000) {
return Response.json({ error: "仅支持8MB以内的图片" }, { status: 400 });
}
const extension =
file.name.split(".").at(-1)?.replace(/[^a-zA-Z0-9]/g, "") || "jpg";
const key = `task-assets/${uid("example")}.${extension}`;
await getUploadBucket().put(key, await file.arrayBuffer(), {
httpMetadata: { contentType: file.type },
customMetadata: { kind: "screenshot-task-example" },
});
return Response.json({ uploaded: true, key });
} catch (error) {
return Response.json(
{ error: error instanceof Error ? error.message : "示例截图上传失败" },
{ status: 500 },
);
}
}

View File

@@ -0,0 +1,49 @@
import { adminForbidden, isAdminRequest } from "../../../lib/admin-auth";
import { ensureSchema, getRawDb, getUploadBucket } from "../../../lib/mvp-db";
import { parseResultScreenshotKeys } from "../../../lib/result-screenshots";
export async function GET(request: Request) {
if (!(await isAdminRequest(request))) return adminForbidden();
try {
await ensureSchema();
const distributionId = new URL(request.url).searchParams
.get("distribution")
?.trim();
const imageIndex = Math.max(
1,
Number(new URL(request.url).searchParams.get("index") || 1),
);
if (!distributionId) {
return Response.json({ error: "缺少任务记录" }, { status: 400 });
}
const row = await getRawDb()
.prepare(
`SELECT result_screenshot_key FROM distributions
WHERE id = ?`,
)
.bind(distributionId)
.first<{ result_screenshot_key: string | null }>();
const screenshotKey = row
? parseResultScreenshotKeys(row.result_screenshot_key)[imageIndex - 1]
: undefined;
if (!screenshotKey) {
return Response.json({ error: "任务截图不存在" }, { status: 404 });
}
const object = await getUploadBucket().get(screenshotKey);
if (!object) {
return Response.json({ error: "任务截图不存在" }, { status: 404 });
}
const headers = new Headers({
"Cache-Control": "private, no-store",
"Content-Disposition": 'inline; filename="task-result-screenshot"',
"X-Content-Type-Options": "nosniff",
});
object.writeHttpMetadata(headers);
return new Response(await object.arrayBuffer(), { headers });
} catch (error) {
return Response.json(
{ error: error instanceof Error ? error.message : "截图读取失败" },
{ status: 500 },
);
}
}

View File

@@ -7,8 +7,6 @@ import {
} from "../../../lib/mvp-db";
import { adminForbidden, isAdminRequest } from "../../../lib/admin-auth";
export const runtime = "edge";
export async function POST(request: Request) {
if (!(await isAdminRequest(request))) return adminForbidden();
try {

View File

@@ -9,8 +9,6 @@ import {
} from "../../../lib/user-auth";
import { ensureSchema, getRawDb } from "../../../lib/mvp-db";
export const runtime = "edge";
type UserRow = {
id: string;
username: string;

File diff suppressed because it is too large Load Diff

View File

@@ -1,18 +1,7 @@
import type { Metadata } from "next";
import { Geist, Geist_Mono } from "next/font/google";
import { headers } from "next/headers";
import "./globals.css";
const geistSans = Geist({
variable: "--font-geist-sans",
subsets: ["latin"],
});
const geistMono = Geist_Mono({
variable: "--font-geist-mono",
subsets: ["latin"],
});
export async function generateMetadata(): Promise<Metadata> {
const requestHeaders = await headers();
const host = requestHeaders.get("x-forwarded-host") || requestHeaders.get("host");
@@ -60,9 +49,7 @@ export default function RootLayout({
}: Readonly<{ children: React.ReactNode }>) {
return (
<html lang="zh-CN">
<body className={`${geistSans.variable} ${geistMono.variable}`}>
{children}
</body>
<body>{children}</body>
</html>
);
}

View File

@@ -1,45 +0,0 @@
import { access, cp, mkdir, rm } from "node:fs/promises";
import { resolve } from "node:path";
import type { Plugin } from "vite";
async function exists(path: string): Promise<boolean> {
try {
await access(path);
return true;
} catch (error) {
if ((error as NodeJS.ErrnoException).code === "ENOENT") {
return false;
}
throw error;
}
}
// Packages Sites metadata and migrations after Vite finishes compiling.
export function sites(): Plugin {
let root = process.cwd();
return {
name: "sites",
apply: "build",
configResolved(config) {
root = config.root;
},
async closeBundle() {
const outputDirectory = resolve(root, "dist", ".openai");
const hostingConfig = resolve(root, ".openai", "hosting.json");
const drizzleSource = resolve(root, "drizzle");
await rm(outputDirectory, { recursive: true, force: true });
await mkdir(outputDirectory, { recursive: true });
if (await exists(hostingConfig)) {
await cp(hostingConfig, resolve(outputDirectory, "hosting.json"));
}
if (await exists(drizzleSource)) {
await cp(drizzleSource, resolve(outputDirectory, "drizzle"), {
recursive: true,
});
}
},
};
}

View File

@@ -1,13 +1,7 @@
import { env } from "cloudflare:workers";
import { drizzle } from "drizzle-orm/d1";
import { drizzle } from "drizzle-orm/mysql2";
import { getPool } from "../lib/database";
import * as schema from "./schema";
export function getDb() {
if (!env.DB) {
throw new Error(
"Cloudflare D1 binding `DB` is unavailable. Set the `d1` field in .openai/hosting.json to `DB` or let your control plane inject the real binding values before using the database."
);
}
return drizzle(env.DB, { schema });
return drizzle({ client: getPool(), schema, mode: "default" });
}

View File

@@ -1,91 +1,115 @@
import { sql } from "drizzle-orm";
import {
datetime,
index,
integer,
sqliteTable,
int,
mysqlTable,
text,
uniqueIndex,
} from "drizzle-orm/sqlite-core";
varchar,
} from "drizzle-orm/mysql-core";
export const partners = sqliteTable("partners", {
id: text("id").primaryKey(),
name: text("name").notNull(),
wecomName: text("wecom_name").notNull(),
owner: text("owner").notNull().default("运营组"),
claimedTotal: integer("claimed_total").notNull().default(0),
completedTotal: integer("completed_total").notNull().default(0),
createdAt: text("created_at").notNull().default(sql`CURRENT_TIMESTAMP`),
const timestamp = (name: string) =>
datetime(name, { mode: "string", fsp: 3 })
.notNull()
.default(sql`CURRENT_TIMESTAMP(3)`);
export const partners = mysqlTable("partners", {
id: varchar("id", { length: 64 }).primaryKey(),
name: varchar("name", { length: 255 }).notNull(),
wecomName: varchar("wecom_name", { length: 255 }).notNull(),
owner: varchar("owner", { length: 255 }).notNull().default("运营组"),
claimedTotal: int("claimed_total").notNull().default(0),
completedTotal: int("completed_total").notNull().default(0),
createdAt: timestamp("created_at"),
});
export const tasks = sqliteTable(
export const tasks = mysqlTable(
"tasks",
{
id: text("id").primaryKey(),
name: text("name").notNull(),
brand: text("brand").notNull(),
quantity: integer("quantity").notNull(),
claimedQuantity: integer("claimed_quantity").notNull().default(0),
dueAt: text("due_at").notNull(),
status: text("status").notNull().default("active"),
sourceUrl: text("source_url").notNull().default(""),
sourceSheetId: text("source_sheet_id").notNull().default(""),
sourceSheetName: text("source_sheet_name").notNull().default(""),
sourceSyncedAt: text("source_synced_at"),
shareToken: text("share_token"),
collectionStartDate: text("collection_start_date"),
collectionDays: text("collection_days").notNull().default("[]"),
collectionScheduleUpdatedAt: text("collection_schedule_updated_at"),
createdAt: text("created_at").notNull().default(sql`CURRENT_TIMESTAMP`),
id: varchar("id", { length: 64 }).primaryKey(),
name: varchar("name", { length: 255 }).notNull(),
brand: varchar("brand", { length: 255 }).notNull(),
quantity: int("quantity").notNull(),
claimedQuantity: int("claimed_quantity").notNull().default(0),
dueAt: varchar("due_at", { length: 32 }).notNull(),
status: varchar("status", { length: 32 }).notNull().default("active"),
taskType: varchar("task_type", { length: 32 })
.notNull()
.default("content_publish"),
platform: varchar("platform", { length: 32 }).notNull().default("小红书"),
contentFormat: varchar("content_format", { length: 32 })
.notNull()
.default("image_text"),
sourceUrl: text("source_url").notNull(),
sourceSheetId: varchar("source_sheet_id", { length: 255 }).notNull().default(""),
sourceSheetName: varchar("source_sheet_name", { length: 255 }).notNull().default(""),
sourceSyncedAt: datetime("source_synced_at", { mode: "string", fsp: 3 }),
shareToken: varchar("share_token", { length: 128 }),
collectionStartDate: varchar("collection_start_date", { length: 32 }),
collectionDays: text("collection_days").notNull(),
collectionScheduleUpdatedAt: datetime("collection_schedule_updated_at", {
mode: "string",
fsp: 3,
}),
createdAt: timestamp("created_at"),
},
(table) => [uniqueIndex("tasks_share_token_idx").on(table.shareToken)],
);
export const contents = sqliteTable("contents", {
id: text("id").primaryKey(),
taskId: text("task_id").notNull(),
export const contents = mysqlTable("contents", {
id: varchar("id", { length: 64 }).primaryKey(),
taskId: varchar("task_id", { length: 64 }).notNull(),
title: text("title").notNull(),
body: text("body").notNull().default(""),
imageAssets: text("image_assets").notNull().default("[]"),
status: text("status").notNull().default("available"),
source: text("source").notNull().default("飞书内容表"),
sourceRow: integer("source_row"),
createdAt: text("created_at").notNull().default(sql`CURRENT_TIMESTAMP`),
body: text("body").notNull(),
imageAssets: text("image_assets").notNull(),
videoAssets: text("video_assets").notNull(),
status: varchar("status", { length: 32 }).notNull().default("available"),
source: varchar("source", { length: 255 }).notNull().default("飞书内容表"),
sourceRow: int("source_row"),
createdAt: timestamp("created_at"),
});
export const accounts = sqliteTable(
export const accounts = mysqlTable(
"accounts",
{
id: text("id").primaryKey(),
platform: text("platform").notNull().default("小红书"),
platformUid: text("platform_uid").notNull(),
publicAccountId: text("public_account_id").notNull().default(""),
nickname: text("nickname").notNull(),
profileUrl: text("profile_url").notNull().default(""),
ipLocation: text("ip_location").notNull().default("待识别"),
followers: integer("followers").notNull().default(0),
postCount: integer("post_count").notNull().default(0),
avgViews: integer("avg_views").notNull().default(0),
firstSeenAt: text("first_seen_at").notNull().default(sql`CURRENT_TIMESTAMP`),
lastSeenAt: text("last_seen_at").notNull().default(sql`CURRENT_TIMESTAMP`),
id: varchar("id", { length: 64 }).primaryKey(),
platform: varchar("platform", { length: 32 }).notNull().default("小红书"),
platformUid: varchar("platform_uid", { length: 255 }).notNull(),
publicAccountId: varchar("public_account_id", { length: 255 }).notNull().default(""),
nickname: varchar("nickname", { length: 255 }).notNull(),
profileUrl: text("profile_url").notNull(),
ipLocation: varchar("ip_location", { length: 255 }).notNull().default("待识别"),
followers: int("followers").notNull().default(0),
gender: varchar("gender", { length: 16 }).notNull().default(""),
bio: text("bio").notNull().default(""),
tags: varchar("tags", { length: 500 }).notNull().default(""),
postCount: int("post_count").notNull().default(0),
avgViews: int("avg_views").notNull().default(0),
cooperationSource: varchar("cooperation_source", { length: 500 })
.notNull()
.default(""),
currentContact: varchar("current_contact", { length: 255 })
.notNull()
.default(""),
firstSeenAt: timestamp("first_seen_at"),
lastSeenAt: timestamp("last_seen_at"),
},
(table) => [
uniqueIndex("accounts_platform_uid_idx").on(
table.platform,
table.platformUid,
),
uniqueIndex("accounts_platform_uid_idx").on(table.platform, table.platformUid),
],
);
export const claims = sqliteTable(
export const claims = mysqlTable(
"claims",
{
id: text("id").primaryKey(),
taskId: text("task_id").notNull(),
partnerId: text("partner_id").notNull(),
claimantName: text("claimant_name").notNull(),
claimToken: text("claim_token").notNull(),
quantity: integer("quantity").notNull(),
createdAt: text("created_at").notNull().default(sql`CURRENT_TIMESTAMP`),
id: varchar("id", { length: 64 }).primaryKey(),
taskId: varchar("task_id", { length: 64 }).notNull(),
partnerId: varchar("partner_id", { length: 64 }).notNull(),
claimantName: varchar("claimant_name", { length: 255 }).notNull(),
claimToken: varchar("claim_token", { length: 128 }).notNull(),
quantity: int("quantity").notNull(),
createdAt: timestamp("created_at"),
},
(table) => [
uniqueIndex("claims_claim_token_idx").on(table.claimToken),
@@ -97,123 +121,118 @@ export const claims = sqliteTable(
],
);
export const delegationBundles = sqliteTable(
export const delegationBundles = mysqlTable(
"delegation_bundles",
{
id: text("id").primaryKey(),
taskId: text("task_id").notNull(),
claimId: text("claim_id").notNull(),
partnerId: text("partner_id").notNull(),
label: text("label").notNull(),
shareToken: text("share_token").notNull(),
quantity: integer("quantity").notNull(),
status: text("status").notNull().default("active"),
createdAt: text("created_at").notNull().default(sql`CURRENT_TIMESTAMP`),
updatedAt: text("updated_at").notNull().default(sql`CURRENT_TIMESTAMP`),
revokedAt: text("revoked_at"),
id: varchar("id", { length: 64 }).primaryKey(),
taskId: varchar("task_id", { length: 64 }).notNull(),
claimId: varchar("claim_id", { length: 64 }).notNull(),
partnerId: varchar("partner_id", { length: 64 }).notNull(),
label: varchar("label", { length: 255 }).notNull(),
shareToken: varchar("share_token", { length: 128 }).notNull(),
quantity: int("quantity").notNull(),
status: varchar("status", { length: 32 }).notNull().default("active"),
createdAt: timestamp("created_at"),
updatedAt: timestamp("updated_at"),
revokedAt: datetime("revoked_at", { mode: "string", fsp: 3 }),
},
(table) => [
uniqueIndex("delegation_bundles_share_token_idx").on(table.shareToken),
index("delegation_bundles_claim_created_idx").on(
table.claimId,
table.createdAt,
),
index("delegation_bundles_claim_created_idx").on(table.claimId, table.createdAt),
],
);
export const distributions = sqliteTable("distributions", {
id: text("id").primaryKey(),
taskId: text("task_id").notNull(),
contentId: text("content_id").notNull(),
partnerId: text("partner_id").notNull(),
claimId: text("claim_id"),
delegationBundleId: text("delegation_bundle_id"),
accountId: text("account_id"),
export const distributions = mysqlTable("distributions", {
id: varchar("id", { length: 64 }).primaryKey(),
taskId: varchar("task_id", { length: 64 }).notNull(),
contentId: varchar("content_id", { length: 64 }).notNull(),
partnerId: varchar("partner_id", { length: 64 }).notNull(),
claimId: varchar("claim_id", { length: 64 }),
delegationBundleId: varchar("delegation_bundle_id", { length: 64 }),
accountId: varchar("account_id", { length: 64 }),
publishUrl: text("publish_url"),
publishTime: text("publish_time"),
publishScreenshotKey: text("publish_screenshot_key"),
status: text("status").notNull().default("claimed"),
claimedAt: text("claimed_at").notNull().default(sql`CURRENT_TIMESTAMP`),
screenshotKey: text("screenshot_key"),
ocrStatus: text("ocr_status").notNull().default("none"),
exposure: integer("exposure"),
views: integer("views"),
d2Likes: integer("d2_likes"),
d2Comments: integer("d2_comments"),
d2Collects: integer("d2_collects"),
d5Likes: integer("d5_likes"),
d5Comments: integer("d5_comments"),
d5Collects: integer("d5_collects"),
d7Likes: integer("d7_likes"),
d7Comments: integer("d7_comments"),
d7Collects: integer("d7_collects"),
latestLikes: integer("latest_likes"),
latestComments: integer("latest_comments"),
latestCollects: integer("latest_collects"),
collectionStatus: text("collection_status").notNull().default("pending"),
publishTime: datetime("publish_time", { mode: "string", fsp: 3 }),
publishScreenshotKey: varchar("publish_screenshot_key", { length: 512 }),
resultScreenshotKey: text("result_screenshot_key"),
resultSubmittedAt: datetime("result_submitted_at", { mode: "string", fsp: 3 }),
status: varchar("status", { length: 32 }).notNull().default("claimed"),
claimedAt: timestamp("claimed_at"),
screenshotKey: varchar("screenshot_key", { length: 512 }),
ocrStatus: varchar("ocr_status", { length: 32 }).notNull().default("none"),
exposure: int("exposure"),
views: int("views"),
d2Likes: int("d2_likes"),
d2Comments: int("d2_comments"),
d2Collects: int("d2_collects"),
d5Likes: int("d5_likes"),
d5Comments: int("d5_comments"),
d5Collects: int("d5_collects"),
d7Likes: int("d7_likes"),
d7Comments: int("d7_comments"),
d7Collects: int("d7_collects"),
latestLikes: int("latest_likes"),
latestComments: int("latest_comments"),
latestCollects: int("latest_collects"),
latestShares: int("latest_shares"),
collectionStatus: varchar("collection_status", { length: 32 })
.notNull()
.default("pending"),
collectionStatusDescription: text("collection_status_description"),
collectionUpdatedAt: text("collection_updated_at"),
lastCollectionDay: integer("last_collection_day"),
updatedAt: text("updated_at").notNull().default(sql`CURRENT_TIMESTAMP`),
collectionUpdatedAt: datetime("collection_updated_at", { mode: "string", fsp: 3 }),
lastCollectionDay: int("last_collection_day"),
updatedAt: timestamp("updated_at"),
});
export const collectionRuns = sqliteTable(
export const collectionRuns = mysqlTable(
"collection_runs",
{
id: text("id").primaryKey(),
taskId: text("task_id").notNull(),
distributionId: text("distribution_id").notNull(),
scheduledDate: text("scheduled_date").notNull(),
scheduleDay: integer("schedule_day"),
scheduledAt: text("scheduled_at").notNull(),
status: text("status").notNull().default("pending"),
likes: integer("likes"),
comments: integer("comments"),
collects: integer("collects"),
id: varchar("id", { length: 64 }).primaryKey(),
taskId: varchar("task_id", { length: 64 }).notNull(),
distributionId: varchar("distribution_id", { length: 64 }).notNull(),
scheduledDate: varchar("scheduled_date", { length: 32 }).notNull(),
scheduleDay: int("schedule_day"),
scheduledAt: datetime("scheduled_at", { mode: "string", fsp: 3 }).notNull(),
status: varchar("status", { length: 32 }).notNull().default("pending"),
likes: int("likes"),
comments: int("comments"),
collects: int("collects"),
shares: int("shares"),
statusDescription: text("status_description"),
startedAt: text("started_at"),
completedAt: text("completed_at"),
createdAt: text("created_at").notNull().default(sql`CURRENT_TIMESTAMP`),
startedAt: datetime("started_at", { mode: "string", fsp: 3 }),
completedAt: datetime("completed_at", { mode: "string", fsp: 3 }),
createdAt: timestamp("created_at"),
},
(table) => [
uniqueIndex("collection_runs_distribution_date_idx").on(
table.distributionId,
table.scheduledDate,
),
index("collection_runs_task_date_idx").on(
table.taskId,
table.scheduledDate,
),
index("collection_runs_task_date_idx").on(table.taskId, table.scheduledDate),
],
);
export const users = sqliteTable(
export const users = mysqlTable(
"users",
{
id: text("id").primaryKey(),
username: text("username").notNull(),
passwordHash: text("password_hash").notNull(),
passwordSalt: text("password_salt").notNull(),
passwordIterations: integer("password_iterations").notNull(),
role: text("role").notNull(),
createdAt: text("created_at").notNull().default(sql`CURRENT_TIMESTAMP`),
updatedAt: text("updated_at").notNull().default(sql`CURRENT_TIMESTAMP`),
id: varchar("id", { length: 64 }).primaryKey(),
username: varchar("username", { length: 255 }).notNull(),
passwordHash: varchar("password_hash", { length: 255 }).notNull(),
passwordSalt: varchar("password_salt", { length: 255 }).notNull(),
passwordIterations: int("password_iterations").notNull(),
role: varchar("role", { length: 32 }).notNull(),
createdAt: timestamp("created_at"),
updatedAt: timestamp("updated_at"),
},
(table) => [
uniqueIndex("users_username_idx").on(table.username),
uniqueIndex("users_single_super_admin_idx")
.on(table.role)
.where(sql`${table.role} = 'super_admin'`),
],
(table) => [uniqueIndex("users_username_idx").on(table.username)],
);
export const authSessions = sqliteTable(
export const authSessions = mysqlTable(
"auth_sessions",
{
tokenHash: text("token_hash").primaryKey(),
userId: text("user_id").notNull(),
expiresAt: text("expires_at").notNull(),
createdAt: text("created_at").notNull().default(sql`CURRENT_TIMESTAMP`),
tokenHash: varchar("token_hash", { length: 128 }).primaryKey(),
userId: varchar("user_id", { length: 64 }).notNull(),
expiresAt: datetime("expires_at", { mode: "string", fsp: 3 }).notNull(),
createdAt: timestamp("created_at"),
},
(table) => [
index("auth_sessions_user_id_idx").on(table.userId),
@@ -221,14 +240,32 @@ export const authSessions = sqliteTable(
],
);
export const mcpExportTokens = sqliteTable(
export const mcpExportTokens = mysqlTable(
"mcp_export_tokens",
{
tokenHash: text("token_hash").primaryKey(),
kind: text("kind").notNull(),
tokenHash: varchar("token_hash", { length: 128 }).primaryKey(),
kind: varchar("kind", { length: 64 }).notNull(),
payload: text("payload").notNull(),
expiresAt: text("expires_at").notNull(),
createdAt: text("created_at").notNull().default(sql`CURRENT_TIMESTAMP`),
expiresAt: datetime("expires_at", { mode: "string", fsp: 3 }).notNull(),
createdAt: timestamp("created_at"),
},
(table) => [index("mcp_export_tokens_expires_at_idx").on(table.expiresAt)],
);
export const backgroundJobs = mysqlTable(
"background_jobs",
{
id: varchar("id", { length: 64 }).primaryKey(),
type: varchar("type", { length: 64 }).notNull(),
payload: text("payload").notNull(),
status: varchar("status", { length: 32 }).notNull().default("pending"),
attempts: int("attempts").notNull().default(0),
availableAt: datetime("available_at", { mode: "string", fsp: 3 }).notNull(),
lockedAt: datetime("locked_at", { mode: "string", fsp: 3 }),
completedAt: datetime("completed_at", { mode: "string", fsp: 3 }),
lastError: text("last_error"),
createdAt: timestamp("created_at"),
updatedAt: timestamp("updated_at"),
},
(table) => [index("background_jobs_pending_idx").on(table.status, table.availableAt)],
);

11
deploy/nginx/Dockerfile Normal file
View File

@@ -0,0 +1,11 @@
FROM node:22.18.0-bookworm-slim AS portal-builder
WORKDIR /portal
ENV NEXT_TELEMETRY_DISABLED=1
COPY koc-portal/package.json koc-portal/package-lock.json ./
RUN npm ci
COPY koc-portal/ ./
RUN npm run build
FROM nginx:1.27-alpine
COPY deploy/nginx/koc-loop.conf /etc/nginx/conf.d/default.conf
COPY --from=portal-builder /portal/out /usr/share/nginx/html/koc

View File

@@ -0,0 +1,42 @@
server {
listen 80;
server_name _;
# 批量回填 Excel 会内嵌多篇笔记原图和截图。
client_max_body_size 85m;
location = /koc {
return 301 /koc/$is_args$args;
}
location /koc/ {
alias /usr/share/nginx/html/koc/;
try_files $uri $uri/ /koc/index.html;
add_header X-Robots-Tag "noindex, nofollow, noarchive" always;
add_header Referrer-Policy "no-referrer" always;
}
location /api/mcp {
proxy_pass http://app:3000;
proxy_http_version 1.1;
proxy_buffering off;
proxy_read_timeout 3600s;
proxy_send_timeout 3600s;
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
location / {
proxy_pass http://app:3000;
proxy_http_version 1.1;
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_read_timeout 180s;
}
}

BIN
design-qa-comparison.png Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 326 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 59 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 397 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 46 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 376 KiB

BIN
design-qa-inline-filter.png Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 56 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 633 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 73 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 228 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 280 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 83 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 44 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 84 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 67 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 76 KiB

View File

@@ -1,50 +1,293 @@
# KOC LOOP 用户管理页设计 QA
# KOC LOOP 数据回收标题跳转设计 QA
- Source visual truth: `/var/folders/0g/8yrmts9s7v5g_xc60sxx0__80000gn/T/codex-clipboard-d06a2293-9e7d-4471-b4ae-5eee942b68e9.png`
- Implementation screenshot: `/private/tmp/koc-user-management-final.jpg`
- Delete confirmation screenshot: `/private/tmp/koc-user-delete-modal.jpg`
- Viewport: desktop `1280 × 720` CSS px; responsive check `680 × 900` CSS px
- Pixels and density: source `2738 × 1382`; implementation `1280 × 720`; browser reported `devicePixelRatio = 2`; comparison used the visible layout and computed CSS geometry rather than pixel-perfect scaling because the requested result intentionally changes the source from side-by-side to stacked sections.
- State: logged-in super administrator on the user-management screen
## 验证对象
## Full-view comparison evidence
- 用户参考截图:`/var/folders/0g/8yrmts9s7v5g_xc60sxx0__80000gn/T/codex-clipboard-2296f03e-e79b-494c-b745-08e939fc20c6.png`
- 浏览器实现截图:`/private/tmp/koc-loop-recovery-title-link-focused.png`
- 参考与实现并排对照:`/private/tmp/koc-loop-recovery-title-link-comparison.png`
- 本地页面:`http://localhost:8080/`
The source places account creation and the account list side by side, forcing a tall narrow form and compressing the list. The implementation intentionally stacks the sections: the account-creation card spans the page and uses one compact horizontal row on desktop; the account list spans the full row below it. Existing KOC LOOP navigation, typography, colors, border treatment, and panel radius remain unchanged.
## 环境与状态
## Focused-region comparison evidence
- CSS 视口1278 × 692桌面布局
- 用户参考截图2556 × 1384按 2:1 密度归一为 1278 × 692
- 实现截图1278 × 692
- 页面状态:数据回收 → 美团医美-备婚 → 数据回收队列
- 交互状态:已回填短链的第一条标题获得键盘焦点;两条未回填记录保持普通文本
The delete confirmation modal was checked separately. It uses the existing modal shell, a restrained destructive color, explicit irreversible-action copy, cancel and confirm actions, and a disabled working state. No new image assets are present on this screen.
## 完整画面对比
## Required fidelity surfaces
- 信息架构、侧栏、采集计划和数据回收表格继续沿用现有界面,没有新增路由或改变表格列宽。
- 已回填标题使用现有绿色交互色,聚焦时显示清晰但克制的描边;未回填标题仍为黑色普通文本。
- 参考图中的目标区域与实现截图在同一张并排对照图中检查,未发现遮挡、换行异常或列错位。
- Fonts and typography: existing product font stack, heading hierarchy, weights, and field labels are preserved; passed.
- Spacing and layout rhythm: panel padding, 16 px vertical section gap, 12 px form gap, and 14 px table rows establish a clearer rhythm; passed after responsive overflow fix.
- Colors and visual tokens: existing green, canvas, line, and panel tokens are preserved; destructive actions use a muted red semantic treatment; passed.
- Image quality and asset fidelity: this screen has no content imagery or custom visual assets; not applicable.
- Copy and content: creation, reset, and deletion copy is concise; deletion clearly states immediate sign-out and irreversibility; passed.
## 聚焦区域检查
## Interaction verification
- 字体与层级:标题字号、字重和省略规则保持不变,仅为可点击标题增加语义色和 hover/focus 状态。
- 间距与布局:链接仍受原有 310px 最大宽度约束,头像、账号副标题和相邻数据列未发生位移。
- 颜色与状态:绿色与现有按钮、成功状态色一致;键盘焦点轮廓可见。
- 图片质量:创作者截图缩略图未受本次改动影响,仍保持原比例显示。
- 文案内容:标题原文、账号名称、平台信息和采集数据均保持不变。
- Created a local ordinary test account.
- Opened the row-level delete confirmation.
- Confirmed deletion and verified the row disappeared.
- Verified the current super-administrator row has no delete action.
- Verified browser console errors: none.
- Verified desktop page horizontal overflow: none (`scrollWidth = innerWidth = 1280`).
- Verified 680 px responsive page horizontal overflow: none (`scrollWidth = innerWidth = 680`); the table scrolls inside its own container.
## 功能验证
## Comparison history
- 当前任务中识别到 1 条可点击标题,`href` 为已回填的 `http://xhslink.cn/o/9qOYiD3Iu8K``target=_blank`
- 点击标题后短链成功跳转并解析为小红书笔记详情页。
- 另外 2 条没有发布链接的标题不是链接,避免误导点击。
- 链接仅允许小红书正式域名和 `xhslink.cn` / `xhslink.com` 短链域名,其他协议或域名不会渲染为链接。
- 后台浏览器控制台无 warning/error。
- Docker 生产构建通过;相关静态验收测试 12 项全部通过。
1. Initial responsive pass found the account table's minimum width expanding the parent grid at 680 px.
2. Added `min-width: 0` to the stacked layout and panels, and constrained the table to its panel.
3. Post-fix evidence: page `scrollWidth` reduced from `714` to `680`, matching the viewport; the table retains an internal `660` px scroll surface.
## 迭代记录
## Findings
No actionable P0, P1, or P2 issues remain.
## Follow-up polish
No blocking polish items. A future iteration may add search when the account count grows substantially.
1. 初次浏览器验收发现第一条发布链接为 `xhslink.cn`,而前端白名单只包含 `xhslink.com`,因此标题仍是普通文本。
2. 补充 `xhslink.cn` 及其子域名白名单,保留 HTTP/HTTPS 与小红书域名边界。
3. 重新构建后,浏览器识别到 1 条可点击标题;点击实际打开对应小红书页面,未回填行仍不可点击。
final result: passed
---
# KOC LOOP 任务筛选常驻搜索框设计 QA
## 验证对象
- 用户目标截图:`/var/folders/0g/8yrmts9s7v5g_xc60sxx0__80000gn/T/codex-clipboard-67b0b080-8ea8-4504-82eb-348e8cf4ec19.png`
- 浏览器实现截图:`/private/tmp/koc-distributions-direct-search.png`
- CSS 视口842 × 778设备像素比 2
- 本地页面:`http://localhost:8080/?nav=distributions`
## 调整结果
1. 品牌/项目、内容类型、平台不再使用“先点下拉、再输入搜索”的两层结构,筛选栏直接展示三个可输入的搜索框。
2. 输入内容时立即进行模糊筛选;聚焦或输入后,下方仅展示匹配候选项,不再重复显示第二个搜索框。
3. 宽屏下四个搜索框保持同一行;当前窄视口下任务名称独占一行,三个分类搜索保持在下一行,不互相遮挡。
4. 保留键盘与无障碍语义,三个分类搜索均使用 `combobox`,候选项使用 `listbox` / `option`
## 功能验证
- 默认展示 6 个任务。
- 品牌/项目直接输入“美团”后,候选项仅显示“美团”和“美团医美”,任务结果同步缩小为 2 个。
- 清空输入后恢复 6 个任务。
- 静态验收测试 14 项全部通过,正式构建通过;本地 Docker 应用已重新构建并加载新页面。
final result: passed
---
# KOC LOOP 任务筛选下拉遮挡设计 QA
## 验证对象
- 用户问题截图:`/var/folders/0g/8yrmts9s7v5g_xc60sxx0__80000gn/T/codex-clipboard-bd2d14b7-472f-43a5-b997-1b6ff9528ba1.png`
- 浏览器实现截图:`/Users/wufp/Documents/koc-loop/design-qa-inline-filter.png`
- 修复前后并排对照:`/Users/wufp/Documents/koc-loop/design-qa-inline-filter-comparison.png`
- CSS 视口1280 × 720设备像素比 2
- 本地页面:`http://localhost:8080/?nav=distributions`
## 问题与调整
1. 之前仍使用浏览器原生 `select`,系统级下拉层会悬浮在页面上,因此即使搜索框本身布局正常,展开菜单仍会遮住搜索区或任务卡片。
2. 品牌/项目、内容类型、平台三个筛选器改为页面内选择面板,面板使用正常文档流布局,展开时把任务卡片向下推。
3. 同一时间只展开一个筛选器;选择选项后自动收起,并保留键盘焦点、`aria-expanded``listbox``option` 语义。
## 布局与功能验证
- 内容类型面板展开后:面板 `position: static`,搜索框底部为 379px面板顶部为 388px两者无重叠。
- 第一张任务卡片顶部为 480px面板底部为 452px任务卡片位于面板下方未被覆盖。
- 选择“视频”后面板自动收起,任务数由 6 个缩小为 1 个;清空筛选后恢复 6 个任务。
- 浏览器实测搜索框、三个筛选按钮和任务卡片均可正常操作,交互过程中未出现页面报错。
- TypeScript 构建、静态验收及共 65 项测试全部通过;本地 Docker 服务已重新构建并通过健康检查。
final result: passed
---
# KOC LOOP 任务分发筛选栏设计 QA
## 验证对象
- 用户问题截图:`/var/folders/0g/8yrmts9s7v5g_xc60sxx0__80000gn/T/codex-clipboard-4c32c7e3-58e0-4262-b625-c02b20ec4490.png`
- 浏览器实现截图:`/Users/wufp/Documents/koc-loop/design-qa-implementation.png`
- 修复前后并排对照:`/Users/wufp/Documents/koc-loop/design-qa-comparison.png`
- CSS 视口1280 × 720
- 本地页面:`http://localhost:8080/?nav=distributions`
## 问题与调整
1. 全局 `label` 布局把搜索图标和输入框纵向堆叠,导致搜索框被拆成上下两层。
2. 搜索框现在显式使用横向布局,并清除继承的外边距;图标使用固定宽度居中对齐。
3. 品牌/项目下拉框加宽,选中值和下拉菜单锚点保持稳定,不再挤压搜索区域。
## 功能验证
- 输入“视频”后,结果从 6 个任务缩小为 1 个任务,筛选栏没有发生错位。
- 清空搜索后恢复展示 6 个任务。
- 浏览器控制台无错误。
- TypeScript、静态验收测试和正式构建均通过。
- 本地 Docker 服务已重建并通过健康检查。
final result: passed
---
# KOC LOOP 可搜索任务筛选浮层设计 QA
## 验证对象
- 参考截图:`/var/folders/0g/8yrmts9s7v5g_xc60sxx0__80000gn/T/codex-clipboard-fc7ce949-d8ea-4a3b-afa6-04a778fdb8dc.png`
- 浏览器实现截图:`/Users/wufp/Documents/koc-loop/design-qa-current-filter.png`
- 参考与实现并排对照:`/Users/wufp/Documents/koc-loop/design-qa-filter-comparison.png`
- 本地页面:`http://localhost:8080/?nav=distributions`
## 布局与交互验证
- 品牌/项目、内容类型、平台均改为紧凑的页面内浮层,不再使用系统原生下拉,也不会把任务卡片整体向下推。
- 浮层锚定在当前筛选按钮下方,宽度与筛选控件协调,未遮挡任务名称搜索框及其他筛选按钮。
- 浮层顶部支持输入模糊查询;输入“医”后只显示“美团医美”,可继续点击完成筛选。
- 同一时间只展开一个筛选器,支持点击页面其他位置或按 Esc 收起。
- 浏览器实测筛选栏、任务卡片和按钮均保持稳定,没有布局跳动。
## 数据修复验证
- 抖音作品重新采集后获得点赞 5,684、收藏 565、转发 6,878、评论 332。
- 抖音主页链接改用短链跳转携带的真实 `sec_uid`,不再用数字内部用户 ID 拼接主页。
- MCP 的公开主页解析对当前账号仍返回资源不存在,因此公开抖音号、粉丝数和 IP 暂时保持待识别,不再写入错误数据。
- 相关自动化测试共 67 项全部通过,本地 Docker 服务已重建并通过健康检查。
final result: passed
# KOC LOOP 任务分发横向搜索框设计 QA
## 验证对象
- 用户问题截图:`codex-clipboard-252bf483-c3d1-4d70-abfb-d54587a300a2.png`
- 浏览器实现截图:`/private/tmp/koc-loop-distribution-filters-fixed.png`
- 本地页面:`http://localhost:8080/?nav=distributions`
## 问题与调整
1. 品牌/项目、内容类型、平台三个筛选框使用通用表单标签,继承了纵向排列样式,导致图标和提示文字上下分离。
2. 筛选框改为独立容器,并显式使用横向排列和垂直居中。
3. 保留原有模糊搜索、筛选逻辑及整体视觉规范。
## 布局与功能验证
- 三个筛选框内图标、输入文字与容器的垂直中心误差均为 0px。
- 四个搜索框保持同排展示,无相互遮挡、无异常换行、无额外浮层。
- 项目构建通过;页面渲染测试 14/14 通过;本地 Docker 运行版本已更新。
最终结果:通过。
---
# KOC LOOP KOC资源卡片密度优化设计 QA
## 验证对象
- 参考卡片:`/var/folders/0g/8yrmts9s7v5g_xc60sxx0__80000gn/T/codex-clipboard-615400a6-0bd1-4110-b280-f7fdc29bf705.png`
- 原页面参考:`/var/folders/0g/8yrmts9s7v5g_xc60sxx0__80000gn/T/codex-clipboard-5bf2bfb5-a50e-4c08-a407-192fdcf20b58.png`
- 第一轮实现截图:`/Users/wufp/Documents/koc-loop/design-qa-resource-cards-v2.png`
- 最终浏览器截图:`/Users/wufp/Documents/koc-loop/design-qa-resource-cards-final.png`
- 聚焦卡片并排对照:`/Users/wufp/Documents/koc-loop/design-qa-resource-card-comparison.png`
- 本地页面:`http://localhost:8080/`KOC资源状态
## 环境与归一化
- CSS 视口1280 × 720设备像素比 2浏览器截图按 1280 × 720 CSS 像素输出。
- 参考卡片像素478 × 700最终完整页面截图1280 × 720。
- 聚焦对照将实现中的 304px 宽卡片等比放大到 478px并与参考卡片并排查看没有把两张独立截图当作同一对比证据。
- 桌面状态为三列卡片;同时验证 960px 两列、640px 一列。
## 完整画面对比
- 信息架构采用参考图的“头像 → 身份信息 → 简介 → 标签”顺序,同时保留 KOC LOOP 原有的粉丝、合作发布、合作来源和主页入口。
- 卡片高度由旧版的大块分区缩短到首屏实测约 274—278px三列宽度均为 304px页面没有横向溢出。
- 小红书使用粉红顶部识别线和柔和粉色标签,抖音使用深色顶部识别线,继续沿用现有平台标识。
## 聚焦区域检查
- 字体与层级:昵称 14px 并提高字重;账号号、属地、标签和数据从第一轮偏小的 8px 提升至 9—11px信息仍紧凑但可读性更好。
- 间距与布局:圆形头像、昵称与性别保持同一视觉组;账号号、平台和属地收进头像右侧;粉丝、合作发布、最近合作合并为一条浅底数据栏。
- 颜色与视觉标记:标签颜色与平台呼应,状态对比足够;卡片 hover 只使用轻微位移和阴影,不改变布局。
- 图片与资产:当前账号数据没有头像 URL因此保留现有首字母头像作为明确的数据缺失状态没有伪造真人头像平台标识继续使用项目已有资产。
- 文案与内容:真实简介最多展示三行;“未填写简介”“还没有简介”等占位内容统一折叠为“暂无简介”;无标签显示“待打标”;合作来源和主页入口合并到底部同一行。
## 交互与响应式验证
- 输入标签“美食探店”后,结果正确缩小为 3 个账号;清空后恢复完整列表。
- 主页入口保持可见并保留现有跳转目标;首屏六张卡片均检测到有效入口文案。
- 960px 视口为两列640px 视口为一列,两个断点均无横向溢出。
- 浏览器控制台无 error本地应用、MySQL、Nginx 均正常运行。
- 正式构建及完整自动化测试通过,共 82 项,无失败。
## 迭代记录
1. 第一轮已完成资料卡层级和高度压缩,但聚焦截图显示辅助文字偏小,无简介账号的底部留白仍较明显,记为 P2。
2. 第二轮提高辅助文字字号,移除固定最小高度,并把合作来源与主页入口合并为同一底栏。
3. 复查后卡片高度稳定在约 274—278px关键内容可读桌面与移动断点无溢出先前 P2 已解决。
## 结论
- 没有遗留 P0、P1 或 P2 问题。
- P3 后续项:如果 MCP 未来提供可靠头像 URL可将首字母头像替换成真实头像进一步接近参考图。
final result: passed
---
# KOC LOOP KOC资源卡片底栏对齐设计 QA
## 验证对象
- 用户标注截图:`/var/folders/0g/8yrmts9s7v5g_xc60sxx0__80000gn/T/codex-clipboard-fa2c031d-52ba-4ef5-bd70-2bf74adfa52d.png`
- 最终浏览器截图:`/Users/wufp/Documents/koc-loop/design-qa-resource-card-alignment-final.png`
- 归一化并排对照:`/Users/wufp/Documents/koc-loop/design-qa-resource-card-alignment-comparison.png`
- 本地页面:`http://localhost:8080/?nav=resources`
## 问题与调整
1. 不同账号的简介和标签数量不一致时,合作来源与主页入口会跟随前方内容上下浮动,导致同一排卡片的底部结构不齐。
2. 卡片保持纵向弹性布局,底栏改为自动占用剩余空间并贴住卡片底部;不增加固定高度,不改变数据或交互逻辑。
3. 三列桌面视口下实测前三张卡片高度均为 267.9px,底栏顶部均为 333.4px,底栏底部均为 365.9px,误差为 0px。
## 验证结果
- CSS 视口1280 × 720三列卡片状态。
- 不同简介、标签数量下,合作来源、来源标签和主页入口保持同一条水平基线。
- 浏览器控制台无 error页面 hover 位移不会改变静止状态的布局基线。
- 页面渲染测试与 TypeScript 检查通过;本地 Docker 已重建。
final result: passed
---
# KOC LOOP KOC资源卡片数据栏对齐设计 QA
## 验证对象
- 用户标注截图:`/var/folders/0g/8yrmts9s7v5g_xc60sxx0__80000gn/T/codex-clipboard-78335120-710a-4a7b-96f9-b1046191c788.png`
- 双列实现截图:`/Users/wufp/Documents/koc-loop/design-qa-resource-card-metrics-alignment-two-column.png`
- 三列实现截图:`/Users/wufp/Documents/koc-loop/design-qa-resource-card-metrics-alignment-final.png`
- 归一化并排对照:`/Users/wufp/Documents/koc-loop/design-qa-resource-card-metrics-alignment-comparison.png`
- 本地页面:`http://localhost:8080/?nav=resources`
## 环境与归一化
- 用户截图为 1674 × 1180px双列实现截图为 837 × 591px。
- 并排对照将用户截图归一化为 837 × 591px与实现截图使用同一双列宽度和页面状态进行聚焦比较。
- 同时在 1280 × 720 三列桌面视口复测,确认调整不依赖双列断点。
## 问题与调整
1. 先前仅把合作来源底栏贴底,数据栏仍跟随简介和标签数量上下浮动,形成 P2 对齐问题。
2. 将弹性留白移动到标签区之后,数据栏与合作来源底栏作为一个完整结构贴住卡片底部;数据内容、标签和交互均未改动。
## 验证结果
- 双列第一排两张卡片的数据栏顶部均为 233.4px、底部均为 267.9px;底栏顶部均为 277.9px、底部均为 310.4px。
- 双列第二排两张卡片的数据栏顶部均为 525.3px、底部均为 559.8px;底栏顶部均为 569.8px、底部均为 602.3px。
- 三列前三张卡片的数据栏与底栏上下边界误差均为 0px。
- 字体、颜色、图片资产和文案未发生变化;浏览器控制台无 error。
- 页面渲染测试 14/14、TypeScript 检查和 Docker 构建均通过。
final result: passed

View File

@@ -0,0 +1,74 @@
services:
mysql:
image: mysql:8.4
restart: unless-stopped
environment:
MYSQL_ROOT_PASSWORD: ${MYSQL_ROOT_PASSWORD:?请在环境文件中设置 MYSQL_ROOT_PASSWORD}
MYSQL_DATABASE: ${MYSQL_DATABASE:-koc_loop}
MYSQL_USER: ${MYSQL_USER:-koc}
MYSQL_PASSWORD: ${MYSQL_PASSWORD:?请在环境文件中设置 MYSQL_PASSWORD}
TZ: Asia/Shanghai
command:
- --character-set-server=utf8mb4
- --collation-server=utf8mb4_0900_ai_ci
- --default-time-zone=+00:00
volumes:
- mysql_data:/var/lib/mysql
healthcheck:
test: ["CMD-SHELL", "mysqladmin ping -h 127.0.0.1 -u root -p$$MYSQL_ROOT_PASSWORD --silent"]
interval: 10s
timeout: 5s
retries: 20
start_period: 30s
app:
build:
context: .
dockerfile: Dockerfile
restart: unless-stopped
environment:
MYSQL_HOST: mysql
MYSQL_PORT: 3306
MYSQL_DATABASE: ${MYSQL_DATABASE:-koc_loop}
MYSQL_USER: ${MYSQL_USER:-koc}
MYSQL_PASSWORD: ${MYSQL_PASSWORD}
UPLOAD_DIR: /data/koc/uploads
APP_ORIGIN: ${APP_ORIGIN}
KOC_PORTAL_URL: ${KOC_PORTAL_URL}
SUPER_ADMIN_USERNAME: ${SUPER_ADMIN_USERNAME}
SUPER_ADMIN_PASSWORD: ${SUPER_ADMIN_PASSWORD}
ADMIN_INTERNAL_TOKEN: ${ADMIN_INTERNAL_TOKEN}
KOC_MCP_API_KEY: ${KOC_MCP_API_KEY}
FEISHU_APP_ID: ${FEISHU_APP_ID:-}
FEISHU_APP_SECRET: ${FEISHU_APP_SECRET:-}
AI_TOOL_CENTER_MCP_URL: ${AI_TOOL_CENTER_MCP_URL:-}
AI_TOOL_CENTER_MCP_KEY: ${AI_TOOL_CENTER_MCP_KEY:-}
ENABLE_SCHEDULER: ${ENABLE_SCHEDULER:-true}
SEED_DEMO_DATA: ${SEED_DEMO_DATA:-false}
TZ: Asia/Shanghai
volumes:
- upload_data:/data/koc/uploads
depends_on:
mysql:
condition: service_healthy
healthcheck:
test: ["CMD-SHELL", "node -e \"fetch('http://127.0.0.1:3000/api/health').then(r=>{if(!r.ok)process.exit(1)}).catch(()=>process.exit(1))\""]
interval: 15s
timeout: 5s
retries: 10
start_period: 45s
nginx:
build:
context: .
dockerfile: deploy/nginx/Dockerfile
restart: unless-stopped
ports:
- "${HTTP_PORT:-80}:80"
depends_on:
app:
condition: service_healthy
volumes:
mysql_data:
upload_data:

View File

@@ -0,0 +1,208 @@
# KOC LOOP 私有化部署指南
本文适用于 `main` 分支。目标架构是运维提出的Nginx 代理 + KOC 服务 + MySQL 数据库,并让后台、外部 KOC 领取页和 Agent MCP 都能通过一个公网域名访问。
## 1. 部署形态
| 组件 | 容器 | 作用 | 持久化 |
| --- | --- | --- | --- |
| Nginx | `nginx` | 公网入口、反向代理、托管 KOC 领取页 | 配置随镜像 |
| KOC 服务 | `app` | Next.js 后台、API、MCP、每天 09:00 自动采集 | 上传目录挂载卷 |
| MySQL 8 | `mysql` | 任务、笔记、领取、回填、账号、采集和用户数据 | MySQL 数据卷 |
访问路径:
- `https://你的域名/`:运营后台;
- `https://你的域名/koc/`:外部 KOC 领取和回填;
- `https://你的域名/api/mcp`Agent MCP
- `https://你的域名/api/health`:服务健康检查。
第一版按单实例 KOC 服务设计。MySQL 和上传目录均持久化。以后需要横向扩容时,可以把上传卷换成共享 NAS对象存储接口已与业务代码分离。
## 2. 服务器要求
- Linux 服务器一台,建议至少 4 核、8 GB 内存、100 GB 数据盘;
- Docker Engine 24+
- Docker Compose v2
- 可解析到服务器或负载均衡的公网域名;
- HTTPS 证书由公司网关、负载均衡或 Nginx 统一终止;
- 服务器可以访问飞书 OpenAPI 和正式数据采集 MCP。
服务器只需对公网开放 80/443。MySQL 不映射公网端口。
## 3. 准备代码与环境变量
```bash
git clone ssh://git@gta.gbotai.cn:42001/wufengping/koc-loop.git
cd koc-loop
git checkout main
cp .env.self-hosted.example .env.self-hosted
```
编辑 `.env.self-hosted`。必须替换所有 `replace-with-*` 占位值:
| 变量 | 用途 |
| --- | --- |
| `MYSQL_ROOT_PASSWORD` | MySQL root 密码,仅数据库容器使用 |
| `MYSQL_PASSWORD` | KOC 服务的数据库密码 |
| `APP_ORIGIN` | 后台公网地址,如 `https://koc.example.com` |
| `KOC_PORTAL_URL` | 领取页完整地址,如 `https://koc.example.com/koc/` |
| `SUPER_ADMIN_USERNAME` | 首次启动创建唯一超级管理员 |
| `SUPER_ADMIN_PASSWORD` | 首次启动的超级管理员初始密码,至少 8 位 |
| `ADMIN_INTERNAL_TOKEN` | 内部管理调用密钥 |
| `KOC_MCP_API_KEY` | Agent 调用 KOC LOOP MCP 的独立 Bearer 密钥 |
| `FEISHU_APP_ID` / `FEISHU_APP_SECRET` | 读取飞书内容表和配图 |
| `AI_TOOL_CENTER_MCP_URL` / `AI_TOOL_CENTER_MCP_KEY` | 小红书公开数据采集服务 |
| `ENABLE_SCHEDULER` | 是否启用每天 09:00 自动采集,生产保持 `true` |
密钥必须由密码管理器生成,禁止提交到 Git、聊天、部署日志或 URL。三个业务密钥 `ADMIN_INTERNAL_TOKEN``KOC_MCP_API_KEY``AI_TOOL_CENTER_MCP_KEY` 不得复用。
`APP_ORIGIN` 必须填写用户实际访问的 HTTPS 公网地址,不能填写 `localhost``app:3000` 或其他容器内部地址。Excel 中的视频下载链接会优先使用这个地址;前置网关还必须把原始 `Host``X-Forwarded-Host``X-Forwarded-Proto` 传给仓库内的 Nginx。
## 4. 首次启动
```bash
docker compose --env-file .env.self-hosted \
-f docker-compose.self-hosted.yml up -d --build
```
应用容器启动时会先执行同一套数据库迁移脚本,成功后才启动 KOC 服务。查看状态:
```bash
docker compose --env-file .env.self-hosted \
-f docker-compose.self-hosted.yml ps
curl -fsS http://127.0.0.1:${HTTP_PORT:-80}/api/health
```
健康检查应返回 `status: ok``database: true`
首次打开后台登录页时,系统会根据环境变量创建唯一超级管理员。创建成功后,可从运行环境移除 `SUPER_ADMIN_PASSWORD` 的明文值并重启应用;后续账号与密码统一在“用户管理”中维护。
## 5. Nginx 与 HTTPS
仓库内 `deploy/nginx/koc-loop.conf` 默认监听容器 80 端口,适合前置公司网关或负载均衡终止 HTTPS。
如果证书直接挂在本机 Nginx
1. 把证书和私钥以只读卷挂载进 `nginx` 容器;
2. 增加 443 `listen ... ssl` 配置;
3. 80 端口只做 301 跳转;
4. 保留 `/koc/` 静态规则、`/api/mcp` 长连接规则和 `/` 反向代理规则。
MCP 路由已经关闭代理缓冲并将超时时间延长到 1 小时,避免 Agent 的长调用被 Nginx 提前截断。仓库内 Nginx 同时将上传限制设为 85 MB用于接收最多 80 MB 的批量回填 Excel公司网关或负载均衡的请求体限制也必须不低于 85 MB。
视频下载接口必须经过 `/api/partner-image` 反向代理,正常响应应包含 `Content-Type: video/mp4` 和带 `.mp4` 文件名的 `Content-Disposition: attachment`。不要在网关层改写该响应类型或移除附件响应头。
## 6. 迁移原 Sites 数据
迁移分为数据库与图片两部分。先在原生产站点保持只读窗口,完成导出后再切换域名,避免新旧系统同时写入。
### 6.1 D1 数据导入 MySQL
把 D1 各表导出成一个 JSON 文件,结构如下:
```json
{
"tables": {
"partners": [{ "id": "...", "name": "..." }],
"tasks": [{ "id": "...", "name": "..." }],
"contents": [],
"accounts": [],
"claims": [],
"delegation_bundles": [],
"distributions": [],
"collection_runs": [],
"users": [],
"auth_sessions": [],
"mcp_export_tokens": []
}
}
```
先确保 MySQL 迁移已完成,再在应用环境中运行:
```bash
npm run db:import-json -- /backup/koc-d1-export.json
```
导入脚本按业务依赖顺序写入,并使用主键/唯一键安全更新已有记录。正式迁移前先在测试库演练并核对任务数、笔记数、领取数、发布数和账号数。
### 6.2 R2 媒体文件导入
把 R2 按原对象 key 导出到一个目录,目录层级必须保留,例如:
```text
content-assets/...
content-videos/...
publish-evidence/...
creator-center/...
```
运行:
```bash
UPLOAD_DIR=/data/koc/uploads \
npm run storage:import -- /backup/koc-r2-export
```
脚本会复制文件,并为缺少元数据的媒体文件生成 Content-Type 元数据。容器部署时也可以在宿主机临时挂载 `upload_data` 卷后执行。
## 7. 上线验收
必须逐项验证:
1. 超级管理员可以登录,普通用户看不到 KOC 资源模块;
2. 用真实飞书表格创建一个 1 篇测试任务;
3. 返回的领取链接以 `/koc/?task=` 开头;
4. 手机公网打开领取页,能查看正文与配图;
5. 回填短链/长链、上传发布截图、刷新后记录仍在;
6. 上传创作者截图并填写曝光量、阅读量;
7. 后台立即采集一篇笔记成功;
8. 保存次日采集计划,确认数据库产生 `collection_runs`
9. 图文任务导出的 Excel 内能直接看到原图和截图;
10. 视频任务导出的 Excel 不含“图片”列,包含“视频”列,点击链接能下载扩展名为 `.mp4` 且可正常播放的文件;
11. 批量回填 Excel 可以上传,发布链接、笔记截图和单篇笔记数据分析截图均能正确回写;
12. Agent 用 `KOC_MCP_API_KEY` 调用 `/api/mcp` 能发现全部工具;
13. 重启全部容器后数据、图片和视频不丢失。
## 8. 备份与恢复
每天至少备份:
- MySQL`mysqldump --single-transaction`
- `mysql_data` 卷;
- `upload_data` 卷;
- 当前 Git 提交号和脱敏后的环境变量清单。
备份必须复制到另一台机器或对象存储,不能只保存在部署服务器。恢复演练至少每季度一次。
## 9. 升级与回滚
升级前先备份数据库和上传卷:
```bash
git pull
docker compose --env-file .env.self-hosted \
-f docker-compose.self-hosted.yml up -d --build
```
应用容器每次启动都会按文件名顺序执行尚未应用的 `mysql/*.sql`。本次版本包含平台/视频字段、账号性别/简介/标签以及“当前联系人”字段的增量迁移;升级后应检查容器日志确认 `0005``0006``0007` 已执行或已被识别为历史迁移。
数据库迁移只允许向前追加新的 `mysql/*.sql` 文件,禁止修改已经在生产执行过的迁移。应用回滚到旧镜像前,要确认旧代码兼容当前数据库结构;涉及不可逆结构变化时,必须同时准备数据库恢复方案。
## 10. 运维排查
| 现象 | 处理 |
| --- | --- |
| `/api/health` 返回 503 | 查看 MySQL 容器健康状态与应用数据库变量 |
| 登录页可开但登录失败 | 确认迁移完成、超级管理员变量仅用于初始化 |
| 配图或截图 404 | 检查 `upload_data` 卷和 `UPLOAD_DIR=/data/koc/uploads` |
| Excel 视频链接出现 localhost 或无法访问 | 检查 `APP_ORIGIN`、公网域名和网关转发的 Host/Proto 请求头 |
| 视频下载后不是 MP4 或无法播放 | 检查 `/api/partner-image` 是否经过应用代理、文件是否完整,以及网关是否保留 Content-Type/Content-Disposition |
| 批量回填表上传返回 413 | 将公司网关、负载均衡和 Nginx 的请求体限制统一提高到至少 85 MB |
| 09:00 未自动采集 | 检查 `ENABLE_SCHEDULER=true`、服务器日志和采集 MCP 网络 |
| MCP 401 | 检查请求头是否为 `Authorization: Bearer <KOC_MCP_API_KEY>` |
| 飞书读取失败 | 检查应用权限、文档授权和服务器到飞书 OpenAPI 的网络 |
生产日志不得打印数据库密码、飞书 Secret、MCP key 或完整带 key 的采集服务 URL。

View File

@@ -1,4 +1,6 @@
# KOC LOOP 部署指南
# KOC LOOP Sites 旧版部署指南
> 此文档仅适用于历史 `codex/sites-release-controls` 分支。`main` 已切换为 Next.js + MySQL + Nginx 私有化部署,正式部署请使用 [KOC LOOP 私有化部署指南](KOC%20LOOP%20私有化部署指南.md),不要按本文把 `main` 发布到 Sites。
KOC LOOP 由两个独立站点组成:

View File

@@ -3,5 +3,8 @@ import { defineConfig } from "drizzle-kit";
export default defineConfig({
out: "./drizzle",
schema: "./db/schema.ts",
dialect: "sqlite",
dialect: "mysql",
dbCredentials: {
url: process.env.DATABASE_URL ?? "mysql://koc:koc@127.0.0.1:3306/koc_loop",
},
});

5
instrumentation.ts Normal file
View File

@@ -0,0 +1,5 @@
export async function register() {
if (process.env.NEXT_RUNTIME !== "nodejs") return;
const { startScheduler } = await import("./lib/scheduler");
startScheduler();
}

View File

@@ -0,0 +1,341 @@
import { strFromU8, unzipSync, zipSync } from "fflate";
export const PARTNER_BATCH_UPLOAD_MAX_BYTES = 80_000_000;
type WorkbookCell = {
reference: string;
row: number;
column: number;
attributes: string;
body: string;
value: string;
};
type ScreenshotColumns = {
headerRow: number;
columns: Set<number>;
};
function decodeXml(value: string) {
return value
.replace(/<[^>]+>/g, "")
.replace(/&lt;/g, "<")
.replace(/&gt;/g, ">")
.replace(/&quot;/g, '"')
.replace(/&apos;/g, "'")
.replace(/&amp;/g, "&")
.replace(/&#(\d+);/g, (_, code) => String.fromCodePoint(Number(code)))
.replace(/&#x([0-9a-f]+);/gi, (_, code) =>
String.fromCodePoint(Number.parseInt(code, 16)),
);
}
function textNodes(xml: string) {
return [...xml.matchAll(/<t\b[^>]*>([\s\S]*?)<\/t>/g)]
.map((match) => decodeXml(match[1]))
.join("");
}
function columnIndex(reference: string) {
const letters = reference.match(/^[A-Z]+/i)?.[0]?.toUpperCase() ?? "";
let result = 0;
for (const letter of letters) result = result * 26 + letter.charCodeAt(0) - 64;
return Math.max(0, result - 1);
}
function normalizeHeader(value: string) {
return value.replace(/[\s_\-()]/g, "").toLocaleLowerCase("zh-CN");
}
function parseCells(worksheetXml: string, sharedStrings: string[]) {
const cells: WorkbookCell[] = [];
for (const match of worksheetXml.matchAll(/<c\b([^>]*)>([\s\S]*?)<\/c>/g)) {
const attributes = match[1];
const body = match[2];
const reference = attributes.match(/\br="([A-Z]+\d+)"/i)?.[1] ?? "";
if (!reference) continue;
const type = attributes.match(/\bt="([^"]+)"/)?.[1] ?? "";
const rawValue = body.match(/<v>([\s\S]*?)<\/v>/)?.[1] ?? "";
const value =
type === "s"
? sharedStrings[Number(rawValue)] ?? ""
: type === "inlineStr"
? textNodes(body)
: decodeXml(rawValue);
cells.push({
reference,
row: Number(reference.match(/\d+$/)?.[0] ?? 0),
column: columnIndex(reference),
attributes,
body,
value: value.trim(),
});
}
return cells;
}
function findScreenshotColumns(cells: WorkbookCell[]): ScreenshotColumns {
for (let row = 1; row <= 8; row += 1) {
const columns = new Set<number>();
for (const cell of cells) {
if (cell.row !== row) continue;
const header = normalizeHeader(cell.value);
if (
header === normalizeHeader("笔记截图") ||
header === normalizeHeader("发布截图") ||
header === normalizeHeader("数据分析截图") ||
header === normalizeHeader("数据分析截图(单篇笔记数据分析截图)") ||
header === normalizeHeader("创作者中心截图")
) {
columns.add(cell.column);
}
}
if (columns.size >= 2) return { headerRow: row, columns };
}
throw new Error("表格结构不正确,请使用本领取页面导出的批量回填表");
}
function relationshipMap(xml: string) {
const relationships = new Map<string, string>();
for (const match of xml.matchAll(/<Relationship\b([^>]*)\/?\s*>/g)) {
const id = match[1].match(/\bId="([^"]+)"/)?.[1] ?? "";
const target = match[1].match(/\bTarget="([^"]+)"/)?.[1] ?? "";
if (id && target) relationships.set(id, decodeXml(target));
}
return relationships;
}
function normalizeZipPath(value: string) {
const result: string[] = [];
for (const part of value.split("/")) {
if (!part || part === ".") continue;
if (part === "..") result.pop();
else result.push(part);
}
return result.join("/");
}
function resolveZipPath(base: string, target: string) {
const slash = base.lastIndexOf("/");
const directory = slash >= 0 ? base.slice(0, slash + 1) : "";
return normalizeZipPath(`${directory}${target}`);
}
function wpsScreenshotMedia(
entries: Record<string, Uint8Array>,
cells: WorkbookCell[],
screenshotColumns: ScreenshotColumns,
) {
const result = new Set<string>();
let expected = 0;
const screenshotIds = new Set<string>();
for (const cell of cells) {
if (
cell.row <= screenshotColumns.headerRow ||
!screenshotColumns.columns.has(cell.column)
) {
continue;
}
const id = decodeXml(cell.body).match(/DISPIMG\("([^"]+)"/i)?.[1];
if (id) {
expected += 1;
screenshotIds.add(id);
}
}
if (screenshotIds.size === 0) return { result, expected, resolved: 0 };
const cellImagesXml = entries["xl/cellimages.xml"]
? strFromU8(entries["xl/cellimages.xml"])
: "";
const relationships = relationshipMap(
entries["xl/_rels/cellimages.xml.rels"]
? strFromU8(entries["xl/_rels/cellimages.xml.rels"])
: "",
);
let resolved = 0;
for (const match of cellImagesXml.matchAll(
/<(?:etc:)?cellImage\b[^>]*>([\s\S]*?)<\/(?:etc:)?cellImage>/g,
)) {
const id = match[1].match(/<(?:xdr:)?cNvPr\b[^>]*\bname="([^"]+)"/)?.[1];
const relationshipId = match[1].match(
/<(?:a:)?blip\b[^>]*\br:embed="([^"]+)"/,
)?.[1];
if (!id || !relationshipId || !screenshotIds.has(id)) continue;
const target = relationships.get(relationshipId);
if (!target) continue;
result.add(resolveZipPath("xl/cellimages.xml", target));
resolved += 1;
}
return { result, expected, resolved };
}
function drawingScreenshotMedia(
entries: Record<string, Uint8Array>,
screenshotColumns: ScreenshotColumns,
) {
const result = new Set<string>();
let expected = 0;
let resolved = 0;
const worksheetXml = entries["xl/worksheets/sheet1.xml"]
? strFromU8(entries["xl/worksheets/sheet1.xml"])
: "";
const sheetRelationships = relationshipMap(
entries["xl/worksheets/_rels/sheet1.xml.rels"]
? strFromU8(entries["xl/worksheets/_rels/sheet1.xml.rels"])
: "",
);
const drawingId = worksheetXml.match(/<drawing\b[^>]*r:id="([^"]+)"/)?.[1];
const drawingTarget = drawingId ? sheetRelationships.get(drawingId) : undefined;
if (!drawingTarget) return { result, expected, resolved };
const drawingPath = resolveZipPath("xl/worksheets/sheet1.xml", drawingTarget);
const drawingXml = entries[drawingPath] ? strFromU8(entries[drawingPath]) : "";
const relationshipPath = `${drawingPath.slice(0, drawingPath.lastIndexOf("/") + 1)}_rels/${drawingPath.slice(drawingPath.lastIndexOf("/") + 1)}.rels`;
const drawingRelationships = relationshipMap(
entries[relationshipPath] ? strFromU8(entries[relationshipPath]) : "",
);
for (const anchor of drawingXml.matchAll(
/<xdr:(?:oneCellAnchor|twoCellAnchor)\b[^>]*>[\s\S]*?<xdr:from>([\s\S]*?)<\/xdr:from>[\s\S]*?<a:blip\b[^>]*r:embed="([^"]+)"[\s\S]*?<\/xdr:(?:oneCellAnchor|twoCellAnchor)>/g,
)) {
const column = Number(anchor[1].match(/<xdr:col>(\d+)<\/xdr:col>/)?.[1]);
const zeroBasedRow = Number(anchor[1].match(/<xdr:row>(\d+)<\/xdr:row>/)?.[1]);
if (
!Number.isInteger(column) ||
!Number.isInteger(zeroBasedRow) ||
zeroBasedRow + 1 <= screenshotColumns.headerRow ||
!screenshotColumns.columns.has(column)
) {
continue;
}
expected += 1;
const target = drawingRelationships.get(anchor[2]);
if (!target) continue;
result.add(resolveZipPath(drawingPath, target));
resolved += 1;
}
return { result, expected, resolved };
}
function richValueScreenshotMedia(
entries: Record<string, Uint8Array>,
cells: WorkbookCell[],
screenshotColumns: ScreenshotColumns,
) {
const result = new Set<string>();
let expected = 0;
let resolved = 0;
const metadataXml = entries["xl/metadata.xml"]
? strFromU8(entries["xl/metadata.xml"])
: "";
const richValueXml = entries["xl/richData/rdrichvalue.xml"]
? strFromU8(entries["xl/richData/rdrichvalue.xml"])
: "";
const richValueRelXml = entries["xl/richData/richValueRel.xml"]
? strFromU8(entries["xl/richData/richValueRel.xml"])
: "";
const relationships = relationshipMap(
entries["xl/richData/_rels/richValueRel.xml.rels"]
? strFromU8(entries["xl/richData/_rels/richValueRel.xml.rels"])
: "",
);
if (!metadataXml || !richValueXml || !richValueRelXml || !relationships.size) {
return { result, expected, resolved };
}
const valueMetadataXml =
metadataXml.match(/<valueMetadata\b[^>]*>([\s\S]*?)<\/valueMetadata>/)?.[1] ??
"";
const metadataToRichValue = [
...valueMetadataXml.matchAll(
/<bk\b[^>]*>[\s\S]*?<rc\b[^>]*\bv="(\d+)"[^>]*\/>[\s\S]*?<\/bk>/g,
),
].map((match) => Number(match[1]));
const richValueToRelationship = [
...richValueXml.matchAll(/<rv\b[^>]*>([\s\S]*?)<\/rv>/g),
].map((match) => Number(match[1].match(/<v>(\d+)<\/v>/)?.[1] ?? -1));
const relationshipIds = [
...richValueRelXml.matchAll(/<rel\b[^>]*\br:id="([^"]+)"[^>]*\/>/g),
].map((match) => match[1]);
for (const cell of cells) {
if (
cell.row <= screenshotColumns.headerRow ||
!screenshotColumns.columns.has(cell.column)
) {
continue;
}
const metadataIndex = Number(cell.attributes.match(/\bvm="(\d+)"/)?.[1] ?? 0);
if (!metadataIndex) continue;
expected += 1;
const richValueIndex = metadataToRichValue[metadataIndex - 1];
const relationshipIndex = richValueToRelationship[richValueIndex];
const relationshipId = relationshipIds[relationshipIndex];
const target = relationships.get(relationshipId);
if (!target) continue;
result.add(resolveZipPath("xl/richData/richValueRel.xml", target));
resolved += 1;
}
return { result, expected, resolved };
}
export type CompactedPartnerBatchWorkbook = {
bytes: Uint8Array;
removedMediaCount: number;
preservedScreenshotCount: number;
};
/**
* Oversized exports are usually caused by full-resolution source images. The
* upload only needs the two screenshot columns, so retain those image entries
* and omit source media from the temporary upload copy.
*/
export function compactPartnerBatchWorkbookForUpload(
input: Uint8Array,
): CompactedPartnerBatchWorkbook {
const isMediaFile = (name: string) =>
name.startsWith("xl/media/") && !name.endsWith("/");
const structure = unzipSync(input, {
filter: (file) => !isMediaFile(file.name),
});
const worksheetXml = structure["xl/worksheets/sheet1.xml"]
? strFromU8(structure["xl/worksheets/sheet1.xml"])
: "";
if (!worksheetXml) throw new Error("Excel 中没有可读取的批量回填工作表");
const sharedXml = structure["xl/sharedStrings.xml"]
? strFromU8(structure["xl/sharedStrings.xml"])
: "";
const sharedStrings = [
...sharedXml.matchAll(/<si\b[^>]*>([\s\S]*?)<\/si>/g),
].map((match) => textNodes(match[1]));
const cells = parseCells(worksheetXml, sharedStrings);
const screenshotColumns = findScreenshotColumns(cells);
const formats = [
wpsScreenshotMedia(structure, cells, screenshotColumns),
drawingScreenshotMedia(structure, screenshotColumns),
richValueScreenshotMedia(structure, cells, screenshotColumns),
];
const screenshotMedia = new Set<string>();
let expectedScreenshotCount = 0;
let resolvedScreenshotCount = 0;
for (const format of formats) {
expectedScreenshotCount += format.expected;
resolvedScreenshotCount += format.resolved;
for (const name of format.result) screenshotMedia.add(name);
}
if (resolvedScreenshotCount < expectedScreenshotCount) {
throw new Error("表格中的回填截图无法完整识别,请重新导出最新版回填表");
}
let mediaCount = 0;
const entries = unzipSync(input, {
filter: (file) => {
if (!isMediaFile(file.name)) return true;
mediaCount += 1;
return screenshotMedia.has(file.name);
},
});
const bytes = zipSync(entries, { level: 6 });
return {
bytes,
removedMediaCount: Math.max(0, mediaCount - screenshotMedia.size),
preservedScreenshotCount: screenshotMedia.size,
};
}

View File

@@ -1,6 +1,7 @@
@import "tailwindcss";
:root {
--font-geist-sans: -apple-system, BlinkMacSystemFont, "Segoe UI", "PingFang SC", "Microsoft YaHei", sans-serif;
--ink: #15221f;
--ink-soft: #354640;
--green: #1e8d68;
@@ -47,6 +48,104 @@ button:disabled {
opacity: 0.5;
}
.platform-badge {
display: inline-flex !important;
width: auto !important;
height: 24px;
align-items: center;
flex: none;
gap: 5px;
margin: 0 !important;
padding: 2px 7px 2px 3px;
border: 1px solid #e1e7e4;
border-radius: 8px;
color: #52615c !important;
background: rgb(255 255 255 / 0.92);
font-size: 9px !important;
font-weight: 720;
line-height: 1 !important;
white-space: nowrap;
}
.platform-badge.compact {
height: 19px;
gap: 4px;
padding: 2px 5px 2px 2px;
border-radius: 6px;
font-size: 8px !important;
}
.platform-logo {
display: grid !important;
width: 18px !important;
height: 18px !important;
place-items: center;
flex: none;
overflow: hidden;
margin: 0 !important;
border-radius: 5px;
line-height: 1 !important;
}
.platform-badge.compact .platform-logo {
width: 14px !important;
height: 14px !important;
border-radius: 4px;
}
.platform-logo.xiaohongshu {
color: white !important;
background: #ff2442;
}
.platform-logo.xiaohongshu b {
color: inherit;
font-size: 5px;
font-weight: 900;
letter-spacing: -0.12em;
transform: translateX(-0.2px);
}
.platform-badge.compact .platform-logo.xiaohongshu b {
font-size: 4px;
}
.platform-logo.douyin {
background: #080b12;
}
.platform-logo.douyin svg {
width: 16px;
height: 16px;
}
.platform-badge.compact .platform-logo.douyin svg {
width: 13px;
height: 13px;
}
.platform-logo.douyin .douyin-cyan { fill: #25f4ee; transform: translate(-0.7px, 0.7px); }
.platform-logo.douyin .douyin-red { fill: #fe2c55; transform: translate(0.7px, -0.3px); }
.platform-logo.douyin .douyin-white { fill: #fff; }
.platform-meta-line,
.hero-platform-line {
display: inline-flex !important;
min-width: 0;
align-items: center;
flex-wrap: wrap;
gap: 6px;
}
.platform-meta-line > span,
.hero-platform-line > * {
margin: 0 !important;
}
.hero-platform-line {
margin-bottom: 10px;
}
.portal-shell {
width: min(100%, 1120px);
min-height: 100vh;
@@ -173,6 +272,140 @@ button:disabled {
box-shadow: 0 1px 2px rgb(15 31 26 / 0.03);
}
.screenshot-task-detail .document-label:first-of-type {
margin-top: 26px;
}
.screenshot-task-detail .note-title-row h1 {
color: var(--green-deep);
}
.task-result-picker {
min-height: 160px;
}
.task-result-label-row {
display: flex;
align-items: center;
justify-content: space-between;
gap: 12px;
}
.task-result-label-row small {
color: #7f8b87;
font-size: 8px;
font-weight: 500;
}
.task-result-gallery {
display: grid;
grid-template-columns: repeat(3, minmax(0, 1fr));
gap: 9px;
margin-top: 8px;
}
.task-result-thumb,
.task-result-add {
position: relative;
min-height: 112px;
overflow: hidden;
border: 1px solid #d8e2dd;
border-radius: 11px;
background: #f6f9f7;
}
.task-result-thumb {
display: block;
width: 100%;
padding: 0;
border: 1px solid #d8e2dd;
color: inherit;
cursor: zoom-in;
}
.task-result-thumb img {
width: 100%;
height: 112px;
display: block;
object-fit: cover;
}
.task-result-thumb > span {
position: absolute;
right: 6px;
bottom: 6px;
padding: 4px 6px;
border-radius: 7px;
color: #267257;
background: rgb(238 249 244 / 0.94);
font-size: 7px;
font-weight: 700;
}
.task-result-thumb.pending-upload > span {
color: #8a643f;
background: rgb(255 245 233 / 0.94);
}
.task-result-thumb button {
position: absolute;
top: 6px;
right: 6px;
display: grid;
width: 24px;
height: 24px;
place-items: center;
padding: 0;
border: 0;
border-radius: 50%;
color: white;
background: rgb(17 48 39 / 0.78);
font-size: 15px;
cursor: pointer;
}
.task-result-add {
display: flex;
align-items: center;
justify-content: center;
flex-direction: column;
border-style: dashed;
cursor: pointer;
}
.task-result-add input {
position: absolute;
inset: 0;
width: 100%;
height: 100%;
opacity: 0;
cursor: pointer;
}
.task-result-add b {
display: grid;
width: 30px;
height: 30px;
place-items: center;
border-radius: 50%;
color: var(--green);
background: var(--green-soft);
font-size: 16px;
}
.task-result-add strong {
margin-top: 8px;
font-size: 8px;
}
.task-result-hint {
display: block;
margin-top: 8px;
color: #8c9793;
font-size: 7px;
line-height: 1.55;
}
.claim-card {
min-height: 430px;
padding: 28px;
@@ -544,6 +777,15 @@ footer {
cursor: not-allowed;
}
.batch-workbook-input {
position: absolute;
width: 1px;
height: 1px;
overflow: hidden;
opacity: 0;
pointer-events: none;
}
.share-composer {
display: grid;
grid-template-columns: minmax(0, 1fr) minmax(190px, 0.45fr) auto;
@@ -708,6 +950,34 @@ footer {
text-align: center;
}
.note-thumb.platform-video-thumb {
display: grid;
place-items: center;
}
.platform-badge.logo-only {
width: 34px !important;
height: 34px;
padding: 0;
border: 0;
background: transparent;
}
.platform-badge.logo-only .platform-logo {
width: 34px !important;
height: 34px !important;
border-radius: 9px;
}
.platform-badge.logo-only .platform-logo.douyin svg {
width: 29px;
height: 29px;
}
.platform-badge.logo-only .platform-logo.xiaohongshu b {
font-size: 8px;
}
.note-index {
display: grid;
width: 34px;
@@ -915,6 +1185,11 @@ footer {
padding: 34px 38px;
}
.mobile-note-summary,
.mobile-note-collapse-trigger {
display: none;
}
.note-document-meta {
display: flex;
gap: 8px;
@@ -995,12 +1270,32 @@ footer {
white-space: pre-wrap;
}
.note-images {
.note-images,
.note-videos {
margin-top: 30px;
padding-top: 24px;
border-top: 1px solid #edf0ee;
}
.note-video-grid {
display: grid;
gap: 14px;
}
.note-video-card {
overflow: hidden;
border: 1px solid #e4e9e6;
border-radius: 12px;
background: #102a22;
}
.note-video-card video {
display: block;
width: 100%;
max-height: 680px;
background: #0b1f19;
}
.note-images-heading {
display: flex;
align-items: flex-end;
@@ -1094,7 +1389,8 @@ footer {
font-weight: 650;
}
.note-image-actions button {
.note-image-actions button,
.note-image-actions a {
height: 28px;
padding: 0 10px;
border: 1px solid #cfe1d9;
@@ -1103,9 +1399,12 @@ footer {
background: #f2f8f5;
font-size: 8px;
font-weight: 680;
line-height: 26px;
text-decoration: none;
}
.note-image-actions button:hover {
.note-image-actions button:hover,
.note-image-actions a:hover {
border-color: #9fc9b8;
background: #eaf5f0;
}
@@ -1171,6 +1470,19 @@ footer {
margin-top: 18px;
}
.evidence-field {
display: block;
margin-top: 18px;
}
.evidence-field > span {
display: block;
margin-bottom: 7px;
color: #52605b;
font-size: 9px;
font-weight: 700;
}
.field-hint {
color: #98a29e;
font-size: 8px;
@@ -1286,6 +1598,142 @@ footer {
min-height: 96px;
}
.screenshot-picker.preview-mode {
align-items: stretch;
justify-content: flex-start;
padding: 8px;
background: #f7faf8;
}
.evidence-empty-state {
display: flex;
min-height: 82px;
align-items: center;
justify-content: center;
flex-direction: column;
}
.evidence-preview-button {
position: relative;
width: 100%;
padding: 0;
overflow: hidden;
border: 0;
border-radius: 9px;
background: #eaf1ed;
cursor: zoom-in;
}
.evidence-preview-button img {
width: 100%;
max-height: 260px;
display: block;
object-fit: contain;
}
.evidence-preview-button.compact-preview img {
max-height: 210px;
}
.evidence-preview-button > span {
position: absolute;
right: 8px;
bottom: 8px;
padding: 5px 8px;
border-radius: 8px;
color: white;
background: rgb(17 48 39 / 0.78);
font-size: 7px;
font-weight: 700;
}
.evidence-upload-action {
position: relative;
width: 100%;
height: 34px;
display: grid;
margin-top: 8px !important;
place-items: center;
overflow: hidden;
border: 1px solid #c5dcd2;
border-radius: 8px;
color: var(--green-deep);
background: #eef7f3;
font-size: 8px;
font-weight: 700;
cursor: pointer;
}
.evidence-upload-action input {
position: absolute;
inset: 0;
width: 100%;
height: 100%;
opacity: 0;
cursor: pointer;
}
.image-lightbox {
position: fixed;
z-index: 1000;
inset: 0;
display: grid;
place-items: center;
padding: 18px;
background: rgb(9 25 20 / 0.78);
}
.image-lightbox-card {
width: min(920px, 100%);
max-height: calc(100vh - 36px);
display: flex;
overflow: hidden;
border: 1px solid rgb(255 255 255 / 0.2);
border-radius: 16px;
background: #f7faf8;
box-shadow: 0 24px 80px rgb(0 0 0 / 0.32);
flex-direction: column;
}
.image-lightbox-heading {
display: flex;
min-height: 48px;
align-items: center;
justify-content: space-between;
gap: 16px;
padding: 10px 14px;
border-bottom: 1px solid #dce5e0;
}
.image-lightbox-heading strong {
overflow: hidden;
color: var(--green-deep);
font-size: 10px;
text-overflow: ellipsis;
white-space: nowrap;
}
.image-lightbox-heading button {
flex: 0 0 auto;
height: 30px;
padding: 0 12px;
border: 1px solid #c7d8d0;
border-radius: 8px;
color: var(--green-deep);
background: white;
font-size: 8px;
font-weight: 700;
}
.image-lightbox-card > img {
width: 100%;
min-height: 0;
max-height: calc(100vh - 102px);
display: block;
object-fit: contain;
background: #14231e;
}
.creator-metric-grid {
display: grid;
grid-template-columns: repeat(2, minmax(0, 1fr));
@@ -1318,6 +1766,10 @@ footer {
}
@media (width <= 520px) {
.task-result-gallery {
grid-template-columns: repeat(2, minmax(0, 1fr));
}
.creator-metric-grid {
grid-template-columns: 1fr;
gap: 0;
@@ -1340,6 +1792,11 @@ footer {
font-weight: 620;
}
.toast.success {
color: var(--green-deep);
font-weight: 720;
}
.loading-shell {
display: flex;
align-items: center;
@@ -1489,9 +1946,14 @@ footer {
.section-actions {
width: 100%;
flex-wrap: wrap;
justify-content: space-between;
}
.section-actions > span {
margin-right: auto;
}
.share-composer {
grid-template-columns: 1fr;
}
@@ -1522,6 +1984,64 @@ footer {
border-radius: 16px;
}
.note-document.mobile-collapsed {
padding: 16px;
}
.note-document.mobile-collapsed .note-document-content {
display: none;
}
.note-document.mobile-collapsed .mobile-note-summary {
display: flex;
align-items: center;
justify-content: space-between;
gap: 14px;
}
.mobile-note-summary > div {
display: grid;
min-width: 0;
gap: 4px;
}
.mobile-note-summary span {
color: var(--green);
font-size: 9px;
font-weight: 750;
}
.mobile-note-summary strong {
overflow: hidden;
color: var(--ink);
font-size: 13px;
text-overflow: ellipsis;
white-space: nowrap;
}
.mobile-note-summary small {
color: #899590;
font-size: 8px;
}
.mobile-note-summary button,
.mobile-note-collapse-trigger {
flex: 0 0 auto;
min-height: 34px;
padding: 0 12px;
border: 1px solid #cfe1d9;
border-radius: 9px;
color: var(--green-deep);
background: #f2f8f5;
font-size: 9px;
font-weight: 700;
}
.mobile-note-collapse-trigger {
display: block;
margin: 14px 0 0 auto;
}
.note-document h1 {
font-size: 28px;
}

View File

@@ -1,66 +1,23 @@
import type { Metadata } from "next";
import { Geist, Geist_Mono } from "next/font/google";
import { headers } from "next/headers";
import "./globals.css";
const geistSans = Geist({
variable: "--font-geist-sans",
subsets: ["latin"],
});
const description =
"领取KOC内容任务逐篇查看笔记详情并一一回填发布账号、链接与截图。";
const geistMono = Geist_Mono({
variable: "--font-geist-mono",
subsets: ["latin"],
});
export async function generateMetadata(): Promise<Metadata> {
const requestHeaders = await headers();
const host = requestHeaders.get("x-forwarded-host") || requestHeaders.get("host");
const protocol =
requestHeaders.get("x-forwarded-proto") ||
(host?.startsWith("localhost") ? "http" : "https");
const metadataBase = host
? new URL(`${protocol}://${host}`)
: new URL("https://koc-task.example.com");
const description =
"领取KOC内容任务逐篇查看笔记详情并一一回填发布账号、链接与截图。";
return {
metadataBase,
export const metadata: Metadata = {
metadataBase: new URL("https://koc-loop.example.com"),
title: "KOC LOOP外部任务领取",
description,
robots: { index: false, follow: false, noarchive: true, nosnippet: true },
referrer: "no-referrer",
icons: { icon: "/koc/favicon.svg", shortcut: "/koc/favicon.svg" },
openGraph: {
title: "KOC LOOP外部任务领取",
description,
robots: {
index: false,
follow: false,
noarchive: true,
nosnippet: true,
},
referrer: "no-referrer",
icons: {
icon: "/favicon.svg",
shortcut: "/favicon.svg",
},
openGraph: {
title: "KOC LOOP外部任务领取",
description,
type: "website",
images: [
{
url: new URL("/og.png", metadataBase).toString(),
width: 1200,
height: 630,
alt: "KOC LOOP 外部任务领取与逐篇回填",
},
],
},
twitter: {
card: "summary_large_image",
title: "KOC LOOP外部任务领取",
description,
images: [new URL("/og.png", metadataBase).toString()],
},
};
}
type: "website",
images: [{ url: "/koc/og.png", width: 1200, height: 630 }],
},
};
export default function RootLayout({
children,
@@ -69,9 +26,7 @@ export default function RootLayout({
}>) {
return (
<html lang="zh-CN">
<body className={`${geistSans.variable} ${geistMono.variable}`}>
{children}
</body>
<body>{children}</body>
</html>
);
}

File diff suppressed because it is too large Load Diff

View File

@@ -1,45 +0,0 @@
import { access, cp, mkdir, rm } from "node:fs/promises";
import { resolve } from "node:path";
import type { Plugin } from "vite";
async function exists(path: string): Promise<boolean> {
try {
await access(path);
return true;
} catch (error) {
if ((error as NodeJS.ErrnoException).code === "ENOENT") {
return false;
}
throw error;
}
}
// Packages Sites metadata and migrations after Vite finishes compiling.
export function sites(): Plugin {
let root = process.cwd();
return {
name: "sites",
apply: "build",
configResolved(config) {
root = config.root;
},
async closeBundle() {
const outputDirectory = resolve(root, "dist", ".openai");
const hostingConfig = resolve(root, ".openai", "hosting.json");
const drizzleSource = resolve(root, "drizzle");
await rm(outputDirectory, { recursive: true, force: true });
await mkdir(outputDirectory, { recursive: true });
if (await exists(hostingConfig)) {
await cp(hostingConfig, resolve(outputDirectory, "hosting.json"));
}
if (await exists(drizzleSource)) {
await cp(drizzleSource, resolve(outputDirectory, "drizzle"), {
recursive: true,
});
}
},
};
}

View File

@@ -1,13 +0,0 @@
import { env } from "cloudflare:workers";
import { drizzle } from "drizzle-orm/d1";
import * as schema from "./schema";
export function getDb() {
if (!env.DB) {
throw new Error(
"Cloudflare D1 binding `DB` is unavailable. Set the `d1` field in .openai/hosting.json to `DB` or let your control plane inject the real binding values before using the database."
);
}
return drizzle(env.DB, { schema });
}

View File

@@ -1,4 +0,0 @@
// Intentionally empty by default.
// Add Drizzle tables here when the site actually needs a database.
// See examples/d1/db/schema.ts for an opt-in example.
export {};

View File

@@ -1,7 +0,0 @@
import { defineConfig } from "drizzle-kit";
export default defineConfig({
out: "./drizzle",
schema: "./db/schema.ts",
dialect: "sqlite",
});

View File

@@ -1,5 +0,0 @@
{
"version": "7",
"dialect": "sqlite",
"entries": []
}

View File

@@ -1,7 +1,13 @@
import type { NextConfig } from "next";
import path from "node:path";
const nextConfig: NextConfig = {
/* config options here */
output: "export",
basePath: "/koc",
assetPrefix: "/koc",
trailingSlash: true,
images: { unoptimized: true },
turbopack: { root: path.resolve(process.cwd()) },
};
export default nextConfig;

File diff suppressed because it is too large Load Diff

View File

@@ -6,37 +6,27 @@
"node": ">=22.13.0"
},
"scripts": {
"dev": "WRANGLER_LOG_PATH=.wrangler/wrangler.log vinext dev",
"build": "WRANGLER_LOG_PATH=.wrangler/wrangler.log vinext build",
"start": "WRANGLER_LOG_PATH=.wrangler/wrangler.log vinext start",
"dev": "next dev -p 3001",
"build": "next build",
"start": "npx serve out -l 3001",
"test": "npm run build && node --test tests/rendered-html.test.mjs",
"lint": "eslint . --ignore-pattern dist --ignore-pattern .next",
"db:generate": "drizzle-kit generate"
"lint": "eslint . --ignore-pattern dist --ignore-pattern .next --ignore-pattern out"
},
"dependencies": {
"drizzle-orm": "0.45.2",
"fflate": "0.7.4",
"next": "16.2.6",
"next": "^16.3.0",
"react": "19.2.6",
"react-dom": "19.2.6"
},
"devDependencies": {
"@cloudflare/vite-plugin": "1.37.1",
"@tailwindcss/postcss": "4.2.1",
"@types/node": "22.19.19",
"@types/react": "19.2.14",
"@types/react-dom": "19.2.3",
"@vitejs/plugin-react": "6.0.2",
"@vitejs/plugin-rsc": "0.5.26",
"drizzle-kit": "0.31.10",
"eslint": "9.39.4",
"eslint-config-next": "16.2.6",
"react-server-dom-webpack": "19.2.6",
"tailwindcss": "4.2.1",
"typescript": "5.9.3",
"vinext": "0.0.50",
"vite": "8.0.13",
"wrangler": "4.92.0"
"typescript": "5.9.3"
},
"type": "module"
}

View File

@@ -14,16 +14,17 @@ test("builds the branded external task shell", async () => {
assert.match(layout, /KOC LOOP外部任务领取/);
assert.match(layout, /og\.png/);
assert.match(page, /正在打开任务/);
await access(new URL("../dist/server/index.js", import.meta.url));
await access(new URL("../out/index.html", import.meta.url));
});
test("keeps claiming minimal and backfill one-to-one", async () => {
const [page, layout, packageJson, hosting] =
const [page, layout, packageJson, nextConfig, styles] =
await Promise.all([
readFile(new URL("../app/page.tsx", import.meta.url), "utf8"),
readFile(new URL("../app/layout.tsx", import.meta.url), "utf8"),
readFile(new URL("../package.json", import.meta.url), "utf8"),
readFile(new URL("../.openai/hosting.json", import.meta.url), "utf8"),
readFile(new URL("../next.config.ts", import.meta.url), "utf8"),
readFile(new URL("../app/globals.css", import.meta.url), "utf8"),
]);
assert.match(page, /微信号\s*\/\s*手机号/);
@@ -36,14 +37,31 @@ test("keeps claiming minimal and backfill one-to-one", async () => {
assert.match(page, /发布链接/);
assert.match(page, /识别发布账号/);
assert.match(page, /inputMode="url"/);
assert.match(page, /长链、短链或整段分享文案/);
assert.match(page, /作品链接或整段分享文案/);
assert.doesNotMatch(page, /type="url"/);
assert.match(page, /发布截图/);
assert.match(page, /发布配图/);
assert.match(page, /复制标题/);
assert.match(page, /复制文案/);
assert.match(page, /下载原图/);
assert.match(page, /下载视频/);
assert.match(page, /download\s*=\s*false/);
assert.match(page, /params\.set\("download", "1"\)/);
assert.match(page, /视频-\$\{index \+ 1\}\.mp4/);
assert.match(page, /function PlatformBadge/);
assert.match(page, /platform-logo/);
assert.match(page, /logoOnly/);
assert.match(page, /platform-video-thumb/);
assert.match(styles, /\.platform-logo\.xiaohongshu/);
assert.match(styles, /\.platform-logo\.douyin/);
assert.match(styles, /\.platform-badge\.logo-only/);
assert.match(page, /批量保存图片/);
assert.match(page, /导出Excel/);
assert.match(page, /上传回填表/);
assert.match(page, /\/api\/partner-batch-workbook/);
assert.match(page, /compactPartnerBatchWorkbookForUpload/);
assert.match(page, /Content-Type/);
assert.match(page, /response\.status === 413/);
assert.match(page, /navigator\.share/);
assert.match(page, /zipSync/);
assert.match(page, /navigator\.clipboard\.writeText/);
@@ -52,10 +70,20 @@ test("keeps claiming minimal and backfill one-to-one", async () => {
assert.match(page, /找回领取记录/);
assert.match(page, /action:\s*"recover"/);
assert.match(page, /同一任务多次领取会分批展示/);
assert.match(page, /这篇笔记已回填,不会与其他笔记错配/);
assert.doesNotMatch(page, /批量回填/);
assert.match(page, /这篇笔记的发布记录回填成功啦~/);
assert.match(page, /toastClassName\(toast\)/);
assert.match(styles, /\.toast\.success/);
assert.match(page, /笔记内容已收起/);
assert.match(page, /展开笔记内容/);
assert.match(page, /收起笔记内容/);
assert.match(page, /isFirstBackfill/);
assert.match(page, /scrollIntoView/);
assert.match(page, /matchMedia\("\(max-width: 620px\)"\)/);
assert.match(styles, /\.note-document\.mobile-collapsed/);
assert.match(page, /批量回填/);
assert.doesNotMatch(page, /复制标题和正文/);
assert.match(page, /PRODUCTION_ADMIN_ORIGIN/);
assert.match(page, /CONFIGURED_ADMIN_ORIGIN/);
assert.match(page, /window\.location\.origin/);
assert.match(page, /\/api\/partner-upload/);
assert.match(page, /"X-KOC-Distribution":\s*selectedItem\.id/);
assert.match(page, /"X-KOC-Upload-Kind":\s*kind/);
@@ -65,22 +93,53 @@ test("keeps claiming minimal and backfill one-to-one", async () => {
assert.match(page, /creatorExposure/);
assert.match(page, /creatorViews/);
assert.match(page, /截图仅用于运营核对不再自动OCR/);
assert.match(page, /evidenceImageUrl/);
assert.match(page, /params\.set\("v", evidenceKey\)/);
assert.match(page, /evidenceImageUrl\(selected, "publish"\)/);
assert.match(page, /evidenceImageUrl\(selected, "creator"\)/);
assert.match(page, /ImageLightbox/);
assert.match(page, /点击查看大图/);
assert.match(page, /publishScreenshotPreview/);
assert.match(page, /creatorScreenshotPreview/);
assert.match(page, /曝光量/);
assert.match(page, /阅读量/);
assert.match(page, /placeholder="填写截图中的曝光量"\s*\/>/);
assert.match(page, /placeholder="填写截图中的阅读量"\s*\/>/);
assert.doesNotMatch(page, /recognizeCreatorMetrics/);
assert.match(page, /note-index \$\{item\.publish_url \? "done" : ""\}/);
assert.match(page, /note-index \$\{\(isScreenshotTask \? item\.result_submitted_at : item\.publish_url\) \? "done" : ""\}/);
assert.match(packageJson, /"fflate":\s*"0\.7\.4"/);
assert.doesNotMatch(packageJson, /tesseract\.js/);
assert.match(layout, /逐篇查看笔记详情并一一回填/);
assert.doesNotMatch(packageJson, /react-loading-skeleton/);
const hostingConfig = JSON.parse(hosting);
assert.equal(hostingConfig.d1, null);
assert.equal(hostingConfig.r2, null);
assert.match(nextConfig, /output: "export"/);
assert.match(nextConfig, /basePath: "\/koc"/);
await access(new URL("../public/og.png", import.meta.url));
await access(new URL("../public/favicon.svg", import.meta.url));
});
test("renders screenshot-only tasks with anonymous delegation and multi-image uploads", async () => {
const [page, styles] = await Promise.all([
readFile(new URL("../app/page.tsx", import.meta.url), "utf8"),
readFile(new URL("../app/globals.css", import.meta.url), "utf8"),
]);
assert.match(page, /screenshot_collect/);
assert.match(page, /小红书搜索关键词/);
assert.match(page, /上传小红书 AI 总结截图/);
assert.match(page, /submit_screenshot_result/);
assert.match(page, /"task-result"/);
assert.match(page, /不需要填写发布链接或其他数据/);
assert.match(page, /multiple/);
assert.match(page, /taskResultScreenshots/);
assert.match(page, /task-result-thumb/);
assert.match(page, /最多9张/);
assert.doesNotMatch(page, /target="_blank"[\s\S]{0,200}task-result-thumb uploaded/);
assert.match(styles, /\.task-result-gallery/);
assert.match(styles, /\.image-lightbox/);
assert.match(styles, /\.evidence-preview-button/);
});
test("shows D1 timestamps in Beijing time", () => {
const stored = "2026-07-29 05:36:00";
assert.equal(parseStoredDate(stored).toISOString(), "2026-07-29T05:36:00.000Z");
@@ -100,6 +159,9 @@ test("creates anonymous delegation bundles and reuses one-to-one backfill", asyn
assert.match(page, /action:\s*"revoke_delegation"/);
assert.match(page, /合作社转派 · 无需登录/);
assert.match(page, /"X-KOC-Delegation"/);
assert.match(page, /const url = new URL\(window\.location\.href\)/);
assert.match(page, /url\.search = ""/);
assert.doesNotMatch(page, /const url = new URL\(window\.location\.origin\)/);
assert.match(page, /url\.searchParams\.set\("share", shareToken\)/);
assert.match(page, /请保存当前分享链接/);
assert.doesNotMatch(page, /底层KOC手机号|底层KOC企微|底层KOC微信/);

View File

@@ -30,5 +30,5 @@
".next/dev/types/**/*.ts",
"**/*.mts"
],
"exclude": ["node_modules"]
"exclude": ["node_modules", "build", "worker", "examples"]
}

View File

@@ -1,59 +0,0 @@
import vinext from "vinext";
import { defineConfig } from "vite";
import hostingConfig from "./.openai/hosting.json";
import { sites } from "./build/sites-vite-plugin";
const SITE_CREATOR_PLACEHOLDER_DATABASE_ID =
"00000000-0000-4000-8000-000000000000";
const { d1, r2 } = hostingConfig;
// macOS Seatbelt blocks FSEvents, so Codex previews need polling for HMR.
const isCodexSeatbeltSandbox = process.env.CODEX_SANDBOX === "seatbelt";
const localBindingConfig = {
main: "./worker/index.ts",
compatibility_flags: ["nodejs_compat"],
d1_databases: d1
? [
{
binding: d1,
database_name: "site-creator-d1",
database_id: SITE_CREATOR_PLACEHOLDER_DATABASE_ID,
},
]
: [],
r2_buckets: r2
? [
{
binding: r2,
bucket_name: "site-creator-r2",
},
]
: [],
};
export default defineConfig(async () => {
// Keep Wrangler and Miniflare state project-local. These are non-secret tool
// settings; application environment belongs in ignored `.env*` files.
process.env.WRANGLER_WRITE_LOGS ??= "false";
process.env.WRANGLER_LOG_PATH ??= ".wrangler/logs";
process.env.MINIFLARE_REGISTRY_PATH ??= ".wrangler/registry";
// Wrangler snapshots its log path while the Cloudflare plugin is imported.
const { cloudflare } = await import("@cloudflare/vite-plugin");
return {
server: isCodexSeatbeltSandbox
? { watch: { useFsEvents: false, usePolling: true } }
: undefined,
plugins: [
vinext(),
sites(),
cloudflare({
viteEnvironment: { name: "rsc", childEnvironments: ["ssr"] },
config: localBindingConfig,
}),
],
};
});

View File

@@ -1,47 +0,0 @@
/** Cloudflare Worker entry point for the vinext-starter template. */
import { handleImageOptimization, DEFAULT_DEVICE_SIZES, DEFAULT_IMAGE_SIZES } from "vinext/server/image-optimization";
import handler from "vinext/server/app-router-entry";
interface Env {
ASSETS: Fetcher;
DB: D1Database;
IMAGES: {
input(stream: ReadableStream): {
transform(options: Record<string, unknown>): {
output(options: { format: string; quality: number }): Promise<{ response(): Response }>;
};
};
};
}
interface ExecutionContext {
waitUntil(promise: Promise<unknown>): void;
passThroughOnException(): void;
}
// Image security config. SVG sources with .svg extension auto-skip the
// optimization endpoint on the client side (served directly, no proxy).
// To route SVGs through the optimizer (with security headers), set
// dangerouslyAllowSVG: true in next.config.js and uncomment below:
// const imageConfig: ImageConfig = { dangerouslyAllowSVG: true };
const worker = {
async fetch(request: Request, env: Env, ctx: ExecutionContext): Promise<Response> {
const url = new URL(request.url);
if (url.pathname === "/_vinext/image") {
const allowedWidths = [...DEFAULT_DEVICE_SIZES, ...DEFAULT_IMAGE_SIZES];
return handleImageOptimization(request, {
fetchAsset: (path) => env.ASSETS.fetch(new Request(new URL(path, request.url))),
transformImage: async (body, { width, format, quality }) => {
const result = await env.IMAGES.input(body).transform(width > 0 ? { width } : {}).output({ format, quality });
return result.response();
},
}, allowedWidths);
}
return handler.fetch(request, env, ctx);
},
};
export default worker;

View File

@@ -1,24 +1,30 @@
import {
resolveXhsPublicAccountDetails,
resolveXhsAccountProfileFromMcp,
resolveXhsProfileDetailsFromMcp,
resolveAccountProfileFromMcp,
resolveProfileDetailsFromMcp,
type CollectionMcpConfig,
} from "./mcp-collection-client";
import { hashText } from "./mvp-db";
import type { DatabaseClient } from "./database";
type DistributionAccountRow = {
id: string;
account_id: string | null;
publish_url: string | null;
platform: string;
claimant_contact: string | null;
};
type BackfillRow = DistributionAccountRow & {
resolved_account_id: string | null;
nickname: string | null;
platform: string | null;
platform_uid: string | null;
public_account_id: string | null;
profile_url: string | null;
followers: number | null;
gender: string | null;
bio: string | null;
tags: string | null;
};
function isVerifiedXhsProfileUrl(value: string | null) {
@@ -36,34 +42,66 @@ function isVerifiedXhsProfileUrl(value: string | null) {
}
}
function isVerifiedDouyinProfileUrl(value: string | null) {
if (!value) return false;
try {
const url = new URL(value);
const secUid = decodeURIComponent(url.pathname.match(/^\/user\/([^/?#]+)/)?.[1] ?? "");
return (
url.protocol === "https:" &&
(url.hostname === "douyin.com" ||
url.hostname.endsWith(".douyin.com")) &&
/^[A-Za-z0-9_-]{20,220}$/.test(secUid) &&
!/^\d+$/.test(secUid)
);
} catch {
return false;
}
}
export async function enrichDistributionAccount(
db: D1Database,
db: DatabaseClient,
distributionId: string,
publishUrl: string,
fallbackNickname: string,
mcpConfig: CollectionMcpConfig,
) {
const profile = await resolveXhsAccountProfileFromMcp(
publishUrl,
fallbackNickname,
mcpConfig,
);
const current = await db
.prepare(
`SELECT id, account_id, publish_url
FROM distributions
WHERE id = ?`,
`SELECT d.id, d.account_id, d.publish_url, t.platform,
cl.claimant_name AS claimant_contact
FROM distributions d
JOIN tasks t ON t.id = d.task_id
LEFT JOIN claims cl ON cl.id = d.claim_id
WHERE d.id = ?`,
)
.bind(distributionId)
.first<DistributionAccountRow>();
if (!current || current.publish_url !== publishUrl) {
return { updated: false, reason: "stale" as const };
}
const platform = current.platform === "抖音" ? "抖音" : "小红书";
const profile = await resolveAccountProfileFromMcp(
publishUrl,
fallbackNickname,
platform,
mcpConfig,
);
const canonicalAccountId = `account-${hashText(
`小红书:${profile.platformUid}`,
`${platform}:${profile.platformUid}`,
)}`;
if (current.account_id === canonicalAccountId) {
const existingAccount = await db
.prepare(
`SELECT id FROM accounts
WHERE platform = ? AND platform_uid = ?
LIMIT 1`,
)
.bind(platform, profile.platformUid)
.first<{ id: string }>();
const targetAccountId = existingAccount?.id || canonicalAccountId;
const currentContact = (current.claimant_contact || "").trim();
if (current.account_id === targetAccountId) {
await db
.prepare(
`UPDATE accounts SET
@@ -81,6 +119,12 @@ export async function enrichDistributionAccount(
WHEN ? IS NOT NULL AND (? > 0 OR followers = 0) THEN ?
ELSE followers
END,
gender = CASE WHEN ? != '' THEN ? ELSE gender END,
bio = CASE WHEN ? != '' THEN ? ELSE bio END,
current_contact = CASE WHEN ? != '' THEN ? ELSE current_contact END,
post_count = (
SELECT COUNT(*) FROM distributions WHERE account_id = ?
),
last_seen_at = CURRENT_TIMESTAMP
WHERE id = ?`,
)
@@ -95,12 +139,19 @@ export async function enrichDistributionAccount(
profile.followers,
profile.followers,
profile.followers,
canonicalAccountId,
profile.gender,
profile.gender,
profile.bio,
profile.bio,
currentContact,
currentContact,
targetAccountId,
targetAccountId,
)
.run();
return {
updated: true,
accountId: canonicalAccountId,
accountId: targetAccountId,
profileUrl: profile.profileUrl,
};
}
@@ -110,8 +161,9 @@ export async function enrichDistributionAccount(
db
.prepare(
`INSERT INTO accounts
(id, platform, platform_uid, public_account_id, nickname, profile_url, ip_location, followers, post_count)
VALUES (?, '小红书', ?, ?, ?, ?, ?, ?, 1)
(id, platform, platform_uid, public_account_id, nickname, profile_url,
ip_location, followers, gender, bio, current_contact, post_count)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, 1)
ON CONFLICT(platform, platform_uid) DO UPDATE SET
public_account_id = CASE
WHEN excluded.public_account_id != ''
@@ -130,17 +182,29 @@ export async function enrichDistributionAccount(
THEN excluded.followers
ELSE accounts.followers
END,
post_count = accounts.post_count + 1,
gender = CASE
WHEN excluded.gender != '' THEN excluded.gender ELSE accounts.gender END,
bio = CASE
WHEN excluded.bio != '' THEN excluded.bio ELSE accounts.bio END,
current_contact = CASE
WHEN excluded.current_contact != ''
THEN excluded.current_contact
ELSE accounts.current_contact
END,
last_seen_at = CURRENT_TIMESTAMP`,
)
.bind(
canonicalAccountId,
targetAccountId,
platform,
profile.platformUid,
profile.redId,
profile.nickname || fallbackNickname,
profile.profileUrl,
profile.ipLocation,
profile.followers ?? 0,
profile.gender,
profile.bio,
currentContact,
),
db
.prepare(
@@ -149,10 +213,25 @@ export async function enrichDistributionAccount(
updated_at = CURRENT_TIMESTAMP
WHERE id = ? AND publish_url = ?`,
)
.bind(canonicalAccountId, distributionId, publishUrl),
.bind(targetAccountId, distributionId, publishUrl),
db
.prepare(
`UPDATE accounts SET post_count = (
SELECT COUNT(*) FROM distributions WHERE account_id = ?
) WHERE id = ?`,
)
.bind(targetAccountId, targetAccountId),
]);
if (provisionalAccountId) {
if (provisionalAccountId && provisionalAccountId !== targetAccountId) {
await db
.prepare(
`UPDATE accounts SET post_count = (
SELECT COUNT(*) FROM distributions WHERE account_id = ?
) WHERE id = ?`,
)
.bind(provisionalAccountId, provisionalAccountId)
.run();
await db
.prepare(
`DELETE FROM accounts
@@ -164,20 +243,20 @@ export async function enrichDistributionAccount(
)
.bind(
provisionalAccountId,
canonicalAccountId,
targetAccountId,
provisionalAccountId,
)
.run();
}
return {
updated: true,
accountId: canonicalAccountId,
accountId: targetAccountId,
profileUrl: profile.profileUrl,
};
}
export async function backfillAccountProfiles(
db: D1Database,
db: DatabaseClient,
mcpConfig: CollectionMcpConfig,
limit = 10,
) {
@@ -187,15 +266,22 @@ export async function backfillAccountProfiles(
d.id,
d.account_id,
d.publish_url,
a.id AS resolved_account_id,
a.nickname,
a.platform,
COALESCE(a.platform, t.platform) AS platform,
a.platform_uid,
a.public_account_id,
a.profile_url,
a.followers
a.followers,
a.gender,
a.bio,
a.tags,
cl.claimant_name AS claimant_contact
FROM distributions d
JOIN tasks t ON t.id = d.task_id
LEFT JOIN accounts a ON a.id = d.account_id
WHERE d.publish_url IS NOT NULL
LEFT JOIN claims cl ON cl.id = d.claim_id
WHERE d.publish_url IS NOT NULL
AND d.publish_url != ''
ORDER BY d.updated_at DESC
LIMIT 100`,
@@ -204,8 +290,34 @@ export async function backfillAccountProfiles(
let attempted = 0;
let updated = 0;
let failed = 0;
const backfilledAccounts = new Set<string>();
for (const row of rows.results) {
if (
row.resolved_account_id &&
!backfilledAccounts.has(row.resolved_account_id)
) {
backfilledAccounts.add(row.resolved_account_id);
const claimantContact = row.claimant_contact?.trim() || "";
await db
.prepare(
`UPDATE accounts
SET current_contact = CASE
WHEN ? != '' THEN ? ELSE current_contact
END,
post_count = (
SELECT COUNT(*) FROM distributions WHERE account_id = ?
)
WHERE id = ?`,
)
.bind(
claimantContact,
claimantContact,
row.resolved_account_id,
row.resolved_account_id,
)
.run();
}
if (attempted >= Math.max(1, Math.min(25, limit))) break;
const noteId = (() => {
try {
@@ -226,19 +338,31 @@ export async function backfillAccountProfiles(
row.platform === "小红书" &&
!isDemoAccount &&
isVerifiedXhsProfileUrl(row.profile_url) &&
(!row.public_account_id || Number(row.followers ?? 0) === 0)
(!row.public_account_id ||
Number(row.followers ?? 0) === 0 ||
!row.gender ||
!row.bio)
) {
attempted += 1;
attemptedThisRow = true;
const details = await resolveXhsProfileDetailsFromMcp(
const details = await resolveProfileDetailsFromMcp(
row.profile_url ?? "",
"小红书",
mcpConfig,
).catch(() =>
resolveXhsPublicAccountDetails(row.profile_url ?? ""),
);
).catch(async () => ({
...(await resolveXhsPublicAccountDetails(row.profile_url ?? "")),
gender: "" as const,
bio: "",
recentNoteTitles: [] as string[],
providerTags: [] as string[],
}));
if (
row.account_id &&
(details.redId || details.followers !== null)
(details.redId ||
details.followers !== null ||
details.gender ||
details.bio ||
details.recentNoteTitles.length > 0)
) {
await db
.prepare(
@@ -255,6 +379,8 @@ export async function backfillAccountProfiles(
WHEN ? != '' AND ? != '待识别' THEN ?
ELSE ip_location
END,
gender = CASE WHEN ? != '' THEN ? ELSE gender END,
bio = CASE WHEN ? != '' THEN ? ELSE bio END,
last_seen_at = CURRENT_TIMESTAMP
WHERE id = ?`,
)
@@ -267,6 +393,10 @@ export async function backfillAccountProfiles(
details.ipLocation ?? "",
details.ipLocation ?? "",
details.ipLocation ?? "",
details.gender,
details.gender,
details.bio,
details.bio,
row.account_id,
)
.run();
@@ -286,12 +416,17 @@ export async function backfillAccountProfiles(
}
const needsEnrichment =
!isDemoAccount &&
(!row.account_id ||
(!row.resolved_account_id ||
(row.platform === "小红书" &&
!isVerifiedXhsProfileUrl(row.profile_url)) ||
(row.platform === "抖音" &&
!isVerifiedDouyinProfileUrl(row.profile_url)) ||
(row.platform === "小红书" && !row.public_account_id) ||
(row.platform === "抖音" && !row.public_account_id) ||
(row.platform === "小红书" &&
Number(row.followers ?? 0) === 0) ||
(row.platform === "抖音" &&
Number(row.followers ?? 0) === 0) ||
row.platform_uid?.startsWith("pending-") ||
Boolean(noteId && row.platform_uid === noteId));
if (!needsEnrichment || !row.publish_url) {

11
lib/background.ts Normal file
View File

@@ -0,0 +1,11 @@
export function runInBackground(
operation: Promise<unknown>,
label: string,
) {
void operation.catch((error) => {
console.error(
`[KOC LOOP] ${label} failed`,
error instanceof Error ? error.message : error,
);
});
}

View File

@@ -1,25 +0,0 @@
function shanghaiDateFromTimestamp(timestamp: number) {
return new Date(timestamp + 8 * 60 * 60 * 1000)
.toISOString()
.slice(0, 10);
}
function shanghaiHourFromTimestamp(timestamp: number) {
return Number(
new Date(timestamp + 8 * 60 * 60 * 1000)
.toISOString()
.slice(11, 13),
);
}
export function isCollectionScheduleDue(
scheduledDate: string,
timestamp: number,
) {
const currentDate = shanghaiDateFromTimestamp(timestamp);
const currentHour = shanghaiHourFromTimestamp(timestamp);
return (
scheduledDate < currentDate ||
(scheduledDate === currentDate && currentHour >= 9)
);
}

View File

@@ -1,17 +1,16 @@
import {
collectXhsMetricsFromMcp,
collectMetricsFromMcp,
type CollectionMcpConfig,
} from "./mcp-collection-client";
import { uid } from "./mvp-db";
import { isCollectionScheduleDue } from "./collection-schedule";
export { isCollectionScheduleDue } from "./collection-schedule";
import type { DatabaseClient, DatabaseStatement } from "./database";
type DistributionForCollection = {
id: string;
task_id: string;
publish_url: string | null;
ocr_status: string;
platform: string;
};
type ScheduledTask = {
@@ -54,6 +53,26 @@ export function shanghaiDateFromTimestamp(timestamp: number) {
.slice(0, 10);
}
function shanghaiHourFromTimestamp(timestamp: number) {
return Number(
new Date(timestamp + 8 * 60 * 60 * 1000)
.toISOString()
.slice(11, 13),
);
}
export function isCollectionScheduleDue(
scheduledDate: string,
timestamp: number,
) {
const currentDate = shanghaiDateFromTimestamp(timestamp);
const currentHour = shanghaiHourFromTimestamp(timestamp);
return (
scheduledDate < currentDate ||
(scheduledDate === currentDate && currentHour >= 9)
);
}
function dueSchedules(
startDate: string,
days: number[],
@@ -76,7 +95,7 @@ function dueSchedules(
}
export async function createCollectionRunTasks(
db: D1Database,
db: DatabaseClient,
taskId: string,
startDate: string,
days: number[],
@@ -103,7 +122,7 @@ export async function createCollectionRunTasks(
.bind(taskId)
.run();
const statements: D1PreparedStatement[] = [];
const statements: DatabaseStatement[] = [];
for (const distribution of distributions.results) {
for (const scheduleDay of normalizedDays) {
const scheduledDate = dateForScheduleDay(startDate, scheduleDay);
@@ -139,7 +158,7 @@ export async function createCollectionRunTasks(
}
export async function collectDistributionMetrics(
db: D1Database,
db: DatabaseClient,
distributionId: string,
scheduledDate: string,
scheduleDay: number | null,
@@ -147,11 +166,16 @@ export async function collectDistributionMetrics(
mcpConfig: CollectionMcpConfig,
) {
const current = await db
.prepare("SELECT * FROM distributions WHERE id = ?")
.prepare(
`SELECT d.*, t.platform
FROM distributions d
JOIN tasks t ON t.id = d.task_id
WHERE d.id = ?`,
)
.bind(distributionId)
.first<DistributionForCollection>();
if (!current) throw new Error("分发记录不存在");
if (!current.publish_url) throw new Error("笔记尚未回填发布链接");
if (!current.publish_url) throw new Error("作品尚未回填发布链接");
const scheduledAt = `${scheduledDate}T09:00:00+08:00`;
const runId = uid("run");
@@ -185,7 +209,12 @@ export async function collectDistributionMetrics(
.bind(distributionId, scheduledDate)
.first<{ id: string; status: string }>();
if (!run) throw new Error("采集任务创建失败");
if (run.status === "success") return { skipped: true };
// Scheduled jobs should remain idempotent, but an operator clicking
// “立即采集” is explicitly asking for a fresh snapshot. Reusing the same
// daily run lets us correct stale or previously mis-mapped platform data.
if (run.status === "success" && source !== "manual") {
return { skipped: true };
}
const collectingDescription =
source === "automatic"
@@ -217,8 +246,12 @@ export async function collectDistributionMetrics(
]);
try {
const { likes, comments, collects } =
await collectXhsMetricsFromMcp(current.publish_url, mcpConfig);
const { likes, comments, collects, shares } =
await collectMetricsFromMcp(
current.publish_url,
current.platform === "抖音" ? "抖音" : "小红书",
mcpConfig,
);
const dayWeight = scheduleDay ?? 1;
const successDescription =
source === "automatic"
@@ -235,6 +268,7 @@ export async function collectDistributionMetrics(
likes = ?,
comments = ?,
collects = ?,
shares = ?,
status_description = ?,
completed_at = CURRENT_TIMESTAMP
WHERE id = ?`,
@@ -243,6 +277,7 @@ export async function collectDistributionMetrics(
likes,
comments,
collects,
shares,
successDescription,
run.id,
),
@@ -252,6 +287,7 @@ export async function collectDistributionMetrics(
SET latest_likes = ?,
latest_comments = ?,
latest_collects = ?,
latest_shares = ?,
collection_status = 'success',
collection_status_description = ?,
collection_updated_at = CURRENT_TIMESTAMP,
@@ -267,12 +303,13 @@ export async function collectDistributionMetrics(
likes,
comments,
collects,
shares,
successDescription,
scheduleDay,
distributionId,
),
]);
return { skipped: false, likes, comments, collects };
return { skipped: false, likes, comments, collects, shares };
} catch (error) {
const message =
error instanceof Error ? error.message : "公开数据采集失败";
@@ -302,7 +339,7 @@ export async function collectDistributionMetrics(
}
export async function runScheduledCollections(
db: D1Database,
db: DatabaseClient,
scheduledTimestamp: number,
mcpConfig: CollectionMcpConfig,
) {
@@ -315,7 +352,7 @@ export async function runScheduledCollections(
}
export async function runDueScheduledCollections(
db: D1Database,
db: DatabaseClient,
timestamp: number,
mcpConfig: CollectionMcpConfig,
source: Extract<CollectionSource, "automatic" | "catchup"> = "catchup",
@@ -396,7 +433,7 @@ export async function runDueScheduledCollections(
}
export async function retryFailedCollections(
db: D1Database,
db: DatabaseClient,
taskId: string,
mcpConfig: CollectionMcpConfig,
) {

227
lib/database.ts Normal file
View File

@@ -0,0 +1,227 @@
import mysql, {
type Pool,
type PoolConnection,
type ResultSetHeader,
type RowDataPacket,
} from "mysql2/promise";
import { getDatabaseUrl } from "./runtime-env";
export type DatabaseResult<T = Record<string, unknown>> = {
results: T[];
success: boolean;
meta: {
changes: number;
last_row_id: number;
};
};
type Executor = Pool | PoolConnection;
type MysqlBindValue =
| string
| number
| bigint
| boolean
| Date
| null
| Blob
| Buffer
| Uint8Array
| MysqlBindValue[]
| { [key: string]: MysqlBindValue };
export function normalizeSqlForMysql(input: string) {
let sql = input.trim();
sql = sql.replace(/^INSERT\s+OR\s+IGNORE\s+INTO\b/i, "INSERT IGNORE INTO");
sql = sql.replace(/\s+ESCAPE\s+'\\\\'/gi, "");
sql = sql.replace(
/datetime\(\s*'now'\s*,\s*'-([0-9]+)\s+minutes?'\s*\)/gi,
"DATE_SUB(UTC_TIMESTAMP(), INTERVAL $1 MINUTE)",
);
sql = sql.replace(
/datetime\(\s*([^,]+?)\s*,\s*'\+([0-9]+)\s+days?'\s*\)/gi,
"DATE_ADD($1, INTERVAL $2 DAY)",
);
const conflict = sql.match(
/\s+ON\s+CONFLICT\s*\(([^)]+)\)\s+DO\s+UPDATE\s+SET\s+([\s\S]+)$/i,
);
if (conflict) {
const assignments = conflict[2].replace(
/\bexcluded\.([a-zA-Z_][a-zA-Z0-9_]*)\b/g,
"VALUES($1)",
);
sql = `${sql.slice(0, conflict.index)} ON DUPLICATE KEY UPDATE ${assignments}`;
}
return sql;
}
function normalizeBindValue(value: unknown): MysqlBindValue {
if (value === undefined) return null;
if (
typeof value === "string" &&
/^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(?:\.\d{1,3})?Z$/.test(value)
) {
return value.replace("T", " ").replace(/Z$/, "");
}
return value as MysqlBindValue;
}
export class DatabaseStatement {
private params: unknown[] = [];
readonly database: DatabaseClient;
readonly sql: string;
constructor(database: DatabaseClient, sql: string) {
this.database = database;
this.sql = sql;
}
bind(...params: unknown[]) {
const statement = new DatabaseStatement(this.database, this.sql);
statement.params = params;
return statement;
}
async all<T = Record<string, unknown>>() {
return this.database.execute<T>(this.sql, this.params);
}
async first<T = Record<string, unknown>>(column?: string) {
const result = await this.all<T>();
const row = result.results[0];
if (!row) return null;
return column ? ((row as Record<string, unknown>)[column] as T) : row;
}
async run() {
return this.database.execute(this.sql, this.params);
}
values() {
return [...this.params];
}
}
export class DatabaseClient {
private readonly executor: Executor;
constructor(executor: Executor) {
this.executor = executor;
}
prepare(sql: string) {
return new DatabaseStatement(this, sql);
}
async execute<T = Record<string, unknown>>(sql: string, params: unknown[] = []) {
const [result] = await this.executor.execute(
normalizeSqlForMysql(sql),
params.map(normalizeBindValue),
);
if (Array.isArray(result)) {
return {
results: result as T[],
success: true,
meta: { changes: 0, last_row_id: 0 },
} satisfies DatabaseResult<T>;
}
const header = result as ResultSetHeader;
return {
results: [],
success: true,
meta: {
changes: header.affectedRows ?? 0,
last_row_id: header.insertId ?? 0,
},
} satisfies DatabaseResult<T>;
}
async batch(statements: DatabaseStatement[]) {
const pool = getPool();
const connection = await pool.getConnection();
try {
await connection.beginTransaction();
const tx = new DatabaseClient(connection);
const results = [];
for (const statement of statements) {
results.push(await tx.execute(statement.sql, statement.values()));
}
await connection.commit();
return results;
} catch (error) {
await connection.rollback();
throw error;
} finally {
connection.release();
}
}
async transaction<T>(operation: (database: DatabaseClient) => Promise<T>) {
const pool = getPool();
const connection = await pool.getConnection();
try {
await connection.beginTransaction();
const transaction = new DatabaseClient(connection);
const result = await operation(transaction);
await connection.commit();
return result;
} catch (error) {
await connection.rollback();
throw error;
} finally {
connection.release();
}
}
}
declare global {
var __kocLoopMysqlPool: Pool | undefined;
}
export function getPool() {
if (!globalThis.__kocLoopMysqlPool) {
globalThis.__kocLoopMysqlPool = mysql.createPool({
uri: getDatabaseUrl(),
connectionLimit: 10,
waitForConnections: true,
queueLimit: 0,
charset: "utf8mb4",
timezone: "Z",
dateStrings: true,
decimalNumbers: true,
enableKeepAlive: true,
});
}
return globalThis.__kocLoopMysqlPool;
}
export function getDatabase() {
return new DatabaseClient(getPool());
}
export async function checkDatabaseConnection() {
const [rows] = await getPool().query<RowDataPacket[]>("SELECT 1 AS healthy");
return rows[0]?.healthy === 1;
}
export async function withDatabaseLock<T>(
name: string,
timeoutSeconds: number,
operation: () => Promise<T>,
) {
const connection = await getPool().getConnection();
try {
const [rows] = await connection.query<RowDataPacket[]>(
"SELECT GET_LOCK(?, ?) AS acquired",
[name, timeoutSeconds],
);
if (Number(rows[0]?.acquired ?? 0) !== 1) return null;
try {
return await operation();
} finally {
await connection.query("SELECT RELEASE_LOCK(?)", [name]);
}
} finally {
connection.release();
}
}

View File

@@ -1,6 +1,9 @@
import type { DatabaseClient } from "./database";
type ReleasableDistribution = {
id: string;
task_id: string;
task_type: string;
content_id: string;
content_title: string;
partner_id: string;
@@ -8,6 +11,7 @@ type ReleasableDistribution = {
claim_id: string | null;
delegation_bundle_id: string | null;
publish_url: string | null;
result_submitted_at: string | null;
};
export class DistributionReleaseError extends Error {
@@ -23,6 +27,7 @@ export class DistributionReleaseError extends Error {
export function distributionReleaseBlockReason(input: {
publishUrl?: string | null;
resultSubmittedAt?: string | null;
taskType?: string | null;
}) {
if (input.publishUrl) return "已回填发布链接的笔记不能释放";
if (input.resultSubmittedAt) return "已提交结果截图的任务不能释放";
@@ -30,90 +35,101 @@ export function distributionReleaseBlockReason(input: {
}
export async function releaseUnfinishedDistribution(
db: D1Database,
database: DatabaseClient,
distributionId: string,
) {
if (!distributionId) {
throw new DistributionReleaseError("请选择需要释放的领取记录", 400);
}
const distribution = await db
.prepare(
`SELECT
d.id,
d.task_id,
d.content_id,
c.title AS content_title,
d.partner_id,
p.name AS partner_name,
d.claim_id,
d.delegation_bundle_id,
d.publish_url
FROM distributions d
JOIN contents c ON c.id = d.content_id
JOIN partners p ON p.id = d.partner_id
WHERE d.id = ?`,
)
.bind(distributionId)
.first<ReleasableDistribution>();
return database.transaction(async (db) => {
const distribution = await db
.prepare(
`SELECT
d.id,
d.task_id,
t.task_type,
d.content_id,
c.title AS content_title,
d.partner_id,
p.name AS partner_name,
d.claim_id,
d.delegation_bundle_id,
d.publish_url,
d.result_submitted_at
FROM distributions d
JOIN tasks t ON t.id = d.task_id
JOIN contents c ON c.id = d.content_id
JOIN partners p ON p.id = d.partner_id
WHERE d.id = ?
FOR UPDATE`,
)
.bind(distributionId)
.first<ReleasableDistribution>();
if (!distribution) {
throw new DistributionReleaseError("领取记录不存在或已被释放", 404);
}
const blocked = distributionReleaseBlockReason({
publishUrl: distribution.publish_url,
});
if (blocked) throw new DistributionReleaseError(blocked, 409);
if (!distribution) {
throw new DistributionReleaseError("领取记录不存在或已被释放", 404);
}
const blocked = distributionReleaseBlockReason({
publishUrl: distribution.publish_url,
resultSubmittedAt: distribution.result_submitted_at,
taskType: distribution.task_type,
});
if (blocked) throw new DistributionReleaseError(blocked, 409);
const statements: D1PreparedStatement[] = [
db
await db
.prepare("DELETE FROM collection_runs WHERE distribution_id = ?")
.bind(distribution.id),
db.prepare("DELETE FROM distributions WHERE id = ?").bind(distribution.id),
db
.bind(distribution.id)
.run();
await db
.prepare("DELETE FROM distributions WHERE id = ?")
.bind(distribution.id)
.run();
await db
.prepare("UPDATE contents SET status = 'available' WHERE id = ?")
.bind(distribution.content_id),
db
.bind(distribution.content_id)
.run();
await db
.prepare(
`UPDATE tasks
SET claimed_quantity = MAX(claimed_quantity - 1, 0)
SET claimed_quantity = GREATEST(claimed_quantity - 1, 0)
WHERE id = ?`,
)
.bind(distribution.task_id),
db
.bind(distribution.task_id)
.run();
await db
.prepare(
`UPDATE partners
SET claimed_total = MAX(claimed_total - 1, 0)
SET claimed_total = GREATEST(claimed_total - 1, 0)
WHERE id = ?`,
)
.bind(distribution.partner_id),
];
.bind(distribution.partner_id)
.run();
if (distribution.delegation_bundle_id) {
statements.push(
db
if (distribution.delegation_bundle_id) {
await db
.prepare(
`UPDATE delegation_bundles
SET quantity = MAX(quantity - 1, 0),
SET quantity = GREATEST(quantity - 1, 0),
status = CASE WHEN quantity <= 1 THEN 'revoked' ELSE status END,
revoked_at = CASE WHEN quantity <= 1 THEN CURRENT_TIMESTAMP ELSE revoked_at END,
updated_at = CURRENT_TIMESTAMP
WHERE id = ?`,
)
.bind(distribution.delegation_bundle_id),
);
}
.bind(distribution.delegation_bundle_id)
.run();
}
if (distribution.claim_id) {
statements.push(
db
if (distribution.claim_id) {
await db
.prepare(
`UPDATE claims
SET quantity = MAX(quantity - 1, 0)
SET quantity = GREATEST(quantity - 1, 0)
WHERE id = ?`,
)
.bind(distribution.claim_id),
db
.bind(distribution.claim_id)
.run();
await db
.prepare(
`DELETE FROM claims
WHERE id = ?
@@ -122,16 +138,16 @@ export async function releaseUnfinishedDistribution(
SELECT 1 FROM distributions WHERE claim_id = ?
)`,
)
.bind(distribution.claim_id, distribution.claim_id),
);
}
.bind(distribution.claim_id, distribution.claim_id)
.run();
}
await db.batch(statements);
return {
distributionId: distribution.id,
taskId: distribution.task_id,
contentId: distribution.content_id,
contentTitle: distribution.content_title,
partnerName: distribution.partner_name,
};
return {
distributionId: distribution.id,
taskId: distribution.task_id,
contentId: distribution.content_id,
contentTitle: distribution.content_title,
partnerName: distribution.partner_name,
};
});
}

View File

@@ -2,7 +2,7 @@ const FEISHU_API_ORIGIN = "https://open.feishu.cn";
const MAX_SHEET_ROWS = 5_000;
const MAX_SHEET_COLUMNS = 100;
const MAX_CONTENT_ROWS = 1_000;
const MAX_MEDIA_BYTES = 20_000_000;
const DEFAULT_MAX_MEDIA_BYTES = 200_000_000;
export type FeishuBindings = {
FEISHU_APP_ID?: string;
@@ -16,11 +16,20 @@ export type FeishuSourceImage = {
height: number | null;
};
export type FeishuSourceVideo = {
index: number;
fileToken: string;
name: string;
mimeType: string;
size: number | null;
};
export type FeishuSourceRow = {
sourceRow: number;
title: string;
body: string;
images: FeishuSourceImage[];
videos: FeishuSourceVideo[];
};
export type FeishuSource = {
@@ -106,12 +115,47 @@ function cellText(value: unknown): string {
.filter(Boolean)
.join("");
}
if (!isRecord(value) || value.type === "embed-image") return "";
if (
!isRecord(value) ||
value.type === "embed-image" ||
value.type === "attachment"
) return "";
if (typeof value.text === "string") return value.text.trim();
if (typeof value.value === "string") return value.value.trim();
return "";
}
function extractVideos(value: unknown, output: FeishuSourceVideo[]) {
if (Array.isArray(value)) {
for (const item of value) extractVideos(item, output);
return;
}
if (!isRecord(value)) return;
const fileToken = bindingValue(value.fileToken ?? value.file_token);
const mimeType = bindingValue(value.mimeType ?? value.mime_type);
const name = bindingValue(value.text ?? value.name ?? value.file_name);
const isVideo =
value.type === "attachment" &&
(mimeType.startsWith("video/") || /\.(?:mp4|mov|m4v|webm)$/i.test(name));
if (isVideo && fileToken) {
output.push({
index: 0,
fileToken,
name: name || "视频",
mimeType: mimeType || "video/mp4",
size:
typeof value.size === "number" && Number.isFinite(value.size)
? value.size
: null,
});
}
for (const child of Object.values(value)) {
if (child !== value.fileToken && child !== value.file_token) {
extractVideos(child, output);
}
}
}
function extractImages(value: unknown, output: FeishuSourceImage[]) {
if (Array.isArray(value)) {
for (const item of value) extractImages(item, output);
@@ -167,6 +211,7 @@ function findHeader(values: unknown[][]) {
titleIndex: number;
bodyIndex: number;
tagsIndex: number;
videoIndex: number;
score: number;
}
| undefined;
@@ -185,9 +230,13 @@ function findHeader(values: unknown[][]) {
const tagsIndex = headers.findIndex((header) =>
headerMatches(header, [/标签/, /话题/, /^tags?$/]),
);
const videoIndex = headers.findIndex((header) =>
headerMatches(header, [/^视频\d*$/, /视频文件/, /视频素材/]),
);
const score =
(titleIndex >= 0 ? 5 : 0) +
(bodyIndex >= 0 ? 5 : 0) +
(videoIndex >= 0 ? 2 : 0) +
(idIndex >= 0 ? 1 : 0) +
(tagsIndex >= 0 ? 1 : 0);
if (!best || score > best.score) {
@@ -197,6 +246,7 @@ function findHeader(values: unknown[][]) {
titleIndex,
bodyIndex,
tagsIndex,
videoIndex,
score,
};
}
@@ -217,6 +267,7 @@ function parseRows(values: unknown[][]) {
const usedSourceRows = new Set<number>();
const rows: FeishuSourceRow[] = [];
let maxImageCount = 0;
let maxVideoCount = 0;
for (
let rowIndex = header.rowIndex + 1;
@@ -253,7 +304,20 @@ function parseRows(values: unknown[][]) {
})
.map((image, imageIndex) => ({ ...image, index: imageIndex + 1 }));
maxImageCount = Math.max(maxImageCount, images.length);
rows.push({ sourceRow, title, body, images });
const collectedVideos: FeishuSourceVideo[] = [];
if (header.videoIndex >= 0) {
extractVideos(row[header.videoIndex], collectedVideos);
}
const seenVideoTokens = new Set<string>();
const videos = collectedVideos
.filter((video) => {
if (seenVideoTokens.has(video.fileToken)) return false;
seenVideoTokens.add(video.fileToken);
return true;
})
.map((video, videoIndex) => ({ ...video, index: videoIndex + 1 }));
maxVideoCount = Math.max(maxVideoCount, videos.length);
rows.push({ sourceRow, title, body, images, videos });
}
if (rows.length === 0) {
@@ -266,6 +330,7 @@ function parseRows(values: unknown[][]) {
header.tagsIndex >= 0 ? cellText(headerRow[header.tagsIndex]) : "",
cellText(headerRow[header.bodyIndex]) || "正文",
...Array.from({ length: maxImageCount }, (_, index) => `图片${index + 1}`),
...Array.from({ length: maxVideoCount }, (_, index) => `视频${index + 1}`),
].filter(Boolean);
return {
@@ -515,34 +580,37 @@ export async function downloadFeishuMedia(
fileToken: string,
bindings: FeishuBindings,
fetchImpl: FetchLike = fetch,
options: { maxBytes?: number; label?: string; timeoutMs?: number } = {},
) {
const maxBytes = Math.max(1, options.maxBytes ?? DEFAULT_MAX_MEDIA_BYTES);
const label = bindingValue(options.label) || "素材";
const normalizedToken = bindingValue(fileToken);
if (!/^[A-Za-z0-9_-]{8,240}$/.test(normalizedToken)) {
throw new FeishuSourceError("飞书图片标识无效", 400);
throw new FeishuSourceError(`飞书${label}标识无效`, 400);
}
const token = await accessToken(bindings, fetchImpl);
const response = await fetchImpl(
`${FEISHU_API_ORIGIN}/open-apis/drive/v1/medias/${encodeURIComponent(normalizedToken)}/download`,
{
headers: { Authorization: `Bearer ${token}` },
signal: AbortSignal.timeout(20_000),
signal: AbortSignal.timeout(options.timeoutMs ?? 120_000),
},
);
const declaredSize = Number(response.headers.get("content-length"));
if (Number.isFinite(declaredSize) && declaredSize > MAX_MEDIA_BYTES) {
throw new FeishuSourceError("飞书图片超过20MB,无法同步", 413);
if (Number.isFinite(declaredSize) && declaredSize > maxBytes) {
throw new FeishuSourceError(`飞书${label}文件过大,无法同步`, 413);
}
if (!response.ok) {
throw new FeishuSourceError(
response.status === 403
? "飞书应用没有这张图片的下载权限"
: `下载飞书图片失败HTTP ${response.status}`,
? `飞书应用没有这${label}的下载权限`
: `下载飞书${label}失败HTTP ${response.status}`,
response.status === 403 ? 403 : 502,
);
}
const bytes = await response.arrayBuffer();
if (bytes.byteLength > MAX_MEDIA_BYTES) {
throw new FeishuSourceError("飞书图片超过20MB,无法同步", 413);
if (bytes.byteLength > maxBytes) {
throw new FeishuSourceError(`飞书${label}文件过大,无法同步`, 413);
}
return {
bytes,

View File

@@ -17,6 +17,7 @@ export type XhsPublicMetrics = {
likes: number;
comments: number;
collects: number;
shares: number;
};
export type XhsAccountProfile = {
@@ -26,6 +27,10 @@ export type XhsAccountProfile = {
redId: string;
ipLocation: string;
followers: number | null;
gender: "" | "男" | "女";
bio: string;
recentNoteTitles: string[];
providerTags: string[];
};
type JsonRpcEnvelope = {
@@ -225,7 +230,7 @@ async function createMcpSession(
timeoutMs,
);
const sessionId = initialize.response.headers.get("mcp-session-id");
if (!sessionId) throw new Error("MCP采集服务未返回会话标识");
if (!sessionId) return undefined;
await postMcp(
fetchImpl,
@@ -242,10 +247,10 @@ async function createMcpSession(
return sessionId;
}
async function callMcpTool(
async function invokeMcpTool(
fetchImpl: typeof fetch,
endpoint: string,
sessionId: string,
sessionId: string | undefined,
timeoutMs: number,
name: string,
args: Record<string, unknown>,
@@ -272,6 +277,12 @@ async function callMcpTool(
try {
payload = JSON.parse(text);
} catch {
if (result.envelope?.result?.isError === true) {
return {
isError: true,
payload: { message: text },
};
}
throw new Error("MCP采集工具返回了无法解析的数据");
}
return {
@@ -280,6 +291,40 @@ async function callMcpTool(
};
}
function isToolArgumentShapeError(result: ToolResult) {
if (!result.isError) return false;
const root = asRecord(result.payload);
const message = String(root?.message ?? "");
return /input validation error/i.test(message);
}
async function callMcpTool(
fetchImpl: typeof fetch,
endpoint: string,
sessionId: string | undefined,
timeoutMs: number,
name: string,
args: Record<string, unknown>,
): Promise<ToolResult> {
const nested = await invokeMcpTool(
fetchImpl,
endpoint,
sessionId,
timeoutMs,
name,
{ request: args },
);
if (!isToolArgumentShapeError(nested)) return nested;
return invokeMcpTool(
fetchImpl,
endpoint,
sessionId,
timeoutMs,
name,
args,
);
}
function metricsFromToolResult(result: ToolResult): XhsPublicMetrics {
const root = asRecord(result.payload);
const response = asRecord(root?.response) ?? root;
@@ -301,16 +346,118 @@ function metricsFromToolResult(result: ToolResult): XhsPublicMetrics {
}
if (!data) throw new Error("采集结果缺少互动数据");
const count = (value: unknown, label: string) =>
value === null || value === undefined || value === ""
? 0
: metricValue(value, label);
return {
likes: metricValue(data.likes, "点赞数"),
comments: metricValue(data.comments, "评论数"),
collects: metricValue(
data.collects ?? data.favorites ?? data.favourites,
likes: count(
data.likes ??
data.liked_count ??
data.likedCount ??
data.like_count ??
data.likeCount ??
data.digg_count ??
data.diggCount,
"点赞数",
),
comments: count(
data.comments ?? data.comment_count ?? data.commentCount,
"评论数",
),
collects: count(
data.collects ??
data.collected_count ??
data.collectedCount ??
data.favorites ??
data.favourites ??
data.collect_count ??
data.collectCount,
"收藏数",
),
shares: count(
data.shares ??
data.share_count ??
data.shareCount ??
data.forwards ??
data.forward_count ??
data.forwardCount,
"转发数",
),
};
}
function usableDouyinSecUid(value: unknown) {
const candidate = stringValue(value);
return candidate && !/^\d+$/.test(candidate) && /^[A-Za-z0-9_-]{20,220}$/.test(candidate)
? candidate
: "";
}
function verifiedDouyinProfileUrl(value: unknown) {
const candidate = stringValue(value);
if (!candidate) return "";
try {
const parsed = new URL(candidate);
const secUid = decodeURIComponent(parsed.pathname.match(/^\/user\/([^/?#]+)/)?.[1] ?? "");
if (
parsed.protocol === "https:" &&
(parsed.hostname === "douyin.com" || parsed.hostname.endsWith(".douyin.com")) &&
usableDouyinSecUid(secUid)
) {
return parsed.toString();
}
} catch {
// The public redirect fallback below can still recover the profile URL.
}
return "";
}
async function douyinProfileFromPublicRedirect(
publishUrl: string,
fetchImpl: typeof fetch,
timeoutMs: number,
) {
let parsed: URL;
try {
parsed = new URL(publishUrl);
} catch {
return null;
}
if (
parsed.protocol !== "https:" ||
!(parsed.hostname === "douyin.com" || parsed.hostname.endsWith(".douyin.com"))
) {
return null;
}
try {
const response = await fetchImpl(parsed.toString(), {
method: "GET",
redirect: "manual",
signal: AbortSignal.timeout(Math.min(timeoutMs, 15_000)),
headers: {
"user-agent":
"Mozilla/5.0 (iPhone; CPU iPhone OS 18_0 like Mac OS X) AppleWebKit/605.1.15 Mobile/15E148",
},
});
const location = response.headers.get("location");
if (!location) return null;
const redirectUrl = new URL(location, parsed);
const secUid = usableDouyinSecUid(
redirectUrl.searchParams.get("sec_uid") ??
redirectUrl.searchParams.get("sec_user_id"),
);
return secUid
? {
platformUid: secUid,
profileUrl: `https://www.douyin.com/user/${encodeURIComponent(secUid)}`,
}
: null;
} catch {
return null;
}
}
function successfulToolData(result: ToolResult, fallbackMessage: string) {
const root = asRecord(result.payload);
const response = asRecord(root?.response) ?? root;
@@ -406,6 +553,64 @@ function findValueByKeys(
return undefined;
}
function profileGender(value: unknown): "" | "男" | "女" {
if (value === 1) return "男";
if (value === 2) return "女";
const normalized = String(value ?? "").trim().toLocaleLowerCase("zh-CN");
if (["男", "男性", "male", "m", "1"].includes(normalized)) return "男";
if (["女", "女性", "female", "f", "2"].includes(normalized)) return "女";
return "";
}
function recentNoteTitlesFromPayload(value: unknown) {
const titles: string[] = [];
const visit = (current: unknown, depth = 0) => {
if (depth > 12 || titles.length >= 20) return;
if (Array.isArray(current)) {
current.forEach((item) => visit(item, depth + 1));
return;
}
const record = asRecord(current);
if (!record) return;
const title = stringValue(record.title ?? record.note_title ?? record.noteTitle);
if (
title &&
(record.note_id || record.noteId || record.url || record.cover) &&
!titles.includes(title)
) {
titles.push(title);
}
Object.values(record).forEach((child) => visit(child, depth + 1));
};
visit(value);
return titles;
}
function providerTagsFromUser(value: unknown) {
const user = findRecord(value, (record) =>
Boolean(
record.gender !== undefined ||
record.desc !== undefined ||
record.signature !== undefined ||
record.fansCount !== undefined ||
record.fans_count !== undefined,
),
);
const raw = user?.tags;
if (!Array.isArray(raw)) return [];
return [
...new Set(
raw
.map((item) =>
typeof item === "string"
? item.trim()
: stringValue(asRecord(item)?.name ?? asRecord(item)?.title),
)
.filter(Boolean),
),
].slice(0, 5);
}
const FOLLOWER_KEYS = new Set([
"fans",
"fans_count",
@@ -489,10 +694,13 @@ async function xhsNoteIdFromShortLink(
) {
return { noteId: "", profile: null };
}
if (
url.hostname !== "xhslink.cn" &&
!url.hostname.endsWith(".xhslink.cn")
) {
const isShortLink =
url.hostname === "xhslink.cn" ||
url.hostname.endsWith(".xhslink.cn");
const isXhsPage =
url.hostname === "xiaohongshu.com" ||
url.hostname.endsWith(".xiaohongshu.com");
if (!isShortLink && !isXhsPage) {
return { noteId: "", profile: null };
}
@@ -568,6 +776,10 @@ function accountProfileFromPublicPage(
redId,
ipLocation: "待识别",
followers: null,
gender: "",
bio: "",
recentNoteTitles: [],
providerTags: [],
};
}
@@ -579,7 +791,7 @@ export async function resolveXhsPublicAccountDetails(
try {
parsed = new URL(profileUrl);
} catch {
return { redId: "", followers: null, ipLocation: "" };
return { nickname: "", redId: "", followers: null, ipLocation: "" };
}
if (
parsed.protocol !== "https:" ||
@@ -589,7 +801,7 @@ export async function resolveXhsPublicAccountDetails(
) ||
!parsed.pathname.startsWith("/user/profile/")
) {
return { redId: "", followers: null, ipLocation: "" };
return { nickname: "", redId: "", followers: null, ipLocation: "" };
}
try {
const response = await fetchImpl(parsed.toString(), {
@@ -602,10 +814,12 @@ export async function resolveXhsPublicAccountDetails(
},
});
if (!response.ok) {
return { redId: "", followers: null, ipLocation: "" };
return { nickname: "", redId: "", followers: null, ipLocation: "" };
}
const html = await response.text();
return {
nickname:
html.match(/"(?:nickname|nickName)":"([^"]+)"/)?.[1] ?? "",
redId:
html.match(/"(?:redId|red_id)":"([^"]+)"/)?.[1] ??
html.match(/小红书号[:]\s*([^<"\s]+)/)?.[1] ??
@@ -620,7 +834,7 @@ export async function resolveXhsPublicAccountDetails(
ipLocation: "",
};
} catch {
return { redId: "", followers: null, ipLocation: "" };
return { nickname: "", redId: "", followers: null, ipLocation: "" };
}
}
@@ -650,18 +864,51 @@ function profileDetailsFromToolResult(result: ToolResult) {
const payload =
asRecord(response?.data) ?? asRecord(root?.data) ?? result.payload;
const followers = followerCountFromPayload(payload);
const nickname =
findStringByKey(payload, "nickname") ||
findStringByKey(payload, "nickName");
const redId =
findStringByKey(payload, "red_id") ||
findStringByKey(payload, "redId") ||
findStringByKey(payload, "unique_id") ||
findStringByKey(payload, "uniqueId") ||
findStringByKey(payload, "short_id") ||
findStringByKey(payload, "shortId") ||
findStringByKey(payload, "douyin_id") ||
findStringByKey(payload, "userId") ||
findStringByKey(payload, "user_id");
const ipLocation =
findStringByKey(payload, "ip_location") ||
findStringByKey(payload, "ipLocation");
if (followers === null && !redId && !ipLocation) {
const gender = profileGender(findValueByKeys(payload, new Set(["gender", "sex"])));
const bio =
findStringByKey(payload, "desc") ||
findStringByKey(payload, "description") ||
findStringByKey(payload, "signature") ||
findStringByKey(payload, "bio");
const recentNoteTitles = recentNoteTitlesFromPayload(payload);
const providerTags = providerTagsFromUser(payload);
if (
followers === null &&
!nickname &&
!redId &&
!ipLocation &&
!gender &&
!bio &&
recentNoteTitles.length === 0
) {
throw new Error("账号主页采集结果缺少可用字段");
}
return { followers, redId, ipLocation };
return {
nickname,
followers,
redId,
ipLocation,
gender,
bio,
recentNoteTitles,
providerTags,
};
}
function accountProfileFromToolResult(
@@ -673,14 +920,18 @@ function accountProfileFromToolResult(
data,
(record) =>
Boolean(
stringValue(record.user_id ?? record.userid) &&
stringValue(record.user_id ?? record.userid ?? record.userId) &&
(stringValue(record.profile_url) ||
stringValue(record.nickname ?? record.name)),
),
);
if (!user) throw new Error("账号主页识别结果缺少作者信息");
const platformUid = stringValue(user.user_id ?? user.userid);
const candidateProfileUrl = stringValue(user.profile_url);
const platformUid = stringValue(
user.user_id ?? user.userid ?? user.userId,
);
const candidateProfileUrl = stringValue(
user.profile_url ?? user.profileUrl,
);
let profileUrl = "";
if (candidateProfileUrl) {
try {
@@ -710,6 +961,10 @@ function accountProfileFromToolResult(
redId: stringValue(user.red_id),
ipLocation: findStringByKey(data, "ip_location") || "待识别",
followers: followerCountFromPayload(data),
gender: "",
bio: "",
recentNoteTitles: [],
providerTags: [],
};
}
@@ -717,7 +972,7 @@ async function completeAccountProfile(
profile: XhsAccountProfile,
fetchImpl: typeof fetch,
endpoint: string,
sessionId: string,
sessionId: string | undefined,
timeoutMs: number,
) {
let completed = profile;
@@ -726,18 +981,7 @@ async function completeAccountProfile(
"parse_xhs_user_summary",
{ url: profile.profileUrl, use_proxy: true },
],
[
"fetch_user_detail",
{ link: profile.profileUrl, plant: "xhs" },
],
] as const) {
if (
completed.followers !== null &&
completed.redId &&
completed.ipLocation !== "待识别"
) {
break;
}
try {
const result = await callMcpTool(
fetchImpl,
@@ -756,6 +1000,16 @@ async function completeAccountProfile(
details.ipLocation && details.ipLocation !== "待识别"
? details.ipLocation
: completed.ipLocation,
gender: details.gender || completed.gender,
bio: details.bio || completed.bio,
recentNoteTitles:
details.recentNoteTitles.length > 0
? details.recentNoteTitles
: completed.recentNoteTitles,
providerTags:
details.providerTags.length > 0
? details.providerTags
: completed.providerTags,
};
} catch (error) {
if (error instanceof McpSessionLostError) throw error;
@@ -807,17 +1061,14 @@ async function resolveAccountInSession(
const endpoint = buildMcpUrl(config);
const timeoutMs = Math.max(5_000, config.timeoutMs ?? 30_000);
let noteId = xhsNoteIdFromUrl(publishUrl);
let publicPageProfile: XhsAccountProfile | null = null;
if (!noteId) {
const shortLink = await xhsNoteIdFromShortLink(
publishUrl,
fallbackNickname,
fetchImpl,
timeoutMs,
);
noteId = shortLink.noteId;
publicPageProfile = shortLink.profile;
}
const linkPage = await xhsNoteIdFromShortLink(
publishUrl,
fallbackNickname,
fetchImpl,
timeoutMs,
);
noteId = noteId || linkPage.noteId;
const publicPageProfile = linkPage.profile;
try {
const sessionId = await createMcpSession(
@@ -825,47 +1076,39 @@ async function resolveAccountInSession(
endpoint,
timeoutMs,
);
if (!noteId) {
const noteResult = await callMcpTool(
fetchImpl,
endpoint,
sessionId,
timeoutMs,
"fetch_content_detail",
{
link: publishUrl,
plant: "xhs",
include_comments: false,
auto_cookie: true,
},
);
const noteData = successfulToolData(
noteResult,
"无法识别小红书笔记",
);
noteId =
stringValue(noteData.noteId ?? noteData.note_id) ||
findStringByKey(noteData, "noteId") ||
findStringByKey(noteData, "note_id");
}
if (!noteId) throw new Error("无法从发布链接识别小红书笔记ID");
const authorResult = await callMcpTool(
const noteResult = await callMcpTool(
fetchImpl,
endpoint,
sessionId,
timeoutMs,
"collect_xhs_wen_note_detail",
"fetch_content_detail",
{
note_id: noteId,
need_desc: false,
include_raw: false,
link: publishUrl,
plant: "xhs",
include_comments: false,
auto_cookie: true,
},
);
const profile = accountProfileFromToolResult(
authorResult,
fallbackNickname,
const noteData = successfulToolData(
noteResult,
"无法识别小红书笔记",
);
noteId =
noteId ||
stringValue(noteData.noteId ?? noteData.note_id) ||
findStringByKey(noteData, "noteId") ||
findStringByKey(noteData, "note_id");
if (!noteId) throw new Error("无法从发布链接识别小红书笔记ID");
let profile: XhsAccountProfile;
try {
profile = accountProfileFromToolResult(
noteResult,
fallbackNickname,
);
} catch {
if (!publicPageProfile) throw new Error("笔记数据缺少公开作者主页");
profile = publicPageProfile;
}
return completeAccountProfile(
profile,
fetchImpl,
@@ -892,6 +1135,7 @@ async function resolveAccountInSession(
async function collectInSession(
publishUrl: string,
platform: "小红书" | "抖音",
config: CollectionMcpConfig,
fetchImpl: typeof fetch,
) {
@@ -907,28 +1151,12 @@ async function collectInSession(
"fetch_content_detail",
{
link: publishUrl,
plant: "xhs",
plant: platform === "抖音" ? "dy" : "xhs",
include_comments: false,
auto_cookie: true,
},
);
try {
return metricsFromToolResult(primary);
} catch {
const fallback = await callMcpTool(
fetchImpl,
endpoint,
sessionId,
timeoutMs,
"parse_xhs_note",
{
url: publishUrl,
include_comments: false,
auto_cookie: true,
},
);
return metricsFromToolResult(fallback);
}
return metricsFromToolResult(primary);
}
export function resolveCollectionMcpConfig(
@@ -960,7 +1188,7 @@ export async function collectXhsMetricsFromMcp(
let lastError: unknown;
for (let attempt = 0; attempt < MAX_MCP_ATTEMPTS; attempt += 1) {
try {
return await collectInSession(parsed.toString(), config, fetchImpl);
return await collectInSession(parsed.toString(), "小红书", config, fetchImpl);
} catch (error) {
lastError = error;
if (!isRetryableTransportError(error)) throw error;
@@ -977,6 +1205,211 @@ export async function collectXhsMetricsFromMcp(
: new Error("MCP采集服务暂时不可用");
}
export async function collectMetricsFromMcp(
publishUrl: string,
platform: "小红书" | "抖音",
config: CollectionMcpConfig,
fetchImpl: typeof fetch = fetch,
) {
if (platform === "小红书") {
return collectXhsMetricsFromMcp(publishUrl, config, fetchImpl);
}
let parsed: URL;
try {
parsed = new URL(publishUrl);
} catch {
throw new Error("发布链接无效");
}
if (!["http:", "https:"].includes(parsed.protocol)) {
throw new Error("发布链接无效");
}
let lastError: unknown;
for (let attempt = 0; attempt < MAX_MCP_ATTEMPTS; attempt += 1) {
try {
return await collectInSession(parsed.toString(), platform, config, fetchImpl);
} catch (error) {
lastError = error;
if (!isRetryableTransportError(error)) throw error;
}
}
throw lastError instanceof Error
? lastError
: new Error("MCP采集服务暂时不可用");
}
function douyinProfileFromToolResult(
result: ToolResult,
fallbackNickname: string,
): XhsAccountProfile {
const data = successfulToolData(result, "无法识别抖音作品");
const author = findRecord(data, (record) =>
Boolean(
stringValue(
record.sec_uid ?? record.secUid ?? record.uid ?? record.user_id ?? record.userId,
) && stringValue(record.nickname ?? record.name ?? record.unique_id ?? record.uniqueId),
),
);
if (!author) throw new Error("抖音作品数据缺少作者信息");
const verifiedSecUid = usableDouyinSecUid(author.sec_uid ?? author.secUid);
const fallbackUid = stringValue(author.uid ?? author.user_id ?? author.userId);
const platformUid = verifiedSecUid || fallbackUid;
const publicId = stringValue(
author.unique_id ?? author.uniqueId ?? author.short_id ?? author.shortId ?? author.douyin_id,
);
const candidateProfileUrl = verifiedDouyinProfileUrl(
author.profile_url ?? author.profileUrl,
);
const profileUrl = candidateProfileUrl ||
(verifiedSecUid
? `https://www.douyin.com/user/${encodeURIComponent(verifiedSecUid)}`
: "");
return {
platformUid,
nickname: stringValue(author.nickname ?? author.name) || fallbackNickname.trim(),
profileUrl,
redId: publicId,
ipLocation:
stringValue(author.ip_location ?? author.ipLocation) ||
findStringByKey(data, "ip_location") ||
findStringByKey(data, "ipLocation") ||
"待识别",
followers: followerCountFromPayload(author) ?? followerCountFromPayload(data),
gender: profileGender(author.gender ?? author.sex),
bio: stringValue(author.desc ?? author.description ?? author.signature ?? author.bio),
recentNoteTitles: recentNoteTitlesFromPayload(data),
providerTags: providerTagsFromUser(data),
};
}
async function resolveDouyinAccountInSession(
publishUrl: string,
fallbackNickname: string,
config: CollectionMcpConfig,
fetchImpl: typeof fetch,
) {
const endpoint = buildMcpUrl(config);
const timeoutMs = Math.max(5_000, config.timeoutMs ?? 30_000);
const sessionId = await createMcpSession(fetchImpl, endpoint, timeoutMs);
const result = await callMcpTool(
fetchImpl,
endpoint,
sessionId,
timeoutMs,
"fetch_content_detail",
{
link: publishUrl,
plant: "dy",
include_comments: false,
auto_cookie: true,
},
);
let profile = douyinProfileFromToolResult(result, fallbackNickname);
if (!verifiedDouyinProfileUrl(profile.profileUrl)) {
const resolved = await douyinProfileFromPublicRedirect(
publishUrl,
fetchImpl,
timeoutMs,
);
if (resolved) {
profile = {
...profile,
platformUid: resolved.platformUid,
profileUrl: resolved.profileUrl,
};
}
}
if (!verifiedDouyinProfileUrl(profile.profileUrl)) return profile;
try {
const detailsResult = await callMcpTool(
fetchImpl,
endpoint,
sessionId,
timeoutMs,
"parse_dy_user_summary",
{ url: profile.profileUrl },
);
const details = profileDetailsFromToolResult(detailsResult);
profile = {
...profile,
nickname: details.nickname || profile.nickname,
redId: details.redId || profile.redId,
followers: details.followers ?? profile.followers,
ipLocation:
details.ipLocation && details.ipLocation !== "待识别"
? details.ipLocation
: profile.ipLocation,
gender: details.gender || profile.gender,
bio: details.bio || profile.bio,
recentNoteTitles:
details.recentNoteTitles.length > 0
? details.recentNoteTitles
: profile.recentNoteTitles,
providerTags:
details.providerTags.length > 0
? details.providerTags
: profile.providerTags,
};
} catch (error) {
if (error instanceof McpSessionLostError) throw error;
}
return profile;
}
export async function resolveAccountProfileFromMcp(
publishUrl: string,
fallbackNickname: string,
platform: "小红书" | "抖音",
config: CollectionMcpConfig,
fetchImpl: typeof fetch = fetch,
) {
if (platform === "小红书") {
return resolveXhsAccountProfileFromMcp(
publishUrl,
fallbackNickname,
config,
fetchImpl,
);
}
let lastError: unknown;
for (let attempt = 0; attempt < MAX_MCP_ATTEMPTS; attempt += 1) {
try {
return await resolveDouyinAccountInSession(
publishUrl,
fallbackNickname,
config,
fetchImpl,
);
} catch (error) {
lastError = error;
if (!isRetryableTransportError(error)) throw error;
}
}
throw lastError instanceof Error ? lastError : new Error("抖音账号识别失败");
}
export async function resolveProfileDetailsFromMcp(
profileUrl: string,
platform: "小红书" | "抖音",
config: CollectionMcpConfig,
fetchImpl: typeof fetch = fetch,
) {
if (platform === "小红书") {
return resolveXhsProfileDetailsFromMcp(profileUrl, config, fetchImpl);
}
const endpoint = buildMcpUrl(config);
const timeoutMs = Math.max(5_000, config.timeoutMs ?? 30_000);
const sessionId = await createMcpSession(fetchImpl, endpoint, timeoutMs);
const result = await callMcpTool(
fetchImpl,
endpoint,
sessionId,
timeoutMs,
"parse_dy_user_summary",
{ url: profileUrl },
);
return profileDetailsFromToolResult(result);
}
export async function resolveXhsAccountProfileFromMcp(
publishUrl: string,
fallbackNickname: string,

View File

@@ -12,7 +12,7 @@ import {
type CollectionMcpBindings,
} from "./mcp-collection-client";
import { ensureSchema, getRawDb } from "./mvp-db";
import { extractXhsPublishUrl } from "./publish-url";
import { extractAnyPublishUrl } from "./publish-url";
import { buildClaimUrl } from "./task-service";
export type McpOperationBindings = CollectionMcpBindings & {
@@ -83,9 +83,9 @@ export async function taskList(
(SELECT COUNT(*) FROM distributions d WHERE d.task_id = t.id AND COALESCE(d.publish_url, '') != '') AS published_count,
(SELECT COUNT(*) FROM distributions d WHERE d.task_id = t.id AND d.exposure IS NOT NULL AND d.views IS NOT NULL) AS day7_count
FROM tasks t ${where}
ORDER BY t.created_at DESC LIMIT ? OFFSET ?`,
ORDER BY t.created_at DESC LIMIT ${limit} OFFSET ${offset}`,
)
.bind(...bindings, limit, offset)
.bind(...bindings)
.all<Record<string, unknown>>(),
db
.prepare(`SELECT COUNT(*) AS total FROM tasks t ${where}`)
@@ -96,14 +96,19 @@ export async function taskList(
total: count?.total ?? 0,
limit,
offset,
tasks: rows.results.map((row) => ({
...row,
collection_days: parseJsonArray(row.collection_days),
claim_url:
portalUrl && row.share_token
? buildClaimUrl(portalUrl, String(row.share_token))
: null,
})),
tasks: rows.results.map(
(row): Record<string, unknown> & {
collection_days: unknown[];
claim_url: string | null;
} => ({
...row,
collection_days: parseJsonArray(row.collection_days),
claim_url:
portalUrl && row.share_token
? buildClaimUrl(portalUrl, String(row.share_token))
: null,
}),
),
};
}
@@ -118,9 +123,9 @@ export async function taskGet(taskId: string, portalUrl: string) {
const [notes, claims, runs] = await Promise.all([
db
.prepare(
`SELECT c.id AS content_id, c.source_row, c.title, c.body, c.image_assets, c.status AS content_status,
`SELECT c.id AS content_id, c.source_row, c.title, c.body, c.image_assets, c.video_assets, c.status AS content_status,
d.id AS distribution_id, d.status AS distribution_status, d.publish_url, d.publish_time,
d.exposure, d.views, d.latest_likes, d.latest_comments, d.latest_collects,
d.exposure, d.views, d.latest_likes, d.latest_comments, d.latest_collects, d.latest_shares,
d.collection_status, d.collection_status_description, d.collection_updated_at,
a.id AS account_id, a.nickname AS account_nickname, a.profile_url, a.followers,
p.name AS cooperation_source, cl.claimant_name
@@ -151,24 +156,30 @@ export async function taskGet(taskId: string, portalUrl: string) {
.bind(taskId)
.all<Record<string, unknown>>(),
]);
return {
task: {
const taskOutput: Record<string, unknown> & {
collection_days: unknown[];
claim_url: string | null;
} = {
...task,
collection_days: parseJsonArray(task.collection_days),
claim_url:
portalUrl && task.share_token
? buildClaimUrl(portalUrl, String(task.share_token))
: null,
},
};
return {
task: taskOutput,
notes: notes.results.map((row) => ({
...row,
image_assets: parseJsonArray(row.image_assets),
video_assets: parseJsonArray(row.video_assets),
total_interactions:
row.latest_likes == null
? null
: Number(row.latest_likes) +
Number(row.latest_comments ?? 0) +
Number(row.latest_collects ?? 0),
Number(row.latest_collects ?? 0) +
Number(row.latest_shares ?? 0),
})),
claims: claims.results,
collection_runs: runs.results,
@@ -211,12 +222,13 @@ export async function recoveryList(
const [rows, count] = await Promise.all([
db
.prepare(
`SELECT d.*, t.name AS task_name, c.source_row, c.title,
`SELECT d.*, t.name AS task_name, t.platform AS task_platform,
t.content_format, c.source_row, c.title,
a.nickname AS account_nickname, a.profile_url, p.name AS cooperation_source,
cl.claimant_name ${base}
ORDER BY COALESCE(d.publish_time, d.claimed_at) DESC LIMIT ? OFFSET ?`,
ORDER BY COALESCE(d.publish_time, d.claimed_at) DESC LIMIT ${limit} OFFSET ${offset}`,
)
.bind(...bindings, limit, offset)
.bind(...bindings)
.all<Record<string, unknown>>(),
db
.prepare(`SELECT COUNT(*) AS total ${base}`)
@@ -232,7 +244,7 @@ export async function recoveryList(
total_interactions:
row.latest_likes == null
? null
: Number(row.latest_likes) + Number(row.latest_comments ?? 0) + Number(row.latest_collects ?? 0),
: Number(row.latest_likes) + Number(row.latest_comments ?? 0) + Number(row.latest_collects ?? 0) + Number(row.latest_shares ?? 0),
})),
};
}
@@ -349,9 +361,11 @@ function resourceWhere(input: ResourceFilters) {
const conditions: string[] = [];
const bindings: unknown[] = [];
if (input.query?.trim()) {
conditions.push("(a.nickname LIKE ? ESCAPE '\\' OR a.public_account_id LIKE ? ESCAPE '\\')");
conditions.push(
"(a.nickname LIKE ? ESCAPE '\\' OR a.public_account_id LIKE ? ESCAPE '\\' OR a.current_contact LIKE ? ESCAPE '\\' OR a.tags LIKE ? ESCAPE '\\' OR a.bio LIKE ? ESCAPE '\\')",
);
const pattern = like(input.query.trim());
bindings.push(pattern, pattern);
bindings.push(pattern, pattern, pattern, pattern, pattern);
}
if (input.ipLocation?.trim()) {
conditions.push("a.ip_location LIKE ? ESCAPE '\\'");
@@ -359,10 +373,16 @@ function resourceWhere(input: ResourceFilters) {
}
if (input.cooperationSource?.trim()) {
conditions.push(
`EXISTS (SELECT 1 FROM distributions dx JOIN partners px ON px.id = dx.partner_id
WHERE dx.account_id = a.id AND px.name LIKE ? ESCAPE '\\')`,
`(a.cooperation_source LIKE ? ESCAPE '\\' OR
EXISTS (SELECT 1 FROM distributions dx
JOIN partners px ON px.id = dx.partner_id
LEFT JOIN claims cx ON cx.id = dx.claim_id
WHERE dx.account_id = a.id
AND (cx.claimant_name IS NULL OR px.name != cx.claimant_name)
AND px.name LIKE ? ESCAPE '\\'))`,
);
bindings.push(like(input.cooperationSource.trim()));
const pattern = like(input.cooperationSource.trim());
bindings.push(pattern, pattern);
}
if (input.platform?.trim() && input.platform !== "all") {
conditions.push("a.platform = ?");
@@ -378,20 +398,34 @@ export async function resourceSearch(input: ResourceFilters) {
const db = getRawDb();
const select = `SELECT a.*,
(SELECT COUNT(*) FROM distributions d WHERE d.account_id = a.id) AS cooperation_count,
(SELECT GROUP_CONCAT(DISTINCT p.name) FROM distributions d JOIN partners p ON p.id = d.partner_id WHERE d.account_id = a.id) AS cooperation_sources`;
(SELECT GROUP_CONCAT(DISTINCT p.name)
FROM distributions d
JOIN partners p ON p.id = d.partner_id
LEFT JOIN claims c ON c.id = d.claim_id
WHERE d.account_id = a.id
AND (c.claimant_name IS NULL OR p.name != c.claimant_name)) AS cooperation_sources`;
const [rows, count] = await Promise.all([
db.prepare(`${select} FROM accounts a ${where} ORDER BY a.last_seen_at DESC LIMIT ? OFFSET ?`)
.bind(...bindings, limit, offset).all<Record<string, unknown>>(),
db.prepare(`${select} FROM accounts a ${where} ORDER BY a.last_seen_at DESC LIMIT ${limit} OFFSET ${offset}`)
.bind(...bindings).all<Record<string, unknown>>(),
db.prepare(`SELECT COUNT(*) AS total FROM accounts a ${where}`).bind(...bindings).first<{ total: number }>(),
]);
return {
total: count?.total ?? 0,
limit,
offset,
accounts: rows.results.map((row) => ({
...row,
cooperation_sources: String(row.cooperation_sources ?? "").split(",").filter(Boolean),
})),
accounts: rows.results.map(
(row): Record<string, unknown> & { cooperation_sources: string[] } => ({
...row,
cooperation_sources: [
...new Set(
`${row.cooperation_sources ?? ""}${row.cooperation_source ?? ""}`
.split(/[、,;|]/)
.map((item) => item.trim())
.filter(Boolean),
),
],
}),
),
};
}
@@ -402,7 +436,7 @@ export async function resourceGet(accountId: string) {
if (!account) throw new Error("账号不存在");
const history = await db.prepare(
`SELECT d.id AS distribution_id, d.task_id, t.name AS task_name, c.title,
d.publish_url, d.publish_time, d.latest_likes, d.latest_comments, d.latest_collects,
d.publish_url, d.publish_time, d.latest_likes, d.latest_comments, d.latest_collects, d.latest_shares,
d.exposure, d.views, p.name AS cooperation_source, cl.claimant_name
FROM distributions d JOIN tasks t ON t.id = d.task_id
JOIN contents c ON c.id = d.content_id JOIN partners p ON p.id = d.partner_id
@@ -418,7 +452,7 @@ export async function backfillResourceProfile(
) {
await ensureSchema();
const db = getRawDb();
const publishUrl = input.publishUrl ? extractXhsPublishUrl(input.publishUrl) : "";
const publishUrl = input.publishUrl ? extractAnyPublishUrl(input.publishUrl) : "";
const row = input.distributionId
? await db.prepare(
`SELECT d.id, d.publish_url, COALESCE(a.nickname, '') AS nickname

View File

@@ -52,7 +52,7 @@ const pagination = {
};
const resourceFilters = {
query: z.string().max(100).optional().describe("账号名称或小红书号/抖音号,支持模糊搜索"),
query: z.string().max(100).optional().describe("账号名称、账号号、简介或标签,支持模糊搜索"),
ip_location: z.string().max(100).optional().describe("IP地区关键词支持模糊搜索"),
cooperation_source: z.string().max(100).optional().describe("历史合作来源关键词,支持模糊搜索"),
platform: z.string().max(30).optional().describe("平台,例如小红书;不传表示全部"),
@@ -159,7 +159,7 @@ export function registerMcpOperationTools(server: McpServer, options: Options) {
"collection_collect_now",
{
title: "立即采集指定笔记",
description: "对指定分发记录立即采集点赞收藏评论数据。",
description: "对指定分发记录立即采集互动数据;小红书为点赞/收藏/评论,抖音另含转发。",
inputSchema: z.object({
distribution_id: z.string().min(1).describe("分发记录ID可从 task_get 或 recovery_list 获取"),
schedule_day: z.number().int().min(1).max(7).optional().describe("标记为第几天采集,可不传"),
@@ -190,7 +190,7 @@ export function registerMcpOperationTools(server: McpServer, options: Options) {
"resource_search",
{
title: "搜索 KOC 账号资源",
description: "按账号名称/账号号、IP地区、合作来源或平台搜索 KOC 资源。",
description: "按账号名称/账号号/标签、IP地区、合作来源或平台搜索 KOC 资源。",
inputSchema: z.object({ ...resourceFilters, ...pagination }),
annotations: { readOnlyHint: true, destructiveHint: false, idempotentHint: true },
},
@@ -204,7 +204,7 @@ export function registerMcpOperationTools(server: McpServer, options: Options) {
"resource_get",
{
title: "查看 KOC 账号详情",
description: "查看账号主页、账号号、粉丝数、IP地区以及全部合作记录。",
description: "查看账号主页、账号号、粉丝数、性别、简介、标签、IP地区以及全部合作记录。",
inputSchema: z.object({ account_id: z.string().min(1).describe("KOC LOOP 账号ID") }),
annotations: { readOnlyHint: true, destructiveHint: false, idempotentHint: true },
},
@@ -218,10 +218,10 @@ export function registerMcpOperationTools(server: McpServer, options: Options) {
"resource_backfill_profile",
{
title: "补全公开账号信息",
description: "根据已回填的发布链接补全账号主页、昵称、小红书号、IP地区和粉丝数。",
description: "根据已回填的小红书或抖音作品链接补全账号主页、昵称、账号号、IP地区和粉丝数。",
inputSchema: z.object({
distribution_id: z.string().optional().describe("分发记录ID和发布链接二选一"),
publish_url: z.string().optional().describe("小红书发布链接或包含链接的分享文案"),
publish_url: z.string().optional().describe("小红书或抖音作品链接,也可传包含链接的分享文案"),
}).refine((value) => Boolean(value.distribution_id || value.publish_url), "请提供分发记录ID或发布链接"),
annotations: { readOnlyHint: false, destructiveHint: false, idempotentHint: true, openWorldHint: true },
},

View File

@@ -1,26 +1,80 @@
import { env } from "cloudflare:workers";
import { type DatabaseClient, getDatabase } from "./database";
import { getObjectStore } from "./object-store";
import { getRuntimeEnv, isEnabled } from "./runtime-env";
import feishuSnapshot from "./feishu-source-snapshot.json";
type D1ResultRow = Record<string, unknown>;
export function getRawDb(): D1Database {
const database = (env as unknown as { DB?: D1Database }).DB;
if (!database) {
throw new Error("数据库尚未连接");
}
return database;
export function getRawDb(): DatabaseClient {
return getDatabase();
}
export function getUploadBucket(): R2Bucket {
const bucket = (env as unknown as { UPLOADS?: R2Bucket }).UPLOADS;
if (!bucket) {
throw new Error("文件存储尚未连接");
}
return bucket;
export function getUploadBucket() {
return getObjectStore();
}
export async function ensureSchema(database?: D1Database) {
export async function ensureSchema(database?: DatabaseClient) {
const db = database ?? getRawDb();
{
await db.prepare("SELECT id FROM tasks LIMIT 1").all();
const tasksWithoutShare = await db
.prepare("SELECT id FROM tasks WHERE share_token IS NULL OR share_token = ''")
.all<{ id: string }>();
for (const task of tasksWithoutShare.results) {
await db
.prepare("UPDATE tasks SET share_token = ? WHERE id = ?")
.bind(crypto.randomUUID().replaceAll("-", ""), task.id)
.run();
}
await db
.prepare(
`UPDATE distributions
SET latest_likes = COALESCE(d7_likes, d5_likes, d2_likes),
latest_comments = COALESCE(d7_comments, d5_comments, d2_comments),
latest_collects = COALESCE(d7_collects, d5_collects, d2_collects),
collection_status = 'success',
collection_status_description = '历史采集数据已迁移',
collection_updated_at = updated_at,
last_collection_day = CASE
WHEN d7_likes IS NOT NULL THEN 7
WHEN d5_likes IS NOT NULL THEN 5
WHEN d2_likes IS NOT NULL THEN 2
ELSE NULL
END
WHERE latest_likes IS NULL
AND COALESCE(d7_likes, d5_likes, d2_likes) IS NOT NULL`,
)
.run();
const tasksNeedingImages = await db
.prepare(
`SELECT DISTINCT t.id
FROM tasks t
JOIN contents c ON c.task_id = t.id
WHERE t.source_sheet_id = ?
AND (c.image_assets IS NULL OR c.image_assets = '' OR c.image_assets = '[]')`,
)
.bind(feishuSnapshot.sheetId)
.all<{ id: string }>();
for (const task of tasksNeedingImages.results) {
const updates = feishuSnapshot.rows
.filter((row) => row.images.length > 0)
.map((row) =>
db
.prepare(
`UPDATE contents SET image_assets = ?
WHERE task_id = ? AND source_row = ?
AND (image_assets IS NULL OR image_assets = '' OR image_assets = '[]')`,
)
.bind(JSON.stringify(row.images), task.id, row.sourceRow),
);
if (updates.length > 0) await db.batch(updates);
}
}
return;
const statements = [
`CREATE TABLE IF NOT EXISTS partners (
id TEXT PRIMARY KEY,
@@ -39,6 +93,9 @@ export async function ensureSchema(database?: D1Database) {
claimed_quantity INTEGER NOT NULL DEFAULT 0,
due_at TEXT NOT NULL,
status TEXT NOT NULL DEFAULT 'active',
task_type TEXT NOT NULL DEFAULT 'content_publish',
platform TEXT NOT NULL DEFAULT '小红书',
content_format TEXT NOT NULL DEFAULT 'image_text',
source_url TEXT NOT NULL DEFAULT '',
source_sheet_id TEXT NOT NULL DEFAULT '',
source_sheet_name TEXT NOT NULL DEFAULT '',
@@ -55,6 +112,7 @@ export async function ensureSchema(database?: D1Database) {
title TEXT NOT NULL,
body TEXT NOT NULL DEFAULT '',
image_assets TEXT NOT NULL DEFAULT '[]',
video_assets TEXT NOT NULL DEFAULT '[]',
status TEXT NOT NULL DEFAULT 'available',
source TEXT NOT NULL DEFAULT '飞书内容表',
source_row INTEGER,
@@ -69,8 +127,13 @@ export async function ensureSchema(database?: D1Database) {
profile_url TEXT NOT NULL DEFAULT '',
ip_location TEXT NOT NULL DEFAULT '待识别',
followers INTEGER NOT NULL DEFAULT 0,
gender TEXT NOT NULL DEFAULT '',
bio TEXT NOT NULL DEFAULT '',
tags TEXT NOT NULL DEFAULT '',
post_count INTEGER NOT NULL DEFAULT 0,
avg_views INTEGER NOT NULL DEFAULT 0,
cooperation_source TEXT NOT NULL DEFAULT '',
current_contact TEXT NOT NULL DEFAULT '',
first_seen_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP,
last_seen_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP
)`,
@@ -109,6 +172,8 @@ export async function ensureSchema(database?: D1Database) {
publish_url TEXT,
publish_time TEXT,
publish_screenshot_key TEXT,
result_screenshot_key TEXT,
result_submitted_at TEXT,
status TEXT NOT NULL DEFAULT 'claimed',
claimed_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP,
screenshot_key TEXT,
@@ -127,6 +192,7 @@ export async function ensureSchema(database?: D1Database) {
latest_likes INTEGER,
latest_comments INTEGER,
latest_collects INTEGER,
latest_shares INTEGER,
collection_status TEXT NOT NULL DEFAULT 'pending',
collection_status_description TEXT,
collection_updated_at TEXT,
@@ -144,6 +210,7 @@ export async function ensureSchema(database?: D1Database) {
likes INTEGER,
comments INTEGER,
collects INTEGER,
shares INTEGER,
status_description TEXT,
started_at TEXT,
completed_at TEXT,
@@ -192,6 +259,11 @@ export async function ensureSchema(database?: D1Database) {
};
await ensureColumn("tasks", "source_url", "source_url TEXT NOT NULL DEFAULT ''");
await ensureColumn(
"tasks",
"task_type",
"task_type TEXT NOT NULL DEFAULT 'content_publish'",
);
await ensureColumn(
"tasks",
"source_sheet_id",
@@ -230,6 +302,14 @@ export async function ensureSchema(database?: D1Database) {
"public_account_id",
"public_account_id TEXT NOT NULL DEFAULT ''",
);
await ensureColumn("accounts", "gender", "gender TEXT NOT NULL DEFAULT ''");
await ensureColumn("accounts", "bio", "bio TEXT NOT NULL DEFAULT ''");
await ensureColumn("accounts", "tags", "tags TEXT NOT NULL DEFAULT ''");
await ensureColumn(
"accounts",
"current_contact",
"current_contact TEXT NOT NULL DEFAULT ''",
);
await ensureColumn("distributions", "claim_id", "claim_id TEXT");
await ensureColumn(
"distributions",
@@ -241,6 +321,16 @@ export async function ensureSchema(database?: D1Database) {
"publish_screenshot_key",
"publish_screenshot_key TEXT",
);
await ensureColumn(
"distributions",
"result_screenshot_key",
"result_screenshot_key TEXT",
);
await ensureColumn(
"distributions",
"result_submitted_at",
"result_submitted_at TEXT",
);
await ensureColumn(
"distributions",
"latest_likes",
@@ -399,6 +489,7 @@ export async function ensureSchema(database?: D1Database) {
}
export async function seedIfEmpty() {
if (!isEnabled(getRuntimeEnv().SEED_DEMO_DATA)) return;
const db = getRawDb();
const row = await db.prepare("SELECT COUNT(*) AS count FROM tasks").first<{
count: number;
@@ -722,21 +813,44 @@ export async function getDashboardData() {
await Promise.all([
db.prepare("SELECT * FROM partners ORDER BY created_at DESC").all(),
db.prepare("SELECT * FROM tasks ORDER BY created_at DESC").all(),
db.prepare("SELECT * FROM accounts ORDER BY last_seen_at DESC").all(),
db
.prepare(
`SELECT
a.*,
COALESCE(
(
SELECT d.publish_url
FROM distributions d
WHERE d.account_id = a.id
AND TRIM(COALESCE(d.publish_url, '')) != ''
ORDER BY d.updated_at DESC, d.claimed_at DESC
LIMIT 1
),
''
) AS latest_publish_url
FROM accounts a
ORDER BY a.last_seen_at DESC`,
)
.all(),
db
.prepare(
`SELECT
d.*,
c.title AS content_title,
p.name AS partner_name,
cl.claimant_name AS claimant_name,
a.nickname AS account_nickname,
a.platform AS account_platform,
t.name AS task_name,
t.brand AS task_brand,
t.task_type AS task_type,
t.platform AS task_platform,
t.content_format AS content_format,
t.due_at AS due_at
FROM distributions d
JOIN contents c ON c.id = d.content_id
JOIN partners p ON p.id = d.partner_id
LEFT JOIN claims cl ON cl.id = d.claim_id
JOIN tasks t ON t.id = d.task_id
LEFT JOIN accounts a ON a.id = d.account_id
ORDER BY d.updated_at DESC, d.claimed_at DESC`,
@@ -749,8 +863,7 @@ export async function getDashboardData() {
tasks: tasksResult.results as D1ResultRow[],
accounts: accountsResult.results as D1ResultRow[],
distributions: distributionsResult.results as D1ResultRow[],
portal_url:
(env as unknown as { KOC_PORTAL_URL?: string }).KOC_PORTAL_URL ?? "",
portal_url: getRuntimeEnv().KOC_PORTAL_URL ?? "",
};
}

125
lib/object-store.ts Normal file
View File

@@ -0,0 +1,125 @@
import { mkdir, readFile, rename, stat, writeFile } from "node:fs/promises";
import path from "node:path";
import { getRuntimeEnv } from "./runtime-env";
type PutOptions = {
httpMetadata?: { contentType?: string };
customMetadata?: Record<string, string>;
};
type StoredMetadata = {
contentType: string;
customMetadata: Record<string, string>;
uploadedAt: string;
};
export class StoredObjectBody {
readonly body: Uint8Array;
private readonly bytes: Buffer;
private readonly metadata: StoredMetadata;
constructor(bytes: Buffer, metadata: StoredMetadata) {
this.bytes = bytes;
this.metadata = metadata;
this.body = new Uint8Array(bytes);
}
async arrayBuffer() {
return this.body.buffer.slice(
this.body.byteOffset,
this.body.byteOffset + this.body.byteLength,
) as ArrayBuffer;
}
writeHttpMetadata(headers: Headers) {
headers.set("Content-Type", this.metadata.contentType);
}
get customMetadata() {
return this.metadata.customMetadata;
}
}
export function normalizeObjectKey(key: string) {
const normalized = key.replaceAll("\\", "/").replace(/^\/+/, "");
if (!normalized || normalized.includes("\0")) {
throw new Error("文件存储键无效");
}
const parts = normalized.split("/");
if (parts.some((part) => !part || part === "." || part === "..")) {
throw new Error("文件存储键不安全");
}
return parts.join("/");
}
function defaultUploadDir() {
return path.join(process.cwd(), ".data", "uploads");
}
export class LocalObjectStore {
readonly root: string;
constructor(root = getRuntimeEnv().UPLOAD_DIR || defaultUploadDir()) {
this.root = root;
}
private resolve(key: string) {
const normalized = normalizeObjectKey(key);
const filePath = path.join(this.root, ...normalized.split("/"));
return { normalized, filePath, metadataPath: `${filePath}.metadata.json` };
}
async put(key: string, input: ArrayBuffer | Uint8Array, options: PutOptions = {}) {
const target = this.resolve(key);
await mkdir(path.dirname(target.filePath), { recursive: true });
const bytes = input instanceof Uint8Array ? input : new Uint8Array(input);
const metadata: StoredMetadata = {
contentType: options.httpMetadata?.contentType || "application/octet-stream",
customMetadata: options.customMetadata ?? {},
uploadedAt: new Date().toISOString(),
};
const suffix = `${process.pid}-${crypto.randomUUID()}`;
const temporaryFile = `${target.filePath}.${suffix}.tmp`;
const temporaryMetadata = `${target.metadataPath}.${suffix}.tmp`;
await Promise.all([
writeFile(temporaryFile, bytes),
writeFile(temporaryMetadata, JSON.stringify(metadata), "utf8"),
]);
await rename(temporaryFile, target.filePath);
await rename(temporaryMetadata, target.metadataPath);
return { key: target.normalized };
}
async get(key: string) {
const target = this.resolve(key);
try {
const [bytes, metadataText] = await Promise.all([
readFile(target.filePath),
readFile(target.metadataPath, "utf8").catch(() => ""),
]);
const fallback: StoredMetadata = {
contentType: "application/octet-stream",
customMetadata: {},
uploadedAt: (await stat(target.filePath)).mtime.toISOString(),
};
const metadata = metadataText
? ({ ...fallback, ...JSON.parse(metadataText) } as StoredMetadata)
: fallback;
return new StoredObjectBody(bytes, metadata);
} catch (error) {
if ((error as NodeJS.ErrnoException).code === "ENOENT") return null;
throw error;
}
}
}
declare global {
var __kocLoopObjectStore: LocalObjectStore | undefined;
}
export function getObjectStore() {
if (!globalThis.__kocLoopObjectStore) {
globalThis.__kocLoopObjectStore = new LocalObjectStore();
}
return globalThis.__kocLoopObjectStore;
}

View File

@@ -0,0 +1,547 @@
import path from "node:path";
import { strFromU8, unzipSync } from "fflate";
export const PARTNER_BATCH_HEADERS = [
"序号(不能改)",
"标题",
"笔记内容(正文+话题)",
"图片",
"发布链接",
"笔记截图",
"数据分析截图(单篇笔记数据分析截图)",
"_系统笔记ID",
"_原笔记截图",
"_原数据分析截图",
] as const;
export const PARTNER_BATCH_VISIBLE_COLUMN_COUNT = 7;
export const PARTNER_BATCH_MAX_BYTES = 80_000_000;
export type PartnerBatchWorkbookColumns = {
headers: string[];
columnWidths: number[];
sourceImageStartColumn: number;
sourceImageCount: number;
sourceVideoStartColumn: number;
sourceVideoCount: number;
publishUrlColumn: number;
publishScreenshotColumn: number;
creatorScreenshotColumn: number;
systemColumn: number;
};
function nonNegativeInteger(value: number) {
return Number.isFinite(value) ? Math.max(0, Math.floor(value)) : 0;
}
export function buildPartnerBatchWorkbookColumns(input: {
contentFormat: "image_text" | "video";
maxSourceImages: number;
maxSourceVideos: number;
}): PartnerBatchWorkbookColumns {
const sourceImageCount =
input.contentFormat === "video"
? 0
: Math.max(1, nonNegativeInteger(input.maxSourceImages));
const sourceVideoCount =
input.contentFormat === "video"
? Math.max(1, nonNegativeInteger(input.maxSourceVideos))
: 0;
const sourceImageStartColumn = 3;
const sourceVideoStartColumn = sourceImageStartColumn + sourceImageCount;
const publishUrlColumn = sourceVideoStartColumn + sourceVideoCount;
const publishScreenshotColumn = publishUrlColumn + 1;
const creatorScreenshotColumn = publishScreenshotColumn + 1;
const systemColumn = creatorScreenshotColumn + 1;
return {
headers: [
"序号(不能改)",
"标题",
"笔记内容(正文+话题)",
...Array.from(
{ length: sourceImageCount },
(_, index) => `图片${index + 1}`,
),
...Array.from(
{ length: sourceVideoCount },
(_, index) => `视频${index + 1}`,
),
"发布链接",
"笔记截图",
"数据分析截图(单篇笔记数据分析截图)",
"_系统笔记ID",
"_原笔记截图",
"_原数据分析截图",
],
columnWidths: [
14,
30,
62,
...Array.from({ length: sourceImageCount }, () => 24),
...Array.from({ length: sourceVideoCount }, () => 20),
45,
28,
32,
22,
22,
22,
],
sourceImageStartColumn,
sourceImageCount,
sourceVideoStartColumn,
sourceVideoCount,
publishUrlColumn,
publishScreenshotColumn,
creatorScreenshotColumn,
systemColumn,
};
}
function firstForwardedValue(value: string | null) {
return value?.split(",")[0]?.trim() ?? "";
}
function httpOrigin(value: string) {
try {
const url = new URL(value);
return url.protocol === "http:" || url.protocol === "https:"
? url.origin
: "";
} catch {
return "";
}
}
export function resolvePartnerWorkbookOrigin(
request: Request,
configuredOrigin = "",
) {
const requestUrl = new URL(request.url);
const host =
firstForwardedValue(request.headers.get("x-forwarded-host")) ||
firstForwardedValue(request.headers.get("host"));
const forwardedProtocol = firstForwardedValue(
request.headers.get("x-forwarded-proto"),
).toLowerCase();
const protocol = ["http", "https"].includes(forwardedProtocol)
? forwardedProtocol
: requestUrl.protocol.replace(":", "");
const proxyOrigin = host ? httpOrigin(`${protocol}://${host}`) : "";
return (
httpOrigin(configuredOrigin) ||
proxyOrigin ||
httpOrigin(requestUrl.origin) ||
requestUrl.origin
);
}
export type PartnerBatchImage = {
bytes: Uint8Array;
contentType: string;
fileName: string;
};
export type PartnerBatchImportRow = {
spreadsheetRow: number;
sequence: string;
title: string;
publishUrl: string;
distributionId: string;
originalPublishScreenshotKey: string;
originalCreatorScreenshotKey: string;
publishScreenshot: PartnerBatchImage | null;
creatorScreenshot: PartnerBatchImage | null;
};
function decodeXml(value: string) {
return value
.replace(/<[^>]+>/g, "")
.replace(/&lt;/g, "<")
.replace(/&gt;/g, ">")
.replace(/&quot;/g, '"')
.replace(/&apos;/g, "'")
.replace(/&amp;/g, "&")
.replace(/&#(\d+);/g, (_, code) => String.fromCodePoint(Number(code)))
.replace(/&#x([0-9a-f]+);/gi, (_, code) =>
String.fromCodePoint(Number.parseInt(code, 16)),
);
}
function xmlAttribute(value: string) {
return decodeXml(value);
}
function textNodes(xml: string) {
return [...xml.matchAll(/<t\b[^>]*>([\s\S]*?)<\/t>/g)]
.map((match) => decodeXml(match[1]))
.join("");
}
function columnIndex(reference: string) {
const letters = reference.match(/^[A-Z]+/i)?.[0]?.toUpperCase() ?? "";
let result = 0;
for (const letter of letters) result = result * 26 + letter.charCodeAt(0) - 64;
return Math.max(0, result - 1);
}
function parseWorksheet(xml: string, sharedStrings: string[]) {
const rows: string[][] = [];
for (const rowMatch of xml.matchAll(/<row\b([^>]*)>([\s\S]*?)<\/row>/g)) {
const rowNumber = Number(
rowMatch[1].match(/\br="(\d+)"/)?.[1] ?? rows.length + 1,
);
const values: string[] = [];
for (const cellMatch of rowMatch[2].matchAll(
/<c\b([^>]*)>([\s\S]*?)<\/c>/g,
)) {
const attributes = cellMatch[1];
const body = cellMatch[2];
const reference = attributes.match(/\br="([A-Z]+\d+)"/i)?.[1] ?? "A1";
const type = attributes.match(/\bt="([^"]+)"/)?.[1] ?? "";
const rawValue = body.match(/<v>([\s\S]*?)<\/v>/)?.[1] ?? "";
let value = "";
if (type === "s") value = sharedStrings[Number(rawValue)] ?? "";
else if (type === "inlineStr") value = textNodes(body);
else if (type === "b") value = rawValue === "1" ? "TRUE" : "FALSE";
else value = decodeXml(rawValue);
values[columnIndex(reference)] = value.trim();
}
while (rows.length < rowNumber - 1) rows.push([]);
rows[rowNumber - 1] = values;
}
return rows;
}
function normalizeHeader(value: string) {
return value.replace(/[\s_\-()]/g, "").toLocaleLowerCase("zh-CN");
}
function isSourceImageHeader(value: string) {
return /^(?:图片|发布配图)\d*$/.test(normalizeHeader(value));
}
function headerAliases(header: (typeof PARTNER_BATCH_HEADERS)[number]) {
const aliases: Record<string, string[]> = {
"序号(不能改)": ["序号(不能改)", "序号"],
: ["标题"],
"笔记内容(正文+话题)": ["笔记内容(正文+话题)", "笔记内容"],
: ["图片", "发布配图"],
: ["发布链接"],
: ["笔记截图", "发布截图"],
"数据分析截图(单篇笔记数据分析截图)": [
"数据分析截图(单篇笔记数据分析截图)",
"数据分析截图",
"创作者中心截图",
],
_系统笔记ID: ["_系统笔记ID", "系统笔记ID"],
_原笔记截图: ["_原笔记截图"],
_原数据分析截图: ["_原数据分析截图"],
};
return aliases[header] ?? [header];
}
function findHeader(rows: string[][]) {
for (let rowIndex = 0; rowIndex < Math.min(rows.length, 8); rowIndex += 1) {
const mapping = new Map<(typeof PARTNER_BATCH_HEADERS)[number], number>();
rows[rowIndex].forEach((value, column) => {
for (const header of PARTNER_BATCH_HEADERS) {
if (header === "图片" && isSourceImageHeader(value)) {
if (!mapping.has(header)) mapping.set(header, column);
break;
}
if (
headerAliases(header).some(
(alias) => normalizeHeader(alias) === normalizeHeader(value),
)
) {
mapping.set(header, column);
break;
}
}
});
if (
mapping.has("序号(不能改)") &&
mapping.has("标题") &&
mapping.has("发布链接") &&
mapping.has("笔记截图") &&
mapping.has("数据分析截图(单篇笔记数据分析截图)") &&
mapping.has("_系统笔记ID")
) {
return { rowIndex, mapping };
}
}
return null;
}
function relationshipMap(xml: string) {
const relationships = new Map<string, string>();
for (const match of xml.matchAll(/<Relationship\b([^>]*)\/?\s*>/g)) {
const attributes = match[1];
const id = attributes.match(/\bId="([^"]+)"/)?.[1] ?? "";
const target = attributes.match(/\bTarget="([^"]+)"/)?.[1] ?? "";
if (id && target) relationships.set(id, xmlAttribute(target));
}
return relationships;
}
function contentType(bytes: Uint8Array, fileName: string) {
if (bytes[0] === 0x89 && bytes[1] === 0x50) return "image/png";
if (bytes[0] === 0xff && bytes[1] === 0xd8) return "image/jpeg";
if (bytes[0] === 0x47 && bytes[1] === 0x49 && bytes[2] === 0x46) {
return "image/gif";
}
if (String.fromCharCode(...bytes.slice(8, 12)) === "WEBP") return "image/webp";
const extension = path.extname(fileName).toLowerCase();
return extension === ".png"
? "image/png"
: extension === ".gif"
? "image/gif"
: extension === ".webp"
? "image/webp"
: "image/jpeg";
}
function resolveZipPath(base: string, target: string) {
return path.posix.normalize(path.posix.join(path.posix.dirname(base), target));
}
function parseDrawingImages(entries: Record<string, Uint8Array>) {
const images = new Map<string, PartnerBatchImage>();
const sheetRelationshipsXml = entries["xl/worksheets/_rels/sheet1.xml.rels"]
? strFromU8(entries["xl/worksheets/_rels/sheet1.xml.rels"])
: "";
const sheetRelationships = relationshipMap(sheetRelationshipsXml);
const sheetXml = entries["xl/worksheets/sheet1.xml"]
? strFromU8(entries["xl/worksheets/sheet1.xml"])
: "";
const drawingId = sheetXml.match(/<drawing\b[^>]*r:id="([^"]+)"/)?.[1] ?? "";
const drawingTarget = sheetRelationships.get(drawingId);
if (!drawingTarget) return images;
const drawingPath = resolveZipPath("xl/worksheets/sheet1.xml", drawingTarget);
const drawingXml = entries[drawingPath] ? strFromU8(entries[drawingPath]) : "";
const drawingRelationshipsPath = path.posix.join(
path.posix.dirname(drawingPath),
"_rels",
`${path.posix.basename(drawingPath)}.rels`,
);
const drawingRelationships = relationshipMap(
entries[drawingRelationshipsPath]
? strFromU8(entries[drawingRelationshipsPath])
: "",
);
for (const anchor of drawingXml.matchAll(
/<xdr:(?:oneCellAnchor|twoCellAnchor)\b[^>]*>[\s\S]*?<xdr:from>([\s\S]*?)<\/xdr:from>[\s\S]*?<a:blip\b[^>]*r:embed="([^"]+)"[\s\S]*?<\/xdr:(?:oneCellAnchor|twoCellAnchor)>/g,
)) {
const column = Number(anchor[1].match(/<xdr:col>(\d+)<\/xdr:col>/)?.[1]);
const row = Number(anchor[1].match(/<xdr:row>(\d+)<\/xdr:row>/)?.[1]);
const mediaTarget = drawingRelationships.get(anchor[2]);
if (!Number.isInteger(column) || !Number.isInteger(row) || !mediaTarget) continue;
const mediaPath = resolveZipPath(drawingPath, mediaTarget);
const bytes = entries[mediaPath];
if (!bytes) continue;
images.set(`${row + 1}:${column}`, {
bytes,
contentType: contentType(bytes, mediaPath),
fileName: path.posix.basename(mediaPath),
});
}
return images;
}
function parseRichValueImages(entries: Record<string, Uint8Array>) {
const images = new Map<string, PartnerBatchImage>();
const worksheetXml = entries["xl/worksheets/sheet1.xml"]
? strFromU8(entries["xl/worksheets/sheet1.xml"])
: "";
const metadataXml = entries["xl/metadata.xml"]
? strFromU8(entries["xl/metadata.xml"])
: "";
const richValueXml = entries["xl/richData/rdrichvalue.xml"]
? strFromU8(entries["xl/richData/rdrichvalue.xml"])
: "";
const richValueRelXml = entries["xl/richData/richValueRel.xml"]
? strFromU8(entries["xl/richData/richValueRel.xml"])
: "";
const richValueRelRelationships = relationshipMap(
entries["xl/richData/_rels/richValueRel.xml.rels"]
? strFromU8(entries["xl/richData/_rels/richValueRel.xml.rels"])
: "",
);
if (
!worksheetXml ||
!metadataXml ||
!richValueXml ||
!richValueRelXml ||
!richValueRelRelationships.size
) {
return images;
}
const valueMetadataXml =
metadataXml.match(/<valueMetadata\b[^>]*>([\s\S]*?)<\/valueMetadata>/)?.[1] ??
"";
const metadataToRichValue = [
...valueMetadataXml.matchAll(/<bk\b[^>]*>[\s\S]*?<rc\b[^>]*\bv="(\d+)"[^>]*\/>[\s\S]*?<\/bk>/g),
].map((match) => Number(match[1]));
const richValueToRelationship = [
...richValueXml.matchAll(/<rv\b[^>]*>([\s\S]*?)<\/rv>/g),
].map((match) => Number(match[1].match(/<v>(\d+)<\/v>/)?.[1] ?? -1));
const relationshipIds = [
...richValueRelXml.matchAll(/<rel\b[^>]*\br:id="([^"]+)"[^>]*\/>/g),
].map((match) => match[1]);
for (const cell of worksheetXml.matchAll(
/<c\b([^>]*)>[\s\S]*?<\/c>/g,
)) {
const reference = cell[1].match(/\br="([A-Z]+\d+)"/i)?.[1] ?? "";
const metadataIndex = Number(cell[1].match(/\bvm="(\d+)"/)?.[1] ?? 0);
const row = Number(reference.match(/\d+$/)?.[0] ?? 0);
const column = columnIndex(reference);
if (!reference || !metadataIndex || !row) continue;
const richValueIndex = metadataToRichValue[metadataIndex - 1];
const relationshipIndex = richValueToRelationship[richValueIndex];
const relationshipId = relationshipIds[relationshipIndex];
const mediaTarget = richValueRelRelationships.get(relationshipId);
if (!mediaTarget) continue;
const mediaPath = resolveZipPath(
"xl/richData/richValueRel.xml",
mediaTarget,
);
const bytes = entries[mediaPath];
if (!bytes) continue;
images.set(`${row}:${column}`, {
bytes,
contentType: contentType(bytes, mediaPath),
fileName: path.posix.basename(mediaPath),
});
}
return images;
}
function parseWpsCellImages(entries: Record<string, Uint8Array>) {
const images = new Map<string, PartnerBatchImage>();
const worksheetXml = entries["xl/worksheets/sheet1.xml"]
? strFromU8(entries["xl/worksheets/sheet1.xml"])
: "";
const cellImagesXml = entries["xl/cellimages.xml"]
? strFromU8(entries["xl/cellimages.xml"])
: "";
const relationships = relationshipMap(
entries["xl/_rels/cellimages.xml.rels"]
? strFromU8(entries["xl/_rels/cellimages.xml.rels"])
: "",
);
if (!worksheetXml || !cellImagesXml || !relationships.size) return images;
const imageIdToRelationship = new Map<string, string>();
for (const match of cellImagesXml.matchAll(
/<(?:etc:)?cellImage\b[^>]*>([\s\S]*?)<\/(?:etc:)?cellImage>/g,
)) {
const imageId = match[1].match(
/<(?:xdr:)?cNvPr\b[^>]*\bname="([^"]+)"/,
)?.[1];
const relationshipId = match[1].match(
/<(?:a:)?blip\b[^>]*\br:embed="([^"]+)"/,
)?.[1];
if (imageId && relationshipId) {
imageIdToRelationship.set(imageId, relationshipId);
}
}
for (const cell of worksheetXml.matchAll(/<c\b([^>]*)>([\s\S]*?)<\/c>/g)) {
const reference = cell[1].match(/\br="([A-Z]+\d+)"/i)?.[1] ?? "";
const imageId = decodeXml(cell[2]).match(/DISPIMG\("([^"]+)"/i)?.[1];
if (!reference || !imageId) continue;
const relationshipId = imageIdToRelationship.get(imageId);
const mediaTarget = relationshipId
? relationships.get(relationshipId)
: undefined;
if (!mediaTarget) continue;
const mediaPath = resolveZipPath("xl/cellimages.xml", mediaTarget);
const bytes = entries[mediaPath];
if (!bytes) continue;
const row = Number(reference.match(/\d+$/)?.[0] ?? 0);
if (!row) continue;
images.set(`${row}:${columnIndex(reference)}`, {
bytes,
contentType: contentType(bytes, mediaPath),
fileName: path.posix.basename(mediaPath),
});
}
return images;
}
function parseImages(entries: Record<string, Uint8Array>) {
const images = parseDrawingImages(entries);
for (const [cell, image] of parseRichValueImages(entries)) {
images.set(cell, image);
}
for (const [cell, image] of parseWpsCellImages(entries)) {
images.set(cell, image);
}
return images;
}
function valueAt(
row: string[],
mapping: Map<(typeof PARTNER_BATCH_HEADERS)[number], number>,
header: (typeof PARTNER_BATCH_HEADERS)[number],
) {
const column = mapping.get(header);
return column === undefined ? "" : String(row[column] ?? "").trim();
}
export function parsePartnerBatchWorkbook(input: ArrayBuffer | Uint8Array) {
const bytes = input instanceof Uint8Array ? input : new Uint8Array(input);
if (bytes.byteLength > PARTNER_BATCH_MAX_BYTES) {
throw new Error("批量回填表不能超过80MB");
}
const entries = unzipSync(bytes);
const worksheetBytes = entries["xl/worksheets/sheet1.xml"];
if (!worksheetBytes) throw new Error("Excel 中没有可读取的批量回填工作表");
const sharedXml = entries["xl/sharedStrings.xml"]
? strFromU8(entries["xl/sharedStrings.xml"])
: "";
const sharedStrings = [
...sharedXml.matchAll(/<si\b[^>]*>([\s\S]*?)<\/si>/g),
].map((match) => textNodes(match[1]));
const rows = parseWorksheet(strFromU8(worksheetBytes), sharedStrings);
const header = findHeader(rows);
if (!header) {
throw new Error("表格结构不正确,请使用本领取页面导出的批量回填表");
}
const images = parseImages(entries);
const publishScreenshotColumn = header.mapping.get("笔记截图")!;
const creatorScreenshotColumn = header.mapping.get(
"数据分析截图(单篇笔记数据分析截图)",
)!;
const result: PartnerBatchImportRow[] = [];
for (let index = header.rowIndex + 1; index < rows.length; index += 1) {
const row = rows[index];
const distributionId = valueAt(row, header.mapping, "_系统笔记ID");
if (!distributionId && !row.some((value) => String(value ?? "").trim())) continue;
result.push({
spreadsheetRow: index + 1,
sequence: valueAt(row, header.mapping, "序号(不能改)"),
title: valueAt(row, header.mapping, "标题"),
publishUrl: valueAt(row, header.mapping, "发布链接"),
distributionId,
originalPublishScreenshotKey: valueAt(
row,
header.mapping,
"_原笔记截图",
),
originalCreatorScreenshotKey: valueAt(
row,
header.mapping,
"_原数据分析截图",
),
publishScreenshot:
images.get(`${index + 1}:${publishScreenshotColumn}`) ?? null,
creatorScreenshot:
images.get(`${index + 1}:${creatorScreenshotColumn}`) ?? null,
});
}
if (result.length === 0) throw new Error("表格中没有可回填的笔记");
return result;
}

View File

@@ -1,12 +1,22 @@
import { env } from "cloudflare:workers";
import { getRuntimeEnv } from "./runtime-env";
const env = getRuntimeEnv();
function allowedOrigin(request: Request) {
const origin = request.headers.get("origin");
if (!origin) return null;
const portalOrigin = String(
const portalUrl = String(
(env as unknown as { KOC_PORTAL_URL?: string }).KOC_PORTAL_URL ?? "",
).replace(/\/$/, "");
return origin === portalOrigin || origin === "http://localhost:3000"
).trim();
let portalOrigin = "";
try {
portalOrigin = portalUrl ? new URL(portalUrl).origin : "";
} catch {
portalOrigin = "";
}
return [portalOrigin, "http://localhost:3000", "http://localhost:3001"].includes(
origin,
)
? origin
: null;
}

View File

@@ -1,22 +1,31 @@
import { hashText } from "./mvp-db";
import {
extractPublishUrl,
extractXhsPublishUrl,
platformFromPublishUrl,
safeHttpUrl,
type SupportedPlatform,
} from "./publish-url";
export { extractXhsPublishUrl } from "./publish-url";
export function accountFromPublishLink(input: string) {
const url = safeHttpUrl(extractXhsPublishUrl(input));
export function accountFromPublishLink(
input: string,
expectedPlatform?: SupportedPlatform,
) {
const extracted = expectedPlatform
? extractPublishUrl(input, expectedPlatform)
: extractXhsPublishUrl(input) || extractPublishUrl(input, "抖音");
const url = safeHttpUrl(extracted);
if (!url) return null;
const platform =
url.hostname.includes("xiaohongshu") || url.hostname.includes("xhslink")
? "小红书"
: "其他平台";
const platform = platformFromPublishUrl(url.toString());
if (!platform || (expectedPlatform && platform !== expectedPlatform)) return null;
const noteId =
url.pathname.match(
/\/(?:discovery\/item|explore)\/([A-Za-z0-9_-]{8,80})/,
)?.[1] ?? "";
platform === "抖音"
? url.pathname.match(/\/(?:video|note)\/([A-Za-z0-9_-]{8,80})/)?.[1] ?? ""
: url.pathname.match(
/\/(?:discovery\/item|explore)\/([A-Za-z0-9_-]{8,80})/,
)?.[1] ?? "";
const platformUid = `pending-${hashText(
noteId || `${url.origin}${url.pathname}`,
)}`;

View File

@@ -25,3 +25,44 @@ export function extractXhsPublishUrl(input: string) {
}
return "";
}
export type SupportedPlatform = "小红书" | "抖音";
function platformMatches(url: URL, platform: SupportedPlatform) {
const hostname = url.hostname.toLowerCase();
if (platform === "抖音") {
return hostname === "douyin.com" || hostname.endsWith(".douyin.com");
}
return (
hostname === "xiaohongshu.com" ||
hostname.endsWith(".xiaohongshu.com") ||
hostname === "xhslink.cn" ||
hostname.endsWith(".xhslink.cn")
);
}
export function extractPublishUrl(
input: string,
platform: SupportedPlatform = "小红书",
) {
const candidates =
input.match(/https?:\/\/[^\s<>"',。!?;:、【】()]+/gi) ?? [];
for (const candidate of candidates) {
const cleaned = candidate.replace(/[.,!?;:~)\]}]+$/g, "");
const url = safeHttpUrl(cleaned);
if (url && platformMatches(url, platform)) return url.toString();
}
return "";
}
export function extractAnyPublishUrl(input: string) {
return extractPublishUrl(input, "小红书") || extractPublishUrl(input, "抖音");
}
export function platformFromPublishUrl(input: string): SupportedPlatform | "" {
const url = safeHttpUrl(input);
if (!url) return "";
if (platformMatches(url, "小红书")) return "小红书";
if (platformMatches(url, "抖音")) return "抖音";
return "";
}

View File

@@ -13,6 +13,9 @@ export type RecoveryWorkbookRow = {
images: Array<{
column: number;
image: RecoveryWorkbookImage;
offsetX?: number;
maxWidth?: number;
maxHeight?: number;
}>;
hyperlinks?: Array<{
column: number;
@@ -25,6 +28,7 @@ type WorkbookOptions = {
headers: string[];
columnWidths: number[];
rows: RecoveryWorkbookRow[];
hiddenColumns?: number[];
};
const XML_HEADER = '<?xml version="1.0" encoding="UTF-8" standalone="yes"?>';
@@ -130,9 +134,17 @@ function imageDimensions(image: RecoveryWorkbookImage) {
return { width: 4, height: 3 };
}
function imageDisplaySize(image: RecoveryWorkbookImage) {
function imageDisplaySize(
image: RecoveryWorkbookImage,
maxWidth = 160,
maxHeight = 150,
) {
const dimensions = imageDimensions(image);
const scale = Math.min(160 / dimensions.width, 150 / dimensions.height, 1);
const scale = Math.min(
maxWidth / dimensions.width,
maxHeight / dimensions.height,
1,
);
return {
width: Math.max(28, Math.round(dimensions.width * scale)),
height: Math.max(28, Math.round(dimensions.height * scale)),
@@ -152,6 +164,14 @@ export function buildRecoveryWorkbook(options: WorkbookOptions) {
const imageEntries = options.rows.flatMap((row, rowIndex) =>
row.images.map((item) => ({ ...item, row: rowIndex + 1 })),
);
const imageCells = new Set<string>();
imageEntries.forEach((entry) => {
const key = `${entry.row}:${entry.column}`;
if (imageCells.has(key)) {
throw new Error("Excel 单元格内只能嵌入一张图片,请为每张图片分配独立列");
}
imageCells.add(key);
});
const hyperlinkEntries = options.rows.flatMap((row, rowIndex) =>
(row.hyperlinks ?? [])
.map((item) => ({
@@ -178,7 +198,7 @@ export function buildRecoveryWorkbook(options: WorkbookOptions) {
const reference = `${columnName(columnIndex)}${number}`;
const value = row.cells[columnIndex] ?? "";
if (imageColumns.has(columnIndex)) {
return inlineCell(reference, value || "见图", 4);
return inlineCell(reference, value, 4);
}
return typeof value === "number"
? numberCell(reference, value, 3)
@@ -196,17 +216,22 @@ export function buildRecoveryWorkbook(options: WorkbookOptions) {
const columns = options.headers
.map((_, index) => {
const width = Math.min(70, Math.max(8, options.columnWidths[index] ?? 14));
return `<col min="${index + 1}" max="${index + 1}" width="${width}" customWidth="1"/>`;
const hidden = options.hiddenColumns?.includes(index) ? ' hidden="1"' : "";
return `<col min="${index + 1}" max="${index + 1}" width="${width}" customWidth="1"${hidden}/>`;
})
.join("");
const drawingXml = imageEntries.length
? `${XML_HEADER}<xdr:wsDr xmlns:xdr="http://schemas.openxmlformats.org/drawingml/2006/spreadsheetDrawing" xmlns:a="http://schemas.openxmlformats.org/drawingml/2006/main" xmlns:r="http://schemas.openxmlformats.org/officeDocument/2006/relationships">${imageEntries
.map((entry, index) => {
const size = imageDisplaySize(entry.image);
const width = size.width * 9525;
const height = size.height * 9525;
return `<xdr:oneCellAnchor><xdr:from><xdr:col>${entry.column}</xdr:col><xdr:colOff>57150</xdr:colOff><xdr:row>${entry.row}</xdr:row><xdr:rowOff>57150</xdr:rowOff></xdr:from><xdr:ext cx="${width}" cy="${height}"/><xdr:pic><xdr:nvPicPr><xdr:cNvPr id="${index + 1}" name="${cleanXmlText(entry.image.description || `图片${index + 1}`)}"/><xdr:cNvPicPr><a:picLocks noChangeAspect="1"/></xdr:cNvPicPr></xdr:nvPicPr><xdr:blipFill><a:blip r:embed="rId${index + 1}"/><a:stretch><a:fillRect/></a:stretch></xdr:blipFill><xdr:spPr><a:prstGeom prst="rect"><a:avLst/></a:prstGeom></xdr:spPr></xdr:pic><xdr:clientData/></xdr:oneCellAnchor>`;
const size = imageDisplaySize(
entry.image,
entry.maxWidth ?? 160,
entry.maxHeight ?? 150,
);
const offsetX = entry.offsetX ?? 6;
const offsetY = 6;
return `<xdr:twoCellAnchor editAs="twoCell"><xdr:from><xdr:col>${entry.column}</xdr:col><xdr:colOff>${offsetX * 9525}</xdr:colOff><xdr:row>${entry.row}</xdr:row><xdr:rowOff>${offsetY * 9525}</xdr:rowOff></xdr:from><xdr:to><xdr:col>${entry.column}</xdr:col><xdr:colOff>${(offsetX + size.width) * 9525}</xdr:colOff><xdr:row>${entry.row}</xdr:row><xdr:rowOff>${(offsetY + size.height) * 9525}</xdr:rowOff></xdr:to><xdr:pic><xdr:nvPicPr><xdr:cNvPr id="${index + 1}" name="${cleanXmlText(entry.image.description || `图片${index + 1}`)}"/><xdr:cNvPicPr><a:picLocks noChangeAspect="1"/></xdr:cNvPicPr></xdr:nvPicPr><xdr:blipFill><a:blip r:embed="rId${index + 1}"/><a:stretch><a:fillRect/></a:stretch></xdr:blipFill><xdr:spPr><a:prstGeom prst="rect"><a:avLst/></a:prstGeom></xdr:spPr></xdr:pic><xdr:clientData/></xdr:twoCellAnchor>`;
})
.join("")}</xdr:wsDr>`
: "";
@@ -227,7 +252,10 @@ export function buildRecoveryWorkbook(options: WorkbookOptions) {
const imageContentTypes = [...imageFormats.entries()]
.map(([extension, contentType]) => `<Default Extension="${extension}" ContentType="${contentType}"/>`)
.join("");
const contentTypes = `${XML_HEADER}<Types xmlns="http://schemas.openxmlformats.org/package/2006/content-types"><Default Extension="rels" ContentType="application/vnd.openxmlformats-package.relationships+xml"/><Default Extension="xml" ContentType="application/xml"/>${imageContentTypes}<Override PartName="/xl/workbook.xml" ContentType="application/vnd.openxmlformats-officedocument.spreadsheetml.sheet.main+xml"/><Override PartName="/xl/worksheets/sheet1.xml" ContentType="application/vnd.openxmlformats-officedocument.spreadsheetml.worksheet+xml"/><Override PartName="/xl/styles.xml" ContentType="application/vnd.openxmlformats-officedocument.spreadsheetml.styles+xml"/>${imageEntries.length ? '<Override PartName="/xl/drawings/drawing1.xml" ContentType="application/vnd.openxmlformats-officedocument.drawing+xml"/>' : ""}<Override PartName="/docProps/core.xml" ContentType="application/vnd.openxmlformats-package.core-properties+xml"/><Override PartName="/docProps/app.xml" ContentType="application/vnd.openxmlformats-officedocument.extended-properties+xml"/></Types>`;
const drawingContentType = imageEntries.length
? '<Override PartName="/xl/drawings/drawing1.xml" ContentType="application/vnd.openxmlformats-officedocument.drawing+xml"/>'
: "";
const contentTypes = `${XML_HEADER}<Types xmlns="http://schemas.openxmlformats.org/package/2006/content-types"><Default Extension="rels" ContentType="application/vnd.openxmlformats-package.relationships+xml"/><Default Extension="xml" ContentType="application/xml"/>${imageContentTypes}<Override PartName="/xl/workbook.xml" ContentType="application/vnd.openxmlformats-officedocument.spreadsheetml.sheet.main+xml"/><Override PartName="/xl/worksheets/sheet1.xml" ContentType="application/vnd.openxmlformats-officedocument.spreadsheetml.worksheet+xml"/><Override PartName="/xl/styles.xml" ContentType="application/vnd.openxmlformats-officedocument.spreadsheetml.styles+xml"/>${drawingContentType}<Override PartName="/docProps/core.xml" ContentType="application/vnd.openxmlformats-package.core-properties+xml"/><Override PartName="/docProps/app.xml" ContentType="application/vnd.openxmlformats-officedocument.extended-properties+xml"/></Types>`;
const hyperlinkRelationshipOffset = imageEntries.length ? 2 : 1;
const hyperlinksXml = hyperlinkEntries.length
? `<hyperlinks>${hyperlinkEntries
@@ -266,7 +294,9 @@ export function buildRecoveryWorkbook(options: WorkbookOptions) {
}
if (imageEntries.length) {
files["xl/drawings/drawing1.xml"] = strToU8(drawingXml);
files["xl/drawings/_rels/drawing1.xml.rels"] = strToU8(drawingRelationships);
files["xl/drawings/_rels/drawing1.xml.rels"] = strToU8(
drawingRelationships,
);
imageEntries.forEach((entry, index) => {
const format = imageFormat(entry.image.contentType, entry.image.bytes);
files[`xl/media/image${index + 1}.${format.extension}`] = entry.image.bytes;

390
lib/resource-import.ts Normal file
View File

@@ -0,0 +1,390 @@
import { strFromU8, unzipSync } from "fflate";
export const RESOURCE_IMPORT_MAX_ROWS = 10_000;
export const RESOURCE_IMPORT_MAX_BYTES = 20 * 1024 * 1024;
export type ResourceImportRow = {
rowNumber: number;
platform: string;
nickname: string;
publicAccountId: string;
profileUrl: string;
ipLocation: string;
followers: number;
followersResolved: boolean;
gender: "" | "男" | "女";
bio: string;
tags: string[];
cooperationSource: string;
errors: string[];
};
const HEADER_ALIASES = {
profileUrl: ["账号链接", "账号主页", "账号主页链接", "主页链接"],
nickname: ["账号昵称", "账号名称", "昵称"],
publicAccountId: ["账号ID", "账号号", "小红书号", "抖音号"],
ipLocation: ["IP属地", "IP地址", "IP地区", "IP所在地"],
followers: ["粉丝数", "粉丝", "粉丝量"],
gender: ["性别"],
bio: ["简介", "账号简介", "个人简介"],
tags: ["标签", "账号标签"],
cooperationSource: ["合作来源", "资源来源", "历史合作来源"],
} as const;
type CanonicalHeader = keyof typeof HEADER_ALIASES;
function decodeXml(value: string) {
return value
.replace(/<[^>]+>/g, "")
.replace(/&lt;/g, "<")
.replace(/&gt;/g, ">")
.replace(/&quot;/g, '"')
.replace(/&apos;/g, "'")
.replace(/&amp;/g, "&")
.replace(/&#(\d+);/g, (_, code) => String.fromCodePoint(Number(code)))
.replace(/&#x([0-9a-f]+);/gi, (_, code) =>
String.fromCodePoint(Number.parseInt(code, 16)),
);
}
function textNodes(xml: string) {
return [...xml.matchAll(/<t\b[^>]*>([\s\S]*?)<\/t>/g)]
.map((match) => decodeXml(match[1]))
.join("");
}
function columnIndex(reference: string) {
const letters = reference.match(/^[A-Z]+/i)?.[0]?.toUpperCase() ?? "";
let result = 0;
for (const letter of letters) result = result * 26 + letter.charCodeAt(0) - 64;
return Math.max(0, result - 1);
}
function parseWorksheet(xml: string, sharedStrings: string[]) {
const rows: string[][] = [];
for (const rowMatch of xml.matchAll(/<row\b([^>]*)>([\s\S]*?)<\/row>/g)) {
const rowAttributes = rowMatch[1];
const rowNumber = Number(rowAttributes.match(/\br="(\d+)"/)?.[1] ?? rows.length + 1);
const values: string[] = [];
for (const cellMatch of rowMatch[2].matchAll(
/<c\b([^>]*?)(?:\/>|>([\s\S]*?)<\/c>)/g,
)) {
const attributes = cellMatch[1];
const body = cellMatch[2] ?? "";
const reference = attributes.match(/\br="([A-Z]+\d+)"/i)?.[1] ?? "A1";
const type = attributes.match(/\bt="([^"]+)"/)?.[1] ?? "";
const rawValue = body.match(/<v>([\s\S]*?)<\/v>/)?.[1] ?? "";
let value = "";
if (type === "s") value = sharedStrings[Number(rawValue)] ?? "";
else if (type === "inlineStr") value = textNodes(body);
else if (type === "b") value = rawValue === "1" ? "TRUE" : "FALSE";
else value = decodeXml(rawValue);
values[columnIndex(reference)] = value.trim();
}
while (rows.length < rowNumber - 1) rows.push([]);
rows[rowNumber - 1] = values;
}
return rows;
}
function parseXlsx(bytes: Uint8Array) {
const entries = unzipSync(bytes);
const sharedXml = entries["xl/sharedStrings.xml"]
? strFromU8(entries["xl/sharedStrings.xml"])
: "";
const sharedStrings = [...sharedXml.matchAll(/<si\b[^>]*>([\s\S]*?)<\/si>/g)].map(
(match) => textNodes(match[1]),
);
const sheets = Object.keys(entries)
.filter((name) => /^xl\/worksheets\/sheet\d+\.xml$/.test(name))
.sort((left, right) => left.localeCompare(right, undefined, { numeric: true }));
if (sheets.length === 0) throw new Error("Excel 中没有可读取的工作表");
return sheets.map((name) => parseWorksheet(strFromU8(entries[name]), sharedStrings));
}
function parseCsv(text: string) {
const rows: string[][] = [];
let row: string[] = [];
let cell = "";
let quoted = false;
for (let index = 0; index < text.length; index += 1) {
const char = text[index];
if (quoted) {
if (char === '"' && text[index + 1] === '"') {
cell += '"';
index += 1;
} else if (char === '"') quoted = false;
else cell += char;
} else if (char === '"') quoted = true;
else if (char === ",") {
row.push(cell.trim());
cell = "";
} else if (char === "\n" || char === "\r") {
if (char === "\r" && text[index + 1] === "\n") index += 1;
row.push(cell.trim());
if (row.some(Boolean)) rows.push(row);
row = [];
cell = "";
} else cell += char;
}
row.push(cell.trim());
if (row.some(Boolean)) rows.push(row);
return rows;
}
function normalizeHeader(value: string) {
return value
.replace(/[\s_\-()]/g, "")
.replace(/必填|选填/g, "")
.toLocaleLowerCase("zh-CN");
}
function canonicalHeader(value: string): CanonicalHeader | null {
const normalized = normalizeHeader(value);
for (const [key, aliases] of Object.entries(HEADER_ALIASES)) {
if (aliases.some((alias) => normalizeHeader(alias) === normalized)) {
return key as CanonicalHeader;
}
}
return null;
}
function findHeader(rows: string[][]) {
for (let index = 0; index < Math.min(rows.length, 12); index += 1) {
const mapping = new Map<CanonicalHeader, number>();
rows[index].forEach((cell, column) => {
const header = canonicalHeader(cell);
if (header && !mapping.has(header)) mapping.set(header, column);
});
if (mapping.has("profileUrl")) {
return { rowIndex: index, mapping };
}
}
return null;
}
export function normalizeProfileUrl(value: string) {
const extracted = value.match(/https?:\/\/[^\s,;]+/i)?.[0] ?? value.trim();
if (!extracted) return "";
try {
const url = new URL(extracted);
if (!(["http:", "https:"].includes(url.protocol))) return "";
url.protocol = "https:";
url.hostname = url.hostname.toLowerCase();
url.search = "";
url.hash = "";
url.pathname = url.pathname.replace(/\/+$/, "") || "/";
return url.toString().replace(/\/$/, "");
} catch {
return "";
}
}
export function platformFromProfileUrl(profileUrl: string) {
if (!profileUrl) return "";
try {
const url = new URL(profileUrl);
if (
(url.hostname === "xiaohongshu.com" ||
url.hostname.endsWith(".xiaohongshu.com")) &&
/^\/user\/profile\/[^/]+/i.test(url.pathname)
) {
return "小红书";
}
if (
((url.hostname === "douyin.com" || url.hostname.endsWith(".douyin.com")) &&
/^\/user\/[^/]+/i.test(url.pathname)) ||
((url.hostname === "iesdouyin.com" ||
url.hostname.endsWith(".iesdouyin.com")) &&
/^\/share\/user\/[^/]+/i.test(url.pathname))
) {
return "抖音";
}
} catch {
// URL validation is reported by normalizeRows.
}
return "";
}
function valueAt(row: string[], mapping: Map<CanonicalHeader, number>, key: CanonicalHeader) {
const index = mapping.get(key);
return index === undefined ? "" : String(row[index] ?? "").trim();
}
export function parseResourceFollowers(value: string) {
const normalized = value.trim().replace(/[,\s]/g, "").replace(/\+$/, "");
if (!normalized) return { value: 0, resolved: false, valid: true };
const match = normalized.match(/^(\d+(?:\.\d+)?)(万|w|W|千|k|K)?$/);
if (!match) return { value: 0, resolved: false, valid: false };
const multiplier =
match[2] === "万" || match[2]?.toLowerCase() === "w"
? 10_000
: match[2] === "千" || match[2]?.toLowerCase() === "k"
? 1_000
: 1;
return {
value: Math.round(Number(match[1]) * multiplier),
resolved: true,
valid: true,
};
}
export function normalizeResourceGender(value: string) {
const normalized = value.trim().toLocaleLowerCase("zh-CN");
if (!normalized || ["未知", "未填写", "待识别", "unknown"].includes(normalized)) {
return { value: "" as const, valid: true };
}
if (["男", "男性", "male", "m"].includes(normalized)) {
return { value: "男" as const, valid: true };
}
if (["女", "女性", "female", "f"].includes(normalized)) {
return { value: "女" as const, valid: true };
}
return { value: "" as const, valid: false };
}
export function normalizeResourceTags(value: string | string[]) {
const source = Array.isArray(value) ? value.join(",") : value;
return [
...new Set(
source
.split(/[,,、;|]/)
.map((item) => item.trim().replace(/^#+/, ""))
.filter(Boolean),
),
];
}
export function resourceImportMissingFields(
row: Pick<
ResourceImportRow,
| "nickname"
| "publicAccountId"
| "ipLocation"
| "followersResolved"
| "gender"
| "bio"
| "tags"
>,
) {
const missing: string[] = [];
if (!row.nickname.trim()) missing.push("nickname");
if (!row.publicAccountId.trim()) missing.push("publicAccountId");
if (!row.ipLocation.trim() || row.ipLocation.trim() === "待识别") {
missing.push("ipLocation");
}
if (!row.followersResolved) missing.push("followers");
if (!row.gender) missing.push("gender");
if (!row.bio.trim()) missing.push("bio");
return missing;
}
function normalizeRows(rows: string[][]) {
const header = findHeader(rows);
if (!header) {
throw new Error("没有找到模板表头,请使用系统提供的导入模板");
}
const result: ResourceImportRow[] = [];
for (let index = header.rowIndex + 1; index < rows.length; index += 1) {
const source = rows[index];
if (!source.some((cell) => String(cell ?? "").trim())) continue;
const rawProfileUrl = valueAt(source, header.mapping, "profileUrl");
const profileUrl = normalizeProfileUrl(rawProfileUrl);
const platform = platformFromProfileUrl(profileUrl);
const rawFollowers = valueAt(source, header.mapping, "followers");
const parsedFollowers = parseResourceFollowers(rawFollowers);
const parsedGender = normalizeResourceGender(
valueAt(source, header.mapping, "gender"),
);
const tags = normalizeResourceTags(valueAt(source, header.mapping, "tags"));
const ipLocation = valueAt(source, header.mapping, "ipLocation");
const errors: string[] = [];
if (!rawProfileUrl) errors.push("账号主页不能为空");
else if (!profileUrl) errors.push("账号主页链接格式不正确");
else if (!platform) errors.push("当前自动解析仅支持小红书或抖音账号主页");
if (!parsedFollowers.valid) {
errors.push("粉丝数格式不正确,请填写数字或如 1.3万、10+");
}
if (!parsedGender.valid) {
errors.push("性别格式不正确,请填写男、女或留空");
}
if (tags.length > 5) {
errors.push("单个账号最多填写 5 个标签,请用逗号分隔");
}
if (/^\d+$/.test(ipLocation)) {
errors.push("IP属地格式不正确请填写省份、地区或国家名称");
}
result.push({
rowNumber: index + 1,
platform,
nickname: valueAt(source, header.mapping, "nickname"),
publicAccountId: valueAt(source, header.mapping, "publicAccountId"),
profileUrl,
ipLocation,
followers: parsedFollowers.value,
followersResolved: parsedFollowers.resolved,
gender: parsedGender.value,
bio: valueAt(source, header.mapping, "bio"),
tags: tags.slice(0, 5),
cooperationSource: valueAt(source, header.mapping, "cooperationSource"),
errors,
});
}
if (result.length === 0) throw new Error("表格中没有可导入的账号数据");
if (result.length > RESOURCE_IMPORT_MAX_ROWS) {
throw new Error(`单次最多导入 ${RESOURCE_IMPORT_MAX_ROWS} 个账号`);
}
return result;
}
export function parseResourceImportFile(fileName: string, bytes: Uint8Array) {
const extension = fileName.toLocaleLowerCase().split(".").pop();
const workbooks =
extension === "csv"
? [parseCsv(new TextDecoder("utf-8").decode(bytes).replace(/^\uFEFF/, ""))]
: extension === "xlsx"
? parseXlsx(bytes)
: null;
if (!workbooks) throw new Error("仅支持 .xlsx 或 .csv 文件");
for (const rows of workbooks) {
if (findHeader(rows)) return normalizeRows(rows);
}
throw new Error("没有找到模板表头,请使用系统提供的导入模板");
}
function shortHash(value: string) {
let hash = 2166136261;
for (const character of value) {
hash ^= character.charCodeAt(0);
hash = Math.imul(hash, 16777619);
}
return Math.abs(hash >>> 0).toString(36);
}
export function resourcePlatformUid(row: Pick<ResourceImportRow, "platform" | "profileUrl" | "publicAccountId">) {
if (row.profileUrl) {
try {
const url = new URL(row.profileUrl);
const candidate =
url.pathname.match(/\/user\/profile\/([^/]+)/i)?.[1] ??
url.pathname.match(/\/(?:user|profile)\/([^/]+)/i)?.[1] ??
url.pathname.split("/").filter(Boolean).at(-1);
if (candidate && candidate.length >= 3) return candidate;
} catch {
// Validation already reports malformed profile links.
}
}
const identity = row.publicAccountId || row.profileUrl;
return `manual-${shortHash(`${row.platform}:${identity}`)}`;
}
export function mergeCooperationSources(existing: string, incoming: string) {
return [
...new Set(
`${existing}${incoming}`
.split(/[、,;|]/)
.map((item) => item.trim())
.filter(Boolean),
),
].join("、");
}

34
lib/result-screenshots.ts Normal file
View File

@@ -0,0 +1,34 @@
export const MAX_RESULT_SCREENSHOTS = 9;
function isResultScreenshotKey(value: unknown): value is string {
return (
typeof value === "string" &&
value.startsWith("task-results/") &&
value.length <= 512
);
}
export function parseResultScreenshotKeys(value: unknown): string[] {
const text = String(value ?? "").trim();
if (!text) return [];
if (isResultScreenshotKey(text)) return [text];
try {
const parsed = JSON.parse(text) as unknown;
if (!Array.isArray(parsed)) return [];
return [...new Set(parsed.filter(isResultScreenshotKey))].slice(
0,
MAX_RESULT_SCREENSHOTS,
);
} catch {
return [];
}
}
export function serializeResultScreenshotKeys(keys: string[]) {
return JSON.stringify(
[...new Set(keys.filter(isResultScreenshotKey))].slice(
0,
MAX_RESULT_SCREENSHOTS,
),
);
}

51
lib/runtime-env.ts Normal file
View File

@@ -0,0 +1,51 @@
export type RuntimeEnv = {
DATABASE_URL?: string;
MYSQL_HOST?: string;
MYSQL_PORT?: string;
MYSQL_USER?: string;
MYSQL_PASSWORD?: string;
MYSQL_DATABASE?: string;
UPLOAD_DIR?: string;
APP_ORIGIN?: string;
KOC_PORTAL_URL?: string;
SUPER_ADMIN_USERNAME?: string;
SUPER_ADMIN_PASSWORD?: string;
ADMIN_INTERNAL_TOKEN?: string;
KOC_MCP_API_KEY?: string;
KOC_LOOP_MCP_API_KEY?: string;
MCP_API_KEY?: string;
FEISHU_APP_ID?: string;
FEISHU_APP_SECRET?: string;
AI_TOOL_CENTER_MCP_URL?: string;
AI_TOOL_CENTER_MCP_KEY?: string;
COLLECTION_MCP_URL?: string;
COLLECTION_MCP_KEY?: string;
WECOM_CORP_ID?: string;
WECOM_AGENT_ID?: string;
WECOM_SECRET?: string;
WECOM_ROBOT_WEBHOOK?: string;
WECOM_NOTIFY_DUE_DAYS?: string;
WECOM_NOTIFY_ENABLED?: string;
SEED_DEMO_DATA?: string;
ENABLE_SCHEDULER?: string;
};
export function getRuntimeEnv(): RuntimeEnv {
return process.env as RuntimeEnv;
}
export function getDatabaseUrl() {
const env = getRuntimeEnv();
if (env.DATABASE_URL) return env.DATABASE_URL;
const host = env.MYSQL_HOST ?? "127.0.0.1";
const port = env.MYSQL_PORT ?? "3306";
const user = encodeURIComponent(env.MYSQL_USER ?? "koc");
const password = encodeURIComponent(env.MYSQL_PASSWORD ?? "");
const database = env.MYSQL_DATABASE ?? "koc_loop";
return `mysql://${user}:${password}@${host}:${port}/${database}`;
}
export function isEnabled(value: string | undefined, defaultValue = false) {
if (value === undefined || value === "") return defaultValue;
return !["0", "false", "no", "off"].includes(value.toLowerCase());
}

65
lib/scheduler.ts Normal file
View File

@@ -0,0 +1,65 @@
import cron, { type ScheduledTask } from "node-cron";
import { backfillAccountProfiles } from "./account-enrichment-service";
import { runScheduledCollections } from "./collection-service";
import { withDatabaseLock } from "./database";
import {
resolveCollectionMcpConfig,
type CollectionMcpBindings,
} from "./mcp-collection-client";
import { ensureSchema, getRawDb } from "./mvp-db";
import { getRuntimeEnv, isEnabled } from "./runtime-env";
import {
resolveWecomConfig,
type WecomBindings,
} from "./wecom-client";
import { runDueSoonWecomNotifications } from "./wecom-notifier-service";
declare global {
var __kocLoopScheduler: ScheduledTask | undefined;
}
async function runDailyJob() {
await withDatabaseLock("koc-loop-daily-collection", 0, async () => {
await ensureSchema();
const db = getRawDb();
const env = getRuntimeEnv();
const config = resolveCollectionMcpConfig(
env as unknown as CollectionMcpBindings,
);
const collections = await runScheduledCollections(db, Date.now(), config);
const accounts = await backfillAccountProfiles(db, config, 10);
let wecom: Awaited<ReturnType<typeof runDueSoonWecomNotifications>> | null =
null;
if (isEnabled(env.WECOM_NOTIFY_ENABLED, true)) {
const wecomConfig = resolveWecomConfig(env as unknown as WecomBindings);
if (
wecomConfig.robotWebhook ||
(wecomConfig.corpId && wecomConfig.agentId && wecomConfig.secret)
) {
try {
wecom = await runDueSoonWecomNotifications(db, wecomConfig);
} catch (error) {
console.error("[KOC LOOP] wecom notify failed", error);
}
}
}
console.info("[KOC LOOP] daily scheduler completed", {
collections,
accounts,
wecom,
});
});
}
export function startScheduler() {
if (!isEnabled(getRuntimeEnv().ENABLE_SCHEDULER, true)) return;
if (globalThis.__kocLoopScheduler) return;
globalThis.__kocLoopScheduler = cron.schedule(
"0 9 * * *",
() => void runDailyJob().catch((error) => {
console.error("[KOC LOOP] daily scheduler failed", error);
}),
{ timezone: "Asia/Shanghai", noOverlap: true },
);
console.info("[KOC LOOP] scheduler enabled at 09:00 Asia/Shanghai");
}

View File

@@ -10,6 +10,18 @@ export type CreateDistributionTaskInput = {
name: string;
brand: string;
dueAt: string;
platform?: "小红书" | "抖音";
contentFormat?: "image_text" | "video";
};
export type CreateScreenshotTaskInput = {
name: string;
brand: string;
dueAt: string;
keyword: string;
instructions: string;
quantity: number;
exampleImageKey?: string;
};
export type DistributionTaskCreation = {
@@ -23,6 +35,18 @@ export type DistributionTaskCreation = {
sheetId: string;
sheetName: string;
sourceUrl: string;
platform: "小红书" | "抖音";
contentFormat: "image_text" | "video";
};
export type ScreenshotTaskCreation = {
created: true;
taskId: string;
shareToken: string;
name: string;
brand: string;
dueAt: string;
quantity: number;
};
type TaskRow = {
@@ -35,6 +59,8 @@ type TaskRow = {
source_url: string;
source_sheet_id: string;
source_sheet_name: string;
platform: "小红书" | "抖音";
content_format: "image_text" | "video";
};
function normalizedValue(value: string) {
@@ -79,17 +105,27 @@ async function findExistingTask(
return db
.prepare(
`SELECT id, share_token, name, brand, due_at, quantity,
source_url, source_sheet_id, source_sheet_name
source_url, source_sheet_id, source_sheet_name,
platform, content_format
FROM tasks
WHERE name = ?
AND brand = ?
AND due_at = ?
AND source_url = ?
AND platform = ?
AND content_format = ?
AND status IN ('active', 'importing')
ORDER BY created_at DESC
LIMIT 1`,
)
.bind(input.name, input.brand, input.dueAt, sourceUrl)
.bind(
input.name,
input.brand,
input.dueAt,
sourceUrl,
input.platform ?? "小红书",
input.contentFormat ?? "image_text",
)
.first<TaskRow>();
}
@@ -104,9 +140,10 @@ async function insertTaskFromSource(
.prepare(
`INSERT INTO tasks
(id, name, brand, quantity, claimed_quantity, due_at, status,
platform, content_format,
source_url, source_sheet_id, source_sheet_name, source_synced_at,
share_token)
VALUES (?, ?, ?, ?, 0, ?, 'importing', ?, ?, ?, ?, ?)`,
VALUES (?, ?, ?, ?, 0, ?, 'importing', ?, ?, ?, ?, ?, ?, ?)`,
)
.bind(
taskId,
@@ -114,6 +151,8 @@ async function insertTaskFromSource(
input.brand,
source.rows.length,
input.dueAt,
input.platform ?? "小红书",
input.contentFormat ?? "image_text",
source.url,
source.sheetId,
source.sheetName,
@@ -129,11 +168,15 @@ async function insertTaskFromSource(
...image,
key: `content-assets/${taskId}/${contentId}/${image.index}`,
}));
const videoAssets = row.videos.map((video) => ({
...video,
key: `content-videos/${taskId}/${contentId}/${video.index}`,
}));
return db
.prepare(
`INSERT INTO contents
(id, task_id, title, body, image_assets, status, source, source_row)
VALUES (?, ?, ?, ?, ?, 'available', ?, ?)`,
(id, task_id, title, body, image_assets, video_assets, status, source, source_row)
VALUES (?, ?, ?, ?, ?, ?, 'available', ?, ?)`,
)
.bind(
contentId,
@@ -141,6 +184,7 @@ async function insertTaskFromSource(
row.title,
row.body,
JSON.stringify(imageAssets),
JSON.stringify(videoAssets),
`飞书 · ${source.sheetName}`,
row.sourceRow,
);
@@ -169,11 +213,13 @@ export async function createDistributionTask(
options: { deduplicate?: boolean } = {},
): Promise<DistributionTaskCreation> {
await ensureSchema();
const input = {
const input: Required<CreateDistributionTaskInput> = {
feishuUrl: normalizedFeishuUrl(rawInput.feishuUrl),
name: normalizedValue(rawInput.name),
brand: normalizedValue(rawInput.brand),
dueAt: normalizedDueDate(rawInput.dueAt),
platform: rawInput.platform === "抖音" ? "抖音" : "小红书",
contentFormat: rawInput.contentFormat === "video" ? "video" : "image_text",
};
if (!input.feishuUrl || !input.name || !input.brand) {
throw new Error("请补全飞书链接、任务名称和品牌/项目");
@@ -193,11 +239,19 @@ export async function createDistributionTask(
sheetId: existing.source_sheet_id,
sheetName: existing.source_sheet_name,
sourceUrl: existing.source_url,
platform: existing.platform,
contentFormat: existing.content_format,
};
}
}
const source = await readFeishuSource(input.feishuUrl, bindings);
if (
input.contentFormat === "video" &&
source.rows.some((row) => row.videos.length === 0)
) {
throw new Error("视频任务中存在未识别到视频的内容行,请检查飞书“视频”列");
}
const inserted = await insertTaskFromSource(source, input);
return {
created: true,
@@ -210,6 +264,96 @@ export async function createDistributionTask(
sheetId: source.sheetId,
sheetName: source.sheetName,
sourceUrl: source.url,
platform: input.platform,
contentFormat: input.contentFormat,
};
}
export async function createScreenshotTask(
rawInput: CreateScreenshotTaskInput,
): Promise<ScreenshotTaskCreation> {
await ensureSchema();
const input = {
name: normalizedValue(rawInput.name),
brand: normalizedValue(rawInput.brand),
dueAt: normalizedDueDate(rawInput.dueAt),
keyword: normalizedValue(rawInput.keyword),
instructions: normalizedValue(rawInput.instructions),
quantity: Math.floor(Number(rawInput.quantity)),
exampleImageKey: normalizedValue(rawInput.exampleImageKey ?? ""),
};
if (!input.name || !input.brand || !input.keyword || !input.instructions) {
throw new Error("请补全任务名称、品牌/项目、搜索关键词和任务说明");
}
if (!Number.isInteger(input.quantity) || input.quantity < 1 || input.quantity > 500) {
throw new Error("任务数量需为 1—500 份");
}
if (input.exampleImageKey && !input.exampleImageKey.startsWith("task-assets/")) {
throw new Error("示例截图无效,请重新上传");
}
const db = getRawDb();
const taskId = uid("task");
const shareToken = crypto.randomUUID().replaceAll("-", "");
const imageAssets = input.exampleImageKey
? JSON.stringify([
{ index: 1, key: input.exampleImageKey, width: null, height: null },
])
: "[]";
await db
.prepare(
`INSERT INTO tasks
(id, name, brand, quantity, claimed_quantity, due_at, status,
task_type, source_url, source_sheet_id, source_sheet_name,
share_token, collection_days)
VALUES (?, ?, ?, ?, 0, ?, 'active', 'screenshot_collect', '', '', '', ?, '[]')`,
)
.bind(
taskId,
input.name,
input.brand,
input.quantity,
input.dueAt,
shareToken,
)
.run();
try {
const statements = Array.from({ length: input.quantity }, (_, index) =>
db
.prepare(
`INSERT INTO contents
(id, task_id, title, body, image_assets, video_assets, status, source, source_row)
VALUES (?, ?, ?, ?, ?, '[]', 'available', '截图回收任务', ?)`,
)
.bind(
uid("content"),
taskId,
input.keyword,
input.instructions,
imageAssets,
index + 1,
),
);
for (let index = 0; index < statements.length; index += 100) {
await db.batch(statements.slice(index, index + 100));
}
} catch (error) {
await db.batch([
db.prepare("DELETE FROM contents WHERE task_id = ?").bind(taskId),
db.prepare("DELETE FROM tasks WHERE id = ?").bind(taskId),
]);
throw error;
}
return {
created: true,
taskId,
shareToken,
name: input.name,
brand: input.brand,
dueAt: input.dueAt,
quantity: input.quantity,
};
}

View File

@@ -1,6 +1,8 @@
import { env } from "cloudflare:workers";
import { getRuntimeEnv } from "./runtime-env";
import { ensureSchema, getRawDb } from "./mvp-db";
const env = getRuntimeEnv();
export type UserRole = "super_admin" | "admin" | "user";
export type AuthUser = {
@@ -15,7 +17,7 @@ export type RequestPrincipal =
const SESSION_COOKIE = "koc_session";
const SESSION_MAX_AGE_SECONDS = 60 * 60 * 24 * 7;
// Cloudflare Workers caps PBKDF2 at 100,000 iterations.
// Keep existing password records compatible while using a production-safe cost.
const PASSWORD_ITERATIONS = 100_000;
function bytesToHex(bytes: Uint8Array) {
@@ -198,6 +200,15 @@ export function clearSessionCookie(secure = true) {
.join("; ");
}
export function requestUsesHttps(request: Request) {
const forwardedProtocol = request.headers
.get("x-forwarded-proto")
?.split(",")[0]
?.trim()
.toLowerCase();
return forwardedProtocol === "https" || new URL(request.url).protocol === "https:";
}
export async function createSession(userId: string) {
const tokenBytes = new Uint8Array(32);
crypto.getRandomValues(tokenBytes);

45
lib/video-file.ts Normal file
View File

@@ -0,0 +1,45 @@
const MP4_BRANDS = new Set([
"avc1",
"dash",
"isom",
"M4A ",
"M4B ",
"M4P ",
"M4V ",
"mp41",
"mp42",
"MSNV",
]);
function fourCharacters(bytes: Uint8Array, offset: number) {
return String.fromCharCode(...bytes.slice(offset, offset + 4));
}
function isMp4Brand(brand: string) {
return (
MP4_BRANDS.has(brand) ||
/^iso[2-9]$/.test(brand) ||
/^3g[2p]$/.test(brand.slice(0, 3))
);
}
/** Rejects HTML/JSON/error payloads and non-MP4 containers before download. */
export function hasMp4FileSignature(input: ArrayBuffer | Uint8Array) {
const bytes = input instanceof Uint8Array ? input : new Uint8Array(input);
if (bytes.byteLength < 12 || fourCharacters(bytes, 4) !== "ftyp") {
return false;
}
const declaredSize = new DataView(
bytes.buffer,
bytes.byteOffset,
bytes.byteLength,
).getUint32(0);
const boxEnd = Math.min(
bytes.byteLength,
declaredSize >= 12 ? declaredSize : bytes.byteLength,
);
for (let offset = 8; offset + 4 <= boxEnd; offset += 4) {
if (isMp4Brand(fourCharacters(bytes, offset))) return true;
}
return false;
}

219
lib/wecom-client.ts Normal file
View File

@@ -0,0 +1,219 @@
const WECOM_API_ORIGIN = "https://qyapi.weixin.qq.com";
const DEFAULT_DUE_DAYS = 3;
export type WecomBindings = {
WECOM_CORP_ID?: string;
WECOM_AGENT_ID?: string;
WECOM_SECRET?: string;
WECOM_ROBOT_WEBHOOK?: string;
WECOM_NOTIFY_DUE_DAYS?: string;
};
export type WecomConfig = {
corpId: string;
agentId: string;
secret: string;
robotWebhook: string;
dueDays: number;
};
type FetchLike = typeof fetch;
type WecomEnvelope = {
errcode?: number;
errmsg?: string;
access_token?: string;
expires_in?: number;
invaliduser?: string;
};
type CachedAccessToken = {
corpId: string;
secret: string;
token: string;
expiresAt: number;
};
let cachedAccessToken: CachedAccessToken | null = null;
export class WecomClientError extends Error {
status: number;
constructor(message: string, status = 502) {
super(message);
this.name = "WecomClientError";
this.status = status;
}
}
function bindingValue(value: unknown) {
return String(value ?? "").trim();
}
function safeMessage(value: unknown) {
return String(value ?? "").trim().slice(0, 240);
}
export function resolveWecomConfig(bindings: WecomBindings): WecomConfig {
return {
corpId: bindingValue(bindings.WECOM_CORP_ID),
agentId: bindingValue(bindings.WECOM_AGENT_ID),
secret: bindingValue(bindings.WECOM_SECRET),
robotWebhook: bindingValue(bindings.WECOM_ROBOT_WEBHOOK),
dueDays: parseDueDays(bindings.WECOM_NOTIFY_DUE_DAYS),
};
}
function parseDueDays(value: string | undefined) {
const parsed = Number(bindingValue(value));
if (!Number.isFinite(parsed) || parsed < 1) return DEFAULT_DUE_DAYS;
return Math.min(30, Math.floor(parsed));
}
function hasAppCredentials(config: WecomConfig) {
return Boolean(config.corpId && config.agentId && config.secret);
}
async function readEnvelope(
response: Response,
fallbackMessage: string,
): Promise<WecomEnvelope> {
const text = await response.text();
try {
return JSON.parse(text) as WecomEnvelope;
} catch {
throw new WecomClientError(
`${fallbackMessage}(企业微信返回了非 JSON 响应)`,
502,
);
}
}
function ensureOk(
payload: WecomEnvelope,
fallbackMessage: string,
) {
const code = Number(payload.errcode ?? 0);
if (code === 0) return;
const message = safeMessage(payload.errmsg) || fallbackMessage;
if (code === 40014 || code === 42001) {
throw new WecomClientError(`企业微信 access_token 无效:${message}`, 401);
}
throw new WecomClientError(`${fallbackMessage}${message}`, 502);
}
async function fetchAccessToken(
config: WecomConfig,
fetchImpl: FetchLike,
) {
if (
cachedAccessToken?.corpId === config.corpId &&
cachedAccessToken?.secret === config.secret &&
cachedAccessToken.expiresAt > Date.now() + 60_000
) {
return cachedAccessToken.token;
}
const url = new URL(`${WECOM_API_ORIGIN}/cgi-bin/gettoken`);
url.searchParams.set("corpid", config.corpId);
url.searchParams.set("corpsecret", config.secret);
const response = await fetchImpl(url.toString(), {
method: "GET",
signal: AbortSignal.timeout(12_000),
});
const payload = await readEnvelope(response, "获取企业微信 access_token 失败");
if (!response.ok) {
throw new WecomClientError(
`获取企业微信 access_token 失败HTTP ${response.status}`,
502,
);
}
ensureOk(payload, "获取企业微信 access_token 失败");
const token = bindingValue(payload.access_token);
if (!token) {
throw new WecomClientError("企业微信未返回有效 access_token", 502);
}
cachedAccessToken = {
corpId: config.corpId,
secret: config.secret,
token,
expiresAt:
Date.now() + Math.max(300, Number(payload.expires_in) || 7_200) * 1_000,
};
return token;
}
export async function sendWecomRobotMessage(
content: string,
config: WecomConfig,
fetchImpl: FetchLike = fetch,
): Promise<void> {
if (!config.robotWebhook) {
console.warn("[KOC LOOP] wecom robot webhook not configured, skipping");
return;
}
const response = await fetchImpl(config.robotWebhook, {
method: "POST",
headers: { "Content-Type": "application/json; charset=utf-8" },
body: JSON.stringify({
msgtype: "text",
text: { content },
}),
signal: AbortSignal.timeout(10_000),
});
const payload = await readEnvelope(response, "企业微信群机器人推送失败");
if (!response.ok) {
throw new WecomClientError(
`企业微信群机器人推送失败HTTP ${response.status}`,
502,
);
}
ensureOk(payload, "企业微信群机器人推送失败");
}
export async function sendWecomAppMessage(
externalUserIds: string[],
content: string,
config: WecomConfig,
fetchImpl: FetchLike = fetch,
): Promise<{ sent: number; failed: number; skipped: boolean }> {
const normalized = externalUserIds
.map((id) => bindingValue(id))
.filter((id) => id.length > 0);
if (normalized.length === 0) {
return { sent: 0, failed: 0, skipped: true };
}
if (!hasAppCredentials(config)) {
return { sent: 0, failed: 0, skipped: true };
}
const token = await fetchAccessToken(config, fetchImpl);
const url = `${WECOM_API_ORIGIN}/cgi-bin/message/send?access_token=${encodeURIComponent(token)}`;
const response = await fetchImpl(url, {
method: "POST",
headers: { "Content-Type": "application/json; charset=utf-8" },
body: JSON.stringify({
touser: normalized.join("|"),
msgtype: "text",
agentid: Number(config.agentId),
text: { content },
}),
signal: AbortSignal.timeout(12_000),
});
const payload = await readEnvelope(response, "企业微信应用消息推送失败");
if (!response.ok) {
throw new WecomClientError(
`企业微信应用消息推送失败HTTP ${response.status}`,
502,
);
}
ensureOk(payload, "企业微信应用消息推送失败");
const invalid = bindingValue(payload.invaliduser).split("|").filter(Boolean);
return {
sent: Math.max(0, normalized.length - invalid.length),
failed: invalid.length,
skipped: false,
};
}
export function clearWecomAccessTokenCacheForTests() {
cachedAccessToken = null;
}

View File

@@ -0,0 +1,171 @@
import type { DatabaseClient } from "./database";
import { shanghaiDateFromTimestamp } from "./collection-service";
import {
sendWecomAppMessage,
sendWecomRobotMessage,
type WecomConfig,
} from "./wecom-client";
import { getRuntimeEnv } from "./runtime-env";
type FetchLike = typeof fetch;
export type WecomNotifySummary = {
dueSoonAttempted: number;
dueSoonSent: number;
dueSoonFailed: number;
dueSoonSkipped: number;
digestSent: boolean;
};
type DueSoonRow = {
distribution_id: string;
partner_id: string;
partner_name: string;
wecom_external_user_id: string | null;
task_name: string;
due_at: string;
content_title: string;
};
export function computeDueCutoff(today: string, dueDays: number) {
const match = /^(\d{4})-(\d{2})-(\d{2})$/.exec(today);
if (!match) return today;
const [, y, m, d] = match;
const date = new Date(
Date.UTC(Number(y), Number(m) - 1, Number(d)) + dueDays * 24 * 60 * 60 * 1_000,
);
return date.toISOString().slice(0, 10);
}
export async function runDueSoonWecomNotifications(
db: DatabaseClient,
config: WecomConfig,
fetchImpl: FetchLike = fetch,
now: number = Date.now(),
): Promise<WecomNotifySummary> {
const today = shanghaiDateFromTimestamp(now);
const cutoff = computeDueCutoff(today, config.dueDays);
const portalUrl = bindingValue(getRuntimeEnv().KOC_PORTAL_URL);
const result = await db
.prepare(
`SELECT
d.id AS distribution_id,
d.partner_id,
p.name AS partner_name,
p.wecom_external_user_id,
t.name AS task_name,
t.due_at,
c.title AS content_title
FROM distributions d
JOIN tasks t ON t.id = d.task_id
JOIN partners p ON p.id = d.partner_id
JOIN contents c ON c.id = d.content_id
WHERE (d.publish_url IS NULL OR d.publish_url = '')
AND t.due_at IS NOT NULL AND t.due_at != ''
AND t.due_at <= ?
ORDER BY t.due_at ASC, p.name ASC`,
)
.bind(cutoff)
.all<DueSoonRow>();
const grouped = new Map<
string,
{
partnerName: string;
externalUserId: string | null;
rows: DueSoonRow[];
}
>();
for (const row of result.results) {
const entry = grouped.get(row.partner_id) ?? {
partnerName: row.partner_name,
externalUserId: row.wecom_external_user_id,
rows: [],
};
entry.rows.push(row);
if (!entry.externalUserId && row.wecom_external_user_id) {
entry.externalUserId = row.wecom_external_user_id;
}
grouped.set(row.partner_id, entry);
}
let dueSoonAttempted = 0;
let dueSoonSent = 0;
let dueSoonFailed = 0;
let dueSoonSkipped = 0;
const digestTasks: string[] = [];
for (const [, entry] of grouped) {
dueSoonAttempted += 1;
const external = entry.externalUserId
? [entry.externalUserId]
: [];
const lines = entry.rows.slice(0, 5).map((row) => {
return `· 《${truncate(row.task_name, 24)}》— ${truncate(row.content_title, 24)}(截止 ${row.due_at}`;
});
const overflow =
entry.rows.length > 5 ? `\n…还有 ${entry.rows.length - 5}` : "";
const link = portalUrl ? `\n领取链接${portalUrl}` : "";
const content =
`${entry.partnerName},你有 ${entry.rows.length} 条内容待发布:\n${lines.join("\n")}${overflow}${link}`;
const appResult = await sendWecomAppMessage(
external,
content,
config,
fetchImpl,
).catch((error: unknown) => {
console.warn(
"[KOC LOOP] wecom app message failed",
{ partner: entry.partnerName, error: safeError(error) },
);
return null;
});
if (appResult === null) {
dueSoonFailed += 1;
} else if (appResult.skipped) {
dueSoonSkipped += 1;
} else {
dueSoonSent += 1;
}
const earliestDue = entry.rows[0]?.due_at ?? "";
digestTasks.push(
`· ${entry.partnerName}${entry.rows.length} 条,最近截止 ${earliestDue}`,
);
}
let digestSent = false;
if (config.robotWebhook && digestTasks.length > 0) {
const digest =
`今日待发布催办(${today},截止 ≤ ${cutoff}\n${digestTasks.join("\n")}`;
try {
await sendWecomRobotMessage(digest, config, fetchImpl);
digestSent = true;
} catch (error) {
console.warn(
"[KOC LOOP] wecom robot digest failed",
{ error: safeError(error) },
);
}
}
return {
dueSoonAttempted,
dueSoonSent,
dueSoonFailed,
dueSoonSkipped,
digestSent,
};
}
function bindingValue(value: unknown) {
return String(value ?? "").trim();
}
function truncate(value: string, max: number) {
return value.length > max ? `${value.slice(0, max)}` : value;
}
function safeError(error: unknown) {
return error instanceof Error ? error.message : String(error);
}

83
lib/workbook-image.ts Normal file
View File

@@ -0,0 +1,83 @@
import sharp from "sharp";
export type WorkbookSourceImage = {
bytes: Uint8Array;
contentType: string;
width?: number | null;
height?: number | null;
description: string;
};
export type WorkbookImageNormalizationOptions = {
maxDimension?: number;
outputFormat?: "png" | "jpeg";
jpegQuality?: number;
};
const NORMALIZABLE_IMAGE = /^image\/(?:jpe?g|png|webp|gif|tiff?|avif|heic|heif)$/i;
function hasImageSignature(bytes: Uint8Array) {
if (bytes.length < 4) return false;
if (bytes[0] === 0xff && bytes[1] === 0xd8) return true;
if (bytes[0] === 0x89 && bytes[1] === 0x50 && bytes[2] === 0x4e && bytes[3] === 0x47) {
return true;
}
if (bytes[0] === 0x47 && bytes[1] === 0x49 && bytes[2] === 0x46) return true;
if (
bytes.length >= 12 &&
String.fromCharCode(...bytes.slice(0, 4)) === "RIFF" &&
String.fromCharCode(...bytes.slice(8, 12)) === "WEBP"
) {
return true;
}
return (
(bytes[0] === 0x49 && bytes[1] === 0x49 && bytes[2] === 0x2a && bytes[3] === 0x00) ||
(bytes[0] === 0x4d && bytes[1] === 0x4d && bytes[2] === 0x00 && bytes[3] === 0x2a)
);
}
/**
* Excel viewers disagree on whether JPEG EXIF orientation should be applied.
* Bake that orientation into the pixels before the image enters the workbook.
* Callers may also resize and encode large source images as JPEG to keep the
* generated workbook within the upload limit.
*/
export async function normalizeWorkbookImage(
image: WorkbookSourceImage,
options: WorkbookImageNormalizationOptions = {},
): Promise<WorkbookSourceImage> {
if (!NORMALIZABLE_IMAGE.test(image.contentType) && !hasImageSignature(image.bytes)) {
return image;
}
try {
let normalized = sharp(image.bytes, { animated: false }).rotate();
if (options.maxDimension && options.maxDimension > 0) {
normalized = normalized.resize({
width: Math.floor(options.maxDimension),
height: Math.floor(options.maxDimension),
fit: "inside",
withoutEnlargement: true,
});
}
normalized =
options.outputFormat === "jpeg"
? normalized
.flatten({ background: "#ffffff" })
.jpeg({
quality: Math.min(95, Math.max(50, options.jpegQuality ?? 82)),
mozjpeg: true,
})
: normalized.png({ compressionLevel: 6 });
const { data, info } = await normalized.toBuffer({ resolveWithObject: true });
return {
...image,
bytes: new Uint8Array(data),
contentType:
options.outputFormat === "jpeg" ? "image/jpeg" : "image/png",
width: info.width,
height: info.height,
};
} catch {
return image;
}
}

198
mysql/0001_init.sql Normal file
View File

@@ -0,0 +1,198 @@
CREATE TABLE IF NOT EXISTS partners (
id VARCHAR(64) PRIMARY KEY,
name VARCHAR(255) NOT NULL,
wecom_name VARCHAR(255) NOT NULL,
owner VARCHAR(255) NOT NULL DEFAULT '运营组',
claimed_total INT NOT NULL DEFAULT 0,
completed_total INT NOT NULL DEFAULT 0,
created_at DATETIME(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci;
-- statement-breakpoint
CREATE TABLE IF NOT EXISTS tasks (
id VARCHAR(64) PRIMARY KEY,
name VARCHAR(255) NOT NULL,
brand VARCHAR(255) NOT NULL,
quantity INT NOT NULL,
claimed_quantity INT NOT NULL DEFAULT 0,
due_at VARCHAR(32) NOT NULL,
status VARCHAR(32) NOT NULL DEFAULT 'active',
source_url TEXT NOT NULL DEFAULT (''),
source_sheet_id VARCHAR(255) NOT NULL DEFAULT '',
source_sheet_name VARCHAR(255) NOT NULL DEFAULT '',
source_synced_at DATETIME(3) NULL,
share_token VARCHAR(128) NULL,
collection_start_date VARCHAR(32) NULL,
collection_days TEXT NOT NULL DEFAULT ('[]'),
collection_schedule_updated_at DATETIME(3) NULL,
created_at DATETIME(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3),
UNIQUE KEY tasks_share_token_idx (share_token)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci;
-- statement-breakpoint
CREATE TABLE IF NOT EXISTS contents (
id VARCHAR(64) PRIMARY KEY,
task_id VARCHAR(64) NOT NULL,
title TEXT NOT NULL,
body LONGTEXT NOT NULL DEFAULT (''),
image_assets LONGTEXT NOT NULL DEFAULT ('[]'),
status VARCHAR(32) NOT NULL DEFAULT 'available',
source VARCHAR(255) NOT NULL DEFAULT '飞书内容表',
source_row INT NULL,
created_at DATETIME(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3),
KEY contents_task_status_idx (task_id, status)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci;
-- statement-breakpoint
CREATE TABLE IF NOT EXISTS accounts (
id VARCHAR(64) PRIMARY KEY,
platform VARCHAR(32) NOT NULL DEFAULT '小红书',
platform_uid VARCHAR(255) NOT NULL,
public_account_id VARCHAR(255) NOT NULL DEFAULT '',
nickname VARCHAR(255) NOT NULL,
profile_url TEXT NOT NULL DEFAULT (''),
ip_location VARCHAR(255) NOT NULL DEFAULT '待识别',
followers INT NOT NULL DEFAULT 0,
gender VARCHAR(16) NOT NULL DEFAULT '',
bio TEXT NOT NULL DEFAULT (''),
tags VARCHAR(500) NOT NULL DEFAULT '',
post_count INT NOT NULL DEFAULT 0,
avg_views INT NOT NULL DEFAULT 0,
first_seen_at DATETIME(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3),
last_seen_at DATETIME(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3),
UNIQUE KEY accounts_platform_uid_idx (platform, platform_uid)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci;
-- statement-breakpoint
CREATE TABLE IF NOT EXISTS claims (
id VARCHAR(64) PRIMARY KEY,
task_id VARCHAR(64) NOT NULL,
partner_id VARCHAR(64) NOT NULL,
claimant_name VARCHAR(255) NOT NULL,
claim_token VARCHAR(128) NOT NULL,
quantity INT NOT NULL,
created_at DATETIME(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3),
UNIQUE KEY claims_claim_token_idx (claim_token),
KEY claims_task_partner_created_idx (task_id, partner_id, created_at)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci;
-- statement-breakpoint
CREATE TABLE IF NOT EXISTS delegation_bundles (
id VARCHAR(64) PRIMARY KEY,
task_id VARCHAR(64) NOT NULL,
claim_id VARCHAR(64) NOT NULL,
partner_id VARCHAR(64) NOT NULL,
label VARCHAR(255) NOT NULL,
share_token VARCHAR(128) NOT NULL,
quantity INT NOT NULL,
status VARCHAR(32) NOT NULL DEFAULT 'active',
created_at DATETIME(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3),
updated_at DATETIME(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3),
revoked_at DATETIME(3) NULL,
UNIQUE KEY delegation_bundles_share_token_idx (share_token),
KEY delegation_bundles_claim_created_idx (claim_id, created_at)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci;
-- statement-breakpoint
CREATE TABLE IF NOT EXISTS distributions (
id VARCHAR(64) PRIMARY KEY,
task_id VARCHAR(64) NOT NULL,
content_id VARCHAR(64) NOT NULL,
partner_id VARCHAR(64) NOT NULL,
claim_id VARCHAR(64) NULL,
delegation_bundle_id VARCHAR(64) NULL,
account_id VARCHAR(64) NULL,
publish_url TEXT NULL,
publish_time DATETIME(3) NULL,
publish_screenshot_key VARCHAR(512) NULL,
status VARCHAR(32) NOT NULL DEFAULT 'claimed',
claimed_at DATETIME(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3),
screenshot_key VARCHAR(512) NULL,
ocr_status VARCHAR(32) NOT NULL DEFAULT 'none',
exposure INT NULL,
views INT NULL,
d2_likes INT NULL,
d2_comments INT NULL,
d2_collects INT NULL,
d5_likes INT NULL,
d5_comments INT NULL,
d5_collects INT NULL,
d7_likes INT NULL,
d7_comments INT NULL,
d7_collects INT NULL,
latest_likes INT NULL,
latest_comments INT NULL,
latest_collects INT NULL,
collection_status VARCHAR(32) NOT NULL DEFAULT 'pending',
collection_status_description TEXT NULL,
collection_updated_at DATETIME(3) NULL,
last_collection_day INT NULL,
updated_at DATETIME(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3),
KEY distributions_task_status_idx (task_id, status),
KEY distributions_content_idx (content_id),
KEY distributions_account_idx (account_id),
KEY distributions_claim_idx (claim_id),
KEY distributions_delegation_idx (delegation_bundle_id)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci;
-- statement-breakpoint
CREATE TABLE IF NOT EXISTS collection_runs (
id VARCHAR(64) PRIMARY KEY,
task_id VARCHAR(64) NOT NULL,
distribution_id VARCHAR(64) NOT NULL,
scheduled_date VARCHAR(32) NOT NULL,
schedule_day INT NULL,
scheduled_at DATETIME(3) NOT NULL,
status VARCHAR(32) NOT NULL DEFAULT 'pending',
likes INT NULL,
comments INT NULL,
collects INT NULL,
status_description TEXT NULL,
started_at DATETIME(3) NULL,
completed_at DATETIME(3) NULL,
created_at DATETIME(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3),
UNIQUE KEY collection_runs_distribution_date_idx (distribution_id, scheduled_date),
KEY collection_runs_task_date_idx (task_id, scheduled_date)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci;
-- statement-breakpoint
CREATE TABLE IF NOT EXISTS users (
id VARCHAR(64) PRIMARY KEY,
username VARCHAR(255) NOT NULL,
password_hash VARCHAR(255) NOT NULL,
password_salt VARCHAR(255) NOT NULL,
password_iterations INT NOT NULL,
role VARCHAR(32) NOT NULL,
super_admin_guard TINYINT GENERATED ALWAYS AS (
CASE WHEN role = 'super_admin' THEN 1 ELSE NULL END
) STORED,
created_at DATETIME(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3),
updated_at DATETIME(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3),
UNIQUE KEY users_username_idx (username),
UNIQUE KEY users_single_super_admin_idx (super_admin_guard)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci;
-- statement-breakpoint
CREATE TABLE IF NOT EXISTS auth_sessions (
token_hash VARCHAR(128) PRIMARY KEY,
user_id VARCHAR(64) NOT NULL,
expires_at DATETIME(3) NOT NULL,
created_at DATETIME(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3),
KEY auth_sessions_user_id_idx (user_id),
KEY auth_sessions_expires_at_idx (expires_at)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci;
-- statement-breakpoint
CREATE TABLE IF NOT EXISTS mcp_export_tokens (
token_hash VARCHAR(128) PRIMARY KEY,
kind VARCHAR(64) NOT NULL,
payload LONGTEXT NOT NULL,
expires_at DATETIME(3) NOT NULL,
created_at DATETIME(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3),
KEY mcp_export_tokens_expires_at_idx (expires_at)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci;
-- statement-breakpoint
CREATE TABLE IF NOT EXISTS background_jobs (
id VARCHAR(64) PRIMARY KEY,
type VARCHAR(64) NOT NULL,
payload LONGTEXT NOT NULL,
status VARCHAR(32) NOT NULL DEFAULT 'pending',
attempts INT NOT NULL DEFAULT 0,
available_at DATETIME(3) NOT NULL,
locked_at DATETIME(3) NULL,
completed_at DATETIME(3) NULL,
last_error TEXT NULL,
created_at DATETIME(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3),
updated_at DATETIME(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3),
KEY background_jobs_pending_idx (status, available_at)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci;

View File

@@ -0,0 +1,2 @@
ALTER TABLE accounts
ADD COLUMN cooperation_source VARCHAR(500) NOT NULL DEFAULT '' AFTER avg_views;

View File

@@ -0,0 +1,8 @@
ALTER TABLE tasks
ADD COLUMN task_type VARCHAR(32) NOT NULL DEFAULT 'content_publish' AFTER status;
-- statement-breakpoint
ALTER TABLE distributions
ADD COLUMN result_screenshot_key VARCHAR(512) NULL AFTER publish_screenshot_key;
-- statement-breakpoint
ALTER TABLE distributions
ADD COLUMN result_submitted_at DATETIME(3) NULL AFTER result_screenshot_key;

Some files were not shown because too many files have changed in this diff Show More