Compare commits

..

2 Commits

Author SHA1 Message Date
CHE LIANG ZHAO
f465faea38 Merge pull request #1 from FadingLight9291117/dev
Some checks failed
Deploy BillAI / Deploy to Production (push) Has been cancelled
feat: implement cross-batch Alipay refund reconciliation
2026-06-16 19:34:17 +08:00
e2e1beb6f7 feat: implement cross-batch Alipay refund reconciliation
When a refund row in an uploaded Alipay bill has no matching expense
row in the same batch (because the original purchase was uploaded in a
prior batch), the refund is now reconciled against the stored record in
bills_cleaned rather than being silently discarded.

Changes:
- analyzer/cleaners/base.py: add unresolved_refunds list to BaseCleaner
- analyzer/cleaners/alipay.py: _aggregate_refunds stores full refund
  metadata (dict); _process_expenses tracks matched keys and populates
  self.unresolved_refunds for unmatched refunds
- analyzer/server.py: thread unresolved_refunds through do_clean,
  CleanResponse, and both /clean endpoints
- server/adapter/adapter.go: add UnresolvedRefund type and field to CleanResult
- server/adapter/http/cleaner.go: deserialize unresolved_refunds from
  Python response and populate CleanResult
- server/repository/repository.go: add ReconcileRefund to BillRepository interface
- server/repository/mongo/repository.go: implement ReconcileRefund —
  full refund soft-deletes the bill, partial refund reduces amount and
  appends remark with original amount and refund order number
- server/handler/upload.go: capture clean result and call ReconcileRefund
  for each unresolved refund after saving cleaned bills
- server/model/response.go: add ReconciledRefundCount to UploadData

Also: add CLAUDE.md (@AGENTS.md), update AGENTS.md, fix DailyTrendChart
missing-date gap by filling zero-expense dates in daily map.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-16 19:29:47 +08:00
12 changed files with 342 additions and 133 deletions

152
AGENTS.md
View File

@@ -2,85 +2,81 @@
Guidelines for AI coding agents working on BillAI - a microservices bill analysis system. Guidelines for AI coding agents working on BillAI - a microservices bill analysis system.
**Current Version:** 1.4.0 | Go: 1.24.0 | Node: 20+ | Python: 3.12+ **Version:** See `CHANGELOG.md` for current version. Latest tag usually matches.
## Architecture ## Architecture
- `web/` - SvelteKit 5 + TailwindCSS 4 + TypeScript (Frontend, port 3000) - `web/` - SvelteKit 5 + TailwindCSS 4 + TypeScript (Frontend, port 3000)
- `server/` - Go 1.24 + Gin + MongoDB (API, port 8080) - `server/` - Go 1.24 + Gin + MongoDB (API, port 8080)
- `analyzer/` - Python 3.12 + FastAPI (Data cleaning, port 8001) - `analyzer/` - Python 3.12 + FastAPI (Data cleaning, port 8001)
SvelteKit proxies `/api/*` requests to Go backend via `web/src/routes/api/[...path]/+server.ts`. **Proxy Caveat:** SvelteKit proxies `/api/*` to Go via `web/src/routes/api/[...path]/+server.ts`, but **only GET and POST are forwarded**. The Go backend intentionally uses `POST` for mutations (update, delete, manual create) to work around this. If you add PUT/PATCH/DELETE endpoints, you must also add them to the proxy.
## Build/Lint/Test Commands ## Build/Lint/Test Commands
### Frontend (web/) ### Frontend (`web/`)
```bash ```bash
npm run dev # Start dev server npm run dev # Start dev server
npm run build # Production build npm run build # Production build
npm run check # TypeScript check npm run check # TypeScript + Svelte check
npm run lint # Prettier + ESLint npm run lint # Prettier --check + ESLint
npm run format # Format with Prettier npm run format # Prettier --write
npm run test # Run all tests (CI mode) npm run test # vitest --run (CI mode)
npx vitest run src/xxx.spec.ts # Run single test file npx vitest run src/xxx.spec.ts # Run single test file
npx vitest run -t "pattern" # Run by name pattern
``` ```
### Backend (server/) **Note:** `web/` has a `yarn.lock` but scripts in `package.json` use `npm run`.
### Backend (`server/`)
```bash ```bash
go run . # Start server go run . # Start server (reads server/config.yaml)
go build -o server . # Build binary go build -o server .
go test ./... # Run all tests go test ./... # Run all tests
go test ./handler/... # Run handler tests
go test -run TestName ./... # Run single test go test -run TestName ./... # Run single test
go test -v ./handler/... # Verbose output go test -v ./handler/... # Verbose output
``` ```
### Analyzer (analyzer/) ### Analyzer (`analyzer/`)
```bash ```bash
python server.py # Start FastAPI python server.py # Start FastAPI (has `if __name__ == "__main__"`)
uvicorn server:app --reload # Hot reload uvicorn server:app --reload # Requires cwd == analyzer/
pytest # Run all tests pytest # Run all tests
pytest test_jd_cleaner.py # Single test file pytest test_jd_cleaner.py # Single test file
pytest -k "pattern" # Run by pattern
``` ```
### Docker ### Docker
```bash ```bash
docker-compose up -d --build # Start/rebuild all services docker compose up -d --build --remove-orphans # Start/rebuild all
docker-compose logs -f server # Follow logs docker compose logs -f server # Follow logs
docker-compose down # Stop services docker compose down # Stop services
``` ```
## Code Style ## Code Style & Conventions
### General ### TypeScript/Svelte (`web/`)
- **Comments:** Chinese common for business logic; English for technical. - **Formatting:** Prettier (tabs, single quotes, `trailingComma: none`, printWidth 100)
- **Conventions:** Follow existing patterns. Check `package.json`/`go.mod`/`requirements.txt` before adding dependencies. - **Imports:** Use `$lib` alias. No relative imports for lib modules.
- **Svelte 5:** Runes (`$state`, `$derived`, `$effect`, `$props`). Events: `onclick={fn}`.
- **Types:** `export interface` for models. Frontend uses `camelCase`, API uses `snake_case`. Converters live in `$lib/models/bill.ts`.
- **Auth:** Token stored in `localStorage` key `auth`. Always use `apiFetch()` from `$lib/api.ts` for authenticated requests.
### TypeScript/Svelte (web/) ### Go (`server/`)
- **Formatting:** Prettier (tabs, single quotes, printWidth 100) - **Module:** `billai-server` (import path). Use this in `go test` / `go build` when outside the directory.
- **Naming:** `PascalCase` for types/components, `camelCase` for variables
- **Imports:** Use `$lib` alias, `$app/*` for SvelteKit builtins. No relative paths for lib modules.
- **Svelte 5:** Use runes (`$state`, `$derived`, `$effect`, `$props`). Event: `onclick={fn}`.
- **Types:** `export interface` for models. Frontend `camelCase`, API `snake_case`. Converters in `$lib/models/bill.ts`.
- **Error Handling:** Check `response.ok`, throw `Error(\`HTTP ${status}\`)`. On 401: `auth.logout()` + redirect.
- **Auth:** `createAuthStore()` in `$lib/stores/auth.ts`. Token in `localStorage` key `auth`. Use `apiFetch()` in `$lib/api.ts`.
### Go Backend (server/)
- **Layer:** `handler``service``adapter`/`repository``model`. No business logic in handlers. - **Layer:** `handler``service``adapter`/`repository``model`. No business logic in handlers.
- **Struct tags:** JSON `snake_case`, `omitempty` optional. Pointer for optional patch fields. Sensitive: `json:"-"`. - **Struct tags:** JSON `snake_case`, `omitempty` for optional. Pointer types for optional patch fields. Sensitive fields: `json:"-"`.
- **Error handling:** 500 for DB errors, 400 for bad requests, 404 not found. Wrap with `fmt.Errorf("context: %w", err)`. - **Response shapes:**
- **Response:** `Result bool`, `Message`, `Data *T`. Auth: `success bool`, `error`, `data`. - Business APIs: `result bool`, `message string`, `data *T`
- **Time:** Use custom `LocalTime` type (serializes as `2006-01-02 15:04:05`). - Auth APIs: `success bool`, `error string`, `data *T` (and `code` for error types)
- **Soft delete:** Never hard-delete. Filter `is_deleted: false` in queries. - **Time:** Custom `LocalTime` type serializes as `"2006-01-02 15:04:05"`.
- **Soft delete:** Never hard-delete. All queries filter `is_deleted: false`.
### Python Analyzer (analyzer/) ### Python (`analyzer/`)
- **Style:** PEP 8. `snake_case` variables, `UPPER_CASE` constants. Prefix private globals with `_`. - **Style:** PEP 8. `snake_case` vars, `UPPER_CASE` constants. Prefix private globals with `_`.
- **Type hints:** Mandatory. Use `Optional[str]` or `str | None`. - **Type hints:** Mandatory. Prefer `str | None` or `Optional[str]`.
- **Models:** `pydantic.BaseModel` for API schemas. - **Models:** `pydantic.BaseModel` for API schemas.
- **Cleaners:** Extend `BaseCleaner(ABC)` from `cleaners/base.py`. Category rules in `config/category.yaml`. - **Cleaners:** Extend `BaseCleaner(ABC)` from `cleaners/base.py`. Category rules in `config/category.yaml`.
## Key Patterns ## Key Patterns & Quirks
### API Flow ### API Flow
``` ```
@@ -89,35 +85,63 @@ Browser → SvelteKit proxy → Go (Gin) → handler → service → adapter →
``` ```
### Authentication ### Authentication
- JWT (HS256). Token in `localStorage` key `auth`. Header: `Authorization: Bearer <token>`. - JWT (HS256). Header: `Authorization: Bearer <token>`.
- `middleware.AuthRequired()` wraps `/api/*` (except `/api/auth/*`). - `middleware.AuthRequired()` guards authed routes. Public routes: `/api/auth/*`, `/api/changelog`, `/health`.
- 401 anywhere`auth.logout()` + redirect `/login`. - Frontend `apiFetch()` intercepts 401`auth.logout()` + redirect `/login`.
### File Processing ### File Processing Pipeline
Upload: ZIP/XLSX → Extract → Convert UTF-8 CSV → Detect bill type → Deduplicate → Clean → Save to MongoDB. Upload: ZIP/XLSX → Extract → Convert UTF-8 CSV → Detect bill type (alipay/wechat/jd) → Deduplicate against MongoDB → Clean via Python → Save cleaned data.
### Adapter (Go ↔ Python) ### Adapter (Go ↔ Python)
`adapter.Cleaner` interface: HTTP (`adapter/http`, default) or subprocess (`adapter/python`). Set via `ANALYZER_MODE` env var. `adapter.Cleaner` interface. Two modes:
- `http` (default): calls FastAPI at `ANALYZER_URL`
- `subprocess`: spawns `python analyzer/clean_bill.py`
Set via `ANALYZER_MODE` env var or `server/config.yaml` `analyzer.mode`.
### Config Precedence
Go backend reads `server/config.yaml`, but Docker compose sets env vars (`ANALYZER_URL`, `MONGO_URI`, `JWT_SECRET`, etc.) that override it.
### SvelteKit Config Notes
- `svelte.config.js` uses `adapter-node` for Docker SSR.
- `csrf.trustedOrigins: ['*']` disables CSRF checks.
- `onwarn` ignores all `a11y_*` warnings (chart components).
### Deployment
- Gitea Actions self-hosted runner (`.gitea/workflows/deploy.yaml`), not GitHub.
- `deploy.sh` is the manual deployment script (same logic as CI).
### Test Coverage
Sparse. Existing tests:
- `web/src/demo.spec.ts` / `page.svelte.spec.ts`
- `server/service/changelog_test.go`
- `analyzer/test_jd_cleaner.py`
## Important Files ## Important Files
| File | Role | | File | Role |
|---|---| |---|---|
| `web/src/lib/api.ts` | Central API client, auth injection | | `web/src/lib/api.ts` | Central API client, auth injection, all API functions |
| `web/src/lib/stores/auth.ts` | Auth state, JWT handling | | `web/src/lib/stores/auth.ts` | Auth state, JWT handling, localStorage key `auth` |
| `web/src/lib/models/bill.ts` | UIBill model + converters | | `web/src/lib/models/bill.ts` | UIBill model + snake_case ↔ camelCase converters |
| `server/main.go` | Entry point | | `web/src/routes/api/[...path]/+server.ts` | SvelteKit → Go proxy (GET/POST only) |
| `server/handler/upload.go` | Full upload pipeline | | `server/main.go` | Entry point, wires adapter + repository + router |
| `server/handler/bills.go` | List/filter bills | | `server/config.yaml` | Go backend config (Mongo, auth, paths, analyzer mode) |
| `server/model/bill.go` | Bill models, LocalTime type | | `server/router/router.go` | Route table, auth group definitions |
| `server/adapter/adapter.go` | Cleaner interface | | `server/handler/upload.go` | Full upload pipeline handler |
| `server/repository/mongo/repository.go` | MongoDB implementation | | `server/handler/bills.go` | List/filter/update/delete bills |
| `analyzer/server.py` | FastAPI entry | | `server/model/bill.go` | Bill models, LocalTime type, BSON/JSON marshaling |
| `server/adapter/adapter.go` | Cleaner interface definition |
| `server/repository/mongo/repository.go` | MongoDB implementation, soft-delete queries |
| `analyzer/server.py` | FastAPI entry, bill detection/clean endpoints |
| `analyzer/cleaners/base.py` | BaseCleaner ABC | | `analyzer/cleaners/base.py` | BaseCleaner ABC |
| `analyzer/category.py` | Category inference | | `analyzer/category.py` | Category inference engine |
| `docker-compose.yaml` | Full stack orchestration |
## Agent Guidelines ## Agent Guidelines
- **Before coding:** Search codebase to understand existing patterns and dependencies
- **Dependencies:** Check `package.json`/`go.mod`/`requirements.txt` before adding new packages - **Before coding:** Search codebase to understand existing patterns and dependencies.
- **Tests:** Always run relevant test suite before committing changes - **Dependencies:** Check `package.json`/`go.mod`/`requirements.txt` before adding new packages.
- **Git commits:** Provide clear messages explaining the "why" of changes - **Tests:** Run the relevant test suite before committing. If no tests exist for your change, verify manually.
- **File references:** Use relative `file_path:line_number` format (e.g., `server/handler/changelog.go:12`) when mentioning code locations - **Git commits:** Provide clear messages explaining the "why" of changes.
- **File references:** Use relative `file_path:line_number` format (e.g., `server/handler/changelog.go:12`) when mentioning code locations.

5
CLAUDE.md Normal file
View File

@@ -0,0 +1,5 @@
# CLAUDE.md
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
@AGENTS.md

View File

@@ -112,8 +112,15 @@ class AlipayCleaner(BaseCleaner):
if key: if key:
if key not in order_refunds: if key not in order_refunds:
order_refunds[key] = Decimal("0") order_refunds[key] = {
order_refunds[key] += refund_amount "amount": Decimal("0"),
"merchant_order_no": refund_merchant_no,
"refund_order_no": refund_order_no,
"time": row[0],
"merchant": row[2],
"description": row[4] if len(row) > 4 else "",
}
order_refunds[key]["amount"] += refund_amount
print(f" 退款记录: {row[0]} | {row[2]} | {refund_amount}") print(f" 退款记录: {row[0]} | {row[2]} | {refund_amount}")
return order_refunds return order_refunds
@@ -121,6 +128,7 @@ class AlipayCleaner(BaseCleaner):
def _process_expenses(self, expense_rows: list, order_refunds: dict) -> list: def _process_expenses(self, expense_rows: list, order_refunds: dict) -> list:
"""处理支出记录""" """处理支出记录"""
final_rows = [] final_rows = []
matched_keys = set()
for row in expense_rows: for row in expense_rows:
if len(row) >= 12: if len(row) >= 12:
@@ -132,13 +140,14 @@ class AlipayCleaner(BaseCleaner):
refund_amount = Decimal("0") refund_amount = Decimal("0")
matched_key = None matched_key = None
for key, amount in order_refunds.items(): for key, refund in order_refunds.items():
if key and (order_no == key or merchant_no == key or order_no.startswith(key)): if key and (order_no == key or merchant_no == key or order_no.startswith(key)):
refund_amount = amount refund_amount = refund["amount"]
matched_key = key matched_key = key
break break
if matched_key: if matched_key:
matched_keys.add(matched_key)
if refund_amount >= expense_amount: if refund_amount >= expense_amount:
# 全额退款,删除 # 全额退款,删除
self.stats["fully_refunded"] += 1 self.stats["fully_refunded"] += 1
@@ -164,6 +173,21 @@ class AlipayCleaner(BaseCleaner):
else: else:
final_rows.append(row) final_rows.append(row)
# 本批次内未匹配到对应支出的退款,交由调用方做跨批次核销
self.unresolved_refunds = [
{
"order_no": key,
"merchant_order_no": refund["merchant_order_no"],
"refund_order_no": refund["refund_order_no"],
"amount": float(format_amount(refund["amount"])),
"time": refund["time"],
"merchant": refund["merchant"],
"description": refund["description"],
}
for key, refund in order_refunds.items()
if key not in matched_keys
]
return final_rows return final_rows
def _is_platform_merchant(self, merchant: str) -> bool: def _is_platform_merchant(self, merchant: str) -> bool:

View File

@@ -221,6 +221,9 @@ class BaseCleaner(ABC):
"final_count": 0, "final_count": 0,
} }
# 本次清理中未能在同批次内匹配到对应支出的退款(跨批次核销用)
self.unresolved_refunds: list[dict] = []
def set_date_range(self, start_date: date | None, end_date: date | None): def set_date_range(self, start_date: date | None, end_date: date | None):
"""设置日期筛选范围""" """设置日期筛选范围"""
self.start_date = start_date self.start_date = start_date

View File

@@ -52,6 +52,7 @@ class CleanResponse(BaseModel):
bill_type: str bill_type: str
message: str message: str
output_path: Optional[str] = None output_path: Optional[str] = None
unresolved_refunds: list[dict] = []
class CategoryRequest(BaseModel): class CategoryRequest(BaseModel):
@@ -138,22 +139,22 @@ def do_clean(
start: str = None, start: str = None,
end: str = None, end: str = None,
output_format: str = "csv" output_format: str = "csv"
) -> tuple[bool, str, str]: ) -> tuple[bool, str, str, list[dict]]:
""" """
执行清洗逻辑 执行清洗逻辑
Returns: Returns:
(success, bill_type, message) (success, bill_type, message, unresolved_refunds)
""" """
# 检查文件是否存在 # 检查文件是否存在
if not Path(input_path).exists(): if not Path(input_path).exists():
return False, "", f"文件不存在: {input_path}" return False, "", f"文件不存在: {input_path}", []
# 检测账单类型 # 检测账单类型
if bill_type == "auto": if bill_type == "auto":
detected_type = detect_bill_type(input_path) detected_type = detect_bill_type(input_path)
if detected_type is None: if detected_type is None:
return False, "", "无法识别账单类型" return False, "", "无法识别账单类型", []
bill_type = detected_type bill_type = detected_type
# 计算日期范围 # 计算日期范围
@@ -172,10 +173,10 @@ def do_clean(
cleaner.clean() cleaner.clean()
type_names = {"alipay": "支付宝", "wechat": "微信", "jd": "京东白条"} type_names = {"alipay": "支付宝", "wechat": "微信", "jd": "京东白条"}
return True, bill_type, f"{type_names.get(bill_type, bill_type)}账单清洗完成" return True, bill_type, f"{type_names.get(bill_type, bill_type)}账单清洗完成", cleaner.unresolved_refunds
except Exception as e: except Exception as e:
return False, bill_type, f"清洗失败: {str(e)}" return False, bill_type, f"清洗失败: {str(e)}", []
# ============================================================================= # =============================================================================
@@ -215,7 +216,7 @@ async def clean_bill(request: CleanRequest):
接收账单文件路径,执行清洗后输出到指定路径 接收账单文件路径,执行清洗后输出到指定路径
""" """
success, bill_type, message = do_clean( success, bill_type, message, unresolved_refunds = do_clean(
input_path=request.input_path, input_path=request.input_path,
output_path=request.output_path, output_path=request.output_path,
bill_type=request.bill_type or "auto", bill_type=request.bill_type or "auto",
@@ -233,7 +234,8 @@ async def clean_bill(request: CleanRequest):
success=True, success=True,
bill_type=bill_type, bill_type=bill_type,
message=message, message=message,
output_path=request.output_path output_path=request.output_path,
unresolved_refunds=unresolved_refunds
) )
@@ -264,7 +266,7 @@ async def clean_bill_upload(
output_path = tmp_output.name output_path = tmp_output.name
try: try:
success, detected_type, message = do_clean( success, detected_type, message, unresolved_refunds = do_clean(
input_path=input_path, input_path=input_path,
output_path=output_path, output_path=output_path,
bill_type=bill_type or "auto", bill_type=bill_type or "auto",
@@ -282,7 +284,8 @@ async def clean_bill_upload(
success=True, success=True,
bill_type=detected_type, bill_type=detected_type,
message=message, message=message,
output_path=output_path output_path=output_path,
unresolved_refunds=unresolved_refunds
) )
finally: finally:

View File

@@ -13,8 +13,20 @@ type CleanOptions struct {
// CleanResult 清洗结果 // CleanResult 清洗结果
type CleanResult struct { type CleanResult struct {
BillType string // 检测到的账单类型: alipay/wechat/jd BillType string // 检测到的账单类型: alipay/wechat/jd
Output string // 脚本输出信息 Output string // 脚本输出信息
UnresolvedRefunds []UnresolvedRefund // 本次清洗未在同批次内匹配到对应支出的退款
}
// UnresolvedRefund 本次清洗未在同批次内匹配到对应支出的退款
type UnresolvedRefund struct {
OrderNo string // 原订单号(去除退款后缀)
MerchantOrderNo string // 商家订单号(备用匹配字段)
RefundOrderNo string // 退款行自身的完整订单号(用于备注追溯)
Amount float64 // 退款金额
Time string // 退款时间
Merchant string // 交易对方
Description string // 商品说明
} }
// ConvertResult 格式转换结果 // ConvertResult 格式转换结果

View File

@@ -29,10 +29,22 @@ type CleanRequest struct {
// CleanResponse HTTP 清洗响应 // CleanResponse HTTP 清洗响应
type CleanResponse struct { type CleanResponse struct {
Success bool `json:"success"` Success bool `json:"success"`
BillType string `json:"bill_type"` BillType string `json:"bill_type"`
Message string `json:"message"` Message string `json:"message"`
OutputPath string `json:"output_path,omitempty"` OutputPath string `json:"output_path,omitempty"`
UnresolvedRefunds []UnresolvedRefund `json:"unresolved_refunds,omitempty"`
}
// UnresolvedRefund 本次清洗未在同批次内匹配到对应支出的退款(与 Python 端 dict 字段对应)
type UnresolvedRefund struct {
OrderNo string `json:"order_no"`
MerchantOrderNo string `json:"merchant_order_no"`
RefundOrderNo string `json:"refund_order_no"`
Amount float64 `json:"amount"`
Time string `json:"time"`
Merchant string `json:"merchant"`
Description string `json:"description"`
} }
// ErrorResponse 错误响应 // ErrorResponse 错误响应
@@ -149,9 +161,23 @@ func (c *Cleaner) Clean(inputPath, outputPath string, opts *adapter.CleanOptions
} }
} }
unresolvedRefunds := make([]adapter.UnresolvedRefund, 0, len(cleanResp.UnresolvedRefunds))
for _, ur := range cleanResp.UnresolvedRefunds {
unresolvedRefunds = append(unresolvedRefunds, adapter.UnresolvedRefund{
OrderNo: ur.OrderNo,
MerchantOrderNo: ur.MerchantOrderNo,
RefundOrderNo: ur.RefundOrderNo,
Amount: ur.Amount,
Time: ur.Time,
Merchant: ur.Merchant,
Description: ur.Description,
})
}
return &adapter.CleanResult{ return &adapter.CleanResult{
BillType: cleanResp.BillType, BillType: cleanResp.BillType,
Output: cleanResp.Message, Output: cleanResp.Message,
UnresolvedRefunds: unresolvedRefunds,
}, nil }, nil
} }

View File

@@ -223,7 +223,7 @@ func Upload(c *gin.Context) {
End: req.End, End: req.End,
Format: req.Format, Format: req.Format,
} }
_, cleanErr := service.RunCleanScript(processFilePath, outputPath, cleanOpts) cleanResult, cleanErr := service.RunCleanScript(processFilePath, outputPath, cleanOpts)
if cleanErr != nil { if cleanErr != nil {
service.CleanupExtractedFiles(extractedFiles) service.CleanupExtractedFiles(extractedFiles)
c.JSON(http.StatusInternalServerError, model.UploadResponse{ c.JSON(http.StatusInternalServerError, model.UploadResponse{
@@ -255,22 +255,37 @@ func Upload(c *gin.Context) {
} }
service.CleanupExtractedFiles(extractedFiles) service.CleanupExtractedFiles(extractedFiles)
repo := repository.GetRepository()
// 13. 如果是京东账单,软删除其他来源中包含"京东-订单编号"的记录 // 13. 如果是京东账单,软删除其他来源中包含"京东-订单编号"的记录
var jdRelatedDeleted int64 var jdRelatedDeleted int64
if billType == "jd" { if billType == "jd" && repo != nil {
repo := repository.GetRepository() deleted, err := repo.SoftDeleteJDRelatedBills()
if repo != nil { if err != nil {
deleted, err := repo.SoftDeleteJDRelatedBills() fmt.Printf("⚠️ 软删除京东关联记录失败: %v\n", err)
if err != nil { } else if deleted > 0 {
fmt.Printf("⚠️ 软删除京东关联记录失败: %v\n", err) jdRelatedDeleted = deleted
} else if deleted > 0 { fmt.Printf("🗑️ 已软删除 %d 条其他来源中的京东关联记录\n", deleted)
jdRelatedDeleted = deleted }
fmt.Printf("🗑️ 已软删除 %d 条其他来源中的京东关联记录\n", deleted) }
// 14. 核销跨批次退款(本次清洗中未在同批次内匹配到对应支出的退款)
var reconciledCount int
if repo != nil && cleanResult != nil {
for _, ur := range cleanResult.UnresolvedRefunds {
matched, rErr := repo.ReconcileRefund(billType, ur.OrderNo, ur.MerchantOrderNo, ur.Amount, ur.Time, ur.Merchant, ur.Description, ur.RefundOrderNo)
if rErr != nil {
fmt.Printf("⚠️ 退款核销失败: %v\n", rErr)
continue
}
if matched {
reconciledCount++
fmt.Printf("💰 已核销退款: 订单%s, 金额%.2f元\n", ur.OrderNo, ur.Amount)
} }
} }
} }
// 14. 返回成功响应 // 15. 返回成功响应
message := fmt.Sprintf("处理成功,新增 %d 条记录", cleanedCount) message := fmt.Sprintf("处理成功,新增 %d 条记录", cleanedCount)
if dedupResult.DuplicateCount > 0 { if dedupResult.DuplicateCount > 0 {
message = fmt.Sprintf("处理成功,新增 %d 条,跳过 %d 条重复记录", cleanedCount, dedupResult.DuplicateCount) message = fmt.Sprintf("处理成功,新增 %d 条,跳过 %d 条重复记录", cleanedCount, dedupResult.DuplicateCount)
@@ -278,18 +293,22 @@ func Upload(c *gin.Context) {
if jdRelatedDeleted > 0 { if jdRelatedDeleted > 0 {
message = fmt.Sprintf("%s标记删除 %d 条重复的京东订单", message, jdRelatedDeleted) message = fmt.Sprintf("%s标记删除 %d 条重复的京东订单", message, jdRelatedDeleted)
} }
if reconciledCount > 0 {
message = fmt.Sprintf("%s核销退款 %d 条", message, reconciledCount)
}
c.JSON(http.StatusOK, model.UploadResponse{ c.JSON(http.StatusOK, model.UploadResponse{
Result: true, Result: true,
Message: message, Message: message,
Data: &model.UploadData{ Data: &model.UploadData{
BillType: billType, BillType: billType,
FileURL: fmt.Sprintf("/download/%s", outputFileName), FileURL: fmt.Sprintf("/download/%s", outputFileName),
FileName: outputFileName, FileName: outputFileName,
RawCount: rawCount, RawCount: rawCount,
CleanedCount: cleanedCount, CleanedCount: cleanedCount,
DuplicateCount: dedupResult.DuplicateCount, DuplicateCount: dedupResult.DuplicateCount,
JDRelatedDeleted: jdRelatedDeleted, JDRelatedDeleted: jdRelatedDeleted,
ReconciledRefundCount: reconciledCount,
}, },
}) })
} }

View File

@@ -2,13 +2,14 @@ package model
// UploadData 上传响应数据 // UploadData 上传响应数据
type UploadData struct { type UploadData struct {
BillType string `json:"bill_type,omitempty"` // alipay/wechat/jd BillType string `json:"bill_type,omitempty"` // alipay/wechat/jd
FileURL string `json:"file_url,omitempty"` // 下载链接 FileURL string `json:"file_url,omitempty"` // 下载链接
FileName string `json:"file_name,omitempty"` // 文件名 FileName string `json:"file_name,omitempty"` // 文件名
RawCount int `json:"raw_count,omitempty"` // 存储到原始数据集合的记录数 RawCount int `json:"raw_count,omitempty"` // 存储到原始数据集合的记录数
CleanedCount int `json:"cleaned_count,omitempty"` // 存储到清洗后数据集合的记录数 CleanedCount int `json:"cleaned_count,omitempty"` // 存储到清洗后数据集合的记录数
DuplicateCount int `json:"duplicate_count,omitempty"` // 重复跳过的记录数 DuplicateCount int `json:"duplicate_count,omitempty"` // 重复跳过的记录数
JDRelatedDeleted int64 `json:"jd_related_deleted,omitempty"` // 软删除的京东关联记录数(其他来源中描述包含京东订单号的记录) JDRelatedDeleted int64 `json:"jd_related_deleted,omitempty"` // 软删除的京东关联记录数(其他来源中描述包含京东订单号的记录)
ReconciledRefundCount int `json:"reconciled_refund_count,omitempty"` // 跨批次核销的退款记录数
} }
// UploadResponse 上传响应 // UploadResponse 上传响应

View File

@@ -4,6 +4,7 @@ package mongo
import ( import (
"context" "context"
"fmt" "fmt"
"math"
"time" "time"
"go.mongodb.org/mongo-driver/bson" "go.mongodb.org/mongo-driver/bson"
@@ -16,6 +17,9 @@ import (
"billai-server/repository" "billai-server/repository"
) )
// refundEpsilon 退款核销后剩余金额的容差阈值,小于该值视为已全额退款
const refundEpsilon = 0.005
// Repository MongoDB 账单存储实现 // Repository MongoDB 账单存储实现
type Repository struct { type Repository struct {
client *mongo.Client client *mongo.Client
@@ -498,6 +502,74 @@ func (r *Repository) SoftDeleteJDRelatedBills() (int64, error) {
return result.ModifiedCount, nil return result.ModifiedCount, nil
} }
// ReconcileRefund 将跨批次退款核销到已存储的清洗后账单
// 按 bill_type + (transaction_id == orderNo 或 merchant_order_no == merchantOrderNo) 查找未删除记录
// 全额退款(剩余金额 <= refundEpsilon则软删除部分退款则扣减 amount 并追加备注
func (r *Repository) ReconcileRefund(billType, orderNo, merchantOrderNo string, refundAmount float64, refundTime, merchant, description, refundOrderNo string) (bool, error) {
if r.cleanedCollection == nil {
return false, fmt.Errorf("cleaned collection not initialized")
}
if orderNo == "" && merchantOrderNo == "" {
return false, nil
}
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
var or []bson.M
if orderNo != "" {
or = append(or, bson.M{"transaction_id": orderNo})
}
if merchantOrderNo != "" {
or = append(or, bson.M{"merchant_order_no": merchantOrderNo})
}
filter := bson.M{
"bill_type": billType,
"is_deleted": bson.M{"$ne": true},
"$or": or,
}
var bill model.CleanedBill
if err := r.cleanedCollection.FindOne(ctx, filter).Decode(&bill); err != nil {
if err == mongo.ErrNoDocuments {
return false, nil
}
return false, fmt.Errorf("查询待核销账单失败: %w", err)
}
remaining := bill.Amount - refundAmount
now := time.Now()
if remaining <= refundEpsilon {
update := bson.M{"$set": bson.M{
"is_deleted": true,
"updated_at": now,
"remark": fmt.Sprintf("[退款核销]全额退款%.2f元(退款单号%s);%s", refundAmount, refundOrderNo, bill.Remark),
}}
_, err := r.cleanedCollection.UpdateOne(ctx, bson.M{"_id": bill.ID}, update)
if err != nil {
return false, fmt.Errorf("核销退款失败: %w", err)
}
return true, nil
}
remaining = math.Round(remaining*100) / 100
update := bson.M{"$set": bson.M{
"amount": remaining,
"updated_at": now,
"remark": fmt.Sprintf("原金额%.2f元,退款%.2f元(退款单号%s);%s", bill.Amount, refundAmount, refundOrderNo, bill.Remark),
}}
_, err := r.cleanedCollection.UpdateOne(ctx, bson.M{"_id": bill.ID}, update)
if err != nil {
return false, fmt.Errorf("核销退款失败: %w", err)
}
return true, nil
}
// 建议: 为提升 ReconcileRefund 查询性能,可为 bills_cleaned 添加索引
// {transaction_id:1, bill_type:1} 和 {merchant_order_no:1, bill_type:1}(与现有"无索引"问题一并处理)
// GetClient 获取 MongoDB 客户端(用于兼容旧代码) // GetClient 获取 MongoDB 客户端(用于兼容旧代码)
func (r *Repository) GetClient() *mongo.Client { func (r *Repository) GetClient() *mongo.Client {
return r.client return r.client

View File

@@ -56,4 +56,10 @@ type BillRepository interface {
// 用于避免京东账单与其他来源(微信、支付宝)账单重复计算 // 用于避免京东账单与其他来源(微信、支付宝)账单重复计算
// 返回: 删除数量、错误 // 返回: 删除数量、错误
SoftDeleteJDRelatedBills() (int64, error) SoftDeleteJDRelatedBills() (int64, error)
// ReconcileRefund 将跨批次退款核销到已存储的清洗后账单
// 按 bill_type + (transaction_id == orderNo 或 merchant_order_no == merchantOrderNo) 查找未删除记录
// 全额退款(剩余金额 <= 0.005)则软删除;部分退款则扣减 amount 并追加备注
// 返回: 是否找到并核销了匹配记录、错误(未找到匹配记录不算错误,返回 matched=false
ReconcileRefund(billType, orderNo, merchantOrderNo string, refundAmount float64, refundTime, merchant, description, refundOrderNo string) (matched bool, err error)
} }

View File

@@ -199,6 +199,20 @@
dayData.set(category, (dayData.get(category) || 0) + amount); dayData.set(category, (dayData.get(category) || 0) + amount);
}); });
// 填充缺失的日期(零支出日期)
const now = new Date();
const allDates: string[] = [];
const cursor = new Date(cutoffDate);
while (cursor <= now) {
allDates.push(formatLocalDate(cursor));
cursor.setDate(cursor.getDate() + 1);
}
allDates.forEach(dateStr => {
if (!dailyMap.has(dateStr)) {
dailyMap.set(dateStr, new Map());
}
});
const dayCount = dailyMap.size; const dayCount = dailyMap.size;
// 根据天数决定聚合粒度 // 根据天数决定聚合粒度