mirror of
https://github.com/ThearchyHelios/AgentLab.git
synced 2026-11-15 00:21:58 +00:00
文档重新梳理:README 精简成介绍和索引,细节按主题拆进 docs/zh-CN 与 docs/en(17 篇一一对应),英文文档不含中文
README.zh-CN.md / README.md 只留:这是什么、为什么用它、主要功能(按领域一行一条,链到文档)、快速开始、部署、文档索引、 开发与测试、许可(原文不动)。原来的每一节按读者和任务分到 17 篇主题文档:ask-data、workflows、evidence、 data-sources(新增:接入数据库、只读保证、上传表格、跨库合并、遮罩列)、data-catalog、tools(新增:工具、审批档位、 能力登记、端点、网络规则——它们管所有工作流,不只应用)、apps、evaluation-and-ledger、schedules(含通知)、deployment、 configuration、postgresql-and-scaling、identity-and-security、architecture、development、upgrade-notes(按时间倒序, 每波先写要做的事)、faq。删掉的只有模块名(api/copilot_gen.py、chat/drive.py、data/schema_view.py)、重复的话和 「以前…现在…」的叙述;操作步骤、命令、配置名、限制、权限边界都有归宿。中文是原文,英文按新的中文写;英文里界面文字 只写英文名,开头注明界面目前只有简体中文,中文占位符换成英文占位符,中文数字的例子改成不依赖中文的说法。 数据剖析可以同时剖析 2–3 张表(baf23998)收进 data-catalog。 新增 scripts/check-docs.py:英文文档(README.md、docs/en/**)不许有中日韩字符和全角符号(代码块也算), README、docs/**、deploy/README.md 的相对链接和锚点都要通,docs/zh-CN 与 docs/en 文件一一对应、标题层级相同。 中文提示里指向 README 章节的地方改指新文档:cluster/guard.py、cluster/version.py、db/base.py 两处报错, data/schema_view.py 文档串,.env.example 三处,scripts/prod.sh、scripts/test-postgres.sh,terms.ts 两处、 DataSourcesTab.tsx、check-manage.mjs 注释,deploy/README.md 三处链接。deploy/agentlab.env 未动。 验收:check-docs 0 问题(英文文档 18 个 0 处中文,相对链接 369 条 0 断链,18 对标题层级一致);相关后端测试 117 通过、7 跳过;tsc -b、check-copy 通过。
This commit is contained in:
@@ -0,0 +1,64 @@
|
||||
English | [Simplified Chinese](../zh-CN/data-sources.md)
|
||||
|
||||
# Data Sources and Spreadsheet Import
|
||||
|
||||
This page covers connecting databases, uploading spreadsheets, and how AgentLab keeps queries read-only. It is for builders and admins. For the data catalog (recording the business meaning of tables and fields), see [Data Catalog](data-catalog.md). The user interface is currently available in Simplified Chinese only.
|
||||
|
||||
## Connect a database
|
||||
|
||||
A data source can be a business database (MySQL / MariaDB, PostgreSQL, or Oracle, with drivers in the `[db]` extra), an uploaded spreadsheet, or a manually registered SQLite file.
|
||||
|
||||
- After connecting, click Introspect Schema: it reads tables, columns, comments, foreign keys, and unique constraints. Column types are stored in each database's full notation, such as `VARCHAR2(20 CHAR)` or `ENUM('a','b')`.
|
||||
- Introspection is a background job, queued with drafting and profiling for the same data source. For a large database with thousands of tables, it first builds an object index and then reads by introspection scope (see [Data Catalog](data-catalog.md#large-databases-introspection-scope-and-on-demand-inclusion)).
|
||||
- One data source corresponds to one schema. Formal introspection doesn't accept `?schema=` for switching schemas temporarily; previews do.
|
||||
- Each data source has a query tool, `db_query__<source>`; at run time, agents look up table schemas with `db_schema__`.
|
||||
- The address of a data source card is `/data/databases#source-<id>`.
|
||||
- AgentLab's own database files (such as the `agentlab.db` left in the data directory after switching to PostgreSQL) can't be registered as data sources.
|
||||
|
||||
Introspection failing and "Not yet introspected" are two different states. Clicking the button won't fix a failure; deal with the timeout or change the schema first. The data source card shows the failure reason and the databases available on that server; agents are told the facts too, including that they can query `information_schema` directly to confirm the table structure.
|
||||
|
||||
## Read-only guarantees
|
||||
|
||||
- The SQL guard combines allowlisted verb checks, body scanning, multi-statement blocking, and a result-set cap.
|
||||
- `DROP` / `TRUNCATE` / `GRANT` and the like are always refused, even on writable data sources.
|
||||
- Statements that begin as queries but write are treated as writes: `WITH … DELETE`, `SELECT … INTO`, `EXPLAIN ANALYZE`, `PRAGMA … =`, `set_config()`, and so on.
|
||||
- Read-only data sources are also read-only at the connection layer, not just through the guard: SQLite is opened with `mode=ro`, and PostgreSQL / MySQL sessions use read-only transactions.
|
||||
- **Oracle has no session-level read-only switch**, which leaves only the guard. Configure a read-only account for read-only Oracle data sources.
|
||||
- On a writable data source, the same `db_query__<source>` executes queries directly, while a write needs confirmation first and is committed only after confirmation. A declaration in the capability registry can't make write statements skip approval.
|
||||
- In Ask Data for users and in personal apps, data sources are always read-only (see [Identity, Permissions, and Security](identity-and-security.md#ask-data-for-users)).
|
||||
|
||||
## Upload spreadsheets
|
||||
|
||||
An uploaded Excel or CSV file becomes a SQLite file, registered as a data source of `kind=sqlite`, with a `db_query__<name>` tool generated automatically. Numbers are computed by SQL rather than read by the model from text fragments.
|
||||
|
||||
- Column types are inferred column by column (INTEGER / REAL / TEXT), and empty cells are stored as NULL. If an integer column were stored as TEXT, `ORDER BY` would fall back to lexicographic order (`'10' < '9'`) without any error.
|
||||
- Why not the knowledge base: there a spreadsheet could only be chunked and retrieved, and the numbers would become whatever the model "reads" from fragments. Output review can catch numbers that appear out of nowhere, but not one cell misread from a pile of fragments.
|
||||
- For data sources imported from spreadsheets, the system writes notes from its verification results; the data catalog neither duplicates nor rewrites them.
|
||||
- An uploaded-spreadsheet cell cited in a report can be traced back to the cell in the original sheet (an inferred source).
|
||||
|
||||
### Complex reports: use an extraction recipe
|
||||
|
||||
- Complex reports are imported with an extraction recipe; CSV works too. Subtotals sandwiched between detail rows are checked and kept out of the data.
|
||||
- AI drafting for recipes uses the Drafting model chosen under Settings → Preferences; when it is empty, it follows Copilot's model.
|
||||
|
||||
### Accumulating by period
|
||||
|
||||
- Monthly detail files can accumulate by period into one table. A new period can trigger a schedule (see [Event triggers](schedules.md#event-triggers)).
|
||||
- Every historical version is a full copy, so disk usage grows with the number of periods and of referenced versions.
|
||||
- Besides the current version and versions referenced by runs, each data source keeps the 3 most recently activated versions; the rest are reclaimed and can't be activated again.
|
||||
|
||||
## Cross-database merge
|
||||
|
||||
Cross-database questions don't use federated queries. Each database first aggregates to the same key, then the merge query node (`merge`) merges by key outside the databases (in-memory SQLite, with the same read-only guard).
|
||||
|
||||
- Truncated inputs are rejected.
|
||||
- Mismatched key types, and row counts that grow after the merge, both trigger warnings.
|
||||
- The numbers in the report trace all the way back to the raw queries on both sides.
|
||||
|
||||
## Masked columns
|
||||
|
||||
List the columns to mask in the data source settings (`mask_columns`); they apply by column name across the whole data source.
|
||||
|
||||
- For masked columns, the model gets only the column name and type: their comments, and the catalog's Chinese name, code values, and description for the column, are withheld.
|
||||
- Profiling doesn't sample masked columns or observe their values.
|
||||
- For users with login on, masking is a boundary; for builders, admins, and everyone when login is off, it only reduces exposure (see [Which roles can see raw values](identity-and-security.md#which-roles-can-see-raw-values)).
|
||||
Reference in New Issue
Block a user