Skip to content

Commit 76fdcdb

Browse files
MonikaFeiglerMonikaFeigler
authored
Merge branch 'main' into devin/1773929518-custom-python-docs
2 parents fd40575 + ee0dc37 commit 76fdcdb

3 files changed

Lines changed: 205 additions & 5 deletions

File tree

‎_data/telemetry_tables.yml‎

Lines changed: 119 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ tables:
33

44
- id: kbc_bucket_snapshot
55
mode: project
6-
description: "This table shows snapshots of buckets in Storage."
6+
description: "**DEPRECATING** – This table is being replaced by the current state in `kbc_bucket`. \n\nThis table shows snapshots of buckets in Storage."
77
is_full_load: false
88
columns:
99
- name: bucket_id
@@ -253,7 +253,7 @@ tables:
253253

254254
- id: kbc_data_app_workspace
255255
mode: project
256-
description: "This table lists runs of [apps](/data-apps/) and their consumption metrics."
256+
description: "**DEPRECATING** – This table is being replaced by `kbc_non_sql_workspace_run`. \n\nThis table lists runs of [apps](/data-apps/) and their consumption metrics."
257257
is_full_load: false
258258
note: "A value of `0` in `data_app_runtime_hours`,`time_credits_used` or `billed_credits_used` means a correction was made. The row couldn't be removed due to incremental loads, so the original value was set to zero."
259259
columns:
@@ -308,7 +308,7 @@ tables:
308308

309309
- id: kbc_data_science_sandbox
310310
mode: project
311-
description: "This table lists Python/R [workspaces](/workspace/) and their consumption metrics."
311+
description: "**DEPRECATING** – This table is being replaced by `kbc_non_sql_workspace_run`. \n\nThis table lists Python/R [workspaces](/workspace/) and their consumption metrics."
312312
is_full_load: false
313313
note: "A value of `0` in `sandbox_runtime_hours`,`time_credits_used` or `billed_credits_used` means a correction was made. The row couldn't be removed due to incremental loads, so the original value was set to zero."
314314
columns:
@@ -737,7 +737,7 @@ tables:
737737

738738
- id: kbc_table_snapshot
739739
mode: project
740-
description: "This table shows [Storage table](/storage/tables/) snapshots."
740+
description: "**DEPRECATING** – This table is being replaced by the current state in `kbc_table`. \n\nThis table shows [Storage table](/storage/tables/) snapshots."
741741
is_full_load: false
742742
columns:
743743
- name: table_id
@@ -2010,7 +2010,7 @@ tables:
20102010

20112011
- id: kbc_workspace
20122012
mode: activity_center
2013-
description: "**DEPRECATING** – This table is being replaced by `kbc_sql_workspace` and `kbc_data_app`. \n\nThis table shows data about existing [workspaces](/workspace/). Unlike SQL and Data Science sandboxes, this table includes all workspaces of the project (i.e., those created by transformations)."
2013+
description: "**DEPRECATING** – This table is being replaced by `kbc_sql_workspace` and `kbc_non_sql_workspace`. \n\nThis table shows data about existing [workspaces](/workspace/). Unlike SQL and Data Science sandboxes, this table includes all workspaces of the project (i.e., those created by transformations)."
20142014
is_full_load: true
20152015
columns:
20162016
- name: kbc_workspace_id
@@ -2135,6 +2135,120 @@ tables:
21352135
description: "Flag indicating if the workspace is shared with other users (`true`, `false`)"
21362136
example: "`true`"
21372137

2138+
- id: kbc_non_sql_workspace
2139+
mode: project
2140+
description: "This table shows data about non-SQL [workspaces](/workspace/), including Data Apps (Streamlit, Python JS) and Data Science sandboxes (Python, R, etc.)."
2141+
is_full_load: false
2142+
columns:
2143+
- name: kbc_non_sql_workspace_id
2144+
pk: true
2145+
description: "Unique non-SQL workspace identifier"
2146+
example: "`985088171_kbc-us-east-1`"
2147+
- name: kbc_data_app_id
2148+
description: "Data App identifier (populated only for Data App workspaces with type `streamlit` or `python-js`; empty for Data Science sandboxes)"
2149+
example: "`7880_kbc-us-east-1_keboola.data-apps_985088171`"
2150+
- name: workspace_id_num
2151+
description: "Numeric part of the workspace identifier (without the stack suffix)"
2152+
example: "`985088171`"
2153+
- name: kbc_project_id
2154+
fk: kbc_project
2155+
description: "Foreign key to the Keboola project"
2156+
example: "`7880_kbc-us-east-1`"
2157+
- name: kbc_workspace_name
2158+
description: "Name of the workspace (derived from component configuration)"
2159+
example: "`My Data App`"
2160+
- name: component_id
2161+
description: "Identifier of the component that created the workspace"
2162+
example: "`keboola.data-apps`"
2163+
- name: kbc_component
2164+
description: "Human-readable component name"
2165+
example: "`Data App`"
2166+
- name: kbc_component_configuration_id
2167+
fk: kbc_component_configuration
2168+
description: "Foreign key to the component configuration that created this workspace"
2169+
example: "`7880_kbc-us-east-1_keboola.data-apps_985088171`"
2170+
- name: workspace_created
2171+
description: "Datetime when the workspace was created"
2172+
example: "`2023-06-13 20:41:42.268`"
2173+
- name: workspace_deleted
2174+
description: "Datetime when the workspace was deleted (empty if still active)"
2175+
example: "`2023-07-18 10:20:56.384`"
2176+
- name: workspace_updated
2177+
description: "Datetime when the workspace was last updated"
2178+
example: "`2023-07-12 12:01:24.181`"
2179+
- name: workspace_hostname
2180+
description: "Hostname of the workspace"
2181+
example: "`https://hello-world-759519.hub.north-europe.azure.keboola.com`"
2182+
- name: workspace_type
2183+
description: "Type of the non-SQL workspace (`streamlit`, `python-js`, `python`, `r`, etc.)"
2184+
example: "`streamlit`"
2185+
- name: kbc_token_id
2186+
fk: kbc_token
2187+
description: "Foreign key to the Keboola token that created or owns the workspace"
2188+
example: "`287689_kbc-us-east-1`"
2189+
- name: kbc_token_name
2190+
description: "Name of the token that created or owns the workspace"
2191+
example: "`john.doe@keboola.com`"
2192+
- name: storage_size_gb
2193+
description: "Size of the workspace storage in GB (used for persistent workspaces)"
2194+
example: "`2`"
2195+
- name: workspace_expire_hours
2196+
description: "Number of hours before the workspace expires due to inactivity"
2197+
example: "`1`"
2198+
- name: workspace_shared
2199+
description: "Flag indicating if the workspace is shared with other users (`true`, `false`)"
2200+
example: "`true`"
2201+
- name: is_persistent
2202+
description: "Flag indicating if the workspace has persistent storage (`true`, `false`)"
2203+
example: "`false`"
2204+
2205+
- id: kbc_non_sql_workspace_run
2206+
mode: project
2207+
description: "This table lists runs of non-SQL [workspaces](/workspace/) (Data Apps and Data Science sandboxes) and their consumption metrics. It replaces `kbc_data_app_workspace` and `kbc_data_science_sandbox` for non-SQL workspace run tracking."
2208+
is_full_load: false
2209+
note: "A value of `0` in `runtime_hours`, `time_credits_used`, or `billed_credits_used` means a correction was made. The row couldn't be removed due to incremental loads, so the original value was set to zero."
2210+
columns:
2211+
- name: kbc_non_sql_workspace_run_id
2212+
pk: true
2213+
description: "Identifier of the workspace active window (between starting and pausing the workspace)"
2214+
example: "`10910_kbc-eu-central-1_8c9e68ac-3a40-4aea-a62c-34ef37d12a5a`"
2215+
- name: kbc_non_sql_workspace_id
2216+
fk: kbc_non_sql_workspace
2217+
description: "Foreign key to the non-SQL workspace"
2218+
example: "`10910_kbc-eu-central-1`"
2219+
- name: kbc_workspace_name
2220+
description: "Name of the workspace (derived from component configuration)"
2221+
example: "`My Data App`"
2222+
- name: kbc_project_id
2223+
fk: kbc_project
2224+
description: "Foreign key to the Keboola project"
2225+
example: "`1075_kbc-eu-central-1`"
2226+
- name: date
2227+
pk: true
2228+
description: "Date for which the runtime hours and credits are calculated"
2229+
example: "`2025-06-30`"
2230+
- name: workspace_start_at
2231+
description: "Datetime when the workspace resume window started"
2232+
example: "`2025-06-30 11:22:08`"
2233+
- name: workspace_sleep_at
2234+
description: "Datetime when the workspace resume window ended"
2235+
example: "`2025-06-30 14:03:40`"
2236+
- name: workspace_type
2237+
description: "Type of the non-SQL workspace (`streamlit`, `python-js`, `python`, `r`, etc.)"
2238+
example: "`streamlit`"
2239+
- name: backend_size
2240+
description: "Size of the workspace backend"
2241+
example: "`Small`"
2242+
- name: runtime_hours
2243+
description: "Runtime of the workspace on the particular date in hours"
2244+
example: "`2.413333`"
2245+
- name: time_credits_used
2246+
description: "Number of time credits consumed by the workspace on the particular date"
2247+
example: "`2.413333`"
2248+
- name: billed_credits_used
2249+
description: "Number of the actually billed credits"
2250+
example: "`2.413333`"
2251+
21382252
- id: kbc_workspace_event
21392253
mode: activity_center
21402254
description: "This table shows data about [workspace](/workspace/) events."

‎workspace/index.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -294,6 +294,10 @@ or [File Output Mapping](/transformations/mappings/#file-output-mapping) (or bot
294294
Unloading data is useful, for example, when your ad-hoc analysis leads to
295295
valuable results, or when you trained a new model which you'd like to use in transformations.
296296

297+
For Snowflake and BigQuery workspaces, you can also export a single table directly from the workspace schema to
298+
[File Storage](/storage/files/) via the Storage API. See
299+
[Exporting Workspace Tables to Files](/workspace/table-export/) for details.
300+
297301
### Data Persistency (beta)
298302
When this feature is enabled in a project, your data in workspaces can be kept. This way you can, when you return, start where you left off without losing data or time by importing the data again or executing scripts to get to the right stage.
299303

‎workspace/table-export.md‎

Lines changed: 82 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,82 @@
1+
---
2+
title: Exporting Workspace Tables to Files
3+
permalink: /workspace/table-export/
4+
---
5+
6+
* TOC
7+
{:toc}
8+
9+
You can export a table that lives in a Snowflake or BigQuery workspace schema to [File Storage](/storage/files/)
10+
via the Storage API. The export runs as an asynchronous storage job, and the job result contains the file ID of the
11+
exported file, which can then be downloaded like any other Keboola file.
12+
13+
This is useful when you build data in a workspace (for example, via the [SQL Editor](/workspace/sql-editor/) or a
14+
custom integration) and need to move the resulting table outside of the workspace without going through Storage
15+
output mapping.
16+
17+
Currently supported backends:
18+
19+
- **Snowflake**
20+
- **BigQuery**
21+
22+
## Endpoint
23+
24+
```
25+
POST https://connection.keboola.com/v2/storage/workspaces/{workspace_id}/table-export
26+
X-StorageApi-Token: your_token
27+
Content-Type: application/json
28+
29+
{
30+
"tableName": "my_table",
31+
"fileName": "custom_export",
32+
"fileType": "csv",
33+
"gzip": true
34+
}
35+
```
36+
37+
### Request Body
38+
39+
| Field | Type | Required | Description |
40+
|-------------|---------|----------|--------------------------------------------------------------------------------------------|
41+
| `tableName` | string | yes | Name of the table (or view) to export from the workspace schema. |
42+
| `fileName` | string | yes | Name that will be used for the resulting file in File Storage. |
43+
| `fileType` | string | no | Output format: `csv` (default) or `parquet`. |
44+
| `gzip` | boolean | no | When `true`, the exported file is gzip-compressed. Default `false`. Ignored for Parquet. |
45+
46+
### Response
47+
48+
The endpoint returns a standard asynchronous [storage job](/overview/#storage-jobs) with HTTP 202. When the job
49+
finishes, its `results` contain the ID of the exported file:
50+
51+
```json
52+
{
53+
"file": {
54+
"id": 12345678
55+
}
56+
}
57+
```
58+
59+
Download the file with the standard [file download](/integrate/storage/api/importer/#download-a-file) flow.
60+
61+
## Backend-Specific Notes
62+
63+
### Snowflake
64+
65+
- Supports **CSV** and **Parquet**.
66+
- Works with all project file storage providers (AWS S3, Azure Blob Storage, Google Cloud Storage).
67+
68+
### BigQuery
69+
70+
- Supports **CSV** and **Parquet**.
71+
- Available for BigQuery projects only; the exported file always lands in the project's GCS file storage.
72+
73+
## Limitations
74+
75+
- The workspace must be a **table workspace**. File/Python/R workspaces are not supported.
76+
- **Reader account** workspaces cannot export data through this endpoint.
77+
- The workspace must use a supported backend (Snowflake or BigQuery).
78+
79+
## API Reference
80+
81+
See the full request/response specification in the
82+
[Storage API reference](https://keboolastorageapi.docs.apiary.io/#reference/workspaces/export-table-from-workspace/export-table-from-workspace).

0 commit comments

Comments
 (0)