Skip to content

Commit bd91730

Browse files
vojtabiberlevojtabiberle
authored
Merge pull request #2500 from keboola/vb/DMD-920/llm-export-processor-lineage
DMD-920 - Add processor and lineage for llm export
2 parents 37a22da + 2653cca commit bd91730

18 files changed

Lines changed: 2673 additions & 462 deletions

File tree

‎CLAUDE.md‎

Lines changed: 74 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -130,6 +130,43 @@ Services auto-reload on code changes using Air.
130130
- Follow KISS principle - clear, straightforward code
131131
- Follow DRY principle - extract common functionality
132132

133+
### Avoid Deep Nesting
134+
- **Maximum 2 levels of nesting** - Extract helper functions for deeper loops
135+
- **Use early returns** - Invert conditions and return/continue early to reduce indentation
136+
- **Extract inner loops** - When you have nested `for` loops, extract the inner loop to a separate function
137+
- **Flatten conditionals** - Replace `if { if { if { }}}` with guard clauses and early returns
138+
139+
Example - instead of:
140+
```go
141+
for _, item := range items {
142+
if item != nil {
143+
for _, sub := range item.Children {
144+
if sub.Valid {
145+
process(sub)
146+
}
147+
}
148+
}
149+
}
150+
```
151+
Use:
152+
```go
153+
for _, item := range items {
154+
if item == nil {
155+
continue
156+
}
157+
processChildren(item.Children)
158+
}
159+
160+
func processChildren(children []Child) {
161+
for _, sub := range children {
162+
if !sub.Valid {
163+
continue
164+
}
165+
process(sub)
166+
}
167+
}
168+
```
169+
133170
### Prohibited Patterns
134171
- **Global variables** - Use dependency injection
135172
- **init() functions** - Use explicit initialization
@@ -145,6 +182,43 @@ Services auto-reload on code changes using Air.
145182
- **Direct "zap" logger** - Use `internal/pkg/log` package
146183
- **Naked returns**
147184
- **Underscores in package names**
185+
- **Deprecated functions** - Remove deprecated functions instead of keeping them; don't mark as deprecated if still needed
186+
- **Variables only used for logging** - Don't create variables solely for debug/info logging; inline or remove them
187+
- **Pseudo-version SDK dependencies** - Use tagged versions (e.g., `v2.12.0`) instead of pseudo-versions (e.g., `v2.9.1-0.20260109014800-d596b2c092e2`)
188+
189+
### Go Naming Conventions
190+
- **No "Get" prefix for getters** - Use `Type()` not `GetType()`, `Name()` not `GetName()`, `Source()` not `GetSource()`
191+
- **Use "Get" only when it fetches external data** - e.g., `GetFromAPI()`, `GetFromDatabase()`
192+
193+
### Function Return Values
194+
- **Wrap multiple return values** - When returning more than 3 values, wrap them in a struct
195+
- **Return errors, don't silently continue** - When an operation fails, return the error instead of continuing with empty/default values
196+
197+
Example - instead of:
198+
```go
199+
func fetchData() (a, b, c, d Type, err error)
200+
```
201+
Use:
202+
```go
203+
type fetchResult struct {
204+
A, B, C, D Type
205+
}
206+
func fetchData() (fetchResult, error)
207+
```
208+
209+
### String Building
210+
- **Use `strings.Builder` for concatenation** - More efficient than `fmt.Sprintf` or `+` for building strings
211+
212+
Example:
213+
```go
214+
func buildUID(prefix, name string) string {
215+
var b strings.Builder
216+
b.WriteString(prefix)
217+
b.WriteByte(':')
218+
b.WriteString(name)
219+
return b.String()
220+
}
221+
```
148222

149223
### Required Patterns
150224
- **Context handling**: Pass context as first parameter; respect cancellation; never store in structs

‎go.mod‎

Lines changed: 56 additions & 57 deletions
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@ require (
1313
entgo.io/ent v0.14.5
1414
github.com/ActiveState/vt10x v1.3.1
1515
github.com/AlecAivazis/survey/v2 v2.3.7
16-
github.com/DataDog/dd-trace-go/v2 v2.5.0
16+
github.com/DataDog/dd-trace-go/v2 v2.4.0
1717
github.com/MichaelMure/go-term-markdown v0.1.4
1818
github.com/Netflix/go-expect v0.0.0-20220104043353-73e0943537d2
1919
github.com/Shopify/toxiproxy/v2 v2.12.0
@@ -28,8 +28,8 @@ require (
2828
github.com/fatih/color v1.18.0
2929
github.com/go-playground/locales v0.14.1
3030
github.com/go-playground/universal-translator v0.18.1
31-
github.com/go-playground/validator/v10 v10.30.1
32-
github.com/go-resty/resty/v2 v2.17.1
31+
github.com/go-playground/validator/v10 v10.28.0
32+
github.com/go-resty/resty/v2 v2.17.0
3333
github.com/gofrs/flock v0.13.0
3434
github.com/gofrs/uuid/v5 v5.4.0
3535
github.com/google/go-cmp v0.7.0
@@ -44,13 +44,13 @@ require (
4444
github.com/keboola/go-cloud-encrypt v0.0.0-20250422071622-41a5d5547c43
4545
github.com/keboola/go-utils v1.4.0
4646
github.com/keboola/keboola-sdk-go/v2 v2.12.0
47-
github.com/klauspost/compress v1.18.3
47+
github.com/klauspost/compress v1.18.1
4848
github.com/klauspost/pgzip v1.2.6
4949
github.com/kylelemons/godebug v1.1.0
5050
github.com/lafikl/consistent v0.0.0-20220512074542-bdd3606bfc3e
5151
github.com/lestrrat-go/strftime v1.1.1
52-
github.com/mattn/go-sqlite3 v1.14.33
53-
github.com/miekg/dns v1.1.70
52+
github.com/mattn/go-sqlite3 v1.14.32
53+
github.com/miekg/dns v1.1.68
5454
github.com/mitchellh/hashstructure/v2 v2.0.2
5555
github.com/oauth2-proxy/mockoidc v0.0.0-20240214162133-caebfff84d25
5656
github.com/oauth2-proxy/oauth2-proxy/v7 v7.13.0
@@ -61,40 +61,40 @@ require (
6161
github.com/relvacode/iso8601 v1.7.0
6262
github.com/rs/zerolog v1.34.0
6363
github.com/santhosh-tekuri/jsonschema/v6 v6.0.2
64-
github.com/schollz/progressbar/v3 v3.19.0
65-
github.com/sirupsen/logrus v1.9.4
64+
github.com/schollz/progressbar/v3 v3.18.0
65+
github.com/sirupsen/logrus v1.9.3
6666
github.com/spf13/afero v1.15.0
6767
github.com/spf13/cast v1.10.0
68-
github.com/spf13/cobra v1.10.2
68+
github.com/spf13/cobra v1.10.1
6969
github.com/spf13/pflag v1.0.10
7070
github.com/spf13/viper v1.21.0
7171
github.com/stretchr/testify v1.11.1
7272
github.com/umisama/go-regexpcache v0.0.0-20150417035358-2444a542492f
7373
github.com/urfave/negroni/v3 v3.1.1
74-
github.com/valyala/fasthttp v1.69.0
75-
github.com/valyala/fastjson v1.6.7
74+
github.com/valyala/fasthttp v1.68.0
75+
github.com/valyala/fastjson v1.6.4
7676
github.com/writeas/go-strip-markdown/v2 v2.1.1
77-
github.com/xtaci/kcp-go/v5 v5.6.61
78-
go.etcd.io/etcd/api/v3 v3.6.7
79-
go.etcd.io/etcd/client/v3 v3.6.7
80-
go.etcd.io/etcd/tests/v3 v3.6.7
77+
github.com/xtaci/kcp-go/v5 v5.6.40
78+
go.etcd.io/etcd/api/v3 v3.6.6
79+
go.etcd.io/etcd/client/v3 v3.6.6
80+
go.etcd.io/etcd/tests/v3 v3.6.6
8181
go.nhat.io/aferocopy/v2 v2.0.3
82-
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.64.0
83-
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.64.0
84-
go.opentelemetry.io/contrib/propagators/b3 v1.39.0
85-
go.opentelemetry.io/otel v1.39.0
86-
go.opentelemetry.io/otel/bridge/opencensus v1.39.0
87-
go.opentelemetry.io/otel/exporters/prometheus v0.61.0
88-
go.opentelemetry.io/otel/metric v1.39.0
89-
go.opentelemetry.io/otel/sdk v1.39.0
90-
go.opentelemetry.io/otel/sdk/metric v1.39.0
91-
go.opentelemetry.io/otel/trace v1.39.0
82+
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.63.0
83+
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.63.0
84+
go.opentelemetry.io/contrib/propagators/b3 v1.38.0
85+
go.opentelemetry.io/otel v1.38.0
86+
go.opentelemetry.io/otel/bridge/opencensus v1.38.0
87+
go.opentelemetry.io/otel/exporters/prometheus v0.60.0
88+
go.opentelemetry.io/otel/metric v1.38.0
89+
go.opentelemetry.io/otel/sdk v1.38.0
90+
go.opentelemetry.io/otel/sdk/metric v1.38.0
91+
go.opentelemetry.io/otel/trace v1.38.0
9292
go.uber.org/zap v1.27.1
93-
goa.design/goa/v3 v3.24.1
94-
goa.design/plugins/v3 v3.24.1
95-
golang.org/x/exp v0.0.0-20260112195511-716be5621a96
96-
golang.org/x/sync v0.19.0
97-
google.golang.org/grpc v1.78.0
93+
goa.design/goa/v3 v3.22.6
94+
goa.design/plugins/v3 v3.22.6
95+
golang.org/x/exp v0.0.0-20251113190631-e25ba8c21ef6
96+
golang.org/x/sync v0.18.0
97+
google.golang.org/grpc v1.77.0
9898
gopkg.in/Knetic/govaluate.v3 v3.0.0
9999
v.io/x/lib v0.1.21
100100
)
@@ -104,7 +104,7 @@ require (
104104
github.com/DataDog/datadog-agent/comp/core/tagger/origindetection v0.71.0 // indirect
105105
github.com/DataDog/datadog-agent/pkg/opentelemetry-mapping-go/otlp/attributes v0.71.0 // indirect
106106
github.com/DataDog/datadog-agent/pkg/version v0.71.0 // indirect
107-
github.com/DataDog/go-libddwaf/v4 v4.8.0 // indirect
107+
github.com/DataDog/go-libddwaf/v4 v4.6.1 // indirect
108108
github.com/VividCortex/ewma v1.2.0 // indirect
109109
github.com/bgentry/speakeasy v0.2.0 // indirect
110110
github.com/bitfield/gotestdox v0.2.2 // indirect
@@ -121,12 +121,11 @@ require (
121121
github.com/shirou/gopsutil/v4 v4.25.8-0.20250809033336-ffcdc2b7662f // indirect
122122
github.com/spiffe/go-spiffe/v2 v2.6.0 // indirect
123123
github.com/theckman/httpforwarded v0.4.0 // indirect
124-
go.etcd.io/etcd/etcdctl/v3 v3.6.7 // indirect
124+
go.etcd.io/etcd/etcdctl/v3 v3.6.6 // indirect
125125
go.etcd.io/gofail v0.2.0 // indirect
126126
go.etcd.io/raft/v3 v3.6.0 // indirect
127-
go.opentelemetry.io/collector/featuregate v1.46.0 // indirect
127+
go.opentelemetry.io/collector/featuregate v1.39.0 // indirect
128128
go.opentelemetry.io/collector/internal/telemetry v0.133.0 // indirect
129-
go.opentelemetry.io/collector/pdata/pprofile v0.140.0 // indirect
130129
go.opentelemetry.io/contrib/bridges/otelzap v0.12.0 // indirect
131130
go.opentelemetry.io/otel/log v0.13.0 // indirect
132131
go.yaml.in/yaml/v2 v2.4.3 // indirect
@@ -166,16 +165,16 @@ require (
166165
github.com/tinylib/msgp v1.3.0 // indirect
167166
go.uber.org/atomic v1.11.0
168167
go.uber.org/multierr v1.11.0 // indirect
169-
golang.org/x/crypto v0.47.0
170-
golang.org/x/mod v0.32.0 // indirect
171-
golang.org/x/net v0.49.0
172-
golang.org/x/sys v0.40.0
173-
golang.org/x/term v0.39.0
174-
golang.org/x/text v0.33.0
168+
golang.org/x/crypto v0.45.0
169+
golang.org/x/mod v0.30.0 // indirect
170+
golang.org/x/net v0.47.0
171+
golang.org/x/sys v0.38.0
172+
golang.org/x/term v0.37.0
173+
golang.org/x/text v0.31.0
175174
golang.org/x/time v0.14.0 // indirect
176-
golang.org/x/tools v0.41.0
175+
golang.org/x/tools v0.39.0
177176
golang.org/x/xerrors v0.0.0-20240903120638-7835f813f4da
178-
google.golang.org/protobuf v1.36.11
177+
google.golang.org/protobuf v1.36.10
179178
gopkg.in/ini.v1 v1.67.0 // indirect
180179
gopkg.in/yaml.v2 v2.4.0 // indirect
181180
gopkg.in/yaml.v3 v3.0.1
@@ -272,9 +271,9 @@ require (
272271
github.com/envoyproxy/go-control-plane/envoy v1.35.0 // indirect
273272
github.com/envoyproxy/protoc-gen-validate v1.2.1 // indirect
274273
github.com/felixge/httpsnoop v1.0.4 // indirect
275-
github.com/gabriel-vasile/mimetype v1.4.12 // indirect
274+
github.com/gabriel-vasile/mimetype v1.4.10 // indirect
276275
github.com/ghodss/yaml v1.0.1-0.20220118164431-d8423dcdf344 // indirect
277-
github.com/go-chi/chi/v5 v5.2.4 // indirect
276+
github.com/go-chi/chi/v5 v5.2.3 // indirect
278277
github.com/go-jose/go-jose/v3 v3.0.4 // indirect
279278
github.com/go-jose/go-jose/v4 v4.1.3 // indirect
280279
github.com/go-logr/logr v1.4.3 // indirect
@@ -302,7 +301,7 @@ require (
302301
github.com/googleapis/gax-go/v2 v2.15.0 // indirect
303302
github.com/gorilla/mux v1.8.1 // indirect
304303
github.com/grpc-ecosystem/go-grpc-middleware v1.4.0 // indirect
305-
github.com/grpc-ecosystem/grpc-gateway/v2 v2.27.2 // indirect
304+
github.com/grpc-ecosystem/grpc-gateway/v2 v2.27.1 // indirect
306305
github.com/hashicorp/go-multierror v1.1.1 // indirect
307306
github.com/hashicorp/go-version v1.7.0 // indirect
308307
github.com/hashicorp/hcl/v2 v2.23.0 // indirect
@@ -336,8 +335,8 @@ require (
336335
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
337336
github.com/power-devops/perfstat v0.0.0-20240221224432-82ca36839d55 // indirect
338337
github.com/prometheus/client_model v0.6.2 // indirect
339-
github.com/prometheus/common v0.67.4 // indirect
340-
github.com/prometheus/procfs v0.19.2 // indirect
338+
github.com/prometheus/common v0.67.1 // indirect
339+
github.com/prometheus/procfs v0.17.0 // indirect
341340
github.com/redis/go-redis/v9 v9.14.0 // indirect
342341
github.com/richardartoul/molecule v1.0.1-0.20240531184615-7ca0df43c0b3 // indirect
343342
github.com/rivo/uniseg v0.4.7 // indirect
@@ -364,25 +363,25 @@ require (
364363
github.com/zclconf/go-cty v1.16.2 // indirect
365364
github.com/zclconf/go-cty-yaml v1.1.0 // indirect
366365
go.etcd.io/bbolt v1.4.3 // indirect
367-
go.etcd.io/etcd/client/pkg/v3 v3.6.7 // indirect
368-
go.etcd.io/etcd/pkg/v3 v3.6.7 // indirect
369-
go.etcd.io/etcd/server/v3 v3.6.7 // indirect
366+
go.etcd.io/etcd/client/pkg/v3 v3.6.6 // indirect
367+
go.etcd.io/etcd/pkg/v3 v3.6.6 // indirect
368+
go.etcd.io/etcd/server/v3 v3.6.6 // indirect
370369
go.opencensus.io v0.24.0 // indirect
371370
go.opentelemetry.io/auto/sdk v1.2.1 // indirect
372371
go.opentelemetry.io/collector/component v1.39.0 // indirect
373-
go.opentelemetry.io/collector/pdata v1.46.0 // indirect
372+
go.opentelemetry.io/collector/pdata v1.39.0 // indirect
374373
go.opentelemetry.io/contrib/detectors/gcp v1.38.0 // indirect
375374
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.37.0 // indirect
376375
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.37.0 // indirect
377-
go.opentelemetry.io/proto/otlp v1.7.1 // indirect
376+
go.opentelemetry.io/proto/otlp v1.7.0 // indirect
378377
gocloud.dev v0.43.0 // indirect
379-
golang.org/x/image v0.35.0 // indirect
380-
golang.org/x/oauth2 v0.34.0 // indirect
381-
golang.org/x/perf v0.0.0-20260112171951-5abaabe9f1bd // indirect
378+
golang.org/x/image v0.33.0 // indirect
379+
golang.org/x/oauth2 v0.33.0 // indirect
380+
golang.org/x/perf v0.0.0-20251112180420-cfbd823f7301 // indirect
382381
google.golang.org/api v0.252.0 // indirect
383382
google.golang.org/genproto v0.0.0-20251007200510-49b9836ed3ff // indirect
384-
google.golang.org/genproto/googleapis/api v0.0.0-20251029180050-ab9386a59fda // indirect
385-
google.golang.org/genproto/googleapis/rpc v0.0.0-20260114163908-3f89685c29c3 // indirect
383+
google.golang.org/genproto/googleapis/api v0.0.0-20251022142026-3a174f9686a8 // indirect
384+
google.golang.org/genproto/googleapis/rpc v0.0.0-20251022142026-3a174f9686a8 // indirect
386385
gopkg.in/natefinch/lumberjack.v2 v2.2.1 // indirect
387386
gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7 // indirect
388387
gotest.tools/gotestsum v1.13.0 // indirect

0 commit comments

Comments
 (0)