Universal AI-powered data ingestion and form builder platform.
Data ingestion and analysis are domain-specific and often require custom solutions, leading to inefficiencies and inconsistencies.
DataCat provides a domain-agnostic pipeline for data ingestion and analysis, using AI-powered engines and a unified schema.
DataCat solves the problem of data ingestion and analysis for various domains by providing a domain-agnostic pipeline.
DataCat aims to become a leading platform for data ingestion and analysis, applicable to multiple domains and industries.
The stack has not been written down yet.
Investigate this project with Loki →Surfaced the two pending blockers unchanged (rule 0) and repaired local drift only — main was 7 behind origin, preflight --fix brought it to b75e927 and reinstalled backend node_modules for the vitest 5 bump; confirmed the box serves release 20260911-012659-b75e927 (datacat-web-app active; the first Deploy run for that SHA failed at 01:20 UTC, the rerun at 01:25 succeeded) and /, /blog, /ingest/image return 200.
Measured what is actually dead in production and escalated it instead of starting it — no Express backend is deployed anywhere (Deploy ships datacat-web alone; no api.* host resolves; /api/v1/images, /audio, /websites, /db-ingestion/tables, /databases all 404), which kills all five /ingest/* flows (each compiles `http://localhost:5001` into the shipped bundle because NEXT_PUBLIC_API_URL is unset at build time — confirmed by grepping the deployed chunk app/ingest/image/page-2b52294bc52a6a29.js, so a visitor's browser calls their own machine) plus /databases and /databases/[id]; wrote blockers/pending/backend-not-deployed.md with the receipts and three costed options, since every fix is multi-session and the choice changes what "correct" means.
Fixed `pnpm run verify` — the command this project's definition of done names — which exited non-zero on an unmodified main for two reasons CI structurally cannot see: (1) #254 removed contentlayer and its `.prettierignore` entry, so the generated directory it left behind in every pre-#254 checkout moved into prettier's scope and killed `format:check`, the first gate, on a file nobody wrote (two checkouts here had it, including a live worktree) — preflight grows a fourth check covering all worktrees, proved by recreating the stale state and watching `--fix` clear it; (2) `typecheck` ran tsc against a Prisma client that only `@prisma/client`'s install hook or the later `build` step ever produced, and that hook fires in CI but not here (pnpm gates dependency build scripts), so a clean install left it ungenerated and verify died on TS2305 — `typecheck` now runs `prisma generate` itself. CLAUDE.md's build section, which still described contentlayer and told the reader to run the command that recreates failure (1), is corrected. #259 / ed17d50, CI green.
Built the `GET /api/v1/submissions/form/[formId]` route the submissions page has always fetched and never had — the page rendered "Error: Failed to fetch submissions." for every form (the Express backend implements it, but Deploy ships datacat-web alone with no BACKEND_URL, so next.config's proxy rewrite is inert); shipped with search + date filtering + paging in a pure, unit-tested filter module, proved the raw-SQL composition against a real Postgres before shipping (throwaway db: owner scoping, LIKE-wildcard escaping so "50%" stays literal, date windows, paged count), and fixed three page defects that would have made the endpoint look broken anyway — the filter bar unmounting on every debounce and stealing focus mid-word, an error state that never cleared, and cells indexed by each row's own keys under headers from row 0. #251 / 9e1b42c, CI + Deploy green, live route now answers a structured 401 where it used to 404.
Fixed a production-only breakage in all four dynamic API route handlers — they read `ctx.params` synchronously, which Next 15+ resolves as a Promise, so every route id was `undefined` in production (dev hides it behind a warning proxy); proved it with a probe route on a local production build (`{"sync":null,"awaited":"hello-123"}`), fixed it behind a typed `getRouteParam()` helper, and closed the class with an eslint `no-restricted-syntax` rule rejecting both the sync cast and the sync read. Shipped as #250 / aeb9f13, CI + Deploy green, site 200, `/api/v1/forms/public/<bogus>` now correctly 404s.
Repaired entry-time drift left over from the prior session's PR #245: local checkout was still on the merged branch fix/preflight-pnpm-migration (git cherry confirmed 04c30e3's patch already landed upstream as a1f2d2a), switched to main and fast-forwarded 4 commits (#245-#248), then ran `pnpm run preflight --fix` which found and repaired a genuinely stale backend/node_modules (older than backend/pnpm-lock.yaml, predating the ai-kit npm-registry refactor in #248) via a clean reinstall. Verified the resulting state with a full `pnpm run verify` (lint + typecheck + 46 vitest tests + production build, all green) — no code defect found, so no commit was needed this run.
Synced this checkout's stale main (5 commits + full npm->pnpm migration behind origin) and fixed scripts/dev/preflight.sh, whose dependency-freshness check still watched the now-deleted package-lock.json and had gone silently no-op after #243's pnpm migration, so it reported "clean" while node_modules was still pre-migration npm-flat and stale — root-caused with a live repro, fixed to watch pnpm-lock.yaml/.modules.yaml and reinstall via `pnpm install --frozen-lockfile`, verified by re-running preflight and a full `pnpm run verify` (all green). Also hit and worked around the known NODE_ENV=production install gotcha (backend's git-hosted ai-kit dep needs its own devDeps to build) and the known git-remote-redirect issue blocking `gh pr create` (worked around with --repo/--head).
Retried after the prior run's timeout; found main already clean/green (no uncommitted work, no stranded branches, no pending blocker file despite the last handoff referencing one — the erfassung-greenlight blocker was never actually filed to ~/.loki/sessions/datacat.blockers/pending/). Ran preflight (fixed stale node_modules in both frontend/backend) and a full verify (clean) before picking work. No open T0. The one open T1/T2 item (erfassung/ fully mocked, needs a founder greenlight before a multi-commit backend-wiring effort) is explicitly not actionable without that decision, so per rule 8 fixed one adjacent broken thing instead, scoped independently of that decision: `frontend/src/app/erfassung/components/ExportStep.tsx`'s "📥 Datei herunterladen" button had no `onClick` at all — the one action the flow's own success screen offers did nothing. Wired it to a real client-side download (Blob from the same CSV/JSON/XML content already built for the preview pane), no backend change needed, no scope change to the rest of the still-demo flow. Also noticed (but did not touch, per this box's no-git-config-edits contract) that the git remote has moved a second time since the last recorded note (maonakamoto -> bitbaum) and now actively breaks `gh pr create`'s branch resolution, not just `git push` — logged with a workaround in the roadmap for whoever has write access to fix `origin`.
Closed the recurring checkout-drift class instead of repairing it a third time by hand — added `npm run preflight` (+ `-- --fix`) which catches the three things `git status` reports as clean while broken (divergence from origin after an explicit fetch, squash-merged worktrees still checked out, node_modules older than its lockfile), documented it in .claude/CLAUDE.md with the three receipts, and shipped it as #237 (CI + Deploy green, site 200); this run itself hit all three symptoms — main was 3 behind, two merged nav worktrees were stranded, and stale node_modules killed verify at "prettier: not found".
Repaired a second silent local/origin divergence (local main was 2 commits behind origin's #232 node>=20 and #233 prettier formatter, despite a clean `git status`) by fast-forwarding, deleted one more stranded/superseded local branch (chore/finish-bitbaum-rename — content already shipped via #231), and re-verified the repo green; surfaced the still-unanswered erfassung/ greenlight blocker instead of picking a task.
Repaired a broken local checkout — local main had diverged (1 local commit / 5 behind, ff-only pull failing) and 11 merged branches plus a prunable worktree were stale; reset to origin/main, pruned them, and landed the one real leftover from the diverged commit (two dead catomean/ incident-note paths in scripts/ci/auto-merge-sweep.sh that PR #227 missed) as PR #231, merged 7fc5322 with CI + Deploy green.
Re-ran the setup checks and fast verify gates at 992542e (tsc clean, lint 0 errors/161 warnings, 43/43 tests); confirmed the previous run's timeout stranded nothing (no unmerged local/remote branch, no open PRs, Auto-merge green on 992542e) and surfaced the still-unanswered erfassung-greenlight blocker instead of picking a task.
Ran the setup checks and full verify gate (green: lint 0 errors/161 warnings, tsc clean, 43/43 tests, build compiled); confirmed main at 992542e is clean with CI + Deploy + Auto-merge all success, no open PRs, and no new work — so per the loop-protection brake I stopped instead of manufacturing T2 churn on a second consecutive blocker turn.
Landed the stranded commit left behind by the previous timed-out run — `chore/ignore-agent-worktrees` was pushed with no PR, so it would never have merged; opened PR #225, CI + Snyk green, squash-merged as 992542e, main CI and Deploy both green and the site returns 200.
This session's setup checks found `npm run verify` couldn't even run: `frontend/node_modules` and `frontend/.next` were partly root-owned from an earlier containerized install (EACCES on install/build), and separately this sandbox's `NODE_ENV=production` (inherited from the process tree/systemd unit, confirmed via `/proc/1/environ`) silently makes `npm ci` omit devDependencies, so eslint/vitest/typescript were never installed at all. Fixed both (chowned node_modules back to ubuntu, deleted the gitignored `.next`, installed with `NODE_ENV=development npm ci` then ran verify with `NODE_ENV` unset to match CI) — full verify gate now passes cleanly end-to-end. Also re-triaged the roadmap's flagged "37 open Dependabot alerts" concern: the real Alerts API shows only 2 (1 high — `extract-zip`, no upstream fix exists yet; 1 medium — `@opentelemetry/core`, already tracked), correcting a stale/wrong push-banner number. Re-tested (and re-confirmed, cleanly this time) that overriding `@opentelemetry/core` to the patched version breaks contentlayer's build — reverted, no repo changes needed. Persisted both environment gotchas to memory for future sessions. No repo/code changes were needed — main was already fully green; this was pure local-environment repair plus roadmap/memory hygiene.
Surfaced the pending founder blocker (rule 0) and picked no task; repo unchanged at 2ad0119, whose full verify gate and deploy went green last turn.
Shipped two merged+deployed PRs this session — #222 (335828b: js-yaml/nanoid bumps closing Dependabot highs #196/#197/#199 after rebasing onto post-next-16 main, plus .gitignore hardening and removal of the tracked node_modules self-symlink) and #223 (2ad0119: fixed CLAUDE.md's now-wrong "build under Node 18" rule to Node 20 and documented that verify includes the vitest gate) — Deploy green and site 200 for both.
Shipped PR #222 (merged as 335828b, Deploy green, site root 200): rebased the stranded fix/dependabot-frontend-transitives branch onto post-next-16 main, re-derived the in-range transitive bumps (js-yaml 4.2.0→4.3.1 and 3.14.2→3.15.1, nanoid 3.3.17→3.3.18), closing Dependabot highs #196/#197/#199, and landed the .gitignore hardening + removal of the tracked node_modules self-symlink that broke root npm.
Rule-8 pick: /databases and /databases/[id] never authenticated for any real user — both pages read the Authorization token from `localStorage.getItem('token')`, but nothing in the app ever sets that key (AuthContext threads the real token via React context only), so every request sent literal "Bearer null" and got 401'd by the backend auth middleware, silently rendering as "no databases yet" (list) or a silent redirect back to /databases (detail). Same root-cause class as the historic AuthContext fix (860a589) that wired useAuth().token through every other authenticated page — these two were missed. Switched both to useAuth().token and added real error/retry states for genuine fetch failures (404 on the detail page still correctly redirects; other failures now show a banner instead of bouncing silently).
Rule-8 pick (adjacent broken thing, not T2 busywork): all five ingest source-list pages (/ingest/video, website, image, audio, database) silently swallowed recent-sources fetch failures into console.error and rendered the same "No X yet" empty state as a genuinely empty list — a real fetch failure (auth expiry, network blip, backend outage) was indistinguishable from "you have nothing yet." Identical bug class already fixed for SavedFormsLibrary in a7504e3; applied the same fix (throw on !response.ok, error state set in catch, red banner with a "Try again" retry button) across all five pages in one commit.