- definition
- version
- dependencies
- conditions
Pipeline graph
A declared DAG owns stage order, dependencies, feature flags, resource requirements, and which nodes are steps or human gates.
Data pipelines & AI systems
Most AI failures are system failures.
AI is real. The surrounding system often is not.
Vyshyvka builds the machinery around the model: ingestion, schemas, extraction, normalization, enrichment, validation, review gates, telemetry, and interfaces where people can inspect, trust, and use the output.
Capabilities
- Graph orchestration
- Typed contracts
- Artifact registry
- LLM orchestration
- ML pipelines
- Source extraction
- Human review
- Runtime validation
- Telemetry
- Deployment
Sources
Pipeline
- Parse
- Normalize
- Enrich
- Validate
Interface
Built for the full AI product path
Deterministic infrastructure for probabilistic systems.
Vyshyvka works across the layers that AI products usually require but rarely get in one place: data engineering, model and prompt architecture, typed contracts, product interfaces, cloud deployment, review workflows, telemetry, and documentation.
System architecture
A robust pipeline is a graph, not a script.
The model call is only one node. The pipeline declares the work, stores evidence, validates boundaries, records decisions, and gives reviewers enough structure to inspect what happened.
- input
- node output
- gate decision
- final result
Typed contracts
Inputs, node outputs, review decisions, validation reports, and final results are parsed before they become run state.
- owner
- schema
- visibility
- retention
Artifact registry
Every artifact has an owner, schema, visibility, retention policy, and deterministic key so the run can be inspected later.
- model
- storage
- matching
- database
Resource adapters
Models, search, matching, storage, persistence, and external APIs sit behind adapters so retries and tests do not leak into pipeline logic.
The actual problem
The model is only one component. The system decides whether it can be trusted.
Many AI initiatives begin with ambition before there is a clear model of the work. What enters the system? What gets transformed? What must be rejected? What needs human review? What becomes product data? What gets measured after launch?
When those answers are missing, the result is predictable: demos, wrappers, dashboards, and claims of transformation with no disciplined pipeline underneath.
- source
- provenance
- status
- confidence
Source integrity
AI output is only as good as the source material, provenance, and context that feed it.
- schema
- required
- repair
- blocked
Typed contracts
Outputs need schemas, validation, repair paths, and failure states before they become product data.
- evidence
- record
- approve
- rollback
Human review
Serious workflows need approval, rejection, comparison, escalation, and rollback.
- cost
- latency
- retries
- drift
Operational visibility
Cost, latency, retries, drift, and quality need to be visible after launch.
Not prompt theater
A model call is not a business system.
Not this
- Chatbot wrappers with no durable data model.
- One-off demos that cannot be audited or replayed.
- Untracked outputs pasted into production workflows by hand.
- Slideware pretending integration is architecture.
This
- Schemas, artifacts, manifests, and source references.
- Validation stages, review queues, and approval boundaries.
- Logs, retries, cost visibility, and failure states.
- AI integrated into the system, not waved over it.
Pipeline anatomy
A disciplined path from source material to product data.
The pipeline decides what enters, what can be trusted, what must be rejected, what needs human review, and what is allowed to become part of the product.
- 01Ingest
Files, records, logs, databases, APIs, staff inputs.
Initialize run, snapshot config, classify sources, preserve provenance.
Source manifest.Missing provenance. - 02Prepare
PDFs, pages, HTML, transcripts, media.
Parse, assess quality, detect structure, split into stable work units.
Chunk manifest.Unreadable input. - 03Extract
Stable chunks, prompts, source references, detection profiles.
Detect candidates, call models, parse outputs, merge cross-source results.
Candidate ledger.Unsupported output. - 04Resolve
Candidates, aliases, normalized fields, existing records.
Normalize, exact-merge, deduplicate, resolve ambiguous matches.
Resolved records.Duplicate conflict. - 05Validate
Structured output, evidence, policies, thresholds.
Schema checks, semantic rules, repair attempts, quality gates.
Rejection ledger.Blocked publish. - 06Review
Evidence, validation state, warnings, proposed records.
Approve, reject, edit, patch config, record decision state.
Approval artifact.Review escalation. - 07Publish
Approved records, write targets, queues.
Writeback to database, CMS, dashboard, API.
Product data.Writeback denied. - 08Monitor
Runs, traces, cost, latency, evals.
Observe drift, retries, quality, failures.
Telemetry trace.Quality regression.
Candidate lifecycle
Good output is earned through checkpoints.
An extraction workflow should not jump straight from prompt to production record. It needs intermediate states so bad items can be rejected, repaired, explained, or dropped.
- 01Detect
Find likely work units before asking a model to produce expensive structured output.
- 02Identify
Give candidates stable IDs, source references, and enough context to trace each result back to its origin.
- 03Normalize
Clean fields, standardize units, collapse obvious aliases, and remove deterministic noise before model-backed decisions.
- 04Materialize
Turn candidate evidence into typed records with source spans, confidence, warnings, and model-call metadata.
- 05Validate
Run schema checks plus semantic quality rules for evidence, unsupported claims, broad spans, duplicates, and missing fields.
- 06Repair
Attempt bounded fixes for rejected items, then keep the repair history and final accept/drop reason.
- 07Finalize
Only accepted records move forward to review, enrichment, publishing, or product persistence.
Review gates
Humans enter at the risky transitions.
Review should not be a vague final pass. It should be a durable artifact that downstream stages consume: approval state, edits, rejected items, config changes, and the reason work is allowed to continue.
Source quality
Stop early when the source is unreadable, incomplete, or missing provenance.
Spend scan
Estimate work, model calls, and cost before the expensive stages begin.
Extraction review
Inspect merged outputs, validation warnings, rejected items, and evidence before resolution continues.
Deduplication
Review uncertain duplicate groups and keep automatic merges explainable.
Enrichment
Approve external matching, reference resolution, and optional enrichment before publish.
Publish readiness
Confirm final records, skipped items, errors, and writeback plan before product persistence.
Artifact anatomy
Every production pipeline should leave evidence.
Reliable AI workflows need intermediate artifacts that can be inspected, replayed, repaired, approved, rejected, measured, and audited. The artifact is not decoration. It is the system's memory.
- source refs
- schema status
- model output
- confidence / failures
- reviewer action
- writeback state
- trace / cost / latency
- source refs
- provenance
- schema status
- contract
- model output
- inference
- confidence / failures
- validation
- reviewer action
- review
- writeback state
- publish
- trace / cost / latency
- telemetry
Operational controls
The pipeline has to fail in recoverable ways.
Once AI work touches product data, operations become part of the architecture: retries, idempotency, rate limits, cost controls, partial failures, and safe final persistence.
Idempotent nodes
Deterministic artifact keys and stable item IDs let the same node run again without duplicating work.
Resumable graph state
Runs persist completed nodes, attempts, timestamps, errors, and declared writes so recovery can skip safe work.
Config snapshots
Approval decisions patch the effective runtime config, and resumed runs read the approved snapshot.
Retry taxonomy
Provider throttles, schema repair, item retries, node retries, and non-retryable failures are handled differently.
Bounded concurrency
Stage concurrency and provider queues keep cost, rate limits, and downstream APIs under control.
Final persistence boundary
Intermediate stages write artifacts; product database writes happen at the end with deterministic source IDs.
Inspection and reporting
If you cannot inspect the run, you cannot trust the run.
The best pipeline dashboard is not a separate reporting product. It is a view over the same graph state, artifacts, manifests, validation results, telemetry, and decisions that the system used to execute.
Run inspector
needs review- nodes
- 42 / 46
- warnings
- 08
- artifacts
- 126
- retries
- 03
- cost
- $12.84
- ingestcomplete
- extractreview
- resolvecomplete
- publishblocked
Definition view
Shows stages, nodes, dependencies, resource requirements, artifact declarations, and authoring warnings.
Run inspector
Summarizes status, completed nodes, attempts, errors, warnings, chunks, accepted records, rejected records, retries, and cost.
Artifact previews
Lets reviewers inspect known run artifacts through a manifest instead of opening arbitrary storage keys.
Service explorer
Drills into one phase with inputs, prompts, validation, dependencies, examples, and chunk-level evidence.
Comparison and evals
Compares runs, fixtures, quality verdicts, model choices, token spend, failure rates, and output trends.
What we build
AI workflows connected to production work.
Governed LLM workflows
Schemas, context builders, output validators, review queues, traceable run history.
Machine learning systems
Feature pipelines, scoring services, evaluation harnesses, inference paths, deployment artifacts.
Data and extraction pipelines
Parsers, canonical records, entity resolution, deduplication, provenance maps.
AI product interfaces
Dashboards, review states, audit trails, confidence displays, retries, publishing controls.
Proof in the work
The same pipeline work shows up in different domains.
The domain changes. The underlying discipline does not: source material becomes structured data, structured data becomes workflow, workflow becomes software people can review and use.
01
Cultural archives and institutional operations
Chamber Music OS- Sources
- Concert history, repertoire, composers, artists, program notes, documents.
- System work
- Import, repair, deduplicate, canonicalize, review, generate, publish.
- Surface
- Public archive, staff tools, composer and repertoire pages, content workflows.
02
Household operations, meals, nutrition, planning
Modern Recipes- Sources
- Recipes, ingredients, pantry records, nutrition data, calendar and task state.
- System work
- Typed AI pipeline stages, artifact logging, validation, rate limits, telemetry.
- Surface
- Recipe generation, image generation, nutrition dashboards, meal planning workflows.
03
Applied AI security and behavioral analytics
Narrative Security- Sources
- Structured security events, user behavior, peer groups, model artifacts.
- System work
- Event normalization, sequence modeling, anomaly scoring, validation, model deployment.
- Surface
- SOC-style dashboard, analyst triage, alert review, operational explanation.
04
Language learning and structured study workflows
Slovka- Sources
- Vocabulary, grammar records, study content, generated exercises.
- System work
- Content modeling, generation, review, sequencing, learning-state management.
- Surface
- Vocabulary review, study flows, generated exercises, learner-facing interfaces.
Selective engagements
Best fit for AI work where correctness matters.
Good fit
- Messy source material needs to become structured records.
- AI output needs review before becoming product data.
- The workflow needs provenance, auditability, or rollback.
- A model needs to connect to a real dashboard, staff tool, or operational system.
- The project needs one studio that can handle data, AI, product, and infrastructure together.
Not a fit
- A chatbot wrapper is enough.
- The goal is only a board-demo prototype.
- Nobody owns the source data.
- There is no review path.
- The budget supports experimentation but not implementation.
Start the work
If the AI ambition is real, the system around it has to be real too.
Send a brief if the project needs more than a demo: source data, structured outputs, review workflows, product interfaces, and infrastructure that can hold up after launch.
Send a project brief