- definition
- version
- dependencies
- conditions
Pipeline graph
A declared DAG defines stage order, dependencies, feature flags, resource requirements, and which nodes require human review.
Data pipelines & AI systems
Turn model output into data people can review and publish.
Model output becomes useful when people can trace, review, and act on it.
Vyshyvka builds the machinery around the model: ingestion, schemas, extraction, normalization, enrichment, validation, review gates, telemetry, and interfaces where people can inspect, trust, and use the output.
Capabilities
- Graph orchestration
- Typed contracts
- Artifact registry
- LLM orchestration
- ML pipelines
- Source extraction
- Human review
- Runtime validation
- Telemetry
- Deployment
Sources
Pipeline
- Parse
- Normalize
- Enrich
- Validate
Interface
Production engineering
Operational controls for model-backed workflows.
The work covers ingestion, typed outputs, review screens, writeback, deployment, monitoring, and the documentation needed to operate the pipeline.
System architecture
Pipeline graph and stage contracts.
The graph defines each stage and dependency. It also stores evidence, validates handoffs, and records the decisions reviewers need to inspect a run.
- input
- node output
- gate decision
- final result
Typed contracts
Inputs, node outputs, review decisions, validation reports, and final results are parsed before they become run state.
- owner
- schema
- visibility
- retention
Artifact registry
Each artifact records its owner, schema, visibility, retention policy, and deterministic key so the run can be inspected later.
- model
- storage
- matching
- database
Resource adapters
Adapters isolate models, search, matching, storage, persistence, and external APIs from the pipeline's retry and test logic.
Production requirements
Every output needs a source, a validation result, and a publication decision.
A viable AI workflow defines its inputs, transformations, rejection rules, review gates, publish boundary, and post-launch measures.
Without those decisions, a demo or dashboard cannot explain how its output was produced, reviewed, or approved for use.
- source
- provenance
- status
- confidence
Source integrity
Each output stays tied to its source material, provenance, and context.
- schema
- required
- repair
- blocked
Typed contracts
Outputs need schemas, validation, repair paths, and failure states before they become product data.
- evidence
- record
- approve
- rollback
Human review
Reviewers need clear ways to approve, reject, compare, escalate, and roll back changes.
- cost
- latency
- retries
- drift
Operational visibility
Teams need to monitor cost, latency, retries, drift, and output quality after launch.
Demo and production comparison
Schemas, source references, approvals, and run logs.
Stops at the demo
- Chatbot wrappers with no durable data model.
- One-off demos that cannot be audited or replayed.
- Untracked outputs pasted into production workflows by hand.
- Architecture diagrams without a working integration path.
Ready for operations
- Schemas, artifacts, manifests, and source references.
- Validation stages, review queues, and approval boundaries.
- Logs, retries, cost visibility, and failure states.
- Model output connected to the operating workflow.
Pipeline anatomy
A disciplined path from source material to product data.
The pipeline defines which sources enter, which records pass validation, which require human review, and which may be published to the product.
- 01Ingest
Files, records, logs, databases, APIs, staff inputs.
Initialize run, snapshot config, classify sources, preserve provenance.
Source manifest.Missing provenance. - 02Prepare
PDFs, pages, HTML, transcripts, media.
Parse, assess quality, detect structure, split into stable work units.
Chunk manifest.Unreadable input. - 03Extract
Stable chunks, prompts, source references, detection profiles.
Detect candidates, call models, parse outputs, merge cross-source results.
Candidate ledger.Unsupported output. - 04Resolve
Candidates, aliases, normalized fields, existing records.
Normalize, exact-merge, deduplicate, resolve ambiguous matches.
Resolved records.Duplicate conflict. - 05Validate
Structured output, evidence, policies, thresholds.
Schema checks, semantic rules, repair attempts, quality gates.
Rejection ledger.Blocked publish. - 06Review
Evidence, validation state, warnings, proposed records.
Approve, reject, edit, patch config, record decision state.
Approval artifact.Review escalation. - 07Publish
Approved records, write targets, queues.
Writeback to database, CMS, dashboard, API.
Product data.Writeback denied. - 08Monitor
Runs, traces, cost, latency, evals.
Observe drift, retries, quality, failures.
Telemetry trace.Quality regression.
Candidate lifecycle
Check each record before publication.
Intermediate states separate a prompt response from a production record. Invalid items can be rejected, repaired, explained, or dropped before they reach the product.
- 01Detect
Find likely work units before asking a model to produce expensive structured output.
- 02Identify
Give candidates stable IDs, source references, and enough context to trace each result back to its origin.
- 03Normalize
Clean fields, standardize units, collapse obvious aliases, and remove deterministic noise before model-backed decisions.
- 04Materialize
Turn candidate evidence into typed records with source spans, confidence, warnings, and model-call metadata.
- 05Validate
Run schema checks plus semantic quality rules for evidence, unsupported claims, broad spans, duplicates, and missing fields.
- 06Repair
Attempt bounded fixes for rejected items, then keep the repair history and final accept/drop reason.
- 07Finalize
Accepted records can proceed to review, enrichment, publishing, or product storage; rejected records remain out of the product.
Review gates
Review gates before resolution, enrichment, and writeback.
Each review creates a durable artifact for downstream stages: approval state, edits, rejected items, config changes, and the reason work may continue.
Source quality
Stop early when the source is unreadable, incomplete, or missing provenance.
Spend scan
Estimate work, model calls, and cost before the expensive stages begin.
Extraction review
Inspect merged outputs, validation warnings, rejected items, and evidence before resolution continues.
Deduplication
Review uncertain duplicate groups and keep automatic merges explainable.
Enrichment
Approve external matching, reference resolution, and optional enrichment before publish.
Publish readiness
Confirm final records, skipped items, errors, and writeback plan before product persistence.
Artifact anatomy
Keep a record of every run.
Intermediate artifacts let teams inspect, replay, repair, approve, reject, measure, and audit a run. They show what happened without reconstructing it from logs and model responses.
- source refs
- schema status
- model output
- confidence / failures
- reviewer action
- writeback state
- trace / cost / latency
- source refs
- provenance
- schema status
- contract
- model output
- inference
- confidence / failures
- validation
- reviewer action
- review
- writeback state
- publish
- trace / cost / latency
- telemetry
Operational controls
Retry, resume, and partial-failure handling.
When model output can change product data, the architecture must account for retries, idempotency, rate limits, cost controls, partial failures, and controlled final writes.
Idempotent nodes
Deterministic artifact keys and stable item IDs let the same node run again without duplicating work.
Resumable graph state
Runs persist completed nodes, attempts, timestamps, errors, and declared writes so recovery can skip safe work.
Config snapshots
Approval decisions patch the effective runtime config, and resumed runs read the approved snapshot.
Retry taxonomy
Provider throttles, schema repair, item retries, node retries, and non-retryable failures are handled differently.
Bounded concurrency
Stage concurrency and provider queues keep cost, rate limits, and downstream APIs under control.
Final persistence boundary
Intermediate stages write artifacts. The final stage writes approved records to the product database using deterministic source IDs.
Inspection and reporting
Run history, artifacts, and validation results.
The pipeline dashboard exposes the graph state, artifacts, manifests, validation results, telemetry, and decisions used during execution.
Run inspector
needs review- nodes
- 42 / 46
- warnings
- 08
- artifacts
- 126
- retries
- 03
- cost
- $12.84
- ingestcomplete
- extractreview
- resolvecomplete
- publishblocked
Definition view
Shows stages, nodes, dependencies, resource requirements, artifact declarations, and authoring warnings.
Run inspector
Summarizes status, completed nodes, attempts, errors, warnings, chunks, accepted records, rejected records, retries, and cost.
Artifact previews
Lets reviewers inspect known run artifacts through a manifest instead of opening arbitrary storage keys.
Service explorer
Drills into one phase with inputs, prompts, validation, dependencies, examples, and chunk-level evidence.
Comparison and evals
Compares runs, fixtures, quality verdicts, model choices, token spend, failure rates, and output trends.
Services
Supported pipeline types.
Governed LLM workflows
Schemas, context builders, output validators, review queues, traceable run history.
Machine learning systems
Feature pipelines, scoring services, evaluation harnesses, inference paths, deployment artifacts.
Data and extraction pipelines
Parsers, canonical records, entity resolution, deduplication, provenance maps.
AI product interfaces
Dashboards, review states, audit trails, confidence displays, retries, publishing controls.
Proof in the work
Four domains, four different source-to-publication paths.
Each case study shows the sources, processing work, and interface used to review or publish the result.
01
Cultural archives and institutional operations
Chamber Music OS- Sources
- Concert history, repertoire, composers, artists, program notes, documents.
- System work
- Import, repair, deduplicate, canonicalize, review, generate, publish.
- Surface
- Public archive, staff tools, composer and repertoire pages, content workflows.
02
Household operations, meals, nutrition, planning
Modern Recipes- Sources
- Recipes, ingredients, pantry records, nutrition data, calendar and task state.
- System work
- Typed AI pipeline stages, artifact logging, validation, rate limits, telemetry.
- Surface
- Recipe generation, image generation, nutrition dashboards, meal planning workflows.
03
Applied AI security and behavioural analytics
Narrative Security- Sources
- Structured security events, user behaviour, peer groups, model artifacts.
- System work
- Event normalization, sequence modelling, anomaly scoring, validation, model deployment.
- Surface
- SOC-style dashboard, analyst triage, alert review, operational explanation.
04
Language learning and structured study workflows
Slovka- Sources
- Vocabulary, grammar records, study content, generated exercises.
- System work
- Content modelling, generation, review, sequencing, learning-state management.
- Surface
- Vocabulary review, study flows, generated exercises, learner-facing interfaces.
Project fit
Suitable AI pipeline projects.
Good fit
- Messy source material needs to become structured records.
- AI output needs review before becoming product data.
- The workflow needs provenance, auditability, or rollback.
- A model needs to connect to a real dashboard, staff tool, or operational system.
- The team needs one owner for the pipeline, review interface, and production writeback.
Not a fit
- The project only requires a simple chatbot interface.
- The goal is a presentation prototype rather than an operational system.
- Nobody owns the source data.
- There is no review path.
- The budget supports experimentation but not implementation.
Project brief
What to include in an AI pipeline brief.
A useful brief names the inputs, the people who review the output, and where an approved result must be written.
Send a project brief