Vibe Coding Discoverjev-curate
View on GitHubHigh-throughput synthetic and pretraining dataset sifter for TypeSafe Jev. Rust streaming core, Parquet and JSONL I/O, typed Choice/Score/Noul judgments, speculative fan-out, 24.0 rows/sec measured.
A Rust CLI and Python-bindable pipeline that streams Parquet or JSONL datasets and uses TypeSafe AI's Jev model to score, filter, and curate examples. Includes presets for reasoning quality, anti-sycophancy, and code correctness.
Use Cases
Filter synthetic reasoning datasetsCurate pretraining corporaScore fine-tuning examplesDetect circular reasoningRemove sycophantic responses and AI disclaimersFind incomplete code examplesProcess Parquet and JSONL datasets
Built With
- Language
- Rust
- Frameworks
- Apache Arrow · Parquet · Tokio · PyO3 · Maturin
Tags
dataset curation · synthetic data · pretraining · fine-tuning · LLM evaluation · reasoning data · Parquet · JSONL · streaming · CLI · Rust · Python