Formats and delivery
Standard delivery is newline-delimited JSON (JSONL) or Apache Parquet, partitioned by board and time period. Records use stable identifiers so that snapshots and incremental deliveries can be joined and deduplicated reliably.
Depending on project requirements, data may be delivered through encrypted object-storage downloads, scheduled snapshot exports, incremental delta feeds, or a dedicated access endpoint. Delivery method, partitioning, update cadence, and filtering are configured per engagement.
Post record schema
{
"post_id": "5ch:news4vip:1712345678:0042",
"thread_id": "5ch:news4vip:1712345678",
"board_id": "news4vip",
"board_name_ja": "ニュー速VIP",
"board_category_ja": "雑談系",
"post_index": 42,
"posted_at": "2024-04-05T21:14:38+09:00",
"posted_at_precision": "second",
"name_field": "以下、5ちゃんねるからVIPがお送りします",
"poster_id": "aB3xY9Zk0",
"body_text": "そういうのはこっちのスレでやってくれ\n>>38 が言ってたやつな",
"body_normalized": "そういうのはこっちのスレでやってくれ >>38 が言ってたやつな",
"reply_to": [38],
"quoted_spans": [
{ "target_post_index": 38, "raw": ">>38" }
],
"char_count": 34,
"contains_ascii_art": false,
"redaction_profile": "standard",
"redactions_applied": 0,
"source_snapshot_id": "snap-2026-08-01",
"record_version": 1
}| Field | Type | Description |
|---|---|---|
| post_id | string | Globally stable post identifier |
| thread_id | string | Parent thread identifier |
| board_id | string | Board short code |
| board_name_ja | string | Board display name |
| board_category_ja | string | Board category grouping |
| post_index | int | 1-based position within thread |
| posted_at | string (RFC 3339) | Post timestamp, JST, timezone-explicit |
| posted_at_precision | enum | second | minute | day — source precision varies by era |
| name_field | string | Name field as submitted |
| poster_id | string | null | Source-provided poster ID where present |
| body_text | string | Post body, source form, newlines preserved |
| body_normalized | string | NFKC-normalised, whitespace-collapsed variant |
| reply_to | int[] | Resolved reply targets by post_index |
| quoted_spans | object[] | Raw anchor text and resolved target |
| char_count | int | Character count of body_text |
| contains_ascii_art | bool | Heuristic ASCII-art flag |
| redaction_profile | enum | raw | standard | strict — see licensing page |
| redactions_applied | int | Count of redactions in this record |
| source_snapshot_id | string | Snapshot this record was exported from |
| record_version | int | Schema version of this record |
Thread record schema
{
"thread_id": "5ch:news4vip:1712345678",
"board_id": "news4vip",
"title": "日本語の口語表現について語るスレ",
"title_normalized": "日本語の口語表現について語るスレ",
"created_at": "2024-04-05T20:47:58+09:00",
"last_post_at": "2024-04-06T03:22:11+09:00",
"post_count": 412,
"unique_poster_id_count": 87,
"status": "archived",
"reply_edges": [[42, 38], [43, 42], [44, 38]],
"source_snapshot_id": "snap-2026-08-01",
"record_version": 1
}reply_edges is an explicit edge list of [from_post_index, to_post_index] pairs — the single clearest demonstration that we are delivering structure rather than scraped text.
| Field | Type | Description |
|---|---|---|
| thread_id | string | Thread identifier |
| board_id | string | Board short code |
| title | string | Thread title |
| title_normalized | string | NFKC-normalised title |
| created_at | string (RFC 3339) | Thread creation timestamp |
| last_post_at | string (RFC 3339) | Last post timestamp |
| post_count | int | Total posts in thread |
| unique_poster_id_count | int | Unique poster IDs |
| status | enum | active | archived | capacity-closed |
| reply_edges | int[][] | [from_post_index, to_post_index] pairs |
| source_snapshot_id | string | Snapshot this record was exported from |
| record_version | int | Schema version |
Encoding and normalisation
- — Output is UTF-8. Legacy source encodings are converted at parse time with a documented mapping; conversion failures are recorded rather than silently dropped.
- — body_text preserves the source form, including full-width characters, kaomoji, and intentional orthographic variation.
- — body_normalized provides an NFKC-normalised, whitespace-collapsed parallel field. Both are delivered; consumers choose.
- — Reply anchors (>>N, ＞＞N, and historical variants) are parsed into structured fields while the original text is retained in body_text.
- — Timestamps are RFC 3339 with explicit JST offset. Where source precision is coarser than one second, posted_at_precision records this rather than fabricating precision.
- — Identifiers are stable across snapshots. A record's post_id does not change between deliveries.
Snapshots and incremental delivery
A snapshot is a point-in-time export of a defined scope (boards, date range, filters), identified by source_snapshot_id and accompanied by a manifest listing every file, its record count, and its SHA-256 checksum.
A delta is the set of records added, modified, or withdrawn between two snapshots, expressed as typed change records so that a downstream store can be brought forward without a full reload.
Deletion and correction events occurring in the source can be propagated through delta feeds where the licence provides for it — an option not available to a one-time crawl.
# JSONL
import json
with open("5ch_posts_news4vip_2024-04.jsonl", encoding="utf-8") as f:
posts = [json.loads(line) for line in f]
# Parquet
import pyarrow.parquet as pq
table = pq.read_table("5ch_posts/board_id=news4vip/year=2024/month=04/")
df = table.to_pandas()
# Reconstruct a thread with its reply graph
thread = [p for p in posts if p["thread_id"] == "5ch:news4vip:1712345678"]
thread.sort(key=lambda p: p["post_index"])
edges = [(p["post_index"], t) for p in thread for t in p["reply_to"]]Delivery options
- Encrypted object storage
- Signed download URLs to encrypted shards with manifest and checksums — Available
- Scheduled snapshots
- Periodic full exports of a defined scope — Available
- Incremental delta feed
- Typed change records between snapshots — Available by arrangement
- Dedicated access endpoint
- Programmatic access scoped to the licence — By engagement
- Private repository delivery
- Delivery to a customer-controlled private repository — By arrangement
| Item | Notes |
|---|---|
| Encrypted object storage | Signed download URLs to encrypted shards with manifest and checksums — Available |
| Scheduled snapshots | Periodic full exports of a defined scope — Available |
| Incremental delta feed | Typed change records between snapshots — Available by arrangement |
| Dedicated access endpoint | Programmatic access scoped to the licence — By engagement |
| Private repository delivery | Delivery to a customer-controlled private repository — By arrangement |
Request a dataset sample.
We provide a representative technical sample, schema documentation, and a dataset brief to qualified organizations.