# DataTap data-spec pack

What to send us, in one file. This is the entire integration for the 90-day
proof of value: export, upload (or POST), read the validation report.

## Option A — trade CSV (preferred)

CSV, UTF-8, header row required. Columns:

    kind,ts_ms,market,account,side,px,qty,outcome[,action,order_id]

| Field    | Values / rules                                                        |
|----------|-----------------------------------------------------------------------|
| kind     | trade · book · settlement · catalyst                                  |
| ts_ms    | integer milliseconds — epoch ms preferred; one time base per file     |
| market   | your market symbol — the first row referencing it creates the market  |
| account  | your account id; for catalyst rows this carries the news source label |
| side     | buy · sell (trade/book rows)                                          |
| px       | decimal within the $0.01–$0.99 binary bounds (trade/book rows)        |
| qty      | positive integer (trade/book rows)                                    |
| outcome  | win · loss (settlement rows)                                          |
| action   | optional, book rows: add · cancel · modify (blank = add)              |
| order_id | optional: your order id, kept on book and trade rows                  |

Example rows:

    trade,1757900000000,CHIEFS-WIN-SB,ACCT-88214,buy,0.58,4000,,,O-1001
    book,1757900000350,CHIEFS-WIN-SB,ACCT-40118,sell,0.97,6000,,add,O-1002
    book,1757900000730,CHIEFS-WIN-SB,ACCT-40118,sell,0.97,6000,,cancel,O-1002
    settlement,1757986400000,CHIEFS-WIN-SB,ACCT-88214,,,,win
    catalyst,1757900012000,CHIEFS-WIN-SB,AP wire,,,,

The two trailing columns are optional, but they are what spoofing, layering
and front-running read: a book row without an action counts as a resting
add, so cancel-heavy behaviour is invisible until you send the actions.

## Option B — FIX drop-copy log

A raw FIX log works as-is: we auto-detect the `8=FIX` preamble and convert
35=8 execution reports with fills (ExecType 150 = F/1/2) using tags
55 (Symbol), 1 (Account), 54 (Side), 31 (LastPx), 32 (LastQty),
60 (TransactTime). Order-lifecycle reports (150 = 0 New, 4 Canceled,
5 Replaced) become book add/cancel/modify rows from 44 (Price) and
38 (OrderQty); 37 (OrderID) or 11 (ClOrdID) rides along as order_id.
SOH or `|` delimiters. Other messages are counted and skipped, not rejected.

## Option C — JSON lines over the API

One object per line with the CSV fields, POSTed to `/api/v1/events` with a
Bearer API key and an `Idempotency-Key` header (retries never double-count).
50MB per batch; chunk larger backfills under distinct keys.

## Packaging

- One file covering the whole window (90 days) is ideal. gzip welcome
  (`.csv.gz`); uploads stream up to 250MB.
- Identical files replay their original report — re-sending is always safe.
- Book (order add/cancel) rows are optional but unlock spoofing/layering
  detection; settlements unlock profitability analysis; catalyst rows (or
  timestamps of when news hit) unlock insider detection.

## Test before you send

Use the **Dry run** button on Portal → Ingestion (or `?dry_run=true` on the
API) to validate an export end to end — full report, rejects with line
numbers, data-quality warnings — without writing anything.

Or validate locally, before anything leaves your network:
https://datatap.org/datatap-validate.py (Python 3, standard library only):

    python3 datatap-validate.py export.csv.gz --map side=Side,account=AcctId

## Your columns, not ours

If your export's headers or vocabularies differ (e.g. `B`/`S` for side,
`W`/`L` for outcome, epoch seconds or ISO-8601 timestamps), set the mapping
once on Portal → Ingestion → column mapper (or `--map` locally). Every
later file converts on ingest.

## Large files (> 250MB)

Ask the API for a presigned upload URL (`POST /api/v1/uploads/presign`),
PUT the file to it, then `POST /api/v1/ingest/from-bucket` with the key —
the file never passes through the web tier.

## Account / KYC companion file (optional, recommended)

    account,identity,funding,device,subnet,address,restricted[,state,country]

account + identity required; funding/device/subnet/address are what entity
resolution joins on; state/country (2–3-letter codes, optional) feed
per-state eligibility and geo-exposure reporting. Uploaded on the same
Ingestion screen, or as JSON lines to `POST /api/v1/accounts`. Wallet↔account
maps (`address,account,chain,flag`) go to the On-Chain screen or
`POST /api/v1/wallets`.

Questions: through your DataTap workspace, or datatap.org/applicants.