Elephant Network CLI

This guide walks Elephant Network oracles through the complete workflow of transforming county data and submitting proofs on-chain using the Elephant CLI.

Overview
Prerequisites
Installation
Create an Encrypted Keystore
Transform Input Requirements
Property Improvement Workflow
Build the Seed Bundle
Fetch Current Source Content
Browser Flow Templates
Browser Flow v2 Handler Packages
Transform v2 Handler Packages
Generate Transformation Scripts
Produce the County Dataset
Hash the County Dataset
Export County Tables
Upload Datagroups to IPFS
Submit Hashes to the Contract
Utility Commands

Overview

The Elephant CLI enables oracles to:

Derive canonical seed files from jurisdiction sourcing metadata (transform).
Download the live county response for reproducible processing (prepare).
Generate and execute extraction scripts for county-specific transformations (generate-transform and transform).
Process Property Improvement data groups with HTML extraction and relationship mapping (transform --data-group "Property Improvement").
Canonicalize outputs, upload to IPFS, and record submissions on the Polygon network (hash, upload, submit-to-contract).

Each section below explains what a command does, the inputs it expects, the resulting artifacts, available options, and a runnable example.

Prerequisites

Node.js 22.15 or later (includes npm); export-tables compresses Parquet pages with the Zstd support built into node:zlib from that release.
Ability to create and extract ZIP archives (zip/unzip).
Access to a Polygon RPC endpoint (e.g., Alchemy, Infura, or internal infrastructure).
Oracle private key to be stored in an encrypted keystore file.
Pinata JWT (PINATA_JWT) for IPFS uploads.
OpenAI API key (OPENAI_API_KEY) for script generation.
Optional: ELEPHANT_SCHEMA_MANIFEST_URL to override the lexicon schema manifest endpoint (default https://lexicon.elephant.xyz/api/manifest), and ELEPHANT_IPFS_GATEWAYS (or the global --ipfs-gateway <origins> option) as a comma-separated list of gateway origins, no trailing slash, used instead of the public defaults when fetching lexicon schemas; list the defaults after your own to keep them as fallback. Defaults: https://ipfs.filebase.io, https://gateway.pinata.cloud, https://trustless-gateway.link. Schemas are fetched as trustless raw blocks (?format=raw) and hash-verified, so any gateway that supports the trustless gateway spec works.
Stable network connection and sufficient disk space for ZIP artifacts.

Installation

Install once and reuse:

npm install -g @elephant-xyz/cli

Or run ad-hoc without installing globally:

npx @elephant-xyz/cli --help

Create an Encrypted Keystore

Use create-keystore to encrypt your Polygon private key for later use with submit-to-contract.

elephant-cli create-keystore \
  --private-key 0xYOUR_PRIVATE_KEY \
  --password "your-strong-password" \
  --output oracle-keystore.json

What it does

Encrypts the supplied private key with the provided password.
Writes an encrypted JSON keystore to disk and prints the derived address.

Inputs

Private key (with or without 0x).
Password (minimum 8 characters).

Output

oracle-keystore.json (or the path provided via --output).

Options

Option	Description	Default
`-k, --private-key <key>`	Private key to encrypt.	Required
`-p, --password <password>`	Password used for encryption.	Required
`-o, --output <path>`	Destination file for the keystore JSON.	`keystore.json`
`-f, --force`	Overwrite the output file if it already exists.	`false`

Transform Input Requirements

The first transform run produces canonical seed files from a county sourcing list. Supply a ZIP that contains a single seed.csv at its top level:

seed-input.zip
└── seed.csv

seed.csv must include the following headers (one property per row):

Column	Required	Purpose
`parcel_id`	✅	Parcel identifier used across Elephant datasets.
`address`	✅	Human-readable street address for logging.
`method`	✅	HTTP method (`GET` or `POST`).
`url`	✅	Base URL to request during `prepare`.
`multiValueQueryString`	➖	JSON string mapping query keys to string arrays (e.g. `{"parcel":["0745"]}`).
`source_identifier`	✅	Stable identifier for the property request (becomes the file stem in later steps).
`county`	✅	County name (case-insensitive; transformed to title case).
`json`	➖	JSON request body (stringified). Mutually exclusive with `body`.
`body`	➖	Raw request payload string. Mutually exclusive with `json`.
`headers`	➖	JSON string of HTTP headers (e.g. `{"content-type":"application/json"}`).

Only one of json or body may be present in a row. Leave optional columns blank when not needed.

Example row:

parcel_id,address,method,url,multiValueQueryString,source_identifier,county,json
074527L1060260060,123 Example Ave,GET,https://county.example.com/search,"{\"parcel\":[\"074527L1060260060\"]}",ALACHUA-074527L1060260060,Alachua,

Property Improvement Workflow

The Property Improvement workflow extracts structured data from permit websites and creates a Property Improvement data group with all related entities and relationships.

Overview

The Property Improvement workflow consists of three main steps:

Prepare: Fetch HTML content from permit websites using HTTP requests
Extract: Run extraction scripts to parse HTML and create structured JSON files
Transform: Create Property Improvement data group with relationships

Step 1: Prepare HTML Content

Create an input.csv file with HTTP request details for the permit website:

source_http_request,request_identifier
"https://egweb1.cityofbonitasprings.org/energov/selfservice#/permit/e7e6ec95-4042-4710-ad00-f946bb30291f",permit-123

Then run the prepare step to fetch the HTML content:

# Create input.zip with input.csv
elephant-cli prepare input.zip --output-zip prepared-property-improvement.zip

What this does:

Reads HTTP request details from input.csv
Fetches HTML content from the permit website
Creates prepared-property-improvement.zip containing both input.csv and the fetched HTML file

Step 2: Create Extraction Scripts

Create a property-improvement-extractor.js script that:

Reads HTML files from the input directory
Extracts Property Improvement data (permit details, contractors, inspections, etc.)
Creates structured JSON files for each entity type
Outputs files to the data directory

Example script structure:

// property-improvement-extractor.js
const fs = require('fs');
const cheerio = require('cheerio');

// Read HTML file
const html = fs.readFileSync('input/permit.html', 'utf8');
const $ = cheerio.load(html);

// Extract Property Improvement data
const propertyImprovement = {
  permit_number: $('.permit-number').text(),
  permit_type: $('.permit-type').text(),
  // ... more fields
};

// Write JSON files
fs.writeFileSync(
  'data/property_improvement.json',
  JSON.stringify(propertyImprovement)
);
fs.writeFileSync(
  'data/property_improvement_has_contractor_1.json',
  JSON.stringify(contractorRelationship)
);

Package your script into property-improvement-scripts.zip:

property-improvement-scripts.zip
└── scripts/
    └── property-improvement-extractor.js

Step 3: Transform to Data Group

Run the transform command to create the Property Improvement data group:

elephant-cli transform \
  --data-group "Property Improvement" \
  --input-zip prepared-property-improvement.zip \
  --scripts-zip property-improvement-scripts.zip \
  --output-zip property-improvement-output.zip

What this does:

Runs extraction script: Executes property-improvement-extractor.js to parse HTML and create JSON files
Copies extracted files: Moves JSON files from data/ to input/ for processing
Creates data group: Builds Property Improvement data group structure with relationships
Validates output: Ensures all files conform to Property Improvement schema

Input Requirements:

input.csv: HTTP request details (same format as seed.csv)
HTML file: Property improvement data from permit website
Extraction script: property-improvement-extractor.js that parses HTML and creates JSON files

Output Structure:

property-improvement-output.zip
└── data/
    ├── <property_improvement_schema_cid>.json    # Data group root file
    ├── property_improvement.json                # Main Property Improvement entity
    ├── company_1.json                           # Contractor/company entities
    ├── file_1.json                              # File attachments
    ├── inspection_1.json                       # Inspection records
    ├── property_improvement_has_contractor_1.json  # Relationships
    ├── property_improvement_has_file_1.json
    └── property_improvement_has_inspection_1.json

Key Differences from County Workflow:

Custom extraction: Uses property-improvement-extractor.js instead of County mapping scripts
HTML-based: Extracts data from HTML content rather than structured CSV data
Single script: Runs one extraction script instead of multiple mapping scripts

Validation:

The output can be validated using:

elephant-cli validate property-improvement-output.zip

This ensures all files conform to the Property Improvement schema and relationships are properly structured.

Inputs

A single property ZIP, a single extracted property directory, or a directory whose immediate children are property ZIPs and/or property subdirectories (one property each, processed in sorted name order; children starting with . or __ are skipped; duplicate names such as 12345.zip next to 12345/ abort the run).
Or a county CAR written by hash --output-car (any input ending in .car). The file is indexed once and checked in place:
- Integrity: every block's bytes re-hash to its CID.
- Root: exactly one root, a CountyIndex version 1 with a numeric properties count and a shards array of links.
- Index closure: every shard decodes to {"properties": [...]}, the entries add up to properties, and every property_cid and data_groups link resolves inside the file.
- Graph closure: every link in every dag-json block reachable from the root resolves inside the file.
- Lexicon: every data-group root listed in a shard validates against the schema it is keyed by, resolving links from the CAR instead of IPFS.
- Orphans: every block is reachable from the root.

Outputs

--output-csv (default submit_errors.csv): validation errors. With a directory of properties, one combined CSV for the whole batch (plus a combined submit_warnings.csv beside it), followed by a Properties processed / succeeded / failed summary; the exit code is non-zero when any property failed.
With a CAR, one row per finding (file_path holds the block CID, error_path the check name or the JSON path for lexicon errors), followed by a summary of blocks, properties, data groups validated, and errors per check; the exit code is non-zero when any check failed.

# Validate every property in a county directory in one invocation
elephant-cli validate ./county-outputs --output-csv county-errors.csv

# Prove a county CAR is intact, closed, and lexicon-valid before registering its root
elephant-cli validate county.car --output-csv car-errors.csv

Build the Seed Bundle

Run transform against the seed ZIP to derive the foundational seed files.

elephant-cli transform \
  --input-zip seed-input.zip \
  --output-zip seed-bundle.zip

What it does

Parses seed.csv and constructs canonical property_seed.json, unnormalized_address.json, and relationship scaffolding.
Generates a seed datagroup JSON (named by the Seed schema CID).
Packages everything inside a top-level data/ directory.

Inputs

ZIP containing seed.csv at the root.

Output

seed-bundle.zip
└── data/
    ├── <seed_schema_cid>.json
    ├── property_seed.json
    ├── relationship_property_to_address.json
    └── unnormalized_address.json

For the next step, extract data/property_seed.json and data/unnormalized_address.json into a new working folder (no subdirectories) and zip them as prepare-input.zip.

Options

Option	Description	Default
`--input-zip <path>`	Seed ZIP containing `seed.csv`.	Required
`--output-zip <path>`	Destination ZIP for generated seed assets.	`transformed-data.zip`
`--scripts-zip <path>`	When provided, runs county scripts instead of seed mode.	None
`--legacy-mode`	Use the legacy AI workflow (not used in modern oracle flow).	`false`

Fetch Current Source Content

Package the extracted seed files into a ZIP that looks like this:

prepare-input.zip
├── property_seed.json
└── unnormalized_address.json

Run prepare to reproduce the county response referenced by the seed.

elephant-cli prepare prepare-input.zip --output-zip prepared-site.zip

For sites requiring browser interaction, choose the appropriate approach:

# Simple disclaimer/agree button only
elephant-cli prepare prepare-input.zip \
  --output-zip prepared-site.zip \
  --continue-button "#acceptDataDisclaimer" \
  --use-browser

# Complex counties requiring multi-step navigation
elephant-cli prepare prepare-input.zip \
  --output-zip prepared-site.zip \
  --browser-flow-template <TEMPLATE_NAME> \
  --browser-flow-parameters '<JSON_PARAMETERS>'

# Sites with CAPTCHA that should be ignored
elephant-cli prepare prepare-input.zip \
  --output-zip prepared-site.zip \
  --use-browser \
  --ignore-captcha

# Using a proxy for browser mode (helpful for bypassing rate limits)
elephant-cli prepare prepare-input.zip \
  --output-zip prepared-site.zip \
  --use-browser \
  --proxy "username:password@192.168.1.1:8080"

# Using multi-request flow for APIs with multiple endpoints
elephant-cli prepare prepare-input.zip \
  --output-zip prepared-site.zip \
  --multi-request-flow-file flow.json

# Using Browser Flow v2 for packaged Puppeteer handlers
elephant-cli prepare prepare-input.zip \
  --output-zip prepared-site.zip \
  --browser-flow-version 2 \
  --browser-flow-zip county-browser-flow-v2.zip

What it does

Reads source_http_request from property_seed.json.
Performs the HTTP request, browser workflow, multi-request flow, or Browser Flow v2 handler package.
Writes the fetched response or captured HTML alongside the seed files.

Inputs

ZIP containing property_seed.json and unnormalized_address.json at the top level.
Browser Flow v2 also accepts the newer top-level names parcel.json and address.json.

Output

Default, browser template, custom browser flow, and multi-request outputs use one response file:

prepared-site.zip
├── property_seed.json
├── unnormalized_address.json
└── <request_identifier>.html | <request_identifier>.json

Browser Flow v2 outputs a capture manifest and one or more cleaned HTML captures:

prepared-site.zip
├── property_seed.json
├── unnormalized_address.json
├── captures.json
└── captures/
    └── <capture-name>.html

Options

Option	Description	Default
`--output-zip <path>`	Destination ZIP containing the fetched response.	Required
`--use-browser`	Fetch GET requests with a headless Chromium browser (needed for dynamic sites).	`false`
`--no-continue`	Skip auto-clicking "Continue" modals when browser mode is active.	`false`
`--continue-button <selector>`	CSS selector for a simple continue/agree button to click.	None
`--ignore-captcha`	Ignore CAPTCHA pages and continue processing.	`false`
`--browser-flow-template <name>`	Use a predefined browser automation template (e.g., `SEARCH_BY_PARCEL_ID`).	None
`--browser-flow-parameters <json>`	JSON parameters for the browser flow template.	None
`--browser-flow-file <path>`	Path to custom browser flow JSON file (takes precedence over template).	None
`--browser-flow-version <version>`	Set to `2` to run a packaged Browser Flow v2 handler.	None
`--browser-flow-zip <path>`	Path to a Browser Flow v2 ZIP containing `handler.js`.	None
`--multi-request-flow-file <path>`	Path to JSON file defining a multi-request flow (sequence of HTTP requests).	None
`--proxy <url>`	Proxy URL with authentication (format: `username:password@ip:port`).	None

Browser Flow Templates

Browser flow templates provide reusable automation patterns for complex county websites that require multi-step navigation. Instead of hardcoding browser interactions, templates allow you to configure automation using CSS selectors as parameters. The URL is automatically extracted from property_seed.json's source_http_request field.

Key Benefits:

Handles modal dialogs and terms acceptance screens
Automates form filling and navigation
Configurable for different county website structures
No code changes required for new counties

For available templates, parameters, and detailed usage examples, see Browser Flow Templates Documentation.

Need Maximum Flexibility?

For complex, site-specific workflows that don't fit standard templates, you can define custom browser flows using JSON files. This gives you complete control over the automation sequence. See Custom Browser Flows Documentation for detailed information and examples.

Quick example using a custom flow:

elephant-cli prepare input.zip \
  --output-zip output.zip \
  --browser-flow-file my-custom-flow.json

Browser Flow v2 Handler Packages

Browser Flow v2 is for counties where the browser workflow is easier to maintain as JavaScript than as a template or JSON workflow. Package an ES module named handler.js into a ZIP, then run prepare with --browser-flow-version 2 and --browser-flow-zip.

Quick example using a Browser Flow v2 package:

elephant-cli prepare prepare-input.zip \
  --output-zip prepared-site.zip \
  --browser-flow-version 2 \
  --browser-flow-zip county-browser-flow-v2.zip

The handler receives a Puppeteer page, normalized seed input, a logger, and helpers for recording the final source URL and saving cleaned HTML captures:

export async function handler({ input, page, saveHtml, saveSourceUrl }) {
  await page.goto(input.url, { waitUntil: 'domcontentloaded' });
  await page.waitForSelector('#property-details', { timeout: 60000 });

  await saveSourceUrl(page.url());
  await saveHtml({ name: 'property-detail' });
}

The output includes captures.json and captures/<name>.html files. See Browser Flow v2 Handler Packages for the complete package contract, handler API, output manifest, and troubleshooting guide.

Transform v2 Handler Packages

Transform v2 consumes Browser Flow v2 prepared ZIPs (address.json, parcel.json, captures.json, and captures/*.html) with a packaged handler.js transform. It is separate from the v1 --scripts-zip runner.

elephant-cli transform \
  --transform-version 2 \
  --transform-zip county-transform-v2.zip \
  --input-zip prepared-site.zip \
  --output-zip transformed-data.zip

Use transform v2 when one transform entrypoint should read one or more named captures and write entity and relationship JSON through the helper API. See Transform v2 Handler Packages for the full package contract, context API, metadata rules, and examples.

Multi-Request Flows

For counties where property data is distributed across multiple API endpoints, use multi-request flows to fetch and combine data from a sequence of HTTP requests into a single JSON output.

When to Use Multi-Request Flows:

Property data is spread across multiple API endpoints (e.g., owner info, sales history, tax data)
Different aspects of property information require separate API calls
The property appraiser provides API access rather than HTML pages
A single page view or request cannot capture all necessary data

Key Features:

Execute multiple HTTP requests in parallel (GET, POST, PUT, PATCH)
Automatically substitute {{=it.request_identifier}} template variables in URLs, headers, bodies, and query parameters using doT.js syntax
Support for JSON and form-encoded request bodies
Automatic response parsing (JSON or string)
Combined output with all request/response pairs in a single JSON file

For detailed documentation, configuration format, validation rules, and troubleshooting, see Multi-Request Flow Documentation.

Quick Example:

Create a flow file manatee-flow.json:

{
  "requests": [
    {
      "key": "OwnersAndGeneralInformation",
      "request": {
        "method": "POST",
        "url": "https://example.com/api/owner",
        "headers": {
          "content-type": "application/x-www-form-urlencoded"
        },
        "body": "parid={{=it.request_identifier}}"
      }
    },
    {
      "key": "Sales",
      "request": {
        "method": "GET",
        "url": "https://example.com/api/sales?parid={{=it.request_identifier}}"
      }
    }
  ]
}

Then run:

elephant-cli prepare prepare-input.zip \
  --output-zip prepared-site.zip \
  --multi-request-flow-file manatee-flow.json

See the examples directory for a complete real-world example.

To use transform with a browser on AWS EC2 instances you need to run Ubuntu 22.04 or later and perfrom the following steps:

# Update package lists
sudo apt update


# Install dependencies
sudo apt install -y \
    dconf-service \
    libasound2t64 \
    libatk1.0-0 \
    libatk-bridge2.0-0 \
    libc6 \
    libcairo2 \
    libcups2 \
    libdbus-1-3 \
    libexpat1 \
    libfontconfig1 \
    libgcc-s1 \
    libgdk-pixbuf2.0-0 \
    libglib2.0-0 \
    libgtk-3-0 \
    libnspr4 \
    libpango-1.0-0 \
    libpangocairo-1.0-0 \
    libstdc++6 \
    libx11-6 \
    libx11-xcb1 \
    libxcb1 \
    libxcomposite1 \
    libxcursor1 \
    libxdamage1 \
    libxext6 \
    libxfixes3 \
    libxi6 \
    libxrandr2 \
    libxrender1 \
    libxss1 \
    libxtst6 \
    ca-certificates \
    fonts-liberation \
    libayatana-appindicator3-1 \
    libnss3 \
    lsb-release \
    xdg-utils \
    wget \
    libgbm1


# Clean up
sudo apt autoremove -y
sudo apt clean

Generate Transformation Scripts

Provide the prepared site bundle to generate-transform to produce county-specific extraction scripts. Set OPENAI_API_KEY beforehand.

export OPENAI_API_KEY=sk-live...
elephant-cli generate-transform prepared-site.zip \
  --output-zip generated-scripts.zip

What it does

Runs an LLM pipeline that reads the seed, address, and downloaded county response.
Generates JavaScript scripts (ownerMapping.js, structureMapping.js, layoutMapping.js, utilityMapping.js, data_extractor.js) plus a manifest.

Inputs

ZIP containing property_seed.json, unnormalized_address.json, and one HTML or JSON county response file at the root. Optionally include a scripts/ directory with prior attempts and CSVs containing previous errors.

Output

generated-scripts.zip
├── data_extractor.js
├── ownerMapping.js
├── structureMapping.js
├── utilityMapping.js
├── layoutMapping.js
└── manifest.json

Options

Option	Description	Default
`-o, --output-zip <path>`	Destination ZIP for generated scripts or repaired scripts.	`generated-scripts.zip`
`-d, --data-dictionary <path>`	Optional reference file fed to the generator.	None
`--scripts-zip <path>`	Existing scripts bundle (must contain `data_extractor.js`) for automatic repair.	None
`-e, --error <string>`	JSON error payload captured from a failed transform run.	None
`--error-csv <path>`	CSV of validation errors produced by `validate` (e.g., `submit_errors.csv`). Deduplicated and used to guide automatic repair (requires `--scripts-zip`).	None

Error repair flow

Supply --scripts-zip and exactly one of:
- --error with a JSON payload like {"type":"error","message":"Unknown enum value X.","path":"Class.property"}
- --error-csv with a validation errors CSV generated by validate (headers: property_cid,data_group_cid,file_path,error_path,error_message,timestamp)
When --error-csv is provided, the CLI:
- Parses all rows, maps each error_path (e.g., /relationships/<rel>/.../(from|to)/<prop>) through the given data_group_cid to resolve the underlying <Class>.<property>.
- Deduplicates errors by <Class>.<property> and aggregates them into a single error payload.
- Fetches the schema fragment for each unique property and provides all fragments to the repair prompt.
The CLI extracts data_extractor.js, passes the current script, aggregated error(s), and schema fragment(s) to the model, and writes back the fixed data_extractor.js into --output-zip.
Only the extraction script is modified; other files remain untouched. The repaired ZIP can be reused with the transform command.

Example invocations:

# Repair using a single JSON error payload
elephant-cli generate-transform prepared-site.zip \
  --scripts-zip generated-scripts.zip \
  --error '{"type":"error","message":"must be one of ...","path":"Owner.first_name"}' \
  --output-zip generated-scripts-fixed.zip

# Repair using a CSV of validation errors produced by `validate`
elephant-cli generate-transform prepared-site.zip \
  --scripts-zip generated-scripts.zip \
  --error-csv submit_errors.csv \
  --output-zip generated-scripts-fixed.zip

Approximate duration: up to one hour per county. The process consumes OpenAI API credits.

Produce the County Dataset

Run transform again, this time supplying both the prepared site ZIP and the generated scripts.

elephant-cli transform \
  --input-zip prepared-site.zip \
  --scripts-zip generated-scripts.zip \
  --output-zip transformed-data.zip

What it does

Normalizes inputs to input.html/input.json, property_seed.json, and unnormalized_address.json in a temporary workspace.
Executes the generated scripts, adding source_http_request metadata to every datagroup.
Builds county relationships, then bundles the results.
Writes the Seed data-group root and its address_has_parcel relationship next to the address.json and parcel.json the scripts produced (keeping any the scripts already wrote), so hash derives the property CID from this bundle alone.

Inputs

prepared-site.zip (from the previous step).
generated-scripts.zip (from the LLM pipeline or a hand-tuned bundle).

Output

transformed-data.zip
└── data/
    ├── property.json
    ├── *.json (cleaned datagroups named by schema CIDs)
    └── relationship_*.json

Options

Option	Description	Default
`--input-zip <path>`	Prepared site ZIP with seed and source response.	Required
`--scripts-zip <path>`	ZIP of scripts to execute.	Required in scripts mode
`--output-zip <path>`	Destination ZIP for the transformed county bundle.	`transformed-data.zip`
`--legacy-mode`	Use the legacy agent flow (not part of the standard pipeline).	`false`

Hash the County Dataset

Feed the transformed bundle to hash to compute content-addressed JSON and produce the submission CSV.

elephant-cli hash transformed-data.zip \
  --output-zip hashed-data.zip \
  --output-csv hash-results.csv

What it does

Canonicalizes every JSON datagroup.
Calculates IPFS-compatible multihash CIDs.
Produces a CSV mapping property, datagroup, and data CIDs, ready for contract submission.

Inputs

ZIP containing a single property directory (such as transformed-data.zip from the previous step). The ZIP may contain either files directly or a data/ folder; both are supported.
Or an extracted property directory, or a directory whose immediate children are property ZIPs and/or property subdirectories. Each child is hashed as one property, in sorted name order, sharing one schema cache and manifest across the batch. Same directory rules as validate.

Outputs

hashed-data.zip
└── <property_cid>/
    ├── <data_cid>.json (canonicalized datagroups)
    └── image files copied from the transform bundle

hash-results.csv
propertyCid,dataGroupCid,dataCid,filePath,uploadedAt
...

The CSV leaves uploadedAt empty (populated after IPFS upload).

With --output-car, every hashed JSON block of the run (image files excluded) is also written into one CAR file whose single root is a county index; a consumer walks root -> shard -> property -> data group:

<index>   {"label":"CountyIndex","version":1,"properties":<count>,"shards":[{"/":"<shard cid>"},...]}
<shard>   {"properties":[{"property_cid":{"/":"<property cid>"},"data_groups":{"<data group schema cid>":{"/":"<data cid>"},...}},...]}

With a directory input, --output-zip is treated as an output directory and --output-csv collects every property's rows into one file:

elephant-cli hash ./county-transformed \
  --output-zip ./county-hashed \
  --output-csv county-hash-results.csv

county-hashed/
├── <property-a>.zip   (stem of the child ZIP or subdirectory name)
└── <property-b>.zip

county-hash-results.csv   (one header, rows from every property)
submit_errors.csv         (combined per-property error reports, beside the CSV)
submit_warnings.csv       (combined per-property warning reports, beside the CSV)

--output-zip must be a directory (or not exist yet); an existing file is rejected. A property that fails does not stop the batch; a Properties processed / succeeded / failed summary is printed and the exit code is non-zero when any property failed.

Add --output-car to collect the whole batch into one CAR alongside the per-property ZIPs:

elephant-cli hash ./county-transformed \
  --output-zip ./county-hashed \
  --output-csv county-hash-results.csv \
  --output-car county.car

Options

Option	Description	Default
`-o, --output-zip <path>`	Destination ZIP containing canonicalized JSON (folder named by property CID).	`hashed-data.zip`
`-c, --output-csv <path>`	CSV file with hash results.	`hash-results.csv`
`--output-car <path>`	Also write all hashed JSON blocks of the run into one CAR rooted at an index.	Not written
`--max-concurrent-tasks <number>`	Target concurrency for hashing (fallback determined automatically).	Auto
`--property-cid <cid>`	Override the property CID used for the output folder and CSV.	Seed CID or inferred value

Export County Tables

Turn a validated county CAR into Parquet tables with export-tables: one table per lexicon class, one per relationship type, and a properties table from the index, plus a tables.car whose single root records every part.

elephant-cli export-tables county.car \
  --output ./county-tables \
  --output-json tables-summary.json

What it does

Walks the CAR from the county index through every shard, property, data-group root and relationship block, in file order.
Places each entity in the table of the lexicon class its relationship schema names for that end (from/to cid -> class schema title, snake_case), never by file name. Every schema is fetched once per CID.
Writes each entity and relationship block once, however many groups or relationships reach it; property_cid and data_group_cid on such a shared block name the first property and data group that reached it in walk order.
Closes a part when the next row would push it past --part-size (default 1g), so small tables are one part and the same CAR always yields byte-identical parts and the same tables root (no timestamps inside the files).
Compresses every Parquet page with Zstd level 3 (the node:zlib implementation); the index records "codec":"zstd".
Computes each part's CID as a UnixFS file with CIDv1, raw leaves and sha2-256, so ipfs add --cid-version 1 --raw-leaves <part> returns the recorded CID.

Inputs

A county CAR written by hash --output-car, ideally after validate county.car passed.
Class, relationship and data-group schemas, resolved through the schema cache and the lexicon manifest.

Outputs

county-tables/
├── tables.car                     (one dag-json CountyTables block, the root)
├── properties/part-00000.parquet  (property_cid, one column per data-group schema CID holding the data-group root CID)
├── property/part-00000.parquet    (one table per class: schema columns, then cid, property_cid, data_group_cid, request_identifier)
├── address/part-00000.parquet
├── property_has_address/part-00000.parquet   (one table per relationship type: relationship_cid, from_cid, to_cid, property_cid, data_group_cid)
└── ...

Column types follow the class schema: string -> UTF8, number -> DOUBLE, integer -> INT64, boolean -> BOOLEAN, every column nullable; source_http_request, source_payload and any other object or array value are UTF8 JSON strings. Every part carries the key-value metadata elephant.county_root, elephant.manifest_url, elephant.table, elephant.part and elephant.part_size_bytes.

The CountyTables root links every part:

{"label":"CountyTables","version":1,"county_root":{"/":"<county index cid>"},"part_size_bytes":1073741824,"codec":"zstd",
 "tables":{"<table>":{"rows":<n>,"parts":[{"cid":{"/":"<unixfs file cid>"},"rows":<n>,"bytes":<n>},...]},...}}

The command prints Tables written: <dir> (<tables> tables, <parts> parts, root <cid>) and, with --output-json, writes the same with per-table row and part counts, the county root and an exportedAt timestamp.

Atlas page

Atlas registers a county as counties/<STATE>/<county>.json. --atlas-page <path> creates or updates that file from the same walk, so nothing is copied by hand:

elephant-cli export-tables county.car \
  --output ./county-tables \
  --atlas-page ./atlas/counties/FL/lee.json --county lee --state FL --fips 12071

{
  "county": "lee",
  "state": "FL",
  "fips": "12071",
  "groups": {
    "county": {
      "cid": "<county index root>",
      "schema": "<data-group schema CID>",
      "tables": "<CountyTables root>"
    }
  }
}

--county, --state and --fips are required when the page does not exist; when it does, any given value must match the file or the command fails.
The group key is the snake_cased title of the archive's data-group schema (County -> county, Property Improvement -> property_improvement), taken from the distinct schema CIDs in the index with the Seed schema dropped. An archive with more than one other data group fails: archive carries N data groups; Atlas registers one group per archive.
groups[<key>] is written as { cid, schema, tables } (county root, schema CID, tables root); other groups in the file are kept, keys are sorted, the file is 2-space indented with a trailing newline. The command prints Atlas page written: <path> group <key>, and the summary (--output-json, library result) carries atlas: { page, group }.

Options

Option	Description	Default
`--output <dir>`	Directory that receives `<table>/part-NNNNN.parquet` files and `tables.car`.	Required
`--part-size <bytes>`	Cap on the bytes of one Parquet part; accepts `k`, `m` and `g` suffixes.	`1g`
`--output-json <path>`	Write the export summary as JSON.	Not written
`--atlas-page <path>`	Create or update the Atlas county page with `groups[<key>] = { cid, schema, tables }`.	Not written
`--county <key>`	Atlas county key; required with `--atlas-page` for a new page, must match an existing one.	From page
`--state <ST>`	Two-letter state code; required with `--atlas-page` for a new page, must match an existing one.	From page
`--fips <code>`	Five-digit county FIPS code; required with `--atlas-page` for a new page, must match an existing one.	From page

Upload Datagroups to IPFS

Upload the hashed bundle to Pinata with the upload command. Provide a Pinata JWT via --pinata-jwt or PINATA_JWT.

export PINATA_JWT=eyJhbGciOi...
elephant-cli upload hashed-data.zip \
  --output-csv upload-results.csv

A .car input (from hash --output-car) is imported through the Kubo RPC API instead (dag/import with pin-roots=true), which a local kubo daemon, Filebase and other pinning providers all speak. Success is reported only after the root has been read back from the gateway (--gateway or ELEPHANT_CAR_GATEWAY, an origin with no trailing slash) and its bytes verified against the root CID; --timeout bounds that readback, not the upload.

A tables directory (from export-tables, recognised by its tables.car) goes through the same API: every Parquet part is added and pinned with add?pin=true&cid-version=1&raw-leaves=true, one request per part, and the CID the node returns must equal the one tables.car records; then tables.car is imported and its root read back exactly as for a county CAR, and the first part is fetched from the gateway (<root>/tables/<table>/parts/0/cid, falling back to its bare CID) and its size compared with the recorded bytes.

# tables directory to Filebase
elephant-cli upload ./county-tables \
  --api https://rpc.filebase.io \
  --output-json tables-upload.json

# local kubo daemon (API on 127.0.0.1:5001, gateway on 127.0.0.1:8080)
elephant-cli upload county.car

# Filebase: the token is composed from the three FILEBASE_* variables
export FILEBASE_ACCESS_KEY=... FILEBASE_SECRET_KEY=... FILEBASE_BUCKET=county-cars
elephant-cli upload county.car \
  --api https://rpc.filebase.io \
  --output-json upload-summary.json

What it does

Extracts the single property directory from the hashed ZIP.
Uploads JSON datagroups (and image assets) to IPFS via Pinata.
Writes a CSV in the same format as hash-results.csv, including upload timestamps.

Inputs

hashed-data.zip containing one property directory named by property CID.

Outputs

IPFS CID for the JSON directory (printed in the CLI).
Optional CID for media files when present.
upload-results.csv mirroring the hash CSV headers with a populated uploadedAt (ISO 8601) column.
For a .car input: the upload summary, printed and (with --output-json) written as JSON:

{
  "api": "https://rpc.filebase.io",
  "root": "baguqeera...",
  "blocks": 133,
  "gatewayUrl": "https://ipfs.filebase.io/ipfs/baguqeera...",
  "uploadedAt": "2025-09-21T00:00:00.000Z"
}

For a tables directory: api, root (the CountyTables CID), countyRoot, parts (part files added), gatewayUrl, uploadedAt.

Options

Option	Description	Default
`--pinata-jwt <jwt>`	Pinata authentication token (falls back to `PINATA_JWT`). ZIP input only.	Required if env var absent
`--api <url>`	Kubo RPC API for a CAR or tables-directory input (falls back to `IPFS_API`).	`http://127.0.0.1:5001`
`--token <bearer>`	Bearer token for the API (falls back to `IPFS_API_TOKEN`, then to base64 of `FILEBASE_ACCESS_KEY:FILEBASE_SECRET_KEY:FILEBASE_BUCKET`).	None
`--gateway <url>`	Gateway origin, no trailing slash, used to read the CAR or tables root back (falls back to `ELEPHANT_CAR_GATEWAY`).	`https://ipfs.filebase.io` for `rpc.filebase.io`, otherwise `http://127.0.0.1:8080`
`--timeout <seconds>`	Seconds to wait for the root (and, for tables, the first part) to resolve on the gateway.	`300`
`--output-json <path>`	Write the CAR or tables upload summary as JSON.	Not written

Submit Hashes to the Contract

Finalize the workflow by submitting the uploaded hashes to the Elephant smart contract on Polygon.

elephant-cli submit-to-contract upload-results.csv \
  --keystore-json oracle-keystore.json \
  --keystore-password "your-strong-password" \
  --rpc-url https://polygon.llamarpc.com \
  --gas-price auto

Use the CSV generated by upload (preferred) or hash (if you operate your own uploader) as the input.

What it does

Validates each row, batches submissions, and sends transactions to the Elephant contract.
Optionally performs dry runs, centralized API submissions, or unsigned transaction export.
Writes transaction IDs to a CSV for auditing.

Inputs

CSV with headers propertyCid,dataGroupCid,dataCid,filePath,uploadedAt.
Encrypted keystore JSON and password, or centralized API credentials.

Outputs

On-chain transactions (unless --dry-run is used).
Updated reports: submit_errors.csv, submit_warnings.csv, and a timestamped transaction-ids-*.csv (override with --transaction-ids-csv).

Options

Option	Description	Default
`--keystore-json <path>`	Encrypted keystore file containing the oracle key.	Required unless using API mode
`--keystore-password <password>`	Password for decrypting the keystore (or set `ELEPHANT_KEYSTORE_PASSWORD`).	Required with keystore
`--rpc-url <url>`	Polygon RPC endpoint.	Env `RPC_URL` or Elephant default
`--contract-address <address>`	Submit contract address.	Env `SUBMIT_CONTRACT_ADDRESS` or default
`--transaction-batch-size <number>`	Number of items per transaction.	`200`
`--gas-price <value>`	Gas price in gwei (`auto` or numeric string).	`30`
`--dry-run`	Validate and produce artifacts without sending transactions.	`false`
`--unsigned-transactions-json <path>`	File to store unsigned transactions (requires `--dry-run`).	None
`--from-address <address>`	Sender address to record in unsigned transactions.	None
`--domain <domain>`	Centralized submission API domain.	None
`--api-key <key>`	API key for centralized submission.	None
`--oracle-key-id <id>`	Oracle key identifier for centralized submission.	None
`--check-eligibility`	Verify consensus and prior submissions before sending.	`false`
`--transaction-ids-csv <path>`	Output CSV for transaction hashes.	`reports/transaction-ids-{timestamp}.csv`

Complete these steps for each property, track generated artifacts, and retain keystore/password information securely. Running the commands in the order above delivers a full seed-to-contract submission for the Elephant Network.

Utility Commands

These helpers support cross-checking hashes, translating identifiers, and auditing previously submitted payloads.

Convert Hex Hashes to CID

elephant-cli hex-to-cid 0x1220e828d7cf579e7a7b2c60cffd66a4663b4857670f2ec16125cb22f1affc6c \
  --validate

What it does

Validates an Ethereum-style 0x-prefixed (or bare) 32-byte hex string.
Converts the hash to a base32 CIDv1 using the raw codec and prints it to stdout.

Input

One 32-byte hex hash (with or without 0x).

Output

CID string on stdout. With --quiet, emits the CID only; otherwise prefixes output with CID:.

Options

Option	Description	Default
`-v, --validate`	Print confirmation that the hex input is valid before conversion.	`false`
`-q, --quiet`	Suppress labels and emit just the CID string.	`false`

Convert CID to Hex Hash

elephant-cli cid-to-hex bafkreicfajrgq6qicnclpbg4qolyhm6co74fcwrkm7n6dyx4qw5bpjvlfe \
  --validate

What it does

Validates a CIDv1 string.
Converts the CID into the 32-byte hex hash expected by on-chain contracts.

Input

One CIDv1 string (multibase base32, usually beginning with b).

Output

32-byte hex hash on stdout (prefixed with Hex: unless --quiet is used).

Options

Option	Description	Default
`-v, --validate`	Print confirmation that the CID input is valid before conversion.	`false`
`-q, --quiet`	Suppress labels and emit just the hex string.	`false`

Check Gas Price

elephant-cli check-gas-price

Or with a custom RPC URL:

elephant-cli check-gas-price --rpc-url https://polygon-mainnet.g.alchemy.com/v2/YOUR_API_KEY

What it does

Fetches current gas prices from the blockchain network.
Displays both legacy (Type 0) and EIP-1559 (Type 2) transaction gas prices.
Shows the current block number and base fee information.

Inputs

None required. Optionally provide an RPC URL.

Output

🐘 Elephant Network CLI - Check Gas Price

📊 Current Gas Prices

Block Number: 79589165

Legacy (Type 0) Transaction:
  Gas Price: 25.285523905 Gwei

EIP-1559 (Type 2) Transaction:
  Max Fee Per Gas: 26.068348884 Gwei
  Max Priority Fee (Tip): 25.789395847 Gwei
  Base Fee Per Gas: 0.281338954 Gwei

✅ Gas price check complete

Options

Option	Description	Default
`--rpc-url <url>`	RPC URL for the blockchain network (falls back to `RPC_URL` env).	`https://polygon-rpc.com`

Use as Library

You can also use this function programmatically in your code:

import { checkGasPrice } from '@elephant-xyz/cli/lib';

const gasPriceInfo = await checkGasPrice({
  rpcUrl: 'https://polygon-mainnet.g.alchemy.com/v2/YOUR_API_KEY',
});

console.log(gasPriceInfo.legacy?.gasPrice); // "25.285523905"
console.log(gasPriceInfo.eip1559?.maxFeePerGas); // "26.068348884"

Check Transaction Status

elephant-cli check-transaction-status transaction-ids.csv

Or with custom options:

elephant-cli check-transaction-status transaction-ids.csv \
  --rpc-url https://polygon-mainnet.g.alchemy.com/v2/YOUR_API_KEY \
  --output-csv checked-transactions.csv \
  --max-concurrent 20

What it does

Reads transaction hashes from a CSV file.
Checks the status of each transaction on the blockchain (success, failed, pending, or not found).
Writes results to a new CSV file with status, block number, gas used, and any errors.

Inputs

CSV file with headers: transactionHash, batchIndex, itemCount, timestamp, status (optional columns are allowed).

Output

transaction-status-checked-20240101120000.csv
transactionHash,batchIndex,itemCount,timestamp,status,blockNumber,gasUsed,checkTimestamp,error
0x123...,0,1,2024-01-01T00:00:00Z,success,12345,100000,2024-01-01T12:00:00Z,
0x456...,1,1,2024-01-01T00:00:00Z,pending,,,2024-01-01T12:00:00Z,

Options

Option	Description	Default
`--rpc-url <url>`	RPC URL for the blockchain network (falls back to `RPC_URL` env).	`https://polygon-rpc.com`
`--output-csv <path>`	Output CSV file path.	`transaction-status-checked-{timestamp}.csv`
`--max-concurrent <num>`	Maximum concurrent status checks.	`10`

Use as Library

You can also use this function programmatically in your code:

import { checkTransactionStatus } from '@elephant-xyz/cli/lib';

// Check a single transaction
const result = await checkTransactionStatus({
  transactionHashes: '0x123...',
  rpcUrl: 'https://polygon-mainnet.g.alchemy.com/v2/YOUR_API_KEY',
});

console.log(result[0].status); // "success", "failed", "pending", or "not found"
console.log(result[0].blockNumber); // 12345 (if mined)
console.log(result[0].gasUsed); // "100000" (if mined)

// Check multiple transactions
const results = await checkTransactionStatus({
  transactionHashes: ['0x111...', '0x222...', '0x333...'],
  rpcUrl: 'https://polygon-mainnet.g.alchemy.com/v2/YOUR_API_KEY',
  maxConcurrent: 20,
});

results.forEach((tx) => {
  console.log(`${tx.transactionHash}: ${tx.status}`);
});

Fetch Data from IPFS or Transactions

elephant-cli fetch-data bafkreicfajrgq6qicnclpbg4qolyhm6co74fcwrkm7n6dyx4qw5bpjvlfe \
  --output-zip fetched-data.zip \
  --gateway https://gateway.pinata.cloud/ipfs

You can also supply a Polygon transaction hash (32-byte hex). When a transaction is provided, the CLI resolves its logged dataset hashes via the configured RPC endpoint before downloading referenced CIDs.

What it does

Traverses an IPFS datagroup tree starting from a CID, following relationship links, and saves the resolved JSON to a ZIP archive.
For transaction hashes, reads on-chain submissions, converts each hex hash back into CID form, and downloads the associated data graph.
Rewrites CID links inside the fetched JSON to point at the relative paths inside the ZIP for easier offline inspection.

Inputs

Either an IPFS CID or a 32-byte transaction hash. Provide one identifier per invocation.

Outputs

fetched-data.zip
└── <property_folder>/
    ├── *.json (datagroups named by schema CID when known)
    └── relationship_*.json (local links between files)

When media assets are referenced and accessible through the gateway, they are downloaded into sibling files in the same property folder.

Options

Option	Description	Default
`-g, --gateway <url>`	IPFS gateway used for downloads (set `IPFS_GATEWAY` to override globally).	`https://gateway.pinata.cloud/ipfs`
`-o, --output-zip <path>`	Destination ZIP that will hold the fetched dataset.	`fetched-data.zip`
`-r, --rpc-url <url>`	Polygon RPC endpoint used when resolving transaction hashes (falls back to `RPC_URL`).	Elephant default

Set --gateway to match the provider used during uploads if you need consistent access controls. Provide an RPC endpoint with access to Elephant submissions when fetching by transaction hash.

Name		Name	Last commit message	Last commit date
.cursor		.cursor
.github		.github
bin		bin
docs		docs
examples		examples
src		src
tests		tests
.eslintignore		.eslintignore
.eslintrc.json		.eslintrc.json
.gitignore		.gitignore
.npmignore		.npmignore
.prettierrc.json		.prettierrc.json
.releaserc.json		.releaserc.json
AGENTS.md		AGENTS.md
CHANGELOG.md		CHANGELOG.md
CLAUDE.md		CLAUDE.md
CONTEXT.md		CONTEXT.md
CONTRIBUTING.md		CONTRIBUTING.md
README.md		README.md
commitlint.config.js		commitlint.config.js
package-lock.json		package-lock.json
package.json		package.json
tsconfig.eslint.json		tsconfig.eslint.json
tsconfig.json		tsconfig.json
vitest.config.ts		vitest.config.ts

Navigation Menu

Repository files navigation

Elephant Network CLI

Table of Contents

Overview

Prerequisites

Installation

Create an Encrypted Keystore

Transform Input Requirements

Property Improvement Workflow

Overview

Step 1: Prepare HTML Content

Step 2: Create Extraction Scripts

Step 3: Transform to Data Group

Build the Seed Bundle

Fetch Current Source Content

Browser Flow Templates

Browser Flow v2 Handler Packages

Transform v2 Handler Packages

Multi-Request Flows

Generate Transformation Scripts

Produce the County Dataset

Hash the County Dataset

Export County Tables

Upload Datagroups to IPFS

Submit Hashes to the Contract

Utility Commands

Convert Hex Hashes to CID

Convert CID to Hex Hash

Check Gas Price

Check Transaction Status

Fetch Data from IPFS or Transactions

About

Releases

Packages

Used by

Contributors

Languages

Latest commit

History

Folders and files

Repository files navigation

Elephant Network CLI

Table of Contents

Overview

Prerequisites

Installation

Create an Encrypted Keystore

Transform Input Requirements

Property Improvement Workflow

Overview

Step 1: Prepare HTML Content

Step 2: Create Extraction Scripts

Step 3: Transform to Data Group

Build the Seed Bundle

Fetch Current Source Content

Browser Flow Templates

Browser Flow v2 Handler Packages

Transform v2 Handler Packages

Multi-Request Flows

Generate Transformation Scripts

Produce the County Dataset

Hash the County Dataset

Export County Tables

Upload Datagroups to IPFS

Submit Hashes to the Contract

Utility Commands

Convert Hex Hashes to CID

Convert CID to Hex Hash

Check Gas Price

Check Transaction Status

Fetch Data from IPFS or Transactions

About

Resources

Contributing

Stars

Watchers

Forks

Releases

Packages

Used by

Contributors

Languages