Data exports

Pull filtered records out as a JSON or CSV file, asynchronously.

Overview

A data export pulls a filtered set of records out of ShipGenius as a file. It covers the cases a search would need many pages for — a nightly warehouse feed, a finance reconciliation, a one-off data pull.

Exports are asynchronous. You start one, it runs server-side, and you poll until it is done. The result is a DataReport carrying a JSON URL and, if you asked for it, a CSV URL.

The shape of a request

An export request is three decisions:

Decision	Field
Which records	`entity_type` + `filter_expression`	Same filter language as search
Which fields	`entity_fragment`	A GraphQL selection set, as a string
What format	`csv_mappings`	Omit for JSON only

Starting an export

mutation StartDataExport($requested_data: DataReportInput!) {
    analytics {
        start_data_export(requested_data: $requested_data) {
            id
            done
            failed
            created_at
        }
    }
}

{
    "requested_data": {
        "entity_type": "ORDER",
        "entity_fragment": "{ id external_order_id created_at shipping_address { city state zip } }",
        "filter_expression": [
            {
                "expression": {
                    "left": { "column": "created_at" },
                    "op": ">=",
                    "right": { "relative_time": { "days": -1 } }
                }
            }
        ]
    }
}

entity_type

One of FilterExpressionEntityType — the same enum the search API and saved filters use. ORDER, ITEM, SHIPMENT_RECORD, INVENTORY, ORDER_LINE, and around 25 more.

entity_fragment

This is the part that has no equivalent in the search API. It is a GraphQL selection set supplied as a string, and it decides which fields land in the file:

{ id external_order_id created_at shipping_address { city state zip } }

Nested selections are allowed, which is how you flatten related records into one export rather than exporting each table separately.

NOTE

The fragment is a string, so your editor will not check it. Build the equivalent selection as a real query against the entity's type page first, confirm it returns what you expect, then paste it in.

filter_expression

Identical to search: a list of FilterExpression, combined with AND. See Searching for the full language, including relative times and aggregations.

Omitting it exports every record of that type.

Asking for CSV

Without csv_mappings you get JSON. Supply DataReportCsvMappingInput and you get a CSV as well.

{
    "requested_data": {
        "entity_type": "ORDER",
        "entity_fragment": "{ id external_order_id created_at }",
        "csv_mappings": {
            "list_method": "AS_JSON",
            "null_text": "",
            "row_mappings": [
                { "field_name": "id", "header_name": "Order ID" },
                { "field_name": "external_order_id", "header_name": "PO Number" },
                { "field_name": "created_at", "header_name": "Created" }
            ]
        }
    }
}

Field	Default	Purpose
`row_mappings`	`[]`	Field to column-header mapping, in output order
`null_text`	`""`	What an absent value prints as
`list_method`	`AS_JSON`	How a nested list becomes rows — see below

Flattening lists into CSV

JSON nests; CSV does not. list_method decides how that is resolved when a record has a list on it, such as an order with several lines:

Value	Result
`AS_JSON`	The list stays in one cell as JSON text. One row per record.
`REPEAT_PARENT`	One row per list element, with the parent's columns repeated on each.
`BLANK_PARENT`	One row per list element, with the parent's columns only on the first.

REPEAT_PARENT gives every row a complete set of parent columns. BLANK_PARENT leaves them blank after the first row of each group. AS_JSON keeps the row count equal to the record count.

Polling for completion

The mutation returns immediately with done: false. Poll the report by id through the search API:

query DataReportStatus($entity_id: ID!) {
    searches {
        data_report {
            by_id(entity_id: $entity_id) {
                id
                done
                failed
                error_message
                fetched_rows
                json_file_url
                csv_file_url
            }
        }
    }
}

Three terminal states, and you must distinguish them:

`done`	`failed`	Meaning
`false`	`false`	Still running. Poll again.
`true`	`false`	Finished. The file URLs are populated.
`true`	`true`	Failed. Read `error_message`.

fetched_rows increments while the export runs, so it doubles as a progress indicator on a long job.

WARNING

done: true does not mean success. Check failed before reading the URLs — on a failed export they are null and error_message holds the reason.

A poll loop with backoff:

async function waitForExport(client, id, { timeout_ms = 15 * 60 * 1000 } = {}) {
    const started = Date.now();
    let delay = 2000;

    for (;;) {
        const { data } = await client.query({
            query: DATA_REPORT_STATUS,
            variables: { entity_id: id },
            fetchPolicy: "no-cache",
        });

        const report = data.searches.data_report.by_id;

        if (report.failed) {
            throw new Error(`Export ${id} failed: ${report.error_message}`);
        }
        if (report.done) {
            return report;
        }
        if (Date.now() - started > timeout_ms) {
            throw new Error(`Export ${id} did not finish within the timeout`);
        }

        await new Promise((resolve) => setTimeout(resolve, delay));
        delay = Math.min(delay * 1.5, 30000);
    }
}

Use fetchPolicy: "no-cache", or a cached first response will make the export look permanently unfinished.

Downloading

A finished report exposes json_file_url and, when CSV was requested, csv_file_url. Fetch them directly — they are already-authorized URLs, so do not attach your API key.

const report = await waitForExport(client, id);
const rows = await fetch(report.json_file_url).then((r) => r.json());

Finding past exports

Reports are a searchable entity like any other, so the search API lists them:

query RecentExports($search: PaginatedSearchQuery) {
    searches {
        data_report {
            search(search: $search) {
                rows {
                    id
                    entity_type
                    created_at
                    done
                    failed
                    fetched_rows
                }
                has_next_page
                next_cursor
            }
        }
    }
}

{
    "search": {
        "limit": 25,
        "order_by": [{ "column": "created_at", "reverse": true }],
        "filter": [
            {
                "expression": {
                    "left": { "column": "created_at" },
                    "op": ">=",
                    "right": { "relative_time": { "days": -7 } }
                }
            }
        ]
    }
}

Choosing between export and search

	Search	Export
Result	Rows in the response	A file URL
Timing	Synchronous	Asynchronous, polled
Field selection	A real GraphQL selection	A selection set as a string
Suits	Interactive listing, small pages	Bulk extraction, scheduled feeds

Walking cursors to accumulate a whole result set into a file is the case exports exist for.

Errors →