Pagination

Read one page with a cursor or walk every page with listAll, cap a walk with max, and resume a walk after a crash from a saved cursor without skipping a record.

Vendors page in different ways: a cursor, an offset, a page number. The SDK shows you one: an opaque cursor string. list reads one page. listAll reads all of them, one at a time, under the rate budget.

The samples on this page read five leads from an adapter that serves two per page:

import type { MemoryData } from "outbound-sdk/memory";

// Five leads in one campaign, enough for three pages of two.
export const seed: MemoryData = {
  campaigns: [{ id: "cmp_1", name: "Paging" }],
  leads: ["ada", "ben", "cy", "dee", "eli"].map((name, index) => ({
    campaignId: "cmp_1",
    email: `${name}@example.com`,
    id: `lead_${index + 1}`,
  })),
};

One page at a time

import { createOutbound, OutboundError } from "outbound-sdk";
import { memory } from "outbound-sdk/memory";

import { seed } from "./seed.ts";

// This adapter serves at most two records per page.
const outbound = createOutbound({
  adapter: memory({ data: seed, pageSize: 2 }),
});

const first = await outbound.leads.list();

console.log(
  first.items.map((lead) => lead.sourceId),
  "complete:",
  first.complete
);

// Pass the cursor back to the same listing for the next page.
const second = await outbound.leads.list({ cursor: first.cursor });

console.log(second.items.map((lead) => lead.sourceId));

const third = await outbound.leads.list({ cursor: second.cursor });

console.log(
  third.items.map((lead) => lead.sourceId),
  "complete:",
  third.complete,
  "cursor:",
  third.cursor
);

// `limit` is a hint. An adapter serves at most what its vendor allows.
const wide = await outbound.leads.list({ limit: 500 });

console.log("limit 500 returned", wide.items.length);

const narrow = await outbound.leads.list({ limit: 1 });

console.log("limit 1 returned", narrow.items.length);

// A cursor only works on the listing it came from.
try {
  await outbound.campaigns.list({ cursor: first.cursor });
} catch (error) {
  if (OutboundError.is(error)) {
    console.log(error.code, "-", error.message);
  }
}

try {
  await outbound.leads.list({ limit: 0 });
} catch (error) {
  if (OutboundError.is(error)) {
    console.log(error.code, "-", error.message);
  }
}

[ 'lead_1', 'lead_2' ] complete: false
[ 'lead_3', 'lead_4' ]
[ 'lead_5' ] complete: true cursor: undefined
limit 500 returned 2
limit 1 returned 1
Invalid - The cursor is not valid for campaigns.list (it belongs to leads.list)
Invalid - limit must be a positive integer, got 0

list returns a PageResult:

Cursors

• A cursor is opaque. It is a string to store and pass back. Its contents can change between versions of the SDK.
• A cursor belongs to one listing. It records the provider, the workspace and the listing it came from. Passing it to another listing, another workspace or another vendor is Invalid, as the output shows.
• Keep the filters the same. A cursor says where the vendor was in a listing. It does not record the filters. Send the same filters with it, or the vendor continues a different listing from the same position.
• A cursor is the vendor’s position, wrapped. How long a position stays valid is up to the vendor. The SDK cannot extend it.

limit

limit is a page size hint. An adapter serves at most what its vendor allows, so limit: 500 above returned two records from an adapter whose page size is two. limit: 1 returned one.

limit must be a positive integer. Anything else is Invalid before any request. It is the size of one page and not a cap on a walk: for that, use max.

A page can also be shorter than limit with more pages to come. That happens when the SDK filters a page for you (a client filter) or a vendor returns a short page. Go by cursor or complete, never by items.length.

Every page

import { createOutbound } from "outbound-sdk";
import { memory } from "outbound-sdk/memory";

import { seed } from "./seed.ts";

const outbound = createOutbound({
  adapter: memory({ data: seed, pageSize: 2 }),
});

// Every page, fetched one at a time as the loop asks for more.
for await (const lead of outbound.leads.listAll()) {
  console.log(lead.sourceId, lead.email);
}

// Or all of them in an array.
const all = await outbound.leads.listAll().toArray();

console.log("toArray:", all.length);

// `max` stops the walk. `truncated` says whether anything was left.
const capped = outbound.leads.listAll({ max: 3 });
const some = await capped.toArray();

console.log("max 3:", some.length, "truncated:", capped.truncated);

const enough = outbound.leads.listAll({ max: 5 });

await enough.toArray();

console.log("max 5:", "truncated:", enough.truncated);

lead_1 ada@example.com
lead_2 ben@example.com
lead_3 cy@example.com
lead_4 dee@example.com
lead_5 eli@example.com
toArray: 5
max 3: 3 truncated: true
max 5: truncated: false

listAll(options?) returns a ListAll. Calling it sends nothing. Pages are fetched when you iterate, one request for a page only once the loop has used up the page before.

listAll takes the same filters as list, plus:

max and truncated

max: 3 above stopped after three records, and truncated says two were left. With max: 5 on five records nothing was left, so truncated is false. When the cap falls exactly on the end of a page and the vendor has more pages, truncated is true as well.

toArray() holds every record in memory at once. On a listing that can be large, iterate, or set max and check truncated.

Records that move between pages

A listing can shift while you walk it: a record is added, and one you already saw slides onto the next page. listAll drops a record it has already handed out in the same walk, by its ref. It cannot show you a record that slid backwards past the page you are on. A walk is a read of a moving list, not a snapshot.

Resume after a crash

walk.cursor moves only after a page has been fully handed out. Save it as you go, and a new process can start from it.

import { createOutbound } from "outbound-sdk";
import { memory } from "outbound-sdk/memory";

import { seed } from "./seed.ts";

const outbound = createOutbound({
  adapter: memory({ data: seed, pageSize: 2 }),
});

// Stands in for your own storage: a database row, a file, a queue message.
let saved: string | undefined;
const handled: string[] = [];

const walk = outbound.leads.listAll();

try {
  for await (const lead of walk) {
    if (lead.sourceId === "lead_4") {
      throw new Error("the process died");
    }

    handled.push(lead.sourceId);
    // Save after each record. The cursor moves only when a page is done.
    saved = walk.cursor;
  }
} catch {
  console.log("handled before the crash:", handled);
}

// A new run starts from the saved cursor.
const again: string[] = [];

for await (const lead of outbound.leads.listAll({ cursor: saved })) {
  again.push(lead.sourceId);
}

console.log("handled after the restart:", again);

handled before the crash: [ 'lead_1', 'lead_2', 'lead_3' ]
handled after the restart: [ 'lead_3', 'lead_4', 'lead_5' ]

The first run handled lead_3 and then died on lead_4, part way through the second page. The saved cursor points at the start of that page, so the second run reads the whole page again: lead_3 is delivered twice.

That is at-least-once delivery. A walk resumed from a saved cursor never skips a record, and it repeats the records of the page it was reading. Make the work you do per record safe to repeat: write to your database with an upsert keyed on ref.

• Before the first page is finished, walk.cursor is undefined. Resuming from undefined starts at the beginning, which is correct.
• After the last page, walk.cursor is undefined again. Record that the walk finished separately, or a finished walk and one that never started look the same.
• The duplicate filter covers one walk. A resumed walk is a new one and does not know what the old one handed out.

The sync plugin does this bookkeeping for you with a store you supply.

What a walk costs

Each page is at least one vendor request, and some adapters need more per page. outbound.estimate({ action: "leads.list", records: 2500 }) returns the number from what the adapter declares, without sending anything: see the overview. walk.meta.requests is what the walk has spent so far.