Skip to content
NewsTeam Documentation Request early access

Creating an importer

An inbound integration captures source data as logfiles, then maps those stored bytes to NewsTeam articles. This example imports one fictional title. It demonstrates the published @newsteam/plugin-sdk@1.0.0-alpha.3 contract, including the required base hooks.

Install Node.js 22.15 or newer. Download all six files into an empty directory:

Terminal window
mkdir newsteam-importer
cd newsteam-importer
for file in package.json package-lock.json tsconfig.json importer.ts importer.test.ts server.ts; do
curl --fail --show-error --silent "https://newsteam.io/examples/inbound-importer/$file" --output "$file"
done
npm ci
npm run check
npm test
npm start

The server listens on 127.0.0.1:3333. npm test runs discovery and capture/replay through an in-process Connect transport; it needs no NewsTeam account, provider credential or source API. The tests assert type routing, cursor advancement, deterministic replay and invalid-input handling.

The package manifest and lockfile pin the published SDK and its tooling. Protocol imports below use version-specific dist paths to avoid loading the package’s convenience server. Recheck those paths when upgrading.

IntegrationService.GetEnvTwo returns a catalogue of plugins, their configuration fields and complete typed services. InboundService provides GetLogfiles and ProcessLogfile. Capability calls carry the typeId header so one service can host multiple integrations.

importer.ts
import { create, type JsonObject } from "@bufbuild/protobuf";
import { Code, ConnectError, type ConnectRouter, type HandlerContext } from "@connectrpc/connect";
import type { InboundPlugin } from "@newsteam/plugin-sdk/dist/plugin.js";
import {
FieldDescriptorSchema, GetEnvResponseSchema, IntegrationService,
} from "@newsteam/plugin-sdk/dist/proto/integration_pb.js";
import {
InboundService, GetLogfilesResponseSchema, ProcessLogfileResponseSchema,
type GetLogfilesRequest, type ProcessLogfileRequest,
} from "@newsteam/plugin-sdk/dist/proto/inbound_pb.js";
import { ArticleSchema } from "@newsteam/plugin-sdk/dist/proto/article_pb.js";
import { Cursor_Status } from "@newsteam/plugin-sdk/dist/proto/cursor_pb.js";
import { SuccessSchema } from "@newsteam/plugin-sdk/dist/proto/shared_pb.js";
function language(attributes?: JsonObject): string {
const value = attributes?.language ?? "en";
if (value !== "en" && value !== "fr") {
throw new ConnectError("Choose en or fr", Code.InvalidArgument);
}
return value;
}
export class ExampleImporter implements InboundPlugin {
readonly typeId = "example-wire";
readonly version = "1.0.0";
readonly name = "Example wire";
readonly description = "One fictional article, captured once and replayed locally.";
readonly fields = [create(FieldDescriptorSchema, {
name: "language", label: "Source language",
data: { case: "string", value: {
defaultValue: "en",
options: [{ value: "en", label: "English" }, { value: "fr", label: "French" }],
} },
})];
// Required alpha.3 base hooks. This example accepts no provider notifications.
async onWebhook(): Promise<never> {
throw new ConnectError("No webhook configured", Code.Unimplemented);
}
async onEvent() { return create(SuccessSchema); }
async getLogfiles({ cursor, attributes }: GetLogfilesRequest) {
language(attributes);
const bytes = new TextEncoder().encode(JSON.stringify({
id: "wire-001", slug: "community-garden", title: "A new community garden",
}));
return create(GetLogfilesResponseSchema, {
data: cursor?.seekPos ? [] : [bytes],
cursor: { status: Cursor_Status.COMPLETE, seekPos: 1, total: 1n },
});
}
async processLogfile({ data, attributes }: ProcessLogfileRequest) {
const locale = language(attributes);
let row: unknown;
try { row = JSON.parse(new TextDecoder().decode(data)); }
catch { throw new ConnectError("Invalid logfile", Code.InvalidArgument); }
if (typeof row !== "object" || row === null ||
!("id" in row) || typeof row.id !== "string" || !row.id ||
!("slug" in row) || typeof row.slug !== "string" || !row.slug ||
!("title" in row) || typeof row.title !== "string" || !row.title) {
throw new ConnectError("Invalid logfile", Code.InvalidArgument);
}
// No source lookups: the stored bytes contain all mapping inputs.
return create(ProcessLogfileResponseSchema, {
articles: [create(ArticleSchema, {
id: row.id, slug: row.slug, title: { text: row.title }, locale,
})],
});
}
}
export function routes(router: ConnectRouter) {
const plugin = new ExampleImporter();
const requireType = (context: HandlerContext) => {
if (context.requestHeader.get("typeId") !== plugin.typeId) {
throw new ConnectError("Expected typeId example-wire", Code.NotFound);
}
};
router.service(IntegrationService, {
getEnvTwo: async () => create(GetEnvResponseSchema, { plugins: [{
typeId: plugin.typeId, version: plugin.version, name: plugin.name,
description: plugin.description, fields: plugin.fields,
services: [{ name: InboundService.typeName,
methods: InboundService.methods.map(method => method.name) }],
}] }),
onWebhook: async (_request, context) => { requireType(context); return plugin.onWebhook(); },
onEvent: async (_request, context) => { requireType(context); return plugin.onEvent(); },
});
router.service(InboundService, {
getLogfiles: async (request, context) => { requireType(context); return plugin.getLogfiles(request); },
processLogfile: async (request, context) => { requireType(context); return plugin.processLogfile(request); },
});
}

The language field is saved integration configuration. The integration validates it when invoked. Keep source URLs, tenant settings and credentials in fields for real providers; keep credentials out of public clients and logfiles.

getLogfiles captures all source data needed by the mapper. This one-item fixture advances seekPos and marks the cursor complete. A real paginated source should preserve cursor identity and metadata, advance progress only after capture, and keep source continuation tokens or boundaries in cursor.attributes where needed. Do not use the fixture’s fixed completion cursor for a live feed.

processLogfile reads only the saved bytes and configuration. Capture supplemental author, category, media and redirect information during capture, so replay never needs a source lookup. Preserve stable IDs, source slugs/URLs, bylines and placements; map body content into native widgets. This deliberately small example returns a title-only article.

The published SDK also requires onEvent and onWebhook. The example acknowledges events without subscribing to them, and rejects webhooks as unconfigured. A method’s presence does not establish provider signature validation or a production notification workflow.

server.ts
import fastify from "fastify";
import { fastifyConnectPlugin } from "@connectrpc/connect-fastify";
import { routes } from "./importer.js";
const server = fastify({ http2: true });
await server.register(fastifyConnectPlugin, { routes, grpcWeb: false });
await server.listen({ host: "127.0.0.1", port: Number(process.env.PORT ?? "3333") });

The server uses HTTP/2 on loopback. Its Connect RPC discovery method is /io.IntegrationService/GetEnvTwo; it is separate from the public V1 JSON API. Deploy a reachable, protected service for NewsTeam to call, then select example-wire in integration settings. Do not expose this unauthenticated fixture server directly on the internet.

importer.test.ts
import assert from "node:assert/strict";
import test from "node:test";
import { createClient, createRouterTransport } from "@connectrpc/connect";
import { IntegrationService } from "@newsteam/plugin-sdk/dist/proto/integration_pb.js";
import { InboundService } from "@newsteam/plugin-sdk/dist/proto/inbound_pb.js";
import { routes } from "./importer.js";
const transport = createRouterTransport(routes);
const inbound = createClient(InboundService, transport);
const integration = createClient(IntegrationService, transport);
const headers = { headers: { typeId: "example-wire" } };
test("typed discovery and type routing", async () => {
const env = await integration.getEnvTwo({});
assert.equal(env.plugins[0]?.typeId, "example-wire");
assert.deepEqual(env.plugins[0]?.services[0]?.methods, ["GetLogfiles", "ProcessLogfile"]);
assert.equal(env.plugins[0]?.fields[0]?.name, "language");
await assert.rejects(inbound.getLogfiles({}), /Expected typeId/);
await assert.rejects(inbound.getLogfiles({}, { headers: { typeId: "wrong" } }), /Expected typeId/);
});
test("capture advances cursor and replay needs no source network", async () => {
const capture = await inbound.getLogfiles({}, headers);
assert.equal(capture.data.length, 1);
assert.equal(capture.cursor?.seekPos, 1);
assert.equal((await inbound.getLogfiles({ cursor: capture.cursor }, headers)).data.length, 0);
const originalFetch = globalThis.fetch;
globalThis.fetch = async () => { throw new Error("Source lookup during replay"); };
try {
const replay = () => inbound.processLogfile({ data: capture.data[0], attributes: { language: "fr" } }, headers);
const first = await replay();
assert.deepEqual(first, await replay());
assert.equal(first.articles[0]?.id, "wire-001");
assert.equal(first.articles[0]?.slug, "community-garden");
assert.equal(first.articles[0]?.locale, "fr");
} finally { globalThis.fetch = originalFetch; }
});
test("invalid input and unconfigured notifications fail explicitly", async () => {
await assert.rejects(inbound.getLogfiles({ attributes: { language: "unknown" } }, headers), /Choose en or fr/);
for (const text of ["not-json", "{}"])
await assert.rejects(inbound.processLogfile({ data: new TextEncoder().encode(text) }, headers), /Invalid logfile/);
await assert.rejects(integration.onWebhook({}, headers), /No webhook configured/);
await assert.rejects(integration.onEvent({}), /Expected typeId/);
await integration.onEvent({}, headers);
});

Before a real import, also test source pagination, interrupted capture/resume, changed and deleted source items, repeated processing, media permissions and article URL preservation. Confirm the target Bucket, workflow status, policy and auto-publish settings with the editor. Provider setup explains the environment checks; the developer pack offers broader examples.