Custom Connector

Write the connector yourself in a Python notebook, for a system nothing else covers.

Category: Other
Source type: CUSTOM
Produces: recorddocumentpagefiletable

Every organisation has the one system nothing supports — an internal API, a mainframe export, a vendor portal, a homegrown store. A custom connector is a small Python notebook you write in the app that turns that system into a normal source: it produces assets, and everything downstream treats them like any other.

Start here: Custom Connectors walks through writing one, and the notebook reference documents every function and field.

What you need to connect

Nothing but the notebook. Whatever your connector needs — a base URL, an account id — goes in variables; whatever must stay secret goes in secrets, encrypted exactly like any other source credential and redacted from logs and cell output.

The notebook defines a handful of plain functions: one to test the connection, one to yield assets, and optionally one to declare relationships.

What Classifyre reads

Whatever you yield. Assets can be records, documents, pages, files or tables, with free-form metadata. Return file bytes and you get the whole file pipeline for free — format detection, text extraction, OCR, archive expansion.

Files your connector yields are read with the shared file pipeline: see Supported File Formats for everything it can open, and OCR & Transcription for reading text out of images, audio and video.

Metadata on every asset

Asset kind · record

Field	Type	Always present	What it is
external_id	string	No	Identifier of the record in the system the notebook read from
content_reference	string	No	Set when the connector recorded this asset without extracting its content (Asset(extract=False)); the value says why. Such an asset is complete and searchable by metadata, takes part in lineage and tags, and is deliberately not text-scanned.
collection	string	No	Container the record came from (table, index, folder, board, ...)
author	string	No	Who created the record, if known

Asset kind · document

Field	Type	Always present	What it is
external_id	string	No	Identifier of the record in the system the notebook read from
content_reference	string	No	Set when the connector recorded this asset without extracting its content (Asset(extract=False)); the value says why. Such an asset is complete and searchable by metadata, takes part in lineage and tags, and is deliberately not text-scanned.
collection	string	No	Container the record came from (table, index, folder, board, ...)
author	string	No	Who created the record, if known
encoding	string	No	Text encoding of the content
word_count	integer	No	Words in the extracted text

Asset kind · page

Field	Type	Always present	What it is
external_id	string	No	Identifier of the record in the system the notebook read from
content_reference	string	No	Set when the connector recorded this asset without extracting its content (Asset(extract=False)); the value says why. Such an asset is complete and searchable by metadata, takes part in lineage and tags, and is deliberately not text-scanned.
collection	string	No	Container the record came from (table, index, folder, board, ...)
author	string	No	Who created the record, if known
encoding	string	No	Text encoding of the content
title	string	No	Page title

Asset kind · file

Field	Type	Always present	What it is
external_id	string	No	Identifier of the record in the system the notebook read from
content_reference	string	No	Set when the connector recorded this asset without extracting its content (Asset(extract=False)); the value says why. Such an asset is complete and searchable by metadata, takes part in lineage and tags, and is deliberately not text-scanned.
collection	string	No	Container the record came from (table, index, folder, board, ...)
author	string	No	Who created the record, if known
size_bytes	integer	No	Size of the file in bytes
mime_type	string	No	Content type of the file

Asset kind · table

Field	Type	Always present	What it is
external_id	string	No	Identifier of the record in the system the notebook read from
content_reference	string	No	Set when the connector recorded this asset without extracting its content (Asset(extract=False)); the value says why. Such an asset is complete and searchable by metadata, takes part in lineage and tags, and is deliberately not text-scanned.
collection	string	No	Container the record came from (table, index, folder, board, ...)
author	string	No	Who created the record, if known
row_count	integer	No	Rows in the result set
columns	string[]	No	Columns as [{name, type}] objects

Lineage

Custom connectors can declare every relationship class — lineage, containment, identity, reference and usage — including column-level field mappings, and including edges that point at objects in systems this connector doesn’t scan. That last part is how a connector for your in-house pipeline can wire its outputs to the warehouse tables another source ingests.

See the relationships section of the notebook reference and Lineage & Relationships.

Worth knowing

Your code runs isolated. The notebook executes in a separate process with a scrubbed environment, so it never sees the platform’s own credentials.
Packages are declared, not installed by hand. List the Python distributions your connector needs and they’re installed before it runs.
Executions are bounded by a timeout, an asset cap, and an output-size limit, so a runaway loop can’t consume a cluster.
Local folders can be exposed to the notebook by name on any deployment that mounts them into the scan.
Facts you already know can be recorded directly with a Tag detector, rather than re-derived by a classifier.

Configuration

Beyond the fields below, every source also has the settings shared by all of them: the sampling strategy, the augmentation notebook that enriches assets after extraction, the detectors to run, the scan schedule, and the compute limits for its scan jobs.

Required

Without these, the source will not save.

Field	Type	Required	What it does	Default
required	object	Yes	The Python notebook that implements this connector.no extra properties	—
notebook	object	Yes	Cells run in document order in one fresh process per execution. There is no persistent kernel: state is rebuilt from the current cell sources every time.no extra properties	—
notebook.cells	array	Yes	Ordered notebook cells. The assembled code cells must define test_connection() and extract().min items 1, max items 200	—
notebook.cells[]	object	Yes	—	—
notebook.revision	integer	No	Monotonic revision, bumped on every save. Used for optimistic locking so two editors cannot silently overwrite each other.min 1	1

Secrets

Stored encrypted and never shown again after you save them. See Configuration & Fields.

Field	Type	Required	What it does	Default
masked	object	No	Encrypted key/value pairs, stored the same way as any other source credential.no extra properties	—
secrets	object	No	Secret values the notebook reads with ctx.secret("name"). Encrypted at rest and redacted from logs and cell output. Keys must be valid Python identifiers.	{}

Optional

Everything you can tune. Sensible defaults apply when you leave them alone.

Field	Type	Required	What it does	Default
optional	object	No	—no extra properties	—
cohort_weights	object	No	How ctx.cohort() splits each run's budget across its bands. auto shifts the split toward the bands whose keys produce HIGH/CRITICAL findings, within guards: the declared split stands until every band has 200 visits, no band drops below the floor, and no band moves more than 15 points per run. fixed always uses the bands the notebook declares, or the ones given here.no extra properties	—
cohort_weights.effective	object	No	Set by the platform on each run's recipe: the split each cohort runs with. Not stored on the source.	—
cohort_weights.fixed	object	No	Per cohort name, the band split to use under fixed, e.g. {"register": {"newest": 60, "oldest": 30, "random": 10}}.	—
cohort_weights.floor	number	No	The least share any band keeps under auto, as a fraction. Never below 0.1.min 0.1, max 0.5	—
cohort_weights.mode	enum	No	auto: measured from what each band yields. fixed: the declared or fixed split, always. Allowed: auto, fixed	auto
limits	object	No	Bounds applied to every notebook execution and scan.no extra properties	—
limits.max_assets	integer	No	Stop a scan after this many assets. Unset means no cap.min 1, max 1000000	—
limits.max_output_bytes	integer	No	Total serialized cell output kept per execution. Larger outputs are truncated rather than stored.min 1024, max 52428800	2097152
limits.timeout_seconds	integer	No	Kill an execution that runs longer than this. Cells cannot be interrupted from inside Python, so this is the real stop button.min 10, max 86400	900
local_folders	array	No	Folders the notebook reads with ctx.folder("name"). Not available in Kubernetes deployments, where files are uploaded to the source instead. This is a convenience, not a sandbox: the notebook process runs as you and can open any path you can.max items 10	[]
local_folders[]	object	No	A folder the notebook can read on the machine that runs the scan: a bind-mounted path in the all-in-one Docker image, or one mounted into CLI jobs by the chart's `api.localFolders` in Kubernetes.no extra properties	—
local_folders[].name	string	Yes	How the notebook refers to it: ctx.folder("name"). Must be a valid Python identifier.pattern ^[A-Za-z_][A-Za-z0-9_]{0,62}$	—
local_folders[].path	string	Yes	Absolute path to the folder, as seen by the machine that runs the scan — for example /data/dumps for a directory bind-mounted into the container, or /mnt/corpora/dumps for a folder mounted by the chart.min length 1, max length 4096	—
packages	array	No	Python packages installed into the run environment before any cell executes. Installed with uv; the base image's own dependencies are always present and do not need listing.max items 50	[]
packages[]	object	No	A Python distribution installed before the notebook runs.no extra properties	—
packages[].name	string	Yes	Distribution name as published on the index (for example 'pandas').pattern ^[A-Za-z0-9][A-Za-z0-9._-]{0,63}$	—
packages[].version	string	No	Optional version or specifier: '2.2.0', '>=2.0', '~=1.4'. Empty means latest.max length 64, pattern ^$\|^(==\|>=\|<=\|~=\|!=\|>\|<).+$\|^[0-9][A-Za-z0-9._+!-]$	—
variables	object	No	Non-secret values the notebook reads with ctx.var("name"), such as a base URL or an account id. Stored in clear text. Keys must be valid Python identifiers.	{}