Utilities

Helper functions for common data transformations

Overview

The ctx.utils object provides utility functions for common data transformations and normalizations. These helpers make it easier to work with URLs, strings, and other data types in your formulas.

Available Utilities

ensureHttp()

Ensure a URL has a protocol (http or https). If the URL already has a protocol, it is returned unchanged. Otherwise, https:// is prepended.

ctx.utils.ensureHttp(url: string): string

Parameters:

• url - The URL to normalize

Returns: The URL with a protocol

Example:

// Add protocol to URLs without one
const url1 = ctx.utils.ensureHttp("example.com");
// Returns: "https://example.com"

const url2 = ctx.utils.ensureHttp("http://example.com");
// Returns: "http://example.com" (unchanged)

const url3 = ctx.utils.ensureHttp("https://example.com");
// Returns: "https://example.com" (unchanged)

Common Use Case:

// Normalize user-provided URLs before scraping
const rawUrl = ctx.thisRow.get("website");
const normalizedUrl = ctx.utils.ensureHttp(rawUrl);

const content = await services.scrape.website({
  url: normalizedUrl,
});

return content.markdown;

normalizeUrl()

Normalize a URL by removing the protocol, www., paths, and query parameters. This is useful for deduplication and comparison.

ctx.utils.normalizeUrl(url: string): string

Parameters:

• url - The URL to normalize

Returns: The normalized URL (domain only, no protocol or www)

Example:

// Normalize various URL formats to the same domain
const url1 = ctx.utils.normalizeUrl("https://www.example.com/about?ref=home");
// Returns: "example.com"

const url2 = ctx.utils.normalizeUrl("http://example.com/contact");
// Returns: "example.com"

const url3 = ctx.utils.normalizeUrl("www.example.com");
// Returns: "example.com"

// All three return the same normalized domain

Common Use Case:

// Deduplicate companies by domain
const website = ctx.thisRow.get("website");
const normalizedDomain = ctx.utils.normalizeUrl(website);

// Check if this domain already exists
const existingCompany = await ctx.getRowByValue(
  "normalized_domain",
  normalizedDomain
);

if (existingCompany) {
  return "duplicate";
}

return normalizedDomain;

Best Practices

URL Normalization Tips:

1. Always normalize for deduplication: Use normalizeUrl() to create a consistent domain format for matching records
2. Ensure protocol before API calls: Use ensureHttp() before passing URLs to services that require full URLs
3. Handle edge cases: Check for null/empty values before normalizing