Skip to content

Development

This guide is for developers who want to contribute to the Substack API client library or understand its internal architecture.

Development Setup

Prerequisites

  • Node.js 16 or higher
  • npm or yarn package manager
  • Git
  • A Substack account for testing

Getting Started

For the most consistent development experience, use the provided dev container configuration:

GitHub Codespaces: 1. Click the "Code" button in the GitHub repository 2. Select "Codespaces" tab
3. Click "Create codespace on main" 4. Dependencies will be installed automatically

VS Code Remote - Containers: 1. Install the "Remote - Containers" extension in VS Code 2. Clone the repository locally 3. Open in VS Code and click "Reopen in Container" when prompted 4. Dependencies will be installed automatically

Option 2: Local Development

  1. Clone the repository: bash git clone https://github.com/jakub-k-slys/substack-api.git cd substack-api

  2. Install dependencies: bash npm install

  3. Build the project: bash npm run build

Environment Setup

Create a .env file for testing:

# .env
SUBSTACK_API_KEY=your-connect-sid-cookie-value
SUBSTACK_HOSTNAME=example.substack.com

To get your substack.sid cookie value: 1. Login to Substack in your browser 2. Open Developer Tools (F12) 3. Go to Application/Storage → Cookies → https://substack.com 4. Copy the substack.sid value

Project Structure

substack-api/
├── src/
│   ├── substack-client.ts    # Main SubstackClient implementation  
│   ├── http-client.ts        # HTTP client with authentication
│   ├── entities/             # Entity classes (Profile, Post, Note, Comment)
│   │   ├── profile.ts
│   │   ├── own-profile.ts  
│   │   ├── post.ts
│   │   ├── note.ts
│   │   └── comment.ts
│   ├── types/               # TypeScript type definitions
│   │   ├── api-types.ts
│   │   ├── entity-types.ts
│   │   └── config-types.ts
│   ├── note-builder.ts      # Helper for building formatted notes
│   └── index.ts             # Public API exports
├── tests/
│   ├── unit/                # Unit tests for individual components
│   ├── integration/         # Integration tests for entity interactions
│   └── e2e/                 # End-to-end tests with real API
├── docs/                    # Comprehensive documentation
│   ├── api-reference.md     # Complete API documentation
│   ├── entity-model.md      # Entity model guide
│   ├── examples.md          # Real-world usage examples
│   ├── quickstart.md        # Getting started guide
│   └── ...
├── samples/                 # Sample applications and scripts
├── dist/                    # Compiled JavaScript files
├── .env.example             # Environment variables template
├── jest.config.js           # Jest configuration for unit tests
├── jest.integration.config.js # Jest configuration for integration tests
├── jest.e2e.config.js       # Jest configuration for E2E tests
├── package.json             # Project configuration
├── tsconfig.json            # TypeScript configuration
└── README.md                # Project overview

Development Workflow

Building

To build the project:

npm run build

This will: 1. Clean the dist directory 2. Compile TypeScript files 3. Generate type definitions

Testing Strategy

The project uses a comprehensive 3-tier testing strategy:

1. Unit Tests (npm test)

  • Purpose: Test individual components in isolation
  • Location: tests/unit/
  • Speed: Very fast (< 1 second)
  • Scope: Functions, classes, utilities
  • Mocking: Heavy use of mocks for external dependencies
npm test              # Run all unit tests
npm run test:watch    # Run in watch mode for development

2. Integration Tests (npm run test:integration)

  • Purpose: Test component interactions and entity relationships
  • Location: tests/integration/
  • Speed: Fast (few seconds)
  • Scope: Entity navigation, async iteration, error handling
  • Mocking: Mock HTTP layer, real entity logic
npm run test:integration          # Run integration tests
npm run test:integration:watch    # Watch mode

3. End-to-End Tests (npm run test:e2e)

  • Purpose: Validate against real Substack API
  • Location: tests/e2e/
  • Speed: Slower (network dependent)
  • Scope: Full workflow validation, API compatibility
  • Mocking: No mocks - real API calls
npm run test:e2e     # Run E2E tests (requires credentials)

End-to-End Testing

The project includes end-to-end (E2E) tests that validate integration with the real Substack server. These tests are located in the tests/e2e/ directory.

Setting Up E2E Tests

  1. Set up credentials: Create a .env file in the project root with your Substack credentials:

bash # Copy the example file cp .env.example .env

Edit the .env file and add your substack.sid cookie:

bash SUBSTACK_API_KEY=your-connect-sid-cookie-value SUBSTACK_HOSTNAME=yoursite.substack.com # optional

Important: Never commit your .env file to version control. It's already included in .gitignore.

  1. Obtain credentials: Get your substack.sid cookie value:
  2. Login to Substack in your browser
  3. Open Developer Tools (F12)
  4. Go to Application/Storage → Cookies → https://substack.com
  5. Copy the substack.sid value

Running E2E Tests

npm run test:e2e              # Run all E2E tests
npm run test:e2e -- --testNamePattern="Profile" # Run specific tests

E2E tests are designed to be: - Safe: Read-only operations where possible, minimal writes - Isolated: Each test cleans up after itself - Conditional: Skip gracefully when credentials are unavailable - Respectful: Include delays to avoid overwhelming the API

Run E2E tests in watch mode:

npm run test:e2e:watch

Run both unit and E2E tests:

npm run test:all

E2E Test Behavior

  • Without credentials: Tests will be automatically skipped with a warning message explaining how to set up credentials.
  • With credentials: Tests will run against the real Substack API using your provided credentials.
  • Test isolation: E2E tests are designed to be read-only and safe to run multiple times without creating unwanted content.
  • Timeout: E2E tests have a 30-second timeout to account for network latency.

E2E Test Coverage

The E2E test suite covers:

  • Authentication: Verifying API key authentication works
  • Publication operations: Getting publication details and metadata
  • Post operations: Fetching posts, pagination, searching, and individual post retrieval
  • Comment operations: Fetching comments for posts and individual comments
  • Notes operations: Fetching notes and pagination (note publishing tests are commented out to avoid creating content)
  • Profile operations: Getting user profiles and public profiles

Creating New E2E Tests

When adding new E2E tests:

  1. Use the conditional test pattern with skipIfNoCredentials()
  2. Handle API errors gracefully (some operations may not be available for all accounts)
  3. Avoid tests that create persistent content unless absolutely necessary
  4. Add logging for skipped operations to help with debugging
  5. Follow the existing test structure and naming conventions

Code Style

The project follows TypeScript best practices:

  • Use explicit types where beneficial
  • Document public APIs with JSDoc comments
  • Follow consistent naming conventions
  • Write unit tests for new functionality

Documentation

Documentation is written in Markdown and built using Sphinx with MyST parser:

  1. Install documentation dependencies: bash pip install sphinx sphinx-rtd-theme myst-parser

  2. Build the documentation: bash cd docs make html

The built documentation will be available in docs/build/html.

Contributing

Contribution Process

  1. Fork the repository
  2. Create a feature branch
  3. Make your changes
  4. Write or update tests
  5. Update documentation
  6. Submit a pull request

Pull Request Guidelines

  • Keep changes focused and atomic
  • Follow existing code style
  • Include tests for new functionality
  • Update documentation as needed
  • Describe your changes in the PR description

Running Checks

Before submitting a PR, ensure:

  1. All tests pass: bash npm test

  2. TypeScript compiles without errors: bash npm run build

  3. Documentation builds successfully: bash cd docs make html

Release Process

  1. Update version in package.json
  2. Update CHANGELOG.md

  3. Build the project: bash npm run build

  4. Run tests: bash npm test

  5. Commit changes: bash git add . git commit -m "Release v1.x.x" git tag v1.x.x git push origin main --tags

  6. Publish to npm: bash npm publish

Development Tips

TypeScript Configuration

The project uses a strict TypeScript configuration. Key settings in tsconfig.json:

{
  "compilerOptions": {
    "strict": true,
    "esModuleInterop": true,
    "declaration": true
  }
}

Testing Tips

  • Use Jest's mock capabilities for testing API calls
  • Test error conditions and edge cases
  • Use TypeScript in test files for better type checking

Example test structure:

import { Substack, SubstackError } from './client';

describe('Substack', () => {
  let client: Substack;

  beforeEach(() => {
    client = new Substack();
  });

  it('should handle successful requests', async () => {
    // Test implementation
  });

  it('should handle errors', async () => {
    // Test implementation
  });
});

Runtime Type Validation

The library uses io-ts for runtime type validation of API responses to ensure data safety beyond TypeScript's compile-time checks.

Why Runtime Validation?

While TypeScript provides excellent static typing, API responses are dynamic and can change without notice. Runtime validation with io-ts provides:

  • Type safety at runtime: Validates that API responses match expected shapes
  • Early error detection: Catches data inconsistencies before they reach domain models
  • Robust error handling: Provides detailed error messages for invalid data
  • Composable validation: Uses composable codecs for complex nested structures

Using io-ts Codecs

The library defines io-ts codecs for key internal types in src/internal/types/io-ts-codecs.ts:

// Raw Post codec
export const RawPostCodec = t.type({
  id: t.number,
  title: t.string,
  slug: t.string,
  post_date: t.string,
  canonical_url: t.string,
  type: t.union([t.literal('newsletter'), t.literal('podcast'), t.literal('thread')])
  // ... other fields
})

export type RawPost = t.TypeOf<typeof RawPostCodec>

Validation in Services

Services use validation utilities to validate API responses:

import { decodeOrThrow } from '../validation'
import { RawPostCodec } from '../types'

async getPostById(id: number): Promise<SubstackPost> {
  const rawResponse = await this.httpClient.get<unknown>(`/api/v1/posts/by-id/${id}`)

  // Validate the response with io-ts before returning
  return decodeOrThrow(RawPostCodec, rawResponse, 'Post response')
}

Validation Utilities

Two main utilities are provided in src/internal/validation.ts:

  • decodeOrThrow: Validates data and throws an error on failure (used in production code)
  • decodeEither: Returns an Either type for error handling (used in tests and error-safe contexts)

Adding New Codecs

When adding new API endpoints or modifying existing ones:

  1. Define io-ts codecs for the expected response shapes
  2. Use decodeOrThrow in service methods to validate responses
  3. Add tests for both successful and failing validation scenarios

Example test:

it('should validate valid post data', () => {
  const validPost = { id: 123, title: 'Test', /* ... */ }
  const result = decodeEither(RawPostCodec, validPost)
  expect(isRight(result)).toBe(true)
})

it('should reject invalid post data', () => {
  const invalidPost = { id: 'not-a-number', /* ... */ }
  expect(() => decodeOrThrow(RawPostCodec, invalidPost, 'test')).toThrow()
})

Debugging

For debugging during development:

  1. Use the debug npm package for logging

  2. Add source maps in tsconfig.json: json { "compilerOptions": { "sourceMap": true } }

  3. Use the VS Code debugger with the following launch configuration: ```json { "type": "node", "request": "launch", "name": "Debug Tests", "program": "${workspaceFolder}/node_modules/.bin/jest", "args": ["--runInBand"], "console": "integratedTerminal", "internalConsoleOptions": "neverOpen" }