Workflow 002: Authenticated Scraping Pipeline

Fresh

Overview

Build a scraping pipeline that authenticates once and reuses credentials across multiple sessions for accessing protected content.

Pipeline Architecture

Choose Your Auth Strategy

Strategy	Best For	Persistence	Setup
Credentials API	Automated login, multi-user	Permanent until deleted	Store once, inject always
Auth Context	Quick session transfer	Per-capture	Capture each time
Browser Profiles	Full browser state	30 days	Persist on release

Implementation: Credentials API Approach

// One-time: Store credentials
await client.credentials.create({
  origin: "https://app.example.com",
  namespace: "scraper:main",
  value: {
    username: "scraper@company.com",
    password: "secure_password",
    totpSecret: "JBSWY3DPEHPK3PXP"  // Optional
  }
});

// Every run: Auto-authenticated session
const session = await client.sessions.create({
  namespace: "scraper:main",
  credentials: { autoSubmit: true },
  useProxy: true,
});

const browser = await chromium.connectOverCDP(
  `wss://connect.steel.dev?apiKey=${process.env.STEEL_API_KEY}&sessionId=${session.id}`
);

const page = browser.contexts()[0].pages()[0];
await page.goto("https://app.example.com/login");
await page.waitForTimeout(3000);  // Wait for auto-login
await page.waitForSelector(".dashboard");  // Confirm auth

// Now scrape protected pages
const data = await scrapePages(page, urls);

await browser.close();
await client.sessions.release(session.id);

Implementation: Profile-Based Approach

// First run: Login and persist
const session1 = await client.sessions.create({ persistProfile: true });
// ... connect, login manually ...
await client.sessions.release(session1.id);

// Subsequent runs: Already authenticated
const session2 = await client.sessions.create({
  profileId: session1.profileId,
  useProxy: true,
});
// ... navigate directly to protected content ...

See Also

• SOP 005: Reuse Auth Context
• SOP 007: Credentials API
• SOP 008: Browser Profiles