Important: This documentation covers Yarn 1 (Classic).
For Yarn 2+ docs and migration guide, see yarnpkg.com.

Package detail

@secret-agent/remote-client

ulixee6MIT1.2.0-alpha.5TypeScript support: included

SecretAgent client that runs remotely

readme

SecretAgent

SecretAgent is a web browser that's built for scraping.

  • <input checked="" disabled="" type="checkbox"> Built for scraping - it's the first modern headless browsers designed specifically for scraping instead of just automated testing.
  • <input checked="" disabled="" type="checkbox"> Designed for web developers - We've recreated a fully compliant DOM directly in NodeJS allowing you bypass the headaches of previous scraper tools.
  • <input checked="" disabled="" type="checkbox"> Powered by Chromium - The powerful Chromium engine sits under the hood, allowing for lightning fast rendering.
  • <input checked="" disabled="" type="checkbox"> Emulates any modern browser - BrowserEmulators make it easy to disguise your script as practically any browser.
  • <input checked="" disabled="" type="checkbox"> Avoids detection along the entire stack - Don't be blocked because of TLS fingerprints in your networking stack.

Check out our website for more details.

Installation

`shell script npm i --save secret-agent


or

```shell script
yarn add secret-agent

Usage

SecretAgent provides access to the W3C DOM specification without the need for Puppeteer's complicated evaluate callbacks and multi-context switching:

const SecretAgent = require('secret-agent');

(async () => {
  const agent = await new SecretAgent();
  await agent.goto('https://example.org');
  const title = await agent.document.title;
  const intro = await agent.document.querySelector('p').textContent;
  await agent.close();
})();

Browse the full API docs.

Contributing

We'd love your help in making SecretAgent a better tool. Please don't hesitate to send a pull request.

License

MIT

changelog

Change Log

All notable changes to this project will be documented in this file. See Conventional Commits for commit guidelines.

1.2.0-alpha.5 (2020-12-29)

Bug Fixes

  • emulate-browsers: fix cross frame tostring (27a69d9)
  • mitm: bug with duplicate header on response (ba3c015), closes #111
  • puppet: workers not tracking devtools calls (b339758)

1.2.0-alpha.4 (2020-12-22)

Bug Fixes

  • replay: replay crashes on win10 during load (b6e517b)
  • replay: shutting down before example load (937fd9b)

1.2.0-alpha.3 (2020-12-16)

Bug Fixes

  • mitm: bubble proxy errors properly to client (b6a72f5), closes #98
  • replay: multiple sessions showing incorrectly (20ba30c)

Features

  • client: update awaited dom to 1.1.8 (a1b9b68)

1.2.0-alpha.2 (2020-12-01)

Bug Fixes

  • core: don’t wait to scroll past bottom (9f8877b)
  • core: fix errors on goto bubbling up (30d4208)
  • docs: secret agent.md url broken (167fe5a)
  • emulate-humans: fix some tests (b1e05d7)
  • eslint: add return types to client code (c2e31cc)
  • mitm: remove auth as separate proxy param (ec14b30)
  • mitm-socket: fix cpu spiking sockets (b71e141)
  • replay: fix stalling loads (7ca1dbd)
  • don’t use vertical screen orientations (7121e94)

Features

  • proxy: configure proxy via client + socks5 (880c938)

1.2.0-alpha.1 (2020-11-20)

Bug Fixes

  • emulators should not use accept-language values from json if using custom locale (fc00660)
  • emulators were failing some double-agent tests (5ae4f55)
  • unwind some dependencies (240bea6)

Features

  • human-emulators: ghost emulator (70bcf27)

1.2.0-alpha.0 (2020-11-11)

Bug Fixes

  • emulator: bug fix for Error.toString (d23704a)

Features

  • awaited-dom: documentation for props (029a1f5)
  • browser-emulators: refactor emulator filenames (b5da042)
  • core: store data files in a single location (c3299b6)

1.1.0-alpha.1 (2020-11-05)

Bug Fixes

  • mitm: should add host to http1 headers (b655ea9)

Features

  • client: get/set/delete cookies + domstorage (2e2de6b)
  • client: split out ISecretAgentClass (8765900)

1.1.0-alpha.0 (2020-11-03)

Bug Fixes

  • puppet: incorrect reuse of executionContextId (e5d8f8d)

chore

  • client: merge Browser/User into SecretAgent (364ed8a)

BREAKING CHANGES

  • client: this change modifies the core interface for interacting with SecretAgent, as createBrowser is removed.

1.0.0-alpha.21 (2020-11-02)

Bug Fixes

  • core: improved waitForElement (4d139b3)
  • mitm: cache content-type (68d7384)
  • replay: cross domain iframes (db65711)
  • replay: loading resources (747f4ff)
  • bugs in replay (2bf8808)

Features

  • core: improve jspath waitForElement perf (435576a)
  • locale: add locale emulation + tests (57cc7ff)
  • replay: add support for iframe and shadows (0978fd5)
  • replay: record heirarchy of elements (89310c0)
  • replay: set screen viewport (f818ff5)
  • session: support CSSOM for recorder/playback (0cbe7c8)
  • session: track frame dom node ids (a41d678)
  • session-state: record and playback shadows (508fd39)

1.0.0-alpha.20 (2020-10-23)

Bug Fixes

  • mitm: tweak stored msg for connect errors (6c819d5)
  • order of session closing (046243b)
  • puppet: stabilize chained nav (7a99f69)
  • mitmRequestAgent tests (d93b4fa)

Features

  • client: add scrollTo shortcut (a1613f1)
  • mitm: dns over tls lookups (8797847)
  • mitm: store ca/keys in network.db (fd69f97)
  • puppet: use mouse wheel events (1efea8a)
  • session-state: record devtools logs (784da7f)
  • session-state: record mitm resource states (08976df)

1.0.0-alpha.19 (2020-10-13)

Bug Fixes

  • replay: install script broke (b79c572)

1.0.0-alpha.18 (2020-10-13)

Bug Fixes

  • replay: bug with monorepo replay versions (05aa786)

1.0.0-alpha.17 (2020-10-13)

Bug Fixes

1.0.0-alpha.16 (2020-10-13)

Bug Fixes

  • extend tests for emulate test (71764b7)
  • core: dont close client on promise rejections (37f1169)
  • core: wait for location change on new tab (0c70d6e)
  • mitm: catch exceptions on closed h2 session (6b5c7d4)
  • mitm: don’t send duplicated headers to h2 (ece1b1f)
  • mitm: duplicate if-none-match header (1cbe1f1)
  • replay: fix command overlays (926dcba)

Features

  • client: xpath support, array index access (c59ccbc)
  • core: isElementVisible - can user see elem (213c351)

1.0.0-alpha.15 (2020-10-06)

Bug Fixes

  • mitm: filter response headers (828dc94)

1.0.0-alpha.14 (2020-10-06)

Bug Fixes

  • client: don’t shutdown on rejected promises (86a331b)
  • mitm: change headers after alpn is set (a21d4ca)
  • mitm: push headers + header arrays (f411b93)
  • replay: fix http2 push headers (755667f)
  • replay: resetting navigation needs to clear (daf9431)
  • replay: use shadow dom for replay elements (b19b382)

Note: Version bump only for package @secret-agent/monorepo

1.0.0-alpha.12 (2020-09-29)

Bug Fixes

  • puppet: chrome 80 test flakiness (9f16cd1)
  • lint and puppet test chrome 80 (0ce09ac)
  • refactor to pause debugger on attach (63a9bd1)
  • mitm: simplify errors, handle not caught (27820ac)
  • replay: playback on page 2 (005bed8)

Features

  • core: back/forward api (805af3d), closes #32
  • docs: Update documentation (2295725), closes #56
  • mitm: switch mitm to use authorization (fade6e8)
  • puppet: add puppet interfaces abstraction (69bae38)
  • puppet: import playwright tests (e2b9bf5)
  • replay: remove ui tabs; nav to session tabs (df8e21c)
  • replay: spawned child tabs (8ae0d75)
  • replay: split session state by tab (9367f2d)
  • import and shrink puppeteer (b1816b8)
  • wait for tab (0961e97)

1.0.0-alpha.11 (2020-08-25)

Bug Fixes

  • copy engine to deployed emulators (98ea24c)

1.0.0-alpha.10 (2020-08-25)

Bug Fixes

1.0.0-alpha.9 (2020-08-25)

Bug Fixes

  • humanoid keys => ids (a30652e)
  • emulators: wait for doc element (c67fbf8)
  • mitm-socket: chrome 83 tls signature (a699212), closes #48 #23
  • replay: fix realtime not loading correctly (29ff447)
  • replay: fix rendering doctype + svg (ac36c79)
  • vue structure for replay (0e38bfa)

Features

  • ci: windows tests (fd5e9db)
  • core: enhance logs (a5b6d58)
  • emulators: add windows runtime polyfills (51ebb11)
  • emulators: enable multi-engine support (1e008c9)
  • mitm: support push streams (1b2af06)
  • replay: stream data and simplify tick tracker (91c350c)
  • restructure frontend to match vue project (f3348a0)

1.0.0-alpha.8 (2020-08-05)

Bug Fixes

  • emulator-plugins: include origin for cors (b1449c1)
  • replay: handle frames and page source changes (fc703d5)
  • circleci configs (7d8e213)
  • pool socket connections per origin (0075f18)
  • ci: circle ci fixes (24596b5)
  • core: core should autoclose if not started (8d46a77), closes #41
  • mitm: windows sockets (dc3cf7d)
  • replay: fix launch path to replay (8d7059b)
  • socket: http2 requests not reusing sockets (3cbf853)
  • use os tmp directory (e1f5a2b)
  • windows tests (c2943e8)

Features

  • mitm: record blocked and cached http (bd47738)
  • replay: fix picker (50d7885)
  • replay: replay individual ticks on interval (e1c29f4)
  • replay: split app/replay in electron backend (3b66eec)
  • session-state: capture requests before send (9607793)
  • circle-ci fixes (aac9a30)

1.0.0-alpha.7 (2020-07-27)

Bug Fixes

Features

1.0.0-alpha.6 (2020-07-22)

Bug Fixes

  • lerna packages (92798e8)
  • packaging issues with mitm and replay (520a912)

1.0.0-alpha.5 (2020-07-21)

Bug Fixes

  • replay: fix replay api usage (c54fe64)

1.0.0-alpha.4 (2020-07-20)

Bug Fixes

  • replay: cover last tick on playbar (baf12e7)
  • replay: record close date when errors occcur (2ce94dd), closes #31
  • change shared package names (d6181a7)
  • publish README.md with next build (4c8b942)

Features

  • replay: add mouse/focus/scroll events (efec55c)
  • replay: add session logs, detect errors (f1865c0), closes #31
  • replay: autoplay replay (bd13ef5)
  • replay: show commands in playbar (58b9f7a)
  • replay: start api from process (403716b)
  • flatten shared workspaces (d53da16)

1.0.0-alpha.3 (2020-07-07)

Bug Fixes

  • mitm: small tweak for mitm tests hanging (c969870)
  • session-state: Improve page recorder perf (14f78b9), closes #8
  • .gitignore was ignoring files that were needed for website (4b9a2e4)
  • mitm timing out large bodies (d38e78f), closes #8

Features

  • dist: improve packaging for double agent (df195b6)
  • emulators: Emulator plugins - set agent (e53cedb), closes #8
  • emulators: improve page logging (cb73806)

1.0.0-alpha.2 (2020-06-27)

Bug Fixes

  • Emulator plugin referencing relative paths (f26feab)
  • missing dependencies (67504f0)

1.0.0-alpha.1 (2020-06-27)

Bug Fixes

  • Emulator plugin referencing relative paths (f26feab)

1.0.0-alpha.0 (2020-06-27)

Note: Version bump only for package secret-agent