I want to rip the contents of a pay website, but I have to log in to their web site on a web page to get access

Does anyone have any good tools for Windows for that?

I’m guessing that any such tools must have a built in browser, or be a browser plugin for it to work.

@zabadoh@lemmy.ml
creator
link
fedilink
English
41Y

I have an account, so that’s not a problem. The problem is how to automate going into every little content page and downloading the content, including the hi-res files.

tekchic
link
fedilink
31Y

I’m on a Mac and use SiteSucker so I know that’s not super helpful but for windows you could try wGet or WebCopy? https://www.cyotek.com/cyotek-webcopy / https://gnuwin32.sourceforge.net/packages/wget.htm

@zabadoh@lemmy.ml
creator
link
fedilink
2
edit-2
1Y

Webcopy looks promising if I can get the crawler part of it to work with this site’s authentication…

edit: I couldn’t get Webcopy’s spider to authenticate correctly.

Webcopy uses the deprecated version of Internet Explorer in Windows 10 as a module, and I can log into the website using the Capture Forms browser dialog, but the cookies or whatever else don’t translate over to the spider.

Piracy: ꜱᴀɪʟ ᴛʜᴇ ʜɪɢʜ ꜱᴇᴀꜱ
!piracy@lemmy.dbzer0.com
Create a post
⚓ Dedicated to the discussion of digital piracy, including ethical problems and legal advancements.

Rules • Full Version

1. Posts must be related to the discussion of digital piracy

2. Don’t request invites, trade, sell, or self-promote

3. Don’t request or link to specific pirated titles, including DMs

4. Don’t submit low-quality posts, be entitled, or harass others



Loot, Pillage, & Plunder

📜 c/Piracy Wiki (Community Edition):


💰 Please help cover server costs.

Ko-Fi Liberapay
Ko-fi Liberapay

  • 1 user online
  • 110 users / day
  • 274 users / week
  • 1.01K users / month
  • 3.51K users / 6 months
  • 1 subscriber
  • 3.39K Posts
  • 82.1K Comments
  • Modlog