@fivefilters WSJ is once again failing extraction, both in my self-hosted deployment and on ftr.fivefilters.org.
I have tried copying the session cookie (djcs_session) from a logged in browser, and while this seems to work with other browsers, it does not work in Postman nor when added to a custom pattern in FTR.
the trick to just redirect the page to wsj.com/amp/articles/foo-bar-bfa123a5 doesn’t work any longer.
a browser plugin, that I use still is able to catch full article. In addition to the redirect it uses a rule, which I could not rebuild for ftr
allow_cookies: 1,
block_regex: /(cdn\.cxense\.com\/|cdn\.ampproject\.org\/v\d\/amp-(access|subscriptions)-.+\.js)/,
useragent: "googlebot"
@fivefilters, any ideas?
Hmm, will have to look into this to see if there’s anything we can do.
@HolgerAusB I had a look and I’m seeing the same as you, no /amp/ support any more (it just redirects). Previously the /amp/ version would contain the full content. Do you know what the browser plugin is doing? I can’t really tell from the snippet you pasted. Adding a user agent string like that makes no difference in my testing.
it seem to block some scripts from the block_regex part. But I do not really know what the trick is.
Any news? I was doing some digging on the Magnolia paywall remover and found this:
I can’t exactly figure out what the javascript is doing to the DOM, but it appears there’s multiple ways to handle the paywall removal depending on the article URL, whether it contains “livecoverage”, or “articles”, etc… It does seem to use amp in some way, but again I’m not proficient in JS enough to tell what it does exactly
