FeedCreator 2.0

I appreciate the recent update to 2.0. unfortunately there are a couple things still missing to make this a killer product:-

  • Pass a name/value for a cookie. Many GDPR sites require a cookie
  • Specify a useragent value. Using Googlebot-News as UA quite often bypasses the cookiecheck above.
  • Handle sites that use JavaScript frameworks, such as React.
  • The ability to specify a proxy for extracting the pages

I haven’t installed this new version, so perhaps I missed a functionality in the doc?
If you need a site to test the above, try https://www.amweb.nl :slight_smile:

Thanks for the feedback, Rene. All good suggestions which we hope to implement soon. Most of the effort in this release went into the new interface to allow users to make use of the current request parameters.

It’s possible to enter proxy server details in the config file. This will then get applied to all requests. The proxy server support is only for servers with cURL or the HTTP PHP extension - but most servers with PHP should have one of these. It might be possible to set the proxy server to a service in the US to see if that bypasses the GDPR check.

We have plans to support JS sites, but that will most likely be a separate solution. The resources required on the server don’t make it feasible for a solution like Feed Creator which processes pages on-demand for a fast response.

Quick update to say that version 2.1 now allows passing cookies and a custom user-agent string. Feel free to test this here: https://createfeed.fivefilters.org

We’ll be releasing it for self-hosting users shortly.

Unfortunately the URL you supplied requires JS parsing for content, so that’s not yet supported.

Great! look forward to the self hosting version, this makes it so much more usable :slight_smile:

Any plans for further integration with FullTextRSS? Would be nice to specify ua=Googlebot-News just once in feedcreator. FTRSS could then use the provided ua as default, if nothing else is specified. Same would be nice for cookie params…

Thanks Rene, will see if we can do something about that in the future.

At the moment Full-Text RSS doesn’t have the ‘cookie’, ‘ua’, and ‘referer’ parameters, but it does allow you to pass all the same HTTP headers via the ‘siteconfig’ parameter. So if it’s something you wanted to attempt yourself, it would be a matter of taking those HTTP header parameters given to Feed Creator and passing them as a single ‘siteconfig’ parameter for Full-Text RSS.

For example, to send the following

User Agent: PHP/7.4
Cookie: euconsent=true

In Feed Creator

Un-encoded (for clarity):

ua=PHP/7.4&cookie=euconsent=true

URL encoded (correct way to send):

ua=PHP%2F7.4&cookie=euconsent%3Dtrue

In Full-Text RSS

Un-encoded (for clarity):

siteconfig=http_header(user-agent):PHP/7.4
http_header(cookie):euconsent=true

URL encoded (correct way to send):

siteconfig=http_header%28user-agent%29%3APHP%2F7.4%0Ahttp_header%28cookie%29%3Aeuconsent%3Dtrue

Thanks, very useful suggestion. I didn’t realize FTRSS siteconfig params could be used in the url!

1 Like