How do you view a webpage without a browser in a console?

[hardworkingnewbie]

Aaron Swartz wrote THE ASCIINATOR for it: http://www.aaronsw.com/2002/html2text/

And there are many scripts around like this one.

 

[SirDice]

If I recall correctly you're fond of Perl, www/p5-libwww is useful. Besides adding some useful Perl modules it also comes with a couple of command line utilities GET(1) and HEAD(1) for example.

 

[a6h]

Zirias No. He asks "How do you view a webpage..." Looking at or downloading the markup using curl or telnet is not viewing a web page. I take that to mean NOT just wanting to look at the source markup.

I think I should said "read" or "fetch" instead of "view".

If I recall correctly you're fond of Perl, www/p5-libwww is useful. Besides adding some useful Perl modules it also comes with a couple of command line utilities GET(1) and HEAD(1) for example.

If perl would be an option then LWP::Simple module from the libwww-perl-6.58 works great.

 

[zirias@]

No.

No.

Please look up the definition of a "representation". In a nutshell, a representation is a format in which the data is presented or transported. There are human-readable ones and non human-readable ones. HTML clearly belongs to the first category (although you can obfuscate it like crazy...)

 

[drhowarddrfine]

Zirias Please look up the definition of "view" which is what he asked for. However, now he says he meant "fetch" and not "view".

One views a web page through a browser or software that interprets the supplied markup. Few have any reason to look at that markup by downloading it.

 

[zirias@]

"View" does not imply a specific representation.

What was meant here was easy to deduce from the given options.

 

[SirDice]

I've made a few 'web scrapers' for work. Needed to download some specific software, and it wasn't available in a 'regular' repository. So I had to scan the web pages for a specific link to a downloadable file. As long as nothing major changes on that particular page the downloader does what it's supposed to do. Used a fairly basic shell script for that to wget(1) the page, parse it somewhat with grep and then fire off another wget(1) to download the latest version of that software.

Now I've used wget(1) in that case because that's what was available to me. On FreeBSD I would probably just use fetch(1) for this.

 

[a6h]

drhowarddrfine and Zirias:
I think I'm the main cause of this mess here. Sorry about that.

 

[drhowarddrfine]

YOU are the cause of ALL our problems vigole !

 

[a6h]

YOU are the cause of ALL our problems vigole !

Indeed.

 

[Hakaba]

To get the content with JavaScript inside a page, you could use node.
To inspect the content, you can use a test library like mocha. Export what you find into a readable format, you can use Babel and Istanbul (nyc, you see the logic ?)

After that you have more libs, dependencies and unknown code than the crappiest browser and you still have no clear view of the webpage...

 

[MeowMan]

When i as a European look at U.S. pages i first need to agree on the applicable law, before i can even see the first page. This is interactive ...
Not all internet pages are as simple as freshports .

Same for me. And I do not know why... I mean, I can understand situation when the viewer discretion is advised and there is a question "are you 18?" However, what about situations when I just want to watch another season of Chernobyl on HBO and no, I am not an old Soviet Spy

God damn it, I think sometimes we have a lot of things to agree with.

P.S. I am from AU

 

[drhowarddrfine]

To get the content with JavaScript inside a page, you could use node.

You can just download it or inspect it and download it with the browser's inspector.

 

[Hakaba]

You can just download it or inspect it and download it with the browser's inspector.

I mean to see the result of the JavaScript execution, not the file.
And to have an inspector, you need a browser.

 

[drhowarddrfine]

Hakaba Same thing. The result of javascript execution is typically displayed on a web browser. Node is not going to do that for you in place of a browser.

 

[trev]

I've made a few 'web scrapers' for work. Needed to download some specific software, and it wasn't available in a 'regular' repository. So I had to scan the web pages for a specific link to a downloadable file. As long as nothing major changes on that particular page the downloader does what it's supposed to do. Used a fairly basic shell script for that to wget(1) the page, parse it somewhat with grep and then fire off another wget(1) to download the latest version of that software.

Now I've used wget(1) in that case because that's what was available to me. On FreeBSD I would probably just use fetch(1) for this.

I too have needed to make "web scrapers" for work and used a combination of wget(1), fetch(1), lynx and w3m. From memory (it was few years ago since I've retired) wget was preferred over fetch when I needed to "save state" so as to be able to retrieve images from some pages.

 

[astyle]

When i as a European look at U.S. pages i first need to agree on the applicable law, before i can even see the first page. This is interactive ...
Not all internet pages are as simple as freshports .

That never happened to me. I was able to view espn.com (a Las Vegas-based site, BTW, even with offices in Connecticut (East Coast US)) just fine. Well, that info is from 2005, which is when I was in EU last time. REALLY need to go back at some point, but there's a LOT of ducks to get in a row for that to happen.