Documentation: what sets amateurs & professionals apart...

[ShelLuser]

Note: this is a somewhat opinionated writing, but I can back up all my statements with evidence (and I will). I also want to point out that my post isn't necessarily meant to talk down on Linux (hence not mentioning it in the topic), but after seeing all this I did feel like outing a critical comment. The reason I do so here... well, why not?

SO yeah... I got into an interesting discussion about operating systems and the topic diverted a little bit from Windows and shifted to Linux and FreeBSD. Where I eventually also pointed out that you can't really compare those because as we all know Linux is mostly a kernel with a userland build around it.

But then we got into a topic which has always fascinated me: documentation. In my opinion documentation is the key asset which makes professionals stand out from the rest. This opinion is also what fuels my drive and admiration for the so called modeling languages but that's an entirely different subject (wanted to mention this because it also got into the discussion I had).

So documentation...

As we know Linux is mostly a kernel which 'starts' at kernel.org. So I went to the website to look for documentation. In specific I'm looking for instructions which explain how to compile the kernel. If you follow their FAQ you'll eventually be pointed to 2 websites: Kernel newbies and the well established Linux Documentation Project (tldp.org).

When you look at kernel newbies (what a name...) then you'll quickly notice that most information evolves around kernel hacking / developing. I started looking at the 'Kernel documentation' section where I found what I was looking for: WorkstationKernel, I quote: "how to set up a workstation kernel". Yeah... See for yourself => an empty page (all it has is an advertisement) containing not much more but "[TBD]". How exactly does this help me build a kernel?

Then I found this link, for people "who want to write and understand the kernel". Unfortunately no link in that list points you to the instructions to actually configure and build the kernel. "The Linux kernel" (hosted on tldp.org) comes close and it's said to also target beginners. It does address some specific issues such as the kernel directory structure and general hierarchy, but no where does it tell you how to actually configure and build it. Mind boggling.

And then I started noticing the main concern which I'm trying to address here.

I scrolled down in the previous list of kernel documentation ("who want to write and understand the kernel") and found the book section which also mentions dates. The most recent book (aka online article) in there dates from 2004. What. the. heck.?

But first... Going back to kernel newbies I scrolled down and finally found what I was looking for: "* Compiling the Linux kernel or OS News Article". Finally I'm getting some results. Well, that is to say... the first link gives me this:

The Kernel-HOWTO has been removed because it don't fitted the LDP standard.

If you qualify for writing a replacement, please contact us, we need you

Note: this is on tldp.org, the official Linux documentation project

Onto the second link.. Expecting to see some kind of OS news article (Linux review) I finally discovered what I was looking for. Just too bad that the article dates from 2001, discusses kernel 2.4.17 (current is 4.13 / 4.14). The reason I mention this is because this article fails to mention that you always have a directory called /usr/src/linux which is used by specific device drivers. Unfortunately this article pre-dates that current standard.

Surely this had to be a fluke right? Just 2 silly examples which gave me a completely wrong impression of the current state of Linux documentation.

Unfortunately (and I really mean that!) not so much.

I started to dig around tldp.org some more and I'm honestly shocked to see the current state of the official Linux documentation. It's a ruin, outdated, obsolete, inaccurate and seemingly totally unmaintained pile of bogus.

Here is a link to "recent TLDP updates" which has 1 entry: the "Unix-and-Internet-Fundamentals-HOWTO" dated from the 5th of January, 2015.

Or how about a nice list of Linux guides, where some articles even go back to the last century. To add insult to injury: The "Linux administrators security guide" was removed from this list at the request of its author because "book is now more "dynamic"". Here is the link. It may be "more dynamic" but would you trust a security guide from 2001 which predates pretty much every new commonly accepted security standard there is today?

Now... sure: sometimes it really isn't much of a problem to have dated sources of information. I'm definitely not trying to claim otherwise. But how the heck can anyone in their right mind call this even remotely professional? Wasn't Linux a professional operating system / environment? This is its official documentation project, the de-facto source for Linux information.

So yeah....

Next time you're having a discussion with someone about the differences between Linux and FreeBSD and the topic happens to shift towards professionalism then don't bother yourself with trying to raise arguments. Just point out the abominable state of the official Linux Documentation Project and I think you've made a point which - at the time of writing - cannot be rebutted in any way, shape, or form.

After all... This is the collection of FreeBSD documentation and you'll notice that it does get its maintenance. The newbie documentation does not contain missing links, outdated documentation and non-existing information (though accidents can always happen).

And for the record: I'm not gloating here, I'm merely shaking my head in disbelief. This isn't a story about how "superior" or "inferior" either one is. This is a sad story about a certain group of people who claim to be IT professionals, yet when you take the effort to look a little further beyond the flashy websites you'll see the ugly truth showing up.

If you claim to be a professional, then you should also pay attention to the 'after math'. And like it or not but documentation is a big part of that.

 

[ronaldlees]

Very interesting. But, to be fair, a lot of "building the kernel" documentation is handled by the individual distos themselves (Debian, etc).

 

[phoenix]

Yeah, there's very little in the way of good, general, all-purpose, non-distro-specific documentation on Linux. You have to pick a distro to use, and then go through the documentation for that distro on how to do things "the distro way". Linux From Scratch would probably have more detailed information on building a Linux kernel without using a specific distro.

 

[chrbr]

We should never forget - somebody must take care about the documentation to have it, and in best case to have it up to date. And this should done by people how understand what they are documenting. This is a good oppertunity to thank wblock@ and all the others I do not know by name. Good documentation is artwork. Fortunately at FreeBSD there are people how keep the art of docs alive.

 

[forquare]

I'm not sure the tldp is the official documentation? I'd have thought it would have something about The Linux Foundation plastered over it if it was.
I think a lot of documentation is done by the distros. I know the Arch Wiki has some good documentation, a lot of which can be applied to other distros/OSs.

Not 100% sure, but I think the documentation you sought hides amongst the source: https://git.kernel.org/pub/scm/linu...git/tree/Documentation/admin-guide/README.rst
I suspect this is somewhat because it may change a little from distro to distro.

 

[sidetone]

Documentation at FreeBSD is difficult. Writing is not hard at all. The difficult part is formatting it according to spacing/tabbing standards, where it gives an error no matter how many times I try to fix it.

 

[chrbr]

The difficult part is formatting it according to spacing/tabbing standards, where it gives an error no matter how many times I try to fix it.

In my opinion textproc/igor is a very good tool for detecting and fixing formatting problems.

 

[bookwormep]

Documentation is a big reason I considered FreeBSD in the first place, so +1.
Of course you know how fast change occurs, so the irony is as soon as your
carefully written "How-to" is fully complete, it needs modification (?!?!)

Edit: And yes, big thank you to @wblock and many others who have written
and revised Handbook pages and articles.

 

[recluce]

True, a lot of the documentation on Linux is provided by the individual distros. But arguably, the quality of that documentation is also quite inconsistent. Yes, Arch is an example of great documentation - but only if you run the default flavour with all the new shinies (like systemd, *barf*). Finding documentation if you run a somewhat different setup (say, OpenRC instead of systemd) is a lot harder. Many other distros, like Ubuntu, are kind of "meh" when it comes to documentation.

FreeBSD is a shining example when it comes to documentation - and this has helped me a great deal, thanks to everybody working on this! Also, the community here on the forum should not go without praise either. Whenever I shoot myself in the foot, friendly and competent advice can be found here. The typical "RTFM" answers on Linux forums are an absolute rarity here. Which is kind of ironic, considering the respective state of documentation between FreeBSD and Linux.

 

[sidetone]

I have to admit that I've used Linux documentation (often Arch Linux) on individual programs to apply it to FreeBSD. This was for window managers, joystick support, desktop settings, xrandr, etc... Often I'm able to look at different documentations (Handbook, manpages, books, 3rd party, related websites etc), and put the information together. Then I have rewritten that information and posted it here, posted it on http://freebsdwiki.net/index.php/Main_Page a few times, or simply used that information myself. I haven't been up to it lately.

 

[wblock@]

I hadn't seen this thread until today. Just a few comments:

First, thanks for the compliments!

If you claim to be a professional, then you should also pay attention to the 'after math'. And like it or not but documentation is a big part of that.

I would like to get people to realize that documentation is not "after math". It should not be considered something you do after the program, or an optional extra. In the best cases, you write the documentation first, and then develop programs from that. Some of the best documentation I've ever seen was for the HP41C programmable calculator. They wrote the instruction manual first, then created the calculator to those specifications. I'd like to get the FreeBSD community to look at documentation as a component. If it's not documented, it's not done.

Writing is not hard at all. The difficult part is formatting it according to spacing/tabbing standards, where it gives an error no matter how many times I try to fix it.

I disagree with this. Writing something is not difficult. Writing something that is useful, anticipates the reader's needs, and fully covers the important parts without diverting into irrelevant minutia is hard. Markup for the FreeBSD project... well, it's not super-easy, but it's not as bad as people think. The FreeBSD documentation project hears "oh, the markup is so hard" quite often. To many of those people, we say "give us the plain text and we'll do the markup for you." Most of the time, we don't hear back after that. It turns out that writing is hard.

I was at MeetBSD in 2016, where they went around the room asking people what brought them to BSD. For most people, documentation was mentioned as the first or second thing. That's a big deal.

It's unfortunate we don't have more people working on it. That's due to a variety of issues, but I encourage people to keep working at it. The docs are a critical part of FreeBSD, and the better they are, the better FreeBSD is.

 

[sidetone]

I understand XML enough to figure out how to carry over markup. It's the error messages, not related to markup that trouble me.

In my opinion textproc/igor is a very good tool for detecting and fixing formatting problems.

That was the program that gave me trouble.

I'll just try again sometime later, perhaps not soon.

As for what's difficult or not about writing, that depends on the person.

 

[sidetone]

I tried again, this time to make the tiniest, but necessary edits that didn't require much differences in spacing. igor again drove me crazy.

 

[Deleted member 30996]

My own writing has always been the hardest to proofread.

I understand XML enough to figure out how to carry over markup. It's the error messages, not related to markup that trouble me.

I have The XML Bible 2nd Edition by Elliot Rusty Harold. I paid $50 for it new and now you can get free in .pdf form. You don't get the disk that comes with the book with the .pdf though.

It's been a long time since I wrote any markup language besides XHTML but have written/can write XML and AIML, or Artificial Intelligence Markup Language, and what Alice runs on.

 

[sidetone]

The publisher or author released it? I used to have an Informit book on XML: it was those learn it in 24 hours books. Then I lost track of it. Once you get a hang of it, especially the definitions, the gist is of XML is easy to remember, then it takes inspecting the doctype you're working on, reapplying it, and keeping track of closing tags.

This is what I get:
igor -R chapter.xml | less -RS

chapter.xml:254:wrap long line:      <application>&xorg;</application> system being installed.  Binary packages
chapter.xml:430:bad tag indent:   </listitem>
chapter.xml:444:bad tag indent:   </listitem>
chapter.xml:458:bad tag indent:   </listitem>
chapter.xml:950:wrap long line:      <para>To install the above Type1 font collections from binary packages,
chapter.xml:955:wrap long line:      <para>Alternatively, to build from the Ports Collection, run the following
chapter.xml:1654:capitalization:      <title>Setting up the &os; nVidia Driver</title>
chapter.xml:1708:capitalization:      <title>Configuring xorg.conf for Desktop Effects</title>
chapter.xml:1790:no comma after "e.g.": manager (e.g. <application>Metacity</application> if you are
chapter.xml:1794:no comma after "i.e.": decorations (i.e. close, minimize, maximize buttons, title
chapter.xml:1798:no comma after "e.g.": at startup automatically (e.g. by adding to

One minute, I had errors only relating to intentional spacing, capitalization and punctuation. Then I compiled it to view it in my browser, when I ran igor again, more errors popped up, having to do with tag indents, and line wrap.

Ok, I checked the lines of those errors, and they weren't in sections I edited. The spacing and indenting is very easy to mess up.

 

[Deleted member 30996]

The publisher or author released it? I used to have an Informit book on XML: it was those learn it in 24 hours books.

I don't know, when I saw this thread I got it out and looked it up on google. The .pdf caught my eye but you can buy used copies online for under $10, some with the book on disk that came with it. It's well worth it.

It is in a word comprehensive at 1200 pages and covers every aspect of using XML. XHTML as a reformulation of HTML 4.0 to valid XML, the use of CSS, XSL, etc.

textproc/igor sure looks handy. I always use a text editor and validate syntax at W3Schools.

 

[hitest]

Next time you're having a discussion with someone about the differences between Linux and FreeBSD and the topic happens to shift towards professionalism then don't bother yourself with trying to raise arguments. Just point out the abominable state of the official Linux Documentation Project and I think you've made a point which - at the time of writing - cannot be rebutted in any way, shape, or form.

I like and use Linux, FreeBSD, and OpenBSD. No argument from me about the state of Linux documentation. FreeBSD documentation is excellent as is the documentation provided by OpenBSD. I don't think Linux is losing any sleep over weak documentation; they're busy dominating the super computer, server, and cellphone markets. I agree that Linux documentation should be improved.

 

[sidetone]

textproc/igor sure looks handy. I always use a text editor and validate syntax at W3Schools.

It's not the XML or Doctype syntax that's giving me a problem. It's the spacing, including indenting, tabbing, and line wrapping. Even after fixing the tabs and spacing in ~/.vimrc to documentation's requirements. The messages are, your spacing is 1 millimeter off. Even when you align the indents, rid of extra spaces after lines, the message is, the indent is wrong. Try to align it, and spacing and indent errors multiply.

I really think others are having trouble understanding that it is not the syntax, or writing that's the problem at all.

bad tag indent:
wrap long line:

It's constant output above from igor, also in the above post, which didn't have to do with the pages I tried to edit. It wasn't there, then appeared, after I didn't update the source to the documentation, unless running make does that. (I can check that out.) Perhaps it can update the source, as how make update, is capable of updating source in /usr/src/. Maybe the command svnlite diff did this; that makes more sense, because my backup file also has the same error, that temporarily wasn't in the file I edited.

XHTML, XSL, Atom, PHP, most XML, etc don't care about this. Spacing in a Makefile matters, but fixing that is simple, and not complicated like this.

 

[fullauto2012]

I am what you might consider a 'newbie'. Well, not true. I AM a newbie. I have always been fascinated with networking and after my stint in the US Air Force, I resolved I would learn it myself seeing as I was/am broke, so there is little chance of a formal education.

I was told some years ago that I should look into linux if I were serious about learning IP networking. So I did, and was nothing but frustrated with the lack of documentation, which lead me to forums and straight out asking for help constantly, only to find the help offered was usually as anemic as the documentation.

I eventually remembered some mutterings I heard from friends back when I worked for PSInet back in the 90's. Something BSD. I checked into it and found a THRIVING community of fans and experts along with GREAT documentation.

So, here I sit. In my 'office' surrounded by old PCs converted into FreeBSD rigs and hosting all those services I told myself I would learn.

TLDR -> FreeBSD is fantastic and the documentation (as well as the community), and that is what ultimately keeps me here. I can only hope to become as fluent in this stuff as some of you. But, I will keep "RTFM'ing".

 

[sidetone]

So, here I sit. In my 'office' surrounded by old PCs converted into FreeBSD rigs and hosting all those services I told myself I would learn.

TLDR -> FreeBSD is fantastic and the documentation (as well as the community), and that is what ultimately keeps me here. I can only hope to become as fluent in this stuff as some of you. But, I will keep "RTFM'ing".

You'll get there. It's, you've seen the configuration files before, and you've edited them. Then later on it clicks, that you know which configuration file you need to edit, without having to look it up. You'll always have to look up the documentation to edit it, but you'll do it less. It takes time, and it happens gradually without you immediately realizing it. Sometimes books, give information that's easier to read, which eventually that information ends up here. Some information can be figured out through (not too much of) trial and error.

 

[Deleted member 30996]

So, here I sit. In my 'office' surrounded by old PCs converted into FreeBSD rigs and hosting all those services I told myself I would learn.

TLDR -> FreeBSD is fantastic and the documentation (as well as the community), and that is what ultimately keeps me here. I can only hope to become as fluent in this stuff as some of you. But, I will keep "RTFM'ing".

Bravo. I'm completely self-taught.

 

[Snurg]

I hadn't seen this thread until today. Just a few comments:
I would like to get people to realize that documentation is not "after math". It should not be considered something you do after the program, or an optional extra. In the best cases, you write the documentation first, and then develop programs from that.

What wblock@ describes as "the best cases" is actually that what distinguishes professionalism from planless hackism.
A professional architect does the design first, to be implemented according the plan which results from the design documentation.

Markup for the FreeBSD project... well, it's not super-easy, but it's not as bad as people think. [...] The FreeBSD documentation project hears "oh, the markup is so hard" quite often. To many of those people, we say "give us the plain text and we'll do the markup for you." Most of the time, we don't hear back after that. It turns out that writing is hard.

Imho the sometimes quite frustrating toolchain problems sidetone described illustratively play a big part in it.
And then there comes the problem that not everybody has the desire to deal with XML.
For my part I do not like it and I do not need it for my work, so I do not want to waste any time into learning XML only to be able to contribute to the FreeBSD documentation.
Neither am I interested in having to deal with that toolchain being used for FreeBSD documentation.
So I haven't contributed to the FreeBSD documentation yet.

It's unfortunate we don't have more people working on it. That's due to a variety of issues, ...

See above.
I think it is essential that the hurdles get lowered.

Thus a few thoughts.
One of the things that make the FreeBSD documentation and the wonkity texts so easy-to-read is their consistent formatting, restricted to make structured reading easier, and luckily not caring about design fads that come and go.
What about adapting one of those wysiwyg javascript applets like TinyMCE to offer just these formatting styles that the FreeBSD doc uses?
This way it could be reasonably easy to make a web frontend which enables people to create/edit documents.
No more knowledge of XML or a difficult-to-handle toolchain required!
And that approach possible would need relatively little manual postprocessing by the FreeBSD documentation maintainers.

I have been thinking of possibly later making a side-project of my main project (web cms) that works about that way:
1. it pulls the FreeBSD documentation
2. It offers people to add to and edit that documentation wysiwyg. (think wikipedia, but even easier)
3. Then the edits and adds could be extracted, filtered from spam, and then sent to the documentation maintainers.
4. The documentation maintainers then can audit/verifiy and integrate into the documentation tree (with comparatively little effort compared to raw text submissions).

I haven't yet found a site that does this. Maybe such already exists?
If such does not exist yet: What do you guys think?
Should there be such a site?

 

[sidetone]

Really, if the text tools would automatically turn the appropriate amount of spaces, that are visually lined up, into the correct tab, and would help word wrap the text according to the Documentations standards, that would fix most of the problem.

It would also be useful to either add proper nouns to the definition list of the XML schema (I was previously calling it doctype), or for igor to show a different type of warning for capitalization of possible brand names or titles.

This may be outside the scope of XML or may not be possible without breaking pages down further, but there needs an anchor for lines to be numbered starting at the top of each page, and for igor, diff and other tools to reflect that. I want to avoid seeing errors outside of pages that I've edited, except for improper XML, and also I don't want to have to worry about recent edits done outside of the page I'm working on when I update the documentation source. Another reason for line numbering at the anchor at the top of each page is, to keep line numbers of applied edits from other pages from interfering with each other in otherwise correct diff file numbering.

I understand why the bar should not be too low. It is reasonable to know how to use XML. XML is not very difficult, especially on the documentaion schema, each element must be closed by itself or another closing tag, and there are not many attributes for this schema. It just takes analyzing the elements for the rest of the schema used on the documentaion pages and reapplying it.

In short, the specific tools would be better as:

• Checking the entire documentation file for XML compliance, possibly also for spelling, and schema with proper names, without anything else.
• Then tools for fixing and checking spacing, tabbing and word wrap alignment for the specified page according to its page by that anchor within that xml file.
• Then the diff and updates need to reflect line number from an anchor from the top of each page until the end of that page, not of the entire xml file.

 

[Snurg]

It is reasonable to know how to use XML. XML is not very difficult, especially on the documentaion schema,... In short, the specific tools...

Yes, and a lot of that can be automated by frontends.

This makes me think about my work at a big newspaper. The reporters wrote their texts, mailed them, and these got typeset by prepress publishing professionals with their (quite complex) toolchain.
It would be just impossible to have the reporters also be competent in layout and all that stuff.

So I think of a simple interface. Not necessarily 100% wysiwyg.
Wouldn't it be sufficient to get the structure, and an approximated presentation?.
Maybe there could be a way to generate according XML files like you describe this way?

would be better as:

• Then the diff and updates need to reflect line number from an anchor from the top of each page until the end of that page, not of the entire xml file.

I haven't thought about that yet... maybe it's possible to produce sort of "diff XML" files which only contain the page(s), chapters or whatever was changed by a contributor, so it's easier to review and import?
And regarding the anchors... I am not sure if the line number approach is the only possibility. Doesn't XML make it possible to structure documentation, so that one could use the structures as anchors, too?
(Sorry for possibly useless brainstorm output...)

 

[sidetone]

This makes me think about my work at a big newspaper. The reporters wrote their texts, mailed them, and these got typeset by prepress publishing professionals with their (quite complex) toolchain.
It would be just impossible to have the reporters also be competent in layout and all that stuff.

It's not difficult, but in their scenario, that takes away from valuable time and concentration from their expertise that must be produced every day. This is about documentation, so it is in line with knowing XML. You know enough about XML to be worried about it.

So I think of a simple interface. Not necessarily 100% wysiwyg.
Wouldn't it be sufficient to get the structure, and an approximated presentation?.
Maybe there could be a way to generate according XML files like you describe this way?

Anything beyond a wysiwyg (what you see is what you get) XML editor would make it too easy. There are a few XML editors in ports, but they are not very good, have heavy dependencies, or are not intuitive at all.

I haven't thought about that yet... maybe it's possible to produce sort of "diff XML" files which only contain the page(s), chapters or whatever was changed by a contributor, so it's easier to review and import?
And regarding the anchors... I am not sure if the line number approach is the only possibility. Doesn't XML make it possible to structure documentation, so that one could use the structures as anchors, too?

The documentation already has #id's that point to its respective #anchors. Links to it send you to that specific section of that page. HTML and XML already have this ability. These anchors need to be used for a more specific purpose. When you hover over the "Top" link at the bottom of the page, "#navigation" is an id for an anchor. Another anchor, is when you look in the address bar at the end that begins at "#post-", or when you hover over the arrow by the quote.