MANTA-3236 Documentation on Addition of New Storage Node

[joyent-automation]

MANTA-3236 Documentation on Addition of New Storage Node

This PR was migrated-from-gerrit, https://cr.joyent.us/#/c/1854/.
The raw archive of this CR is here.
See MANTA-4594 for info on Joyent Eng's migration from Gerrit.

CR discussion

@chudley commented at 2017-04-27T08:47:43

Patch Set 1:

(23 comments)

Thanks for writing this up, Jay! It's looking great.

I've added a lot of notes about specific things here but most of them are pretty picky small stuff. As well as that, here's a few general notes about the document:

• Is there a MANTA ticket for this change? We should have a ticket here, and then alter the commit message to be "$ticket_id $ticket_summary"
• We tend to use "*"s as bullet points, but I don't know how much of a requirement that is. Regardless, it would be good to change the "-"s we have here to be more consistent across the document
• There's no spacing after the headings in most cases which affects readability when working with the raw markdown. Some additional spacing after each heading (and also double spacing before, if appropriate) would be great to add
• "Manta" seems to be all lower case in most places, but the rest of the document has it capitalised
• I've noted this in the comments, but we use the four-space indentation method of writing code blocks. I'm not certain, but I think it helps with rendering code blocks under bulleted lists

I haven't pulled this down yet to see the rendered version, but it would be really useful if you could also link somewhere that we can see this rendered. I don't think there's a great answer for this, but I tend to push to my fork and link to the document (see https://cr.joyent.us/#/c/865/, if I can figure out how to get the link to display properly...). In this case I'll probably end up trying to either pull it down myself or copy/paste it into a gist or something.

I'll keep an eye out for these changes and take another run-through when they're done. If you need help with the git pieces on squashing your commits then we have info in the "New User Guide", or you can ping me.

Patch Set 1 code comments

docs/operator-guide/index.md#1734 @chudley

I know this is a really common task, but maybe worth linking to the documentation here for completeness. I understand docs.joyent.com has undergone some major changes over the past few years, so if we're not sure about the lifetime of the links to those docs then maybe just a link to CNAPI or something:

https://github.com/joyent/sdc-cnapi/blob/master/docs/index.md#setting-up-a-new-server

docs/operator-guide/index.md#1734 @qdzlug

Done

docs/operator-guide/index.md#1739 @chudley

I think it's all caps for "VXLAN".

docs/operator-guide/index.md#1739 @qdzlug

Done

docs/operator-guide/index.md#1744 @chudley

Totally possible to add these tags manually, but these tags can be added by manta-net.sh. I believe we also provides some flexibility to their naming (although I'm not sure in practice how much this ever changes between installations).

docs/operator-guide/index.md#1744 @qdzlug

Added a note about manta-net.sh - I wasn't sure if I should open the can of worms that is having different tag names here or not, so I just went with an "assuming you've used the standard nictags" to note that here.

docs/operator-guide/index.md#1746 @chudley

Just as a heads up, I'd love to get these formalised, which I've proposed in RFD 76.

No change required here.

docs/operator-guide/index.md#1746 @qdzlug

Done

docs/operator-guide/index.md#1761 @chudley

This applies to a few places in the document, but we tend to use the "indent by four spaces" method of writing code blocks.

A good example of this is the joyent/sdc-manta README, where this method looks to play nicely with code blocks inside bullet points:

https://github.com/joyent/sdc-manta/blob/master/README.md

docs/operator-guide/index.md#1761 @qdzlug

Done

docs/operator-guide/index.md#1773 @chudley

I think these headings might read better if they were bullet points, at least so long as the code blocks read well enough.

For an example of how a complicated, multi-step process with code blocks is documented already, check out the "Deploying Manta" section:

https://github.com/joyent/manta/blob/master/docs/operator-guide/index.md#deploying-manta

docs/operator-guide/index.md#1773 @qdzlug

Done

docs/operator-guide/index.md#1782 @chudley

I don't know how useful this is here, as we're asking the user to run it but not saving the results for later use. I think when it comes to build out the config below, we could maybe just refer to these details as something along the lines of "MAC addresses can be obtained from CNAPI".

docs/operator-guide/index.md#1782 @qdzlug

I put a note above to save the data on the networks / mac addresses; I think it may be a bit easier to have the user grab it, note it, then refer to it? Maybe?

docs/operator-guide/index.md#1788 @chudley

Referring to a comment above, I don't think these NIC tags are necessarily going to have these names in every installation.

docs/operator-guide/index.md#1788 @qdzlug

Put a note in that this assumes you are using the standard nictag names.

docs/operator-guide/index.md#1789 @chudley

Looks like a typo here and on the line above.

docs/operator-guide/index.md#1789 @qdzlug

Done

docs/operator-guide/index.md#1804 @chudley

Same note about bullet points (number list in this case) as above here.

Actually, after reading all of the steps, I think we could probably break it down further. Something like:

1. Apply networking

• steps 1 - 9 as bullet points

2. Deploy new services

• steps 10 - 11 as bullet points

3. Updates and verification

• remaining steps

docs/operator-guide/index.md#1804 @qdzlug

Agreed; doing that now and making sure the bullets and indentation plays nice.

docs/operator-guide/index.md#1811 @chudley

I'll be honest, I wasn't sure this was required at any point past initial installation. I could certainly be wrong here. Do we have any docs that point to this step being required as part of the Manta deployment zone update?

docs/operator-guide/index.md#1811 @qdzlug

I don't know if we have docs, but when I was upgrading OTEX and couldn't find manta-oneach, dap said "oh, you need to re-run the setup to deploy those bits" - I think it was in chat, but I can track this down. See https://github.com/joyent/sdc-headnode/blob/4fcf5d5465b15953e1492b9fe00232595b905725/scripts/setup_manta_zone.sh#L158

docs/operator-guide/index.md#1811 @chudley

I wonder if it's required when there are new tools added to the Manta deployment zone. I think we should leave this out of this shrimp-specific piece of the documentation, in favour of adding a separate section (in a separate change) about missing tools in the GZ.

docs/operator-guide/index.md#1823 @chudley

I think this section might be best replaced with a link to the "Networking configuration" section of the operator guide (https://github.com/joyent/manta/blob/master/docs/operator-guide/index.md#networking-configuration), but you could still do with including an example in this section outlining a single new node.

This is because adding tags to aggregations has got better support with MANTA-2587, which now has had some use in SPC (and NET-367 is fixed) is due to be added to the docs (https://cr.joyent.us/#/c/865/). This may also make a few other sections of this change unnecessary (e.g. the previous sdc-cnapi call for sysinfo details).

docs/operator-guide/index.md#1823 @qdzlug

Added a link in for now; would it make sense to leave things are are and then do an update re: aggregation once CR 865 lands? I'd like to be able to link back up to that change in mine.

docs/operator-guide/index.md#1823 @chudley

Yes, that seems fine to me (including for your L1831 comment).

docs/operator-guide/index.md#1824 @chudley

I think this should be in all caps.

docs/operator-guide/index.md#1824 @qdzlug

Done

docs/operator-guide/index.md#1831 @chudley

See note above re: MANTA-2587.

docs/operator-guide/index.md#1831 @qdzlug

Same note as above - re: shall we wait for cr 865 to land so we can link back up to that?

docs/operator-guide/index.md#1835 @chudley

The annotations in this JSON make it quite hard to follow, but I understand it's useful to at least say something in these properties. I think we could go a long way by saying something like "The following is an example of adding a single new storage node to a single AZ deployment called 'staging-1'. For more information see 'Network configuration' section."

I think the way it's described in the "Network configuration" section i a good example.

docs/operator-guide/index.md#1835 @qdzlug

Agreed; will re-work this to show a sample and then link back up to Networking Configuration.

docs/operator-guide/index.md#1837 @chudley

See note above, but if we still make use of this example then I think this should be "DC-NAME". Same as in the "azs" section.

docs/operator-guide/index.md#1837 @qdzlug

Done

docs/operator-guide/index.md#1886 @chudley

manta-net.sh will validate the config and return any issues with it (including JSON parsing errors, afaik), so I think this step is unnecessary.

docs/operator-guide/index.md#1886 @qdzlug

Done

docs/operator-guide/index.md#1894 @chudley

I think steps 5 through 8 can be replaced with "Follow step 4 of the 'Deploying Manta' section."

docs/operator-guide/index.md#1894 @qdzlug

Done

docs/operator-guide/index.md#1921 @chudley

I think this is actually done via manta-net.sh.

We can probably remove this step, but if there's a reason why this is still required then we should fix it.

docs/operator-guide/index.md#1921 @qdzlug

Done

docs/operator-guide/index.md#1939 @chudley

TIL about -l.

Would be good to have this as a code span (backticks).

docs/operator-guide/index.md#1939 @qdzlug

Done

docs/operator-guide/index.md#1942 @chudley

Earlier we used ... show -s -j, which is the same thing, but would be good to be consistent.

I have no preference on how it's written in this guide, but I tend to use -sj.

docs/operator-guide/index.md#1942 @qdzlug

Done

docs/operator-guide/index.md#1998 @chudley

I think if we're following along in the guide then we're already in the Manta deployment zone.

docs/operator-guide/index.md#1998 @qdzlug

Done

docs/operator-guide/index.md#2008 @chudley

This is useful information, but I don't think it belongs in this guide. Basically, I'm not sure what an operator could do with information when following along.

docs/operator-guide/index.md#2008 @qdzlug

Done

@chudley commented at 2017-04-28T08:29:39

Patch Set 4:

(8 comments)

This is looking good! Describing how to write JSON and manipulate all these files is a complicated procedure when written down, but I think we're getting close to this being a concise set of steps.

There's a fair amount of trailing whitespace that has been introduced which will show up in gerrit as red blocks. We should remove these, and you should be able to set your editor to make noise about this stuff to help you spot them.

A number of the code blocks also appear to be done with backticks on their own line, where they should be indented by 4 spaces. I've marked a few of them as comments. Also, some of the code blocks under bullet points appear to display incorrectly, where I can see whitespace at the start of some lines when it's rendered (such as the block at L1954). I've found that getting this indentation right is a little bit finicky, and in this case it looks like there should be 6 spaces to get the indentation to work.

The document reads a lot better now, but some of the bullet points are difficult to read because it looks like the heading has been maintained (e.g. L1812). I think we should remove this heading style of introducing the bullet points and just word them out a bit. Check out this section of the operator guide as an example: https://github.com/joyent/manta/blob/master/docs/operator-guide/index.md#prerequisites. It can be quite wordy, but it fits with the rest of the guide.

In L1812's case, we could just remove the line break and I feel it would read a lot better.

Another example on L1929 would be something like:

• Run manta-adm update /var/tmp/newstorage.json to apply these changes. This command will call...

What do you think?

Patch Set 4 code comments

docs/operator-guide/index.md#1765 @chudley

I think this line should be indented by 4 spaces with the backticks removed.

I've just noticed this, but it seems we use a dollar sign here when displaying a prompt (headnode$ ...). There's a good example here:

https://github.com/joyent/manta/blob/master/docs/operator-guide/index.md#amon-alarm-updates

docs/operator-guide/index.md#1765 @qdzlug

Done

docs/operator-guide/index.md#1767 @chudley

Now that we refer to the "Network configuration" section after the example config (below), I think this means we can do away with this section and go straight to "Steps to Add New Storage Node".

docs/operator-guide/index.md#1767 @qdzlug

Done

docs/operator-guide/index.md#1805 @chudley

Should this be removed? It doesn't look like the other headings contain "Step #:".

docs/operator-guide/index.md#1805 @qdzlug

Done

docs/operator-guide/index.md#1821 @chudley

I think this is similar to the first comment. Probably should be 4 space indented with backticks removed.

docs/operator-guide/index.md#1821 @qdzlug

Done

docs/operator-guide/index.md#1834 @chudley

The indentation in this JSON looks off.

docs/operator-guide/index.md#1834 @qdzlug

Done

docs/operator-guide/index.md#1893 @chudley

4 space indent here, too.

docs/operator-guide/index.md#1893 @qdzlug

Done

docs/operator-guide/index.md#1895 @chudley

Should this be an additional bullet point?

docs/operator-guide/index.md#1895 @qdzlug

Done

docs/operator-guide/index.md#1920 @chudley

Given that we gave an example server UUID in the networking config JSON, I think we can skip this example piece.

docs/operator-guide/index.md#1920 @qdzlug

Done

@qdzlug commented at 2017-04-28T13:38:21

Patch Set 5:

(31 comments)

Hi Rich,

I think I got everything - take a look and see how the bullets / code spacing looks now - I have a rendered version here: https://gist.github.com/qdzlug/6b28f394259c45ad76e905e1bdae3336#common-operational-procedures

It's much tighter now, but let me know if you see more that needs to be spiffed up - I'm a bit too close to it at this point to make any judgments...

Thanks for the help here!

Jay

@chudley commented at 2017-04-28T14:55:53

Patch Set 5:

(10 comments)

Thanks for the gist! This is coming along nicely.

Patch Set 5 code comments

docs/operator-guide/index.md#1749 @chudley

I think it might be worth removing this piece, as ensuring the NIC is tagged is one of manta-net.sh's jobs.

docs/operator-guide/index.md#1749 @qdzlug

Done

docs/operator-guide/index.md#1789 @chudley

Can we put this on one line?

docs/operator-guide/index.md#1789 @qdzlug

Done

docs/operator-guide/index.md#1846 @chudley

Whitespace.

docs/operator-guide/index.md#1846 @qdzlug

Done

docs/operator-guide/index.md#1883 @chudley

The capitalisation here looks out of place. Can we also re-wrap this paragraph to ensure we make use of the 80 chars for readability?

docs/operator-guide/index.md#1883 @qdzlug

Done

docs/operator-guide/index.md#1891 @chudley

Can we re-wrap these lines to improve raw readability?

docs/operator-guide/index.md#1895 @chudley

Last bit of whitespace.

docs/operator-guide/index.md#1895 @qdzlug

Done

docs/operator-guide/index.md#1903 @chudley

I think this line might be >80 chars. It doesn't seem worth it to place the "| bunyan" part on another line though, so I'd say this is OK as it is.

docs/operator-guide/index.md#1931 @chudley

Can this be put onto one line like the other points?

@qdzlug commented at 2017-04-29T12:58:22

Patch Set 8:

(5 comments)

Hey Rich,

Rendered version here - https://gist.github.com/qdzlug/6b28f394259c45ad76e905e1bdae3336#common-operational-procedures - at this point I can now add Shrimp in my sleep. Not sure how useful that is.

Agreed on the setup-manta bit (to add the tools) - I took that out, and figured I would look to see if there is a logical place to put it elsewhere in the document. Keep tripping up on whitespace, but think I finally got it through my head to check before I push.

Jay

@chudley commented at 2017-05-02T10:57:13

Patch Set 8:

(4 comments)

Thanks, Jay! Some additional styling comments added now that I've read it again fully, but otherwise content is looking good.

Sorry these changes have come in bits, but I'm picking up little things as the flow of the document is changing.

Patch Set 8 code comments

docs/operator-guide/index.md#1759 @chudley

It would be nice to do away with this heading and make the next 3 headings H3 (instead of H4).

docs/operator-guide/index.md#1759 @qdzlug

Done

docs/operator-guide/index.md#1763 @chudley

Given that the points in this list should be done in series, can we make them numbered lists instead? This also applies to the "Deploy Services" section.

For numbered lists every item should start with "1." so we don't need to change them when a new item is added. Markdown will make sure these turn into the proper number order when it's rendered.

docs/operator-guide/index.md#1763 @qdzlug

Done

docs/operator-guide/index.md#1894 @chudley

Does this alarm section specifically relate to adding a new shrimp? I wonder if we should be recommending that this should be done at all because there might be important errors that the operator should handle.

Same with the drop/add steps. I wonder if these are here because we've recommended an update to the manta-deployment zone?

docs/operator-guide/index.md#1894 @qdzlug

I'm going to ask dap on this one today - I had thought this was done in order to make sure you had probes on the new object that you are clearing. See #5 in this block - https://gist.github.com/qdzlug/ae6e8ab743ce0d1244ded0578e82bc0d#procedure and also https://gist.github.com/qdzlug/ae6e8ab743ce0d1244ded0578e82bc0d#amon-alarm-updates

My gut feel here would be for to flesh out that second bit and then refer to it from the other places (ie, here are the procedures to remediate, drop, and add new probes if you've done something that requires it). That way it's in one place.

docs/operator-guide/index.md#1894 @chudley

I think if we just mention in this list that we need to handle the alarms and reference the other section in the guide then that will be plenty.

docs/operator-guide/index.md#1907 @chudley

The top-level items in this list are fine left unordered, but I think this item should be outside of the list on a paragraph on its own. This way it would read as if the "madtom" and "marlin-dashboard" updates can be done out of order, then once they're complete we can verify.

docs/operator-guide/index.md#1907 @qdzlug

Done

@qdzlug commented at 2017-05-02T12:48:37

Patch Set 9:

(4 comments)

Hi Rich,

Rendered version here - https://gist.github.com/qdzlug/ae6e8ab743ce0d1244ded0578e82bc0d#amon-alarm-updates - see my notes below for the mantamon probes bit. I'll follow up with dap this afternoon on that, and apply whatever he recommends.

Jay

@chudley commented at 2017-05-03T12:56:50

Patch Set 9:

(1 comment)

@chudley commented at 2017-05-16T09:09:37

Patch Set 9:

Hey Jay, just circling back to this one and I think so long as my comment on the alarms (https://cr.joyent.us/#/c/1854/8/docs/operator-guide/index.md@1894) makes sense to you then I think this will be plenty. Dave mentioned that the alarms pieces will be changing soon, so it makes sense to refer to a separate section in the guide so that we don't have to update multiple places of the guide for this procedure.

@davepacheco commented at 2017-05-16T16:31:13

Patch Set 9:

(14 comments)

Thanks again for doing this, and sorry it's taken me a while to take a closer look. I've got a few nits and suggestions below.

Patch Set 9 code comments

/COMMIT_MSG#7 @davepacheco

The colon shouldn't be here.

docs/operator-guide/index.md#1736 @davepacheco

Is this the canonical documentation for this? What about https://docs.joyent.com/private-cloud/install/compute-node-setup?

docs/operator-guide/index.md#1736 @qdzlug

I was torn here between linking to the doc in github versus the docs on docs.jo, and in the end I decided that the update cycle for changes would be shorter on the github side, since we really don't have anyone working on this level of docs on the docs.jo side currently. I'm not married to that however - we could do it either way. Thoughts?

docs/operator-guide/index.md#1736 @davepacheco

That's a fair question. This doc is supposed to be more akin to product documentation (i.e., for operators, not exclusively developers), so that's why I thought the docs.joyent.com one would be more appropriate, but I don't really know either.

docs/operator-guide/index.md#1741 @davepacheco

What exactly does "install VXLAN" refer to?

docs/operator-guide/index.md#1741 @qdzlug

This is the installation of the software defined networking - where we add the underlay nic and such to the compute nodes.

docs/operator-guide/index.md#1757 @davepacheco

It might seem confusing that we don't mention any of these during the initial Manta setup. Does Triton use the traits? Or is that something Joyent-Ops-specific? If Triton uses them, can we instead refer people to Triton documentation indicating that they should trait the CNs if they want to keep Manta services isolated? We probably should mention that during initial setup as well.

docs/operator-guide/index.md#1757 @qdzlug

This is intended to keep them out of the provisioning pool - so in this sense any trait will work. This is primarily something that grew out of JPC, but on prem installs are also done this way (ie, no customers have wanted to co-locate manta nodes w/ other CN's). My thought - maybe we update the initial setup to allow for this option (isolation) and then refer back to it here?

docs/operator-guide/index.md#1757 @chudley

Your last point sounds like the best bet, but would require changes elsewhere. I wonder if for now we could simply say:

... installation process as outlined [here], including any link aggregations and traits. Note that fabrics are not supported on Manta nodes.

I don't know if we explicitly don't support fabrics on Manta nodes, or just recommend against it (seeing as Manta doesn't use them). I also don't know the official Triton product name for fabrics/vxlan/underlays.

FWIW, MANTA-2047 intends to standardise these traits.

docs/operator-guide/index.md#1757 @qdzlug

OK, so...I think that helps things immensely. I just collapsed the traits into this and went into a variant of rich's suggestion.

docs/operator-guide/index.md#1762 @davepacheco

I'd suggest referring to the "Manta deployment zone upgrades" section. Although the section is currently pretty bare, I'd rather avoid duplicating it in case that later involves other steps or caveats.

docs/operator-guide/index.md#1762 @qdzlug

Done

docs/operator-guide/index.md#1766 @davepacheco

This should be lowercase.

docs/operator-guide/index.md#1766 @qdzlug

Done

docs/operator-guide/index.md#1769 @davepacheco

Which phase is that?

docs/operator-guide/index.md#1769 @qdzlug

Referred to an earlier version - removed becuase it makes no sense now.
Done.

docs/operator-guide/index.md#1827 @davepacheco

I'd suggest adding something about "which is to run manta-net.sh" in case the steps get renumbered.

docs/operator-guide/index.md#1827 @qdzlug

Done

docs/operator-guide/index.md#1837 @davepacheco

This section (a few lines below as well) should be rewrapped for 80 columns.

docs/operator-guide/index.md#1837 @qdzlug

Done

docs/operator-guide/index.md#1853 @davepacheco

I would suggest referring people to the recommendations in "Choosing how to lay out zones". Again, that section could use more information, but fleshing that out will keep the details in one place.

docs/operator-guide/index.md#1853 @qdzlug

Done

docs/operator-guide/index.md#1869 @davepacheco

Maybe remove the square brackets? That notation sometimes means an argument is optional, and it's definitely not here. You could make SERVER_UUID all caps to clarify that you have to plug that in.

docs/operator-guide/index.md#1869 @qdzlug

Done

docs/operator-guide/index.md#1872 @davepacheco

That's not the current documented Marlin update procedure. We should refer people to that section above.

docs/operator-guide/index.md#1872 @qdzlug

Done

docs/operator-guide/index.md#1890 @davepacheco

I think these should refer to a separate section (e.g., "Updating the Marlin Dashboard and Madtom zones after zone deployments"). There are other cases where you might want to do this.

docs/operator-guide/index.md#1890 @qdzlug

OK - so split this out to a new section for now (madtom and marlin-dash) at this same level (common operational tasks), then I link to it here. I also refer to your rewritten amon alarm updates (and took out the add/drop bits).

docs/operator-guide/index.md#1890 @davepacheco

I think you updated those parts, but still kept the copy of the instructions here?

docs/operator-guide/index.md#1903 @davepacheco

As I think was mentioned, this should probably refer to an existing section about updating alarms. I'm rewriting that in a separate change, but if you just refer to it by name, this will still be valid.

docs/operator-guide/index.md#1903 @qdzlug

I've updated this by removing my stuff and linking to your new mantamon bits.

@qdzlug commented at 2017-06-13T14:56:07

Patch Set 10:

(13 comments)

Hi Dap/Rich,

Thanks for the review, and sorry for the delay in getting back.

Rendered version is here - https://gist.github.com/qdzlug/92c0dbf85ea4532de851a5e9b9140122

Some answers/questions to the questions posed in your review Dap - I can adjust those as required - just need a little guidance.

Thanks,

Jay

@chudley commented at 2017-06-13T15:56:23

Patch Set 10:

(1 comment)

Thanks, Jay! Along with my comment on L1757 there are some trailing spaces introduced around L1838 which would need to be cleaned up.

@qdzlug commented at 2017-06-13T22:34:37

Patch Set 11:

(1 comment)

Rich - thanks! I put your suggestions into play. Sometime this week, if you could take 5 mins and show me your editor setup and how you avoid the trailling whitespace issue that I seem to have I'd be eternally in your debt....

@davepacheco commented at 2017-07-19T22:37:11

Patch Set 11:

(5 comments)

Patch Set 11 code comments

docs/operator-guide/index.md#1857 @davepacheco

I think change this to "Manta", now that mantamon is gone.

docs/operator-guide/index.md#1863 @davepacheco

How about "compute zones"?

docs/operator-guide/index.md#1868 @davepacheco

Rewrap?

[chudley]

@qdzlug I'm not sure where this one was left off, sorry. I'd be happy to go through this and re-review if you're still keen on the change!