@apurva @christina.hendricks thanks for the details – I can reproduce it on MacOS as well.
Though I’m surprised it only shows up with Firefox on MacOS, I’ve confirmed that it is a known, rather subtle issue discussed at https://groups.google.com/d/msg/mathjax-users/v3W-daBz87k/xjxFFdfQBQAJ: the surrounding webfont (Crimson Text) has bad metadata which trips up MathJax.
To explain, MathJax tries to match its output to the surrounding context, in particular by alignign not just font-size but ex-heights. Crimson Text seems to be one of many Google Fons that has bad metadata regarding its ex-height, causing MathJax’s measuring logic to get it wrong.
You can test this by temporarily disabling Crimson Text which fixes it for me.
It might be that FIrefox considers this a bug or that Pressbooks is willing to (let you) disable font-matching (as described in the MathJax User Group link).
Regarding diagrams, MathJax includes support for the amsCD package http://docs.mathjax.org/en/v2.7-latest/tex.html#amscd but that’s a bit limited and diagonal arrows are difficult.
I’d personally say that those kinds of diagrams don’t fall into the “equation” bucket (which MathJax focuses on) so rendering them in SVG is more appropriate.
You can still make them accessible either in themselves (e.g., using a title/description tags) or by using aria-labelledby/aria-describedby to point to a part of the page that labels/describes the diagram. (And if you want to go all out, there are more complex ARIA techniques if you want to make them “explorable” but those won’t work in epub or kindle formats so it’s likely not worth the trouble.)