Deep Dive into Markdown: AST Parsing, CommonMark, and HTML Conversion.
Understand the architecture of Markdown compilers, Abstract Syntax Trees (ASTs), CommonMark specifications, and secure client-side HTML rendering.
1. The Evolution of Markdown
Created in 2004 by John Gruber and Aaron Swartz, Markdown was designed with a fundamental principle: plain text formatting should be completely readable as-is without looking like markup language tags. Over two decades, Markdown became the standard format for documentation, developer issue trackers, static site generators, and notes apps.
Because the original specification left ambiguous parsing behaviors, the community created CommonMark, a rigorous, unambiguous standard specification, along with GitHub Flavored Markdown (GFM), which introduced tables, task lists, autolinks, and strikethroughs.
2. Inside the Markdown Compilation Engine
Modern Markdown parsers (such as Unified/Remark, Marked, and Markdown-it) do not rely on fragile regular expression substitutions. Instead, they operate as true recursive descent compilers:
- Block Parsing: The input text stream is split into distinct structural blocks: paragraphs, headings, blockquotes, code blocks, and list items.
- Inline Parsing: Inside each block, inline tokenizers scan for span elements: bold, italics, links, inline code, and images.
- Abstract Syntax Tree (AST) Generation: A nested tree representation of the document is generated (e.g., mdast), separating content from output formatting.
- HTML Rendering: The AST nodes are transformed into semantic HTML tags with designated class names and attributes.
3. Preventing Cross-Site Scripting (XSS) in Rendered Markdown
By design, raw Markdown permits embedded HTML tags. For instance, writing <div class="note">...</div> is completely legal Markdown syntax. However, this flexibility presents severe security risks if user-generated Markdown is rendered without sanitization:
To neutralize XSS vectors, production web applications must pass rendered HTML through sanitization libraries like DOMPurify or DOM-level escaping algorithms that strip malicious protocols (javascript:, data:) and dangerous event handlers (onload, onerror). Test and preview your formatted documents live using our free Markdown Preview Tool.
4. Stripping HTML and Converting Back to Plain Text
When ingesting rich text or HTML for indexing, summarization, or text-to-speech processing, stripping markup tags without breaking words or spacing is essential. You can easily strip markup elements and inspect clean content with our Remove HTML Tags Utility.