Overview

A regex that tokenizes CSS.

var cssTokens = require("css-tokens").default

var cssString = ".foo{prop: foo;}\n..."

cssString.match(cssTokens)
// [".foo", "{", "prop", ":", " ", "foo", ";", "}", "\n", ...]

Installation

• npm install css-tokens

import cssTokens from "css-tokens"
// or:
var cssTokens = require("css-tokens").default

Usage

cssTokens

A regex with the g flag that matches CSS tokens.

The regex always matches, even invalid CSS and the empty string.

The next match is always directly after the previous.

var token = matchToToken(match)

import {matchToToken} from "css-tokens"
// or:
var matchToToken = require("css-tokens").matchToToken

Takes a match returned by cssTokens.exec(string), and returns a {type: String, value: String} object. The following types are available:

• string
• comment
• number
• unquotedUrl
• name
• punctuator
• whitespace
• invalid

Comments and strings also have a closed property indicating if the token was closed or not (see below).

Strings come in two flavors. To distinguish them, check if the token starts with ' or ".

Names may start with @ (as in at-rule names), . (as in class selectors) and # (as in id selectors and hex colors).

Invalid code handling

Unterminated strings are still matched as strings. CSS strings cannot contain (unescaped) newlines, so unterminated strings simply end at the end of the line.

Unterminated multi-line comments are also still matched as comments. They simply go on to the end of the string.

Unterminated unquoted urls are also still matched as unquoted urls. They continue as long as there are valid characters.

Invalid ASCII characters have their own capturing group.

Limitations

Tokenizing CSS using regexes—in fact, one single regex—won’t be perfect. But that’s not the point either.

Quoted vs. unquoted urls

The following is hardly a “limitation”, but could be mentioned:

url(http://www.w3.org/2000/svg)
url('http://www.w3.org/2000/svg')

The first line is matched as one single token (unquotedUrl), while the second is matched as four (name + punctuator + string + punctuator). This could be fixed, but isn’t to simplify the regex.

License

MIT.

Version 2.0.0 (2017-01-11)

This release contains one breaking change, that should improve performance in V8:

So how can you, as a JavaScript developer, ensure that your RegExps are fast? If you are not interested in hooking into RegExp internals, make sure that neither the RegExp instance, nor its prototype is modified in order to get the best performance:

var re = /./g;
re.exec('');  // Fast path.
re.new_property = 'slow';

This module used to export a single regex, with .matchToToken bolted on, just like in the above example. This release changes the exports of the module to avoid this issue.

Before:

import cssTokens from "css-tokens"
// or:
var cssTokens = require("css-tokens")
var matchToToken = cssTokens.matchToToken

After:

import cssTokens, {matchToToken} from "css-tokens"
// or:
var cssTokens = require("css-tokens").default
var matchToToken = require("css-tokens").matchToToken

Version 1.0.1 (2015-06-20)

• Fixed: Declared an undeclared variable.

Version 1.0.0 (2015-02-26)

• Changed: Merged the 'operator' and 'punctuation' types into 'punctuator', for consistency with js-tokens. (Backwards-incompatible change.)

Version 0.4.2 (2015-02-21)

• Improved: cssTokens.matchToToken performance.

Version 0.4.1 (2015-01-08)

• Fixed: \f is now recognized as a newline.

Version 0.4.0 (2014-12-19)

• Changed: The cssTokens.names array has been replaced with the cssTokens.matchToToken function. The capturing groups of cssTokens are no longer part of the public API; instead use said function. See this gist for an example. (Backwards-incompatible change.)
• Changed: The empty string is now considered an “invalid” token, instead an “empty” token (its own group). (Backwards-incompatible change.)
• Removed: component support. (Backwards-incompatible change.)

Version 0.3.0 (2014-06-19)

• Added: Support for --custom-properties. (Backwards-incompatible change.)
• Fixed: @- and .- (followed by a non-name character) are now matched as invalid + operator, instead of name. Note that #- is actually allowed by the spec. That used to be matched as a name, but is now matched as invalid + operator, too. It doesn’t matter. (Backwards-incompatible change.)

Version 0.2.0 (2014-03-11)

• Names may now start with @, # and .. This makes it easier to work with at-rules, ids, classes and hex colors. (Backwards incompatible change.)

  Previously those three characters were matched by themselves as punctuation and were followed by names. Now, if any of those characters are on their own (not followed by a name), they’re matched as invalid.

  Previously, hex colors were matched as either punctuation + name or punctuation + number + name. Now they’re always names. (That might not be ideal, but consider that #f00 is both a valid id and a valid hex color, which are impossible to distinguish. Think “names of colors”.)

Version 0.1.0 (2014-03-09)

• Initial release.