code: function nextToken() {
              if (curOffset >= length) {
                  return false;
              }
              if (stream) {
                  // Set the start of the next token to the current stream position.
                  stream.start = stream.pos;
              }
              while (!stream || stream.eol()) {
                  if (!nextLine()) {
                      return false;
                  }
              }
              style = mode.token(stream, state);
              token = stream.current();
              curOffset = lineStart + stream.pos;
              return true;
          }
docs: Get the next token, updating the style and token to refer to the current token, and updating the curOffset to point to the end of the token (relative to the start of the original text).