parameters: (name, the_dict)
docstring: Process regular expression and exclusion-based filters on lists.
  
    An exclusion list is in a dict key named with a trailing "!", like
    "sources!".  Every item in such a list is removed from the associated
    main list, which in this example, would be "sources".  Removed items are
    placed into a "sources_excluded" list in the dict.
  
    Regular expression (regex) filters are contained in dict keys named with a
    trailing "/", such as "sources/" to operate on the "sources" list.  Regex
    filters in a dict take the form:
      'sources/': [ ['exclude', '_(linux|mac|win)\\.cc$'],
                    ['include', '_mac\\.cc$'] ],
    The first filter says to exclude all files ending in _linux.cc, _mac.cc, and
    _win.cc.  The second filter then includes all files ending in _mac.cc that
    are now or were once in the "sources" list.  Items matching an "exclude"
    filter are subject to the same processing as would occur if they were listed
    by name in an exclusion list (ending in "!").  Items matching an "include"
    filter are brought back into the main list if previously excluded by an
    exclusion list or exclusion regex filter.  Subsequent matching "exclude"
    patterns can still cause items to be excluded after matching an "include".
docstring_summary: Process regular expression and exclusion-based filters on lists.
docstring_tokens: Process regular expression and exclusion - based filters on lists .
function: def ProcessListFiltersInDict(name, the_dict):
    """Process regular expression and exclusion-based filters on lists.
  
    An exclusion list is in a dict key named with a trailing "!", like
    "sources!".  Every item in such a list is removed from the associated
    main list, which in this example, would be "sources".  Removed items are
    placed into a "sources_excluded" list in the dict.
  
    Regular expression (regex) filters are contained in dict keys named with a
    trailing "/", such as "sources/" to operate on the "sources" list.  Regex
    filters in a dict take the form:
      'sources/': [ ['exclude', '_(linux|mac|win)\\.cc$'],
                    ['include', '_mac\\.cc$'] ],
    The first filter says to exclude all files ending in _linux.cc, _mac.cc, and
    _win.cc.  The second filter then includes all files ending in _mac.cc that
    are now or were once in the "sources" list.  Items matching an "exclude"
    filter are subject to the same processing as would occur if they were listed
    by name in an exclusion list (ending in "!").  Items matching an "include"
    filter are brought back into the main list if previously excluded by an
    exclusion list or exclusion regex filter.  Subsequent matching "exclude"
    patterns can still cause items to be excluded after matching an "include".
    """
  
    # Look through the dictionary for any lists whose keys end in "!" or "/".
    # These are lists that will be treated as exclude lists and regular
    # expression-based exclude/include lists.  Collect the lists that are
    # needed first, looking for the lists that they operate on, and assemble
    # then into |lists|.  This is done in a separate loop up front, because
    # the _included and _excluded keys need to be added to the_dict, and that
    # can't be done while iterating through it.
  
    lists = []
    del_lists = []
    for key, value in the_dict.iteritems():
      operation = key[-1]
      if operation != '!' and operation != '/':
        continue
  
      if type(value) is not list:
        raise ValueError(name + ' key ' + key + ' must be list, not ' + \
                         value.__class__.__name__)
  
      list_key = key[:-1]
      if list_key not in the_dict:
        # This happens when there's a list like "sources!" but no corresponding
        # "sources" list.  Since there's nothing for it to operate on, queue up
        # the "sources!" list for deletion now.
        del_lists.append(key)
        continue
  
      if type(the_dict[list_key]) is not list:
        value = the_dict[list_key]
        raise ValueError(name + ' key ' + list_key + \
                         ' must be list, not ' + \
                         value.__class__.__name__ + ' when applying ' + \
                         {'!': 'exclusion', '/': 'regex'}[operation])
  
      if not list_key in lists:
        lists.append(list_key)
  
    # Delete the lists that are known to be unneeded at this point.
    for del_list in del_lists:
      del the_dict[del_list]
  
    for list_key in lists:
      the_list = the_dict[list_key]
  
      # Initialize the list_actions list, which is parallel to the_list.  Each
      # item in list_actions identifies whether the corresponding item in
      # the_list should be excluded, unconditionally preserved (included), or
      # whether no exclusion or inclusion has been applied.  Items for which
      # no exclusion or inclusion has been applied (yet) have value -1, items
      # excluded have value 0, and items included have value 1.  Includes and
      # excludes override previous actions.  All items in list_actions are
      # initialized to -1 because no excludes or includes have been processed
      # yet.
      list_actions = list((-1,) * len(the_list))
  
      exclude_key = list_key + '!'
      if exclude_key in the_dict:
        for exclude_item in the_dict[exclude_key]:
          for index in xrange(0, len(the_list)):
            if exclude_item == the_list[index]:
              # This item matches the exclude_item, so set its action to 0
              # (exclude).
              list_actions[index] = 0
  
        # The "whatever!" list is no longer needed, dump it.
        del the_dict[exclude_key]
  
      regex_key = list_key + '/'
      if regex_key in the_dict:
        for regex_item in the_dict[regex_key]:
          [action, pattern] = regex_item
          pattern_re = re.compile(pattern)
  
          if action == 'exclude':
            # This item matches an exclude regex, so set its value to 0 (exclude).
            action_value = 0
          elif action == 'include':
            # This item matches an include regex, so set its value to 1 (include).
            action_value = 1
          else:
            # This is an action that doesn't make any sense.
            raise ValueError('Unrecognized action ' + action + ' in ' + name + \
                             ' key ' + regex_key)
  
          for index in xrange(0, len(the_list)):
            list_item = the_list[index]
            if list_actions[index] == action_value:
              # Even if the regex matches, nothing will change so continue (regex
              # searches are expensive).
              continue
            if pattern_re.search(list_item):
              # Regular expression match.
              list_actions[index] = action_value
  
        # The "whatever/" list is no longer needed, dump it.
        del the_dict[regex_key]
  
      # Add excluded items to the excluded list.
      #
      # Note that exclude_key ("sources!") is different from excluded_key
      # ("sources_excluded").  The exclude_key list is input and it was already
      # processed and deleted; the excluded_key list is output and it's about
      # to be created.
      excluded_key = list_key + '_excluded'
      if excluded_key in the_dict:
        raise GypError(name + ' key ' + excluded_key +
                       ' must not be present prior '
                       ' to applying exclusion/regex filters for ' + list_key)
  
      excluded_list = []
  
      # Go backwards through the list_actions list so that as items are deleted,
      # the indices of items that haven't been seen yet don't shift.  That means
      # that things need to be prepended to excluded_list to maintain them in the
      # same order that they existed in the_list.
      for index in xrange(len(list_actions) - 1, -1, -1):
        if list_actions[index] == 0:
          # Dump anything with action 0 (exclude).  Keep anything with action 1
          # (include) or -1 (no include or exclude seen for the item).
          excluded_list.insert(0, the_list[index])
          del the_list[index]
  
      # If anything was excluded, put the excluded list into the_dict at
      # excluded_key.
      if len(excluded_list) > 0:
        the_dict[excluded_key] = excluded_list
  
    # Now recurse into subdicts and lists that may contain dicts.
    for key, value in the_dict.iteritems():
      if type(value) is dict:
        ProcessListFiltersInDict(key, value)
      elif type(value) is list:
        ProcessListFiltersInList(key, value)
function_tokens: def ProcessListFiltersInDict ( name , the_dict ) : # Look through the dictionary for any lists whose keys end in "!" or "/". # These are lists that will be treated as exclude lists and regular # expression-based exclude/include lists.  Collect the lists that are # needed first, looking for the lists that they operate on, and assemble # then into |lists|.  This is done in a separate loop up front, because # the _included and _excluded keys need to be added to the_dict, and that # can't be done while iterating through it. lists = [ ] del_lists = [ ] for key , value in the_dict . iteritems ( ) : operation = key [ - 1 ] if operation != '!' and operation != '/' : continue if type ( value ) is not list : raise ValueError ( name + ' key ' + key + ' must be list, not ' + value . __class__ . __name__ ) list_key = key [ : - 1 ] if list_key not in the_dict : # This happens when there's a list like "sources!" but no corresponding # "sources" list.  Since there's nothing for it to operate on, queue up # the "sources!" list for deletion now. del_lists . append ( key ) continue if type ( the_dict [ list_key ] ) is not list : value = the_dict [ list_key ] raise ValueError ( name + ' key ' + list_key + ' must be list, not ' + value . __class__ . __name__ + ' when applying ' + { '!' : 'exclusion' , '/' : 'regex' } [ operation ] ) if not list_key in lists : lists . append ( list_key ) # Delete the lists that are known to be unneeded at this point. for del_list in del_lists : del the_dict [ del_list ] for list_key in lists : the_list = the_dict [ list_key ] # Initialize the list_actions list, which is parallel to the_list.  Each # item in list_actions identifies whether the corresponding item in # the_list should be excluded, unconditionally preserved (included), or # whether no exclusion or inclusion has been applied.  Items for which # no exclusion or inclusion has been applied (yet) have value -1, items # excluded have value 0, and items included have value 1.  Includes and # excludes override previous actions.  All items in list_actions are # initialized to -1 because no excludes or includes have been processed # yet. list_actions = list ( ( - 1 , ) * len ( the_list ) ) exclude_key = list_key + '!' if exclude_key in the_dict : for exclude_item in the_dict [ exclude_key ] : for index in xrange ( 0 , len ( the_list ) ) : if exclude_item == the_list [ index ] : # This item matches the exclude_item, so set its action to 0 # (exclude). list_actions [ index ] = 0 # The "whatever!" list is no longer needed, dump it. del the_dict [ exclude_key ] regex_key = list_key + '/' if regex_key in the_dict : for regex_item in the_dict [ regex_key ] : [ action , pattern ] = regex_item pattern_re = re . compile ( pattern ) if action == 'exclude' : # This item matches an exclude regex, so set its value to 0 (exclude). action_value = 0 elif action == 'include' : # This item matches an include regex, so set its value to 1 (include). action_value = 1 else : # This is an action that doesn't make any sense. raise ValueError ( 'Unrecognized action ' + action + ' in ' + name + ' key ' + regex_key ) for index in xrange ( 0 , len ( the_list ) ) : list_item = the_list [ index ] if list_actions [ index ] == action_value : # Even if the regex matches, nothing will change so continue (regex # searches are expensive). continue if pattern_re . search ( list_item ) : # Regular expression match. list_actions [ index ] = action_value # The "whatever/" list is no longer needed, dump it. del the_dict [ regex_key ] # Add excluded items to the excluded list. # # Note that exclude_key ("sources!") is different from excluded_key # ("sources_excluded").  The exclude_key list is input and it was already # processed and deleted; the excluded_key list is output and it's about # to be created. excluded_key = list_key + '_excluded' if excluded_key in the_dict : raise GypError ( name + ' key ' + excluded_key + ' must not be present prior ' ' to applying exclusion/regex filters for ' + list_key ) excluded_list = [ ] # Go backwards through the list_actions list so that as items are deleted, # the indices of items that haven't been seen yet don't shift.  That means # that things need to be prepended to excluded_list to maintain them in the # same order that they existed in the_list. for index in xrange ( len ( list_actions ) - 1 , - 1 , - 1 ) : if list_actions [ index ] == 0 : # Dump anything with action 0 (exclude).  Keep anything with action 1 # (include) or -1 (no include or exclude seen for the item). excluded_list . insert ( 0 , the_list [ index ] ) del the_list [ index ] # If anything was excluded, put the excluded list into the_dict at # excluded_key. if len ( excluded_list ) > 0 : the_dict [ excluded_key ] = excluded_list # Now recurse into subdicts and lists that may contain dicts. for key , value in the_dict . iteritems ( ) : if type ( value ) is dict : ProcessListFiltersInDict ( key , value ) elif type ( value ) is list : ProcessListFiltersInList ( key , value )