Scan Text into Tokens and Spans

Task

Write scan_tokens(text, kinds). kinds is a list with one entry for each character in text. Every entry is exactly one of "word", "space", or "punctuation".

Scan from left to right using these rules:

• Consecutive characters whose kind is "word" form one token.
• A character whose kind is "punctuation" forms one token by itself.
• Characters whose kind is "space" are skipped.

Return a list of four-item tuples (token, start, stop, kind). start and stop are zero-based, half-open character positions in the original text, so token == text[start:stop]. The kind is the kind of the token. Word tokens may contain several characters; punctuation tokens contain one character.

The returned spans must account for every non-space character exactly once. Skipped spaces remain inferable as the gaps between neighboring spans. Do not change kinds.

Example

The exact character kinds are supplied, so no regular expressions or character-class library is needed. A punctuation character next to another punctuation character still produces a separate token for each character.

Your implementation

Edit solution.py and keep this function name and signature:

Return the required list; do not print it or ask for input. You may assume that len(kinds) == len(text) and that every kind is one of the three listed strings.