Encode a Bounded Token Sequence

Task

Write encode_tokens(tokens, vocabulary, bos_token, eos_token, unknown_token). tokens is a list of token strings. vocabulary maps token strings to integer IDs. The three named special tokens are guaranteed to be in the vocabulary.

Return (ids, unknown_positions). Put the beginning-of-sequence ID first, follow the input token order, and put the end-of-sequence ID last. A token in the vocabulary uses its mapped ID. Any other token uses the ID for unknown_token, and its original zero-based position in tokens is appended to unknown_positions.

The special boundary IDs count as output IDs but do not have positions in unknown_positions. Empty input still produces the two boundary IDs. No input list or dictionary may be changed.

Example

blue is absent, so position 1 is recorded while the sequence keeps its place with the unknown ID.

Your implementation

Edit solution.py and keep this function name and signature:

Return exactly a two-item tuple containing two new lists: encoded integer IDs and zero-based unknown positions. Do not print, ask for input, or mutate any argument.