Redesigning the mannequin
TimesFM is a patched decoder that tokenizes each 32 contiguous timepoints (a patch) as an enter token and applies a transformer stack on high of the sequence of enter tokens to generate the output tokens. It then applies a shared multilayer perceptron (MLP) to translate every output token again to a time sequence of 128 timepoints.
To create TimesFM-ICF (In-Context Nice-tuning), we begin with the bottom TimesFM mannequin and proceed the pre-training with new context: the forecast historical past plus all in-context examples. Step one is to verify the mannequin doesn’t confuse or conflate the forecasting historical past and the in-context examples. Think about you are giving the mannequin an inventory of numbers that symbolize a couple of various things, possibly sun shades gross sales figures from one retailer, then umbrella gross sales figures from one other. Should you simply merge all these numbers collectively, the mannequin would possibly get confused, considering it is one steady stream of knowledge. For instance, if the primary retailer’s gross sales have been going up and the second retailer’s gross sales have been happening, the mannequin would possibly incorrectly see it as a single up-and-down sample, moderately than two separate, easy tendencies.
To repair this, we put a particular, learnable “widespread separator token” — like a digital “cease signal” or a “new paragraph” image — after every set of numbers. With these separators in place, as quickly because the mannequin attends to the separator token of an instance it has seen earlier than, it will not combine it up with the info it is presently attempting to foretell. This theoretically permits the mannequin to study from patterns in these previous examples and apply that data to the present forecast. As an example, the mannequin may study that “all the shop gross sales are displaying constant, directional tendencies currently, so I ought to predict an upward pattern for my new retailer’s sunscreen gross sales.”