each token is 4 characters long, with spaces included. They are going to improve the maximum number of tokens the model can handle in a conversation in order to give better responses. Currently, the free model (ChatGPT 3.5) can handle up to 4,096 tokens. This means that if you input a longer text, only the last 4,096 words will be considered for generating the response. The same applies to long conversations. The initial messages will make sense, and the model will use them along with your requests to generate subsequent responses. However, once the conversation reaches 4,096 tokens, the beginning will be forgotten.
The paid model (ChatGPT-4) doubles the amount of text it can remember as context for generating responses. Currently, it can remember up to 8,192 tokens. However, in the near future, this will be increased to 32,768. To put it into perspective, that's roughly equivalent to 50 pages. That's the context window.
OpenAI's GPT models have been trained on an extensive amount of text. However, the way these models process text is different from how we traditionally think of words, letters, or symbols. Instead, they focus on identifying common sequences of characters known as tokens. Tokens can be thought of as "pieces of words" and, for English text, each token typically represents around 4 characters, including symbols, numbers, letters, and spaces.
To help users understand this new term, OpenAI provides a free tool called the "Tokenizer", as the name suggest, it assists in determining how many tokens are present within a specific text snippet. That 6k input in your test needs to be 'translated' in this tool to know exactly the amount of tokens that are being processed. If the input is greater than 8,768, it will probably start forgetting the first part of the text and will not include it in the output.
In the FAQ of their pricing page they describe most of this info and there is also a link to the Tokenizer, but if you need to fully understand how this models work, I recommend you to look for the tiny details in their published papers on LLMs. Very interesting stuff.
From André Aparicio
(https://www.facebook.com/andre.tiranosaurio/)
each token is 4 characters long, with spaces included. They are going to improve the maximum number of tokens the model can handle in a conversation in order to give better responses. Currently, the free model (ChatGPT 3.5) can handle up to 4,096 tokens. This means that if you input a longer text, only the last 4,096 words will be considered for generating the response. The same applies to long conversations. The initial messages will make sense, and the model will use them along with your requests to generate subsequent responses. However, once the conversation reaches 4,096 tokens, the beginning will be forgotten.
The paid model (ChatGPT-4) doubles the amount of text it can remember as context for generating responses. Currently, it can remember up to 8,192 tokens. However, in the near future, this will be increased to 32,768. To put it into perspective, that's roughly equivalent to 50 pages. That's the context window.
You can check this at https://platform.openai.com/docs/models/continuous-model-upgrades
OpenAI's GPT models have been trained on an extensive amount of text. However, the way these models process text is different from how we traditionally think of words, letters, or symbols. Instead, they focus on identifying common sequences of characters known as tokens. Tokens can be thought of as "pieces of words" and, for English text, each token typically represents around 4 characters, including symbols, numbers, letters, and spaces.
To help users understand this new term, OpenAI provides a free tool called the "Tokenizer", as the name suggest, it assists in determining how many tokens are present within a specific text snippet. That 6k input in your test needs to be 'translated' in this tool to know exactly the amount of tokens that are being processed. If the input is greater than 8,768, it will probably start forgetting the first part of the text and will not include it in the output.
In the FAQ of their pricing page they describe most of this info and there is also a link to the Tokenizer, but if you need to fully understand how this models work, I recommend you to look for the tiny details in their published papers on LLMs. Very interesting stuff.
https://openai.com/pricing