VScode continue plugin in agent mode fails after a while

[YanickNoblanc]

Hi all, I am using VSCode Continue plugin (version 1.3.24) on a Windows computer. It can successfully communicate with my vLLM offline server (server has not access to internet). The vLLM server can serve several language models. (Llama-3.1-8B-Instruct, Mistral-7B-Instruct-v0.2, ...). At VSCode side, I have set a local Continue config file at our project root folder. All of this is working fine and we are able to ask questions in a chat prompt, when used in agent mode, Continue plugin is able to locate our source code files, suggest modifications, etc ... But systematicaly, and whatever the language model used is, after a while using agent, Continue plugin fails communicating with our server. vllm logs at server side show that max token have been reached, log:
ValueError: 'max_tokens' or 'max_completion_tokens' is too large: 4096. This model's maximum context length is 32768 tokens and your request has 28739 input tokens (4096 > 32768 - 28739).

The problem only seems to be ate VSCode/Continue side because my model still properly answers questions when going through something else than VSCode (curl requests). Hence my question: Is there a way in agent mode to avoid reaching max_tokens? When this happens, how can we reset or dump history at VSCode side to recover?

[YanickNoblanc]

Copying VSCode console logs when problem happens:

[Violation] Forced reflow while executing JavaScript took 58ms
console.ts:139 [Extension Host] [@continuedev] error: 400 'max_tokens' or 'max_completion_tokens' is too large: 4096. This model's maximum context length is 32768 tokens and your request has 28739 input tokens (4096 > 32768 - 28739). None {"context":"llm_stream_chat","model":"/models/Llama-3.1-8B-Instruct","provider":"openai","useOpenAIAdapter":true,"streamEnabled":true,"templateMessages":false}
localProcessExtensionHost.ts:280 Extension Host
localProcessExtensionHost.ts:281 [@continuedev] error: 400 'max_tokens' or 'max_completion_tokens' is too large: 4096. This model's maximum context length is 32768 tokens and your request has 28739 input tokens (4096 > 32768 - 28739). None {"context":"llm_stream_chat","model":"/models/Llama-3.1-8B-Instruct","provider":"openai","useOpenAIAdapter":true,"streamEnabled":true,"templateMessages":false}

console.ts:139 [Extension Host] Error: 400 'max_tokens' or 'max_completion_tokens' is too large: 4096. This model's maximum context length is 32768 tokens and your request has 28739 input tokens (4096 > 32768 - 28739). None
	at APIError.generate (c:\Users\g590587\.vscode\extensions\continue.continue-1.2.11-win32-x64\out\extension.js:151941:18)
	at OpenAI.makeStatusError (c:\Users\g590587\.vscode\extensions\continue.continue-1.2.11-win32-x64\out\extension.js:158838:25)
	at OpenAI.makeRequest (c:\Users\g590587\.vscode\extensions\continue.continue-1.2.11-win32-x64\out\extension.js:158996:29)
	at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
	at async OpenAIApi.chatCompletionStream (c:\Users\g590587\.vscode\extensions\continue.continue-1.2.11-win32-x64\out\extension.js:159892:26)
	at async OpenAI2.openAIAdapterStream (c:\Users\g590587\.vscode\extensions\continue.continue-1.2.11-win32-x64\out\extension.js:245116:26)
	at async OpenAI2.streamChat (c:\Users\g590587\.vscode\extensions\continue.continue-1.2.11-win32-x64\out\extension.js:245237:32)
	at async llmStreamChat (c:\Users\g590587\.vscode\extensions\continue.continue-1.2.11-win32-x64\out\extension.js:724116:19)
	at async Wd.handleMessage [as value] (c:\Users\g590587\.vscode\extensions\continue.continue-1.2.11-win32-x64\out\extension.js:762400:29)
log.ts:460   ERR [Extension Host] Unable to log remote console arguments Output omitted for a large object that exceeds the limits
error @ log.ts:460
error @ log.ts:565
error @ logService.ts:51
mCs @ remoteConsoleUtil.ts:58
$logExtensionHostMessage @ mainThreadConsole.ts:38
S @ rpcProtocol.ts:458
Q @ rpcProtocol.ts:443
M @ rpcProtocol.ts:373
L @ rpcProtocol.ts:299
(anonymous) @ rpcProtocol.ts:161
C @ event.ts:1201
fire @ event.ts:1232
fire @ ipc.net.ts:652
l.onmessage @ localProcessExtensionHost.ts:383
console.ts:139 [Extension Host] Unable to log remote console arguments Output omitted for a large object that exceeds the limits
gCs @ console.ts:139
$logExtensionHostMessage @ mainThreadConsole.ts:39
S @ rpcProtocol.ts:458
Q @ rpcProtocol.ts:443
M @ rpcProtocol.ts:373
L @ rpcProtocol.ts:299
(anonymous) @ rpcProtocol.ts:161
C @ event.ts:1201
fire @ event.ts:1232
fire @ ipc.net.ts:652
l.onmessage @ localProcessExtensionHost.ts:383
XCircleIcon.js:40 [Violation] 'setTimeout' handler took 112ms

[YanickNoblanc]

I tried : restarting plugin, uninstalling/re-installing plugin, switching between Pre-release and Release plugin versions, Problem persists whatever I do.