Session Management During Pod Autoscaling with Cookie Affinity in Kubernetes

Author: yijtanCreated Aug 17, 2024Updated Sep 29, 2026

Labelsenhancementcloudtoinvestigate

Describe the bug

I have deployed a Gradio-based chatbot within a Kubernetes environment, with Nginx set up for load balancing and cookie affinity enabled. The system works perfectly when running a predefined number of replicas, thanks to the fixed cookie affinity setup in Nginx. However, I am encountering issues when I enable an autoscaler to dynamically adjust the number of pod replicas based on the number of users.

Problems:

Scaling Up: When the number of pods scales up, new requests are sometimes routed to a new pod that does not recognize the user_id. This results in an error due to the lack of context or session information for the user.
Persistent Cookie Affinity: A potential solution I tried was to set Nginx’s affinity mode to persistent. While this fixes the scaling-up issue, it introduces problems during pod scaling down. When a pod is terminated, requests that are still tied to that pod via the persistent affinity throw an error as the pod is no longer available to handle the request.

Proposed Solution:

I want to implement session management using a session cookie stored in the user’s browser. This cookie would contain a session_id, which can be used to retrieve the user’s context whenever they interact with the chatbot, even if the request is routed to a new pod.

Issue:

I need to retrieve the session cookie from the browser and pass it to my backend at the bot function level to handle the session management. However, I am unable to use gr.Request inside the nested loops in the bot function. This is blocking my implementation of session management.

I thought about moving the bot function outside the init_gradio loop so that I could use gr.Request. However, the bot function uses elements defined within the nested loop, such as input, submit, stop, and clear, which makes me unsure if moving it would be feasible.

Have you searched existing issues?

I have searched and found no existing issues

Reproduction

python

import gradio as gr
def init_gradio():
            with gr.Row() as input_row:
                        input = gr.MultimodalTextbox()
                        submit = gr.Button(value='Submit')
                        stop = gr.Button(value="Stop")
                        clear = gr.ClearButton()


            def bot(_input, _chatbot, _logs, c_id, u_id):
                if not c_id:
                    conversation = create_conversation(u_id, _input['text'])
                    c_id = conversation.conversation_id
                else:
                    update_conversation_time(c_id)
                latest_conversation_ids, chs = fetch_top_conversations(u_id)
                chs = [[c] for c in chs]
              
                    yield {
                            input: gr.update(value=None, interactive=False), 
                            chatbot: _chatbot, 
                            logs: _logs, 
                            export: None,
                            submit: gr.update(visible=False),
                            stop: gr.update(visible=True),
                            clear: gr.update(visible=False),
                            running: gr.update(value=True),
                            conversation_id: c_id,
                            conversation_history_summaries: gr.update(samples=chs, visible=True, samples_per_page=len(chs))
                        }

   
                yield {
                    input: gr.update(interactive=True), 
                    chatbot: gr.update(value=_chatbot, likeable=True, show_copy_button=True),
                    logs: gr.update(value=logs_df, visible=True), 
                    export: gr.update(value=logs_file_name, visible=True),
                    submit: gr.update(visible=False),
                    stop: gr.update(visible=False),
                    clear: gr.update(visible=False),
                    running: gr.update(value=False),
                    conversation_id:c_id,
                    conversation_history_summaries: gr.update(samples=chs, visible=True)
                }

            clicked_enter = input.submit(
                fn=bot,
                inputs=[input, chatbot, logs, conversation_id, user_id],
                outputs=[input, chatbot, logs, export, submit, stop, clear, running, conversation_id, conversation_history_summaries],
                show_progress='hidden',
                scroll_to_output=True,
                js=js
            )

            clicked_submit = submit.click(
                fn=bot,
                inputs=[input, chatbot, logs, conversation_id, user_id],
                outputs=[input, chatbot, logs, export, submit, stop, clear, running, conversation_id, conversation_history_summaries],
                show_progress='hidden',
                scroll_to_output=True,
                js=js
            )

            demo.load(load, None, [login_chatbot, menu_state, menu_pane, loginout_btn, profile_pic, conversation_history_summaries, chat_pane, chatbot, input_row, user_pane, user_id], api_name="greet", show_progress='hidden')

    return demo        

demo = init_gradio()

Screenshot

No response

Logs

No response

System Info

bash

Operating System: Darwin
gradio version: 4.41.0
gradio_client version: 1.3.0

Severity

I can work around it

Source: gradio-app/gradio

View original on GitHub View discussion on GitHub