Using MathChat to Solve Math Problems

AutoGen offers conversable agents powered by LLM, tool or human, which can be used to perform tasks collectively via automated chat. This framework allows tool use and human participation through multi-agent conversation. Please find documentation about this feature here.

MathChat is an experimental conversational framework for math problem solving. In this notebook, we demonstrate how to use MathChat to solve math problems. MathChat uses the AssistantAgent and MathUserProxyAgent, which is similar to the usage of AssistantAgent and UserProxyAgent in other notebooks (e.g., Automated Task Solving with Code Generation, Execution & Debugging). Essentially, MathUserProxyAgent implements a different auto reply mechanism corresponding to the MathChat prompts. You can find more details in the paper An Empirical Study on Challenging Math Problem Solving with GPT-4 or the blogpost.

Requirements

AutoGen requires Python>=3.8. To run this notebook example, please install the [mathchat] option.

pip install "pyautogen[mathchat]"

Set your API Endpoint

The config_list_from_json function loads a list of configurations from an environment variable or a json file.

It first looks for environment variable “OAI_CONFIG_LIST” which needs to be a valid json string. If that variable is not found, it then looks for a json file named “OAI_CONFIG_LIST”. It filters the configs by models (you can filter by other keys as well).

The config list looks like the following:

config_list = [
    {
        'model': 'gpt-4',
        'api_key': '<your OpenAI API key here>',
    },
    {
        'model': 'gpt-4',
        'api_key': '<your Azure OpenAI API key here>',
        'base_url': '<your Azure OpenAI API base here>',
        'api_type': 'azure',
        'api_version': '2023-06-01-preview',
    },
    {
        'model': 'gpt-3.5-turbo',
        'api_key': '<your Azure OpenAI API key here>',
        'base_url': '<your Azure OpenAI API base here>',
        'api_type': 'azure',
        'api_version': '2023-06-01-preview',
    },
]

If you open this notebook in colab, you can upload your files by clicking the file icon on the left panel and then choose “upload file” icon.

You can set the value of config_list in other ways you prefer, e.g., loading from a YAML file.

Construct agents for MathChat

We start by initializing the AssistantAgent and MathUserProxyAgent. The system message needs to be set to “You are a helpful assistant.” for MathChat. The detailed instructions are given in the user message. Later we will use the MathUserProxyAgent.generate_init_message to combine the instructions and a math problem for an initial message to be sent to the LLM assistant.

Example 1

Problem: Find all � that satisfy the inequality (2�+10)(�+3)<(3�+9)(�+8). Express your answer in interval notation.

Correct Solution: We have(2�+10)(�+3)<(3�+9)(�+8)⇒2(�+5)(�+3)<3(�+3)(�+8)⇒2(�+5)(�+3)−3(�+3)(�+8)<0⇒(2�+10−(3�+24))(�+3)<0⇒(−�−14)(�+3)<0⇒(�+14)(�+3)>0.This inequality is satisfied if and only if (�+14) and (�+3) are either both positive or both negative. Both factors are positive for �>−3 and both factors are negative for �<−14. When −14<�<−3, one factor is positive and the other negative, so their product is negative. Therefore, the range of � that satisfies the inequality is (−∞,−14)∪(−3,∞).

Example 2

Problem: For what negative value of � is there exactly one solution to the system of equations�=2�2+��+6�=−�+4?

Correct Solution: Setting the two expressions for � equal to each other, it follows that 2�2+��+6=−�+4. Re-arranging, 2�2+(�+1)�+2=0. For there to be exactly one solution for �, then the discriminant of the given quadratic must be equal to zero. Thus, (�+1)2−4⋅2⋅2=(�+1)2−16=0, so �+1=±4. Taking the negative value, �=−5.

Example 3

Problem: Find all positive integer values of � such that the equation �2−7�+�=0 only has roots that are real and rational. Express them in decreasing order, separated by commas.

Correct Solution: For the roots to be real and rational, the discriminant must be a perfect square. Therefore, (−7)2−4⋅1⋅�=49−4� must be a perfect square. The only positive perfect squares less than 49 are 1, 4, 9, 16, 25, and 36. The perfect squares that give a integer value of � are 1, 9, and 25. Thus, we have the equations 49−4�=1, 49−4�=9, and 49−4�=25. Solving, we get that the positive integer values of c are 12,10,6.

MathChat allows different prompts that instruct the assistant to solve the problem.

Check out MathUserProxyAgent.generate_init_message(problem, prompt_type='default', customized_prompt=None):

• You may choose from ['default', 'python', 'two_tools'] for parameter prompt_type. We include two more prompts in the paper:
  1. 'python' is a simplified prompt from the default prompt that uses Python only.
  2. 'two_tools' further allows the selection of Python or Wolfram Alpha based on this simplified python prompt. Note that this option requires a Wolfram Alpha API key and put it in wolfram.txt.
• You can also input your customized prompt if needed: mathproxyagent.generate_init_message(problem, customized_prompt="Your customized prompt"). Since this mathproxyagent detects ‘\boxed{}’ as termination, you need to have a similar termination sentence in the prompt: “If you get the answer, put the answer in \boxed{}.”. If the customized is provided, the prompt_type will be ignored.

Example 4 (Use the “python” prompt):

Problem: If 725�+727�=1500 and 729�+731�=1508, what is the value of �−� ?

Correct Solution: Subtracting the two equations gives:(729�+731�)−(725�+727�)=1508−1500⇒4�+4�=8⇒�+�=2.

Multiplying this equation by 725 and subtracting this equation from 725�+727�=1500 gives(725�+727�)−725(�+�)=1500−725(�+�)⟹2�=50.So we can write �−� as (�+�)−2�, which equals 2−50=−48.

Example 5 (Use the “two_tools” prompt)

Problem: Find all numbers � for which the graph of �=�2+� and the graph of �=�� intersect. Express your answer in interval notation.

Correct Solution: If these two graphs intersect then the points of intersection occur when [x^2+a=ax,] or [x^2-ax+a=0.] This quadratic has solutions exactly when the discriminant is nonnegative: [(-a)^2-4\cdot1\cdot a\geq0.] This simplifies to [a(a-4)\geq0.] This quadratic (in �) is nonnegative when � and �−4 are either both ≥0 or both ≤0. This is true for � in(−∞,0]∪[4,∞).Therefore the line and quadratic intersect exactly when � is in (−∞,0]∪[4,∞).