Reverse Engineering Responses from Google Gemini / Chatgpt

By Chris Byrne. Published 31.6.25 (updated 10.7.25).

I'm somewhere in a galaxy far far away from being a LLM / AI / Machine Learning expert, but (inspired by many posts I've seen on Linkedin) I've been doing some relatively simple prompting of Google Gemini / ChatGPT to try to (partially) "reverse engineer" the raw tools requests / "query fan outs" / "grounded queries", personalisation & localisation etc used in generating responses for some prompts (for Gemini without using any 3rd party tools / plugins or looking at Chrome Developer Tools to analyse JSON files etc) . I'm also looking at the information contained in the LLM's response, e.g. as part of my research r.e. ChatGPT in its feedback stated that it CAN but does not critically analyse by default the web content that it may synthesise to create a response unless if their is a clear indication that it is promotional e.g. a blog marked as "Advertisement" .

I'm looking mainly at "best"-type searches in my research hitherto (including ecom / services-related searches etc). This is all research in progress...

It appears that if you ask Gemini about how it "would" generate a response to a prompt, it can tell you with a certain degree of similarity to the "real response" in relation to the query fan outs (which you can uncover with the Chrome plugin "AI Search Fan-out Tracker" to see the query fan outs used in the "real response"). And similar for Chatgpt, in terms of recommended products for a "best" type search.

For Chatgpt there is Chatgpt Path, a free Chrome plugin which can reveal how ChatGPT generates responses, including the live searches it makes (with Bing), its 'thought processes', and the data source(s) it used. You can put the output from this plugin into Chatgpt to make it more comprehensible!

My 'blink' test comparing my method with using the AI Search Fan-out Tracker's response r.e. query fan outs for one prompt in Google Gemini suggests my method can list c.70%+ the "query fan out" searches used in generating "real" responses. Whether all "query fan out" searches are used in generating the response I don't know - my research is ongoing!

Try the sample prompts below and let me know what you think! These tools can be erratic at times from what I see...

The Techniques

Google Gemini

It appears from my research that the "Gemini model, in its fundamental training process, does not retain direct "access" to the original raw data, including specific names or URLs of sources, in a way that it can directly retrieve them like a database." Thus reverse engineering the information used in responses from the training corpus of Gemini is not possible to the best of my knowledge.

Example prompt: 'show me all raw tool requests / query fan outs / web searches as well as any personalisation / geolocation you would use to generate the answer to the prompt “best marketing agencies” including all webpages you would visit to generate the answer' . You can run the prompt in different models & modes e.g. "Deep Research" to compare and contrast.

I've compared the Query Fan-Out Analysis output from a similar prompt & response in Gemini to the info from the Google Chrome plugin "AI Search Fan-out Tracker" (https://chromewebstore.google.com/detail/ai-search-fan-out-tracker/nflpppciongpooakaahfdjgioideblkd - which does not appear to be working 100% properly all of the time for me at time of writing for Gemini) in Gemini to the response to the prompt on its own. There were 18/ 25 "query fan out" search queries listed in the "real response" by the plugin using my direct prompt in the Gemini interface - Gemini appears to answer directly some of the info using my method which the aforementioned plugin can help uncover by examining the code of the response to the simple prompt e.g. "best marketing agency" which suggests the information from the answer my prompt correlates to a large degree (in some aspects at least) to the "real answer" to the "real prompt".

Note the my prompt and response it contains more detail about Google's response than just the query fan out info from the Chrome plugin alone as my prompt is a more holistic question about the info used to generate the response.

Chatgpt

Note that "After the training process is complete, the AI model does not retain access to data analyzed in training", thus reverse engineering the information used in responses from the training corpus of Chatgpt is not possible to the best of my knowledge.

To the best of my knowledge a good way at the moment to try to reverse engineer Chatgpt's responses from what I can see is as follows. You can use Chatgpt / Google Gemini help you understand Chatgpt's output in a few quick steps:

1. Install "ChatGPT Path", a new free Google Chrome browser plugin which can reveal how ChatGPT generates a response, including the live searches it makes, its 'thought process', and the data sources it used. https://chromewebstore.google.com/detail/chatgpt-path/kiopibcjdnlpamdcdcnphaajccobkban?authuser=0&hl=en

2. Input a prompt into ChatGPT which you like to better understand how ChatGPT forms its response to

3. Then download the output as a spreadsheet from the "ChatGPT Path" plugin

4. prompt Chatgpt (in deep research mode if you so desire) as follows:

'analyse for the attachment output for the query "[insert prompt]" from Chatpt including any "grounded queries" / keywords sent to Bing, ChatGPT's thought process and all the Sources (Web Pages) used to build the response.' Then upload the spreadsheet to Chatgpt before submitting the prompt.
or prompt Gemini (in deep research mode if you so desire) to:

'analyse the following output for the query "[insert prompt]" from Chatpt including any "grounded queries" / keywords sent to Bing, ChatGPT's thought process and all the Sources (Web Pages) used to build the response.' Then upload the spreadsheet to Gemini before submitting the prompt.

You can request the analysis of the ChatGPT Path plugin output in whatever format works for you e.g. bullet points.

The response can give you a more comprehensible version of the spreadsheet analysed by a LLM – one must be mindful of the usual caveats about content generated by AI.

Another method (with less accurate data from what I see) is to submit a prompt, then respond with "show me how the previous answer was generated, including all ("grounded") queries sent to Bing, ChatGPT's thought process and all the Sources (Web Pages) and any personalisation / localisation used to build the response.”

Remove from Chrom

Another related prompt (suggested in part by Chatgpt to assist my research!) that may be of use is:

"simulate a full ChatGPT-style answer for for the query "[insert prompt]" using grounded Bing searches (with simulated search strings, site evaluations, ChatGPT's thought process and all the Sources (Web Pages) and any personalisation / localisation used to build the response and ranking logic"

My test shows the above method can shows a certain degree of correlation with the real results in terms of recommended products for a "best" query: see below (List A being the actual response for the direct prompt in the test, list B the results given by using my method above): list A had 9 results in total and list B 7 results in total

This is all a work in progress - I'm trying to learn / (& learn how to) test as I go. Let me know if I'm barking up the wrong tree in the wrong forest (hallucinating wildly like the LLMs I'm using)...