Reverse Engineering Responses from Google Gemini / Chatgpt

      I'm in a galaxy far far away from being a LLM / AI / Machine Learning  expert, but (inspired by many posts I've seen on Linkedin) I've been doing some relatively simple prompting of Google Gemini / ChatGPT to try to (partially) "reverse engineer" the raw tools requests / "query fan outs" / "grounded" queries, personalisation & localisation etc used in generating responses for some prompts (for Gemini without using any 3rd party tools / plugins or looking at Chrome Developer Tools to analyse JSON files etc) . I'm looking mainly at "best"-type searches in my research hitherto (for ecom related searches etc).

   It appears that if you ask Gemini about how it "would" generate a response to a prompt, it will tell you with a certain degree of similarity to the actual response which you can uncover with the Chrome plugin "AI Search Fan-out Tracker" to see the query fan outs used by it . My 'blink' test comparing my method with using the AI Search Fan-out Tracker's response r.e. query fan outs in Google Gemini suggests my method can list 70%+ the "query fan out" searches used in generating "real" responses. Whether all "query fan out" searches are used in generating the response I don't know - my research is ongoing!

    Try the sample prompts below and let me know what you think! These tools can be erratic at times from what I see ...

The Techniques

Google Gemini

    Example prompt: 'show me all raw tool requests / query fan outs / web searches as well as any personalisation / geolocation you would use to generate the answer to the prompt “best marketing agencies” including all webpages you would visit to generate the answer' . You can run the prompt in different models & modes e.g. "Deep Research" to compare and contrast.

     I've compared the Query Fan-Out Analysis output from a similar prompt & response in Gemini to the info from the Google Chrome plugin "AI Search Fan-out Tracker" (https://chromewebstore.google.com/detail/ai-search-fan-out-tracker/nflpppciongpooakaahfdjgioideblkd - which does not appear to be working 100% properly for me at time of writing for Gemini) in Gemini to the response to the prompt on its own. There were 18/ 25 "query fan out" search queries reported by the plugin using my direct prompt in the Gemini interface - Gemini appears to answer directly some of the info using my method which the plugin can help uncover by examining the code of the response to the simple prompt e.g. "best marketing agency". Note the prompt and  response in the direct prompt contains more detail about Google's response than just the query fan out info from the Chrome plugin alone as the prompt is a more holistic question about the info used to generate the response.

    It is possible to do some similar prompting of Chatgpt, but my tests do not show a great degree of similarity to the 'real' responses tested with AI Search Fan-out Tracker etc. Again I'm not sure AI Search Fan-out Tracker is working 100% for me with Gemini and Chatgpt. This is research in progress...

Chatgpt

    The best way at the moment to try to reverse engineer Chatgpt's responses from what I can see is as follows. You can use Chatgpt / Google Gemini help you understand Chatgpt's output in a few quick steps:

1. Install "ChatGPT Path", a new free Google Chrome browser plugin which can reveal how ChatGPT generates a response, including the live searches it may make, its thought process, and  the data sources it uses.  https://chromewebstore.google.com/detail/chatgpt-path/kiopibcjdnlpamdcdcnphaajccobkban?authuser=0&hl=en

2. Input a prompt into ChatGPT which you like to better understand how ChatGPT forms its response to

3. Then download the output as a spreadsheet from the "ChatGPT Path" plugin

4.  prompt Chatgpt as follows:

• 'analyse for the attachment output for the query "[insert prompt]" from Chatpt including any"grounded" queries/keywords sent to Bing, ChatGPT's thought process and all the Sources (Web Pages) used to build the response.' Then upload the spreadsheet to Chatgpt before submitting the prompt.

• or prompt Google Gemini (in deep research mode if you so desire) to:

'analyse the following output for the query "[insert prompt]" from Chatpt including any "grounded" queries/keywords sent to Bing, ChatGPT's thought process and all the Sources (Web Pages) used to build the response.' Then upload the spreadsheet to Gemini before submitting the prompt.

    The response will give you a more comprehensible version of the spreadsheet analysed by a LLM – one must be mindful of the usual caveats about content generated by AI.

• Another related prompt (suggested by Chatgpt to assist my research!) that may be of use is: 

"simulate a full ChatGPT-style answer for for the query "[insert prompt]" using grounded Bing searches (with simulated search strings, site evaluations, and ranking logic)"

   You can request the analysis of the ChatGPT Path plugin output in whatever format works for you e.g. bullet points. 

   This is all a work in progress. Let know if I'm barking up the wrong tree (hallucinating wildly like the LLMs I'm using)...