As I child I was called preguntón (too inquisitive) and I had the ability to end anyone´s patience. My chained “why"s could seem endless. I´d say that hasn´t changed much but I´ve managed to control it. I know when I have to stop asking others and myself to not sink into an infinite loop. It happens when I work, too. “Why is Ansible throwing a MaxConcurrentOperationsPerUser error? Why is that MaxConcurrentOperationsPerUser behaving like a sequential counter? How is that counter defined in Windows?” though in this case I´m asking myself and have to find my own answers.

Using LLMs as study assistants, my experience

As a technology enthusiast, SRE and Tech lead I´m exposed to the current frenzy with generative AI in the form of the well-known LLMs. I don´t want to come accross as a guru trying to tell anybody how much of it its actually ground breaking and how much its hype. Certainly I´m not going to delve into the philosophical side of what is consciousness and the Chinese Room experiment. I´m just going to share my experience in a field that I don´t see discussed as much from the IT perspective.

Yes, of course GPT 3.5 has helped me with the basics of scripts in bash and pwsh. And I´ve changed REGEX or JSON validators for prompts where I ask it to do them for me so I achieve the desired end result. I´ve also asked it about things to see if I can reduce the time I spend googling or to just have to google a confirmation. But another use that I´m getting out of it is in my studying.

DISCLAIMER: Mi experience for this post is based entirely on GPT 3.5 as a representative LLM.

Leveraging a tool

Filled with curiosity I decided to try using LLMs as a study aid, to save me time and use it better. I want to see what this tool can do. I don´t want to be afraid of it, I want to see how it can make things easier for me. Mind you, notes and official documentation are still my primary source. I still verify and expand on more specific knowledge by going to search engines. I´ve used GPT-3.5 for three vluable things: starting from more fuzzy concepts and delving deeper, with as much backtracking as necessary (that dreaded chain of “whys” from children), asking for simple and custom exercises for practice, and using it as a conversational tool like a 3.5 version Rubber Duck that answers back to further solidify knowledge.

From fuzzy to concrete, with as many “whys” as needed

I said it, I think the “inquisitive” part of me comes from a tendency of wanting to dig into the most elemental aspect every time I gaing new knowledge. I learned many years ago that the limit of “why?” at the end of each explanation someone gave me was between three and four before they lost patience. GPT-3.5 has not lost its patience with me yet. Have you ever been studying something and realized you want more theoretical basis? Welcome to my life, too often. With GPT-3.5 I can ask questions and, at the very least, points me in a direction to search further.

It also works, as I said, to delve into the specifics of a topic. For example you can start asking about overriding in Python and end with examples of operator overloading with overriding their magic functions, through a dialogue in natural language rather than a string of searches or a click-fest on Stack Overflow links.

Custom practice exercises

I haven´t tried it with complex exercises because I don´t fully trust it and because for those I want humans that “go for the kill”. But when I want to solodify a concept after studying it, there´s nothing better than requesting a few customized exercises. “GPT-3.5, give me 10 exercises with lambda functions. Okay, make them a bit more complicated, mix them with the use of map()or filter(). I´m stuck on this one, help me solve it”.

LLM exercises

Rubberduck

It resembles the first point, but I want to make a disctintion because the idea is to engage in a dialogue to improve understanding of a concept or to solve a problem without directly asking for the solution. Something like: “I´m going to paste a code snippet, I don´t want you to tell me how it works yet. I´ll explain it step by step, and you´ll ask me questions to help me improve my explanation”

LLM rubber duck 1

LLM rubber duck 2

Here it behaves like a good rubber duck, which also provides corrections. We can se that I´m not such a good teacher, and it has encouraged me to be more careful with the terms I use. It´s right, in list comprehensions there are no “function calls”, there are operations, which may include function calls. Also, it quickly returns to its role of “I am the one who gives the answers”, if it was a human, it would even look arrogant.

The risks

All that sounds great, but besides the advantages we need to talk about what GPT-3.5 doesn´t do that well. I´ve come across some risk that can detract from the value of learning with the aid of LLMs. I´ve divided them in a couple of categories:

The LLM is plain wrong

Not all answers are correct or real, or the ones it would produce if you ask again or tune the prompt. That´s why I think its ideal to use it to solidify and deepen concepts studied in other sources. It can also make mistakes when correcting your answers, so its best to ask for exercises that can be checked in another way (this is quite simple when studying programming languages, fortunately)

LLM hallucination1

Most of the time, with a prompt requesting a reevaluation of the answer, it will correct itself. But I wouldn´t fully trust it!

The LLM propagates biases from the content it´s been trained on

Why would GPT-3.5 use examples of less-than-ideal practices in its examples? My answer to that is that its based on the content the LLM has been trained on. That is, human-generated content, and I´m afraid much of that content does may not follow best practices.

LLM bias

Here we can see GPT-3.5 uses from module import class in the example, which is totally valid but not recommended since it´s imported into the current namespace and can lead to confusion or even collisions (what if we had defined our own reduce?) It works, but what is clear is that its not the best example to give tosomeone who is getting acquainted with the language.

In the recent Tecnologica Santa Cruz, Mariano Sigman stated as the sole tesis of his talk that AI is eminently human creation, and I agree. Unintentionally, due to the training material we provide them, LLMs show a very human side with our same cognitive biases and errors.

Excessive dependency

In this I´m going to sound old-school, but the biggest risk for me is relying too much on a specific technology. It can help us complete tasksfaster, or at least get them started faster. That´s good. But we have to continue appreciating our own ability to think, to get bored of studying and to endure feeling stuck until we understand something or find the solution to a problem. That time is valuable. If it serves no other purpose, it at least trains us into patience and tolerance for frustration. And that´s valuable for any job, but I dare to think it´s even more so for IT. This risk borders on the philosophical, I know. But its something that seems to be happening as technology changes our habits and even our abilities. There are plenty of clues that we are losing attention span due to technology.

Conclusions

The main point is that this post is not intended to be a comparison or in-depth investigation and is based solely on my experience with a fairly limited (yet representative) subset of text-based LLMs. The second point is that this is moving fast. Still, I´ll try to comment on some things that I believe can be generalized. If you want to learn from scratch or reinforce knowledge, it works, but in both cases you´ll have to verify everything. For me, ideally it´ll be used to solidify concepts that are already studied elsewhere more… reliable. Its also great to generate primers as well as exercises that help solidify concepts.

And lastly, I´ll leave here a nerdy movie quote that sums up what I think abouy being able to ask “whys” to a machine in natural language ad nauseaum without anyone or anything losing patience:

Watching John with the machine, it was suddenly so clear. The terminator would never stop. It would never leave him, and it would never hurt him, never shout at him, or get drunk and hit him, or say it was too busy to spend time with him. It would always be there. And it would die to protect him. Of all the would-be fathers who came and went over the years, this thing, this machine, was the only one who measured up. In an insane world, it was the sanest choice –Sarah Connor (Terminator 2)