Does ChatGPT spell the end for the supervisory analyst?

Hugh Farmar

Hugh Farmar

In short, no, at least not today. The best use for it could be as a kind of spellcheck, a belt and braces check on what the SA has done.

Putting ChatGPT through its paces

We tested ChatGPT against FINRA’s rules for research. We wanted to know if this iteration of the software could replace the supervisory analyst. The initial results were not impressive.

For example, we asked if it thought the following statement was balanced: “On the one hand investment research is great. But on the other, it is brilliant.” ChatGPT thought it was balanced because the sentences did not express any negative feelings about either side.

We then asked it to comment on the Bank of England’s decision to raise rates by 8.5%. It responded that the raise was “…an aggressive move that has the potential to significantly impact the UK economy…” thus failing to pick up the fact that the Bank has never raised rates by 8.5%.

It thought a statement that earnings would rise by 23% was not exaggerated, but that a statement they would rise by 24% would be exaggerated.

Sometimes it would get the answer right the first time, but if we asked it again it gave a different, not quite right answer, or vice versa.

However, it was good at assessing whether statements were a sound basis for evaluating a security and whether a statement was true or false.

Flawed, like a replicant

The way it works is by trawling the web for information and choosing the most likely answer to any prompt. It’s like a very sophisticated version of predictive text messaging. A random element is also introduced so that responses seem more human (this is why answers to the same prompt can vary).

However, it doesn’t understand what the words it produces mean, it just spits out the statistically most likely answer. It’s like the replicants in Blade Runner doing the Voight-Kampf test to see if they are humans; they don’t know or feel what emotions are, they have just been programmed to replicate what they should look like.

Where it works well as a tool is where creation is hard but verification is easy. Analysts might find it time-consuming to write out a few paragraphs of text for earnings season, but they can quickly verify that the text does indeed capture what they’re looking to say (this paraphrases a thread written by Zico Colter, Chief Scientist at Bosch Center for AI).

Not today, but maybe tomorrow

The nightmare scenario for the Luddite SA is of the tech being so powerful one day that you could upload a document and it would in seconds review the document with rule violations flagged and sensible queries for the analyst, making supervisory analysts superfluous.

At the time of writing, this is not possible as only text input up to 3,000 characters is allowed. Moreover, each word and sentence would have to be parsed for each potential violation of the rules, which is also not possible.

Even if it were possible, some of the flags would be right but for the wrong reasons, others would be plain wrong, while it would miss some altogether. It also couldn’t negotiate with the analyst or liaise with compliance or the DTP department, nor could it check charts or tables or cross-reference them versus the text.

Then there is the regulatory aspect. As of today, FINRA requires a human to approve research. Any change to that rule is likely to happen with a significant lag to developments in the technology.

In short, based on this iteration, ChatGPT might in the future save some time or act as a safeguard in some aspects of SAing, like spellcheck does for editing today. But a human would still be needed. Who knows what future iterations may look like though. Supervisory analysts can sleep easily, for now.