Experimenters passed 517 questions from Stack Overflow to ChatGPT to see what output itwould respond with. Around half theanswers, when interpreted semantically by humans. turned out to be incorrect.

This is to be expected, because ChatGPT is not intelligent and does not understand themeaning of text that it operates on. It can do more sophisticated text manipulation thatQuery Replace, but it has no more knowledge or understanding than Query Replace.