It's not just edge cases where it fails. It fails all the time at all kinds of things.
I've been using chatgpt in my work, but I have to essentially know the answer it's going to give me because I have to catch all of its mistakes. It really, really nice for certain kinds of drudge work.
Using northwind is probably not a good thing to use to evaluate chatgpt's general capability. It is very commonly used for examples of almost anything database-related, which means it's extremely well represented in chatgpt's training data. Chatgpt probably doesn't need to generalize or understand much of anything about northwind to answer your questions in terms of it. You need to try it on wacky problems specific to you.
> Using northwind is probably not a good thing to use to evaluate chatgpt's general capability.
I did think of that, which is why I modified the schema, and removed any reference to Northwind (normally products are prefixed with "Northwind Traders"). That said, it isn't a particularly complex schema in my example, but it's a reasonable starting point for something a small business might use.
I've been using chatgpt in my work, but I have to essentially know the answer it's going to give me because I have to catch all of its mistakes. It really, really nice for certain kinds of drudge work.
Using northwind is probably not a good thing to use to evaluate chatgpt's general capability. It is very commonly used for examples of almost anything database-related, which means it's extremely well represented in chatgpt's training data. Chatgpt probably doesn't need to generalize or understand much of anything about northwind to answer your questions in terms of it. You need to try it on wacky problems specific to you.