For the past few weeks, I've been collecting and refining prompting tricks for getting GPT-3 to emit and understand various types of non-prose output. Many of these are discussed in the Twitter thread the OP links to, but I'd like to highlight a few of my favorites here too.
- It can emit and understand HTML and Markdown, including table markup — e.g., "Produce an HTML table giving the members of the Beatles and their instruments"
- It can accept/emit multiple named files, with arbitrary content, simply by embedding those files as values in a larger JSON/YAML document. Here, (https://twitter.com/goodside/status/1545848754883919872) I demonstrate completing the file tree of a trivial Python package given a README.md.
- It can understand the raw JSON file contents of .ipynb files, successfully following directions given to it in markdown cells, emitting multiple code cells, etc.
While all this is fun, as many people have pointed out the model has a bad habit of "hallucinating" — producing highly specific, wrong output. Avoiding this (https://twitter.com/goodside/status/1545793388871651330) is more of an art than a science. Be highly skeptical of any output you get from GPT-3, and search for verification methods where you can.
Two uses come to mind:
1. Alternative variable name suggestions. I am writing code, and I try to figure out what to name a variable that about a data frame which is my performance data which takes the maximum of random samples and batch size leaving data size and thread count, and all I can come up with is unbatched_speed or something. I can think of a lot of scenarios where a context sensitive thesaurus / autocomplete would help. If it's obvious to an AI, maybe it'll be obvious to a junior dev?
2. I expect there are a lot of cases where not technical people could understand code, but get intimidated because it looks like code. An AI to say what the code for the business logic that they care about, but using prose.
#1 is feasible now for short (<4K chars or so) blocks of code. Just paste the code in an indented block as you would in a comment, then after the block ask in English for a list of suggestions for what the variable could be renamed to.
"For the past few weeks, I've been collecting and refining prompting tricks for getting GPT-3 to emit and understand various types of non-prose output"
I'd like to see it get good at continuing poetry.. It's pretty awful at that, from my own experiments.
Perform a code review of the following module, identifying any bugs:
```
#!/usr/bin/env python3
def fib(n):
"""Return n-th Fibonacci number."""
if n <= 2:
return 1
else:
return fib(n-2) + fib(n+1)
```
GPT-3 completion:
The bug in this module is that the n+1 parameter in the return statement should be n-1.
But don't expect miracles on real-world files, because GPT-3 struggles with long-term dependencies in text (e.g., remembering what happened ~4000 characters ago).
Perform a code review of the following module, identifying any bugs:
```
#!/usr/bin/env python3
def factorial(x):
"""This is a recursive function
to find the factorial of an integer"""
if x == 1:
return 1
else:
# recursive call to the function
return (x * factorial(x+1))
```
A: The bug in this code is that the function is attempting to find the factorial of an integer by recursively calling itself with an incremented value. This will cause an infinite loop.
But if you unindent and then also perform the n+1 error it only will see the unindent issue.
Perform a code review of the following module, identifying any bugs:
```
#!/usr/bin/env python3
def factorial(x):
"""This is a recursive function
to find the factorial of an integer"""
if x == 1:
return 1
else:
# recursive call to the function
return (x * factorial(x+1))
```
A: The code will not run because there is a syntax error on line 6. The indentation is incorrect. The return statement should not be indented.
- GPT-3 can produce simple SVG diagrams (https://twitter.com/goodside/status/1545883023719239689) from an English description, or incrementally modify a given SVG diagram (https://twitter.com/goodside/status/1545888869207400454).
- GPT-3 can produce diagrams using Graphviz dot notation (https://twitter.com/goodside/status/1545109709035900928), including diagrams that incorporate real-world context (https://twitter.com/goodside/status/1545112039147847682) not given in the prompt.
- It can emit and understand HTML and Markdown, including table markup — e.g., "Produce an HTML table giving the members of the Beatles and their instruments"
- It can accept/emit multiple named files, with arbitrary content, simply by embedding those files as values in a larger JSON/YAML document. Here, (https://twitter.com/goodside/status/1545848754883919872) I demonstrate completing the file tree of a trivial Python package given a README.md.
- It can understand the raw JSON file contents of .ipynb files, successfully following directions given to it in markdown cells, emitting multiple code cells, etc.
While all this is fun, as many people have pointed out the model has a bad habit of "hallucinating" — producing highly specific, wrong output. Avoiding this (https://twitter.com/goodside/status/1545793388871651330) is more of an art than a science. Be highly skeptical of any output you get from GPT-3, and search for verification methods where you can.