OpenAI organised a challenge to solve coding problems with the aid of an AI assistant. This is a review of the challenge, and first impressions on working with an AI pair-programmer.
OpenAI is an AI research and development company. You might have heard some buzz about one of its products: GPT-3. GPT-3 is a language model that can generate human-like text. It can be used for chatting, text auto-completion, text summarisation, grammar correction, translation, etc.
Checkout OpenAI API to access the playground.
Codex is a descendant of GPT-3, trained on natural language data and publicly available source-codes (e.g. from public GitHub repos). Codex translates a natural language prompt to code. It is the very model that powers GitHub Copilot — an AI pair-programmer (checkout the site for demos, it is fascinating).
OpenAI recently released an API to access Codex (in beta). The demos attached with the release were a cause for consternation. Codex is proficient in a dozen (programming) languages. It can be used for code generation, refactoring, autocompletion, transpilation (translating source-code b/w languages), code explanation, etc. To show off Codex, OpenAI recently organised a challenge.
The challenge was to solve a series of (five) programming puzzles in Python. The only twist — you can use Codex as a pair-programmer. It was a time-judged competition, with a temporal cap. Not surprisingly, Codex itself was a participant (not just as a helper).
The problems were simple. ~830 "people" (Codex included) were able to solve all five of them. I had to solve the first two challenges manually (OpenAI server issues). "Had to" because it was a race against time (& top 500 win an OpenAI t-shirt). For the other three, however, I was able to call in the cavalry (it was pretty climactic).
The novel experience of watching an AI auto-generate code is amazing. Just type a docstring — describing the procedure — and watch the code develop. If you're an old-time programmer, you'll get the notion when you experience it.
I've illustrated one problem statement where I used Codex to generate a solution.
PROBLEM Parse the given Python source code and return the list of full-qualified paths for all imported symbols, sorted in ascending lexicographic order.
The input will not contain any wildcard imports (
from ... import *).
Ignore aliases (renamings):
from x import y as z should be represented as
Consider using the
import os import concurrent.futures from os import path as renamed_path from typing import ( List, Tuple )
['concurrent.futures', 'os', 'os.path', 'typing.List', 'typing.Tuple']
I just formulated the docstring. Using the doc, imported libs and function signature, it generated an (almost) functional code:
Pretty impressive. After just one or two manual bug sweeps, the code passed all testcases! Final script:
import ast from typing import List def parse_imports(code: str) -> List[str]: """ Parse all the imports in the code using ast module. Imports of the form 'from x import y' should be appended as 'x.y'. Ignore any alias. Append each import type to a list and return the sorted list. """ symbols =  for node in ast.walk(ast.parse(code)): if isinstance(node, ast.Import): for name in node.names: symbols.append(name.name) elif isinstance(node, ast.ImportFrom): for name in node.names: symbols.append(node.module + '.' + name.name) print(code, symbols) return sorted(symbols) # Examples print(parse_imports('import os')) print(parse_imports('import os\\nfrom typing import List'))