OpenAI has introduced a special version of GPT-3 that can create images based on descriptions. DALL-E uses a data set of text-image pairs and should be able to correctly interpret more or a few arbitrary combinations. The system can create images from scratch or modify existing images.

The project name is a suitcase word from the surname of the Spanish artist Salvador Dali and des Pixar film “WALL-E”. It is a version of the Generative Pre-trained Transformer 3 (GPT-3) language model with 12 billion parameters.

Wild combinations and humanization The blog post on DALL-E shows some examples, some of which have impressive results , but also point out errors. The inputs range from obvious texts such as “a shop window with the writing openai” (“a store front that has the word ‘openai’ written on it”) or “a small red building block that lies on a large green building block” (” a small red block sitting on a large green block “) to bizarre descriptions like” a drawing of a baby radish in a pink tutu walking a dog “(” an illustration of a baby daikon radish in a tutu walking a dog ” ) or “a snail made of harp”.

With the snail harp, DALL-E proves definitely AI creativity.

(Image: OpenAI)

Some of the descriptions use anthropomorphism, i.e. the humanization of animals or objects, in order to achieve artificial to artistic results. In addition, DALL-E is probably able to translate images into another form, for example to create a sketch from a photo of a cat.

DALL-E offers access to a 3D rendering Engine uses natural language and can precisely control the lighting conditions or angles. For more complex descriptions like “An emoji of a baby penguin wearing a blue hat, red gloves, green shirt, and yellow pants” ) the system is probably wrong with the correct color assignment for some issues.

Automatically classified with CLIP The blog post shows numerous examples and uses another new tool from OpenAI for the optimal selection of images: CLIP (Contrastive Language-Image Pre-training) is an artificial neural network that converts visual concepts into categories. Like GPT-3, it relies on zero-shot learning (ZSL) to detect objects that were not classified during network training.

OpenAI has CLIP for a reranking of those with DALL-E Created images used to create the top to investigate. According to the article on DALL-E, however, manual “cherry picking” did not influence the selection.

DALL-Es imagining a radish with Tutu and dog

(Image: OpenAI)

In addition to the creative experiments, DALL-E probably took some basic geographic understanding with him during the training, for example to create plausible chains of houses in San Francisco that do not exist in reality. The system can also correctly assign flags or dishes. The blog post admits, however, that it falls back on individual stereotypes, especially with culinary dishes and the fauna of certain countries.

Further details and a list of the work on which the project is based can be obtained from the OpenAI -Blog. For the near future, the team behind DALL-E is planning a more detailed study of how the model affects social issues. It also wants to examine the problem of bias, which occurs again and again in machine learning, i.e. distortions caused by prejudices in training and the ethical challenges of technology. (rme)