Thesis Description & Objectives
Despite advancements in visual fidelity, current generative text-to-image models struggle with precise control over generated images, especially in response to detailed prompts and directional cues. While current generative text-to-image latent diffusion models have reached unprecedented results in terms of visual fidelity, there are still open issues to be addressed in exerting precise control over the generated images. On the one hand, generative models have difficulty in creating correct images when the textual prompt contains many details and often with object placement and spatial awareness. Recent text-to-image latent diffusion models have shown substantial improvements in prompt following, yet still struggle with the use of words such as “left” or “behind”. One possible reason lies in the inherent limitations of the text embedding employed to condition the generation process, that fails to learn sufficiently detailed and disentangled representation.
We are developing a new set of architectures based on scene-graph-to-image conditioning, to validate if a structured input overcomes some of the limitations of unstructured text. The proposed architecture exploit graph neural networks instead of visual language models, such as CLIP, to guide the generative process.
Thesis topics available in this context follow three research lines. First, investigate the impact of data quality on scene-graph-to-image generation. Particularly relevant is the adaptation of text-to-image benchmark to the SG-to-image setting by automatically translating text captions to SG, solving a bottleneck of the current approach. The thesis activities will involve one or more of the following: i) develop or train LLMs to convert textual descriptions to scene graph representations; ii) create and extend data pipelines, including data augmentation, for training generative models, iii) validate experimentally the impact of data quality on the generative models through established benchmarks for visual reasoning.
The second topic investigates the impact of different architectural choices and their impact on the quality of the generated images. The thesis activities will involve one or more of the following: i) develop variants of the proposed architecture to, e.g., understand the properties of different encoding, GNN architectures, in collaboration with other students; ii) perform ablation studies and iii) adapt the pipeline to other benchmarks, e.g., for visual and spatial reasoning.
The third topic will compare layout-free with layout-based methods for SG-to-image generation. Layout-based methods condition the image generation on a spatial layout formed by bounding boxes of objects. It forces stronger constraints on the spatial arrangements of objects, but disregard relationships and attributes. The thesis activities will involve one or more of the following: i) automatic layout generation from scene graph or textual description; ii) Implementing and testing existing layout-to-image architectures on our datasets; and iii) extending our current SG-to-image pipeline integrating bounding boxes in the scene graph representation.
