GenAlign: Visual Knowledge Graph Extraction for Design Generation

TL;DR

We ground design generation in which concrete properties a designer means by a word like “modern” — not a single guess from the model.

Designers describe goals with ambiguous adjectives (“modern”, “minimal”), but a generative model must pick one interpretation among many — and it silently reinterprets from turn to turn, so intent drifts. GenAlign externalizes each abstract attribute as inspectable segment–property bindings in a personal Visual Knowledge Graph (VKG) that updates in real time. The bindings act as stable anchors, so body→geometry→angular means the same thing in turn 1 and turn 10 — turning guess-and-check prompting into systematic, personalized property editing.

GenAlign in action: segment-guided exploration, VKG-conditioned generation, and property editing. — GenAlign in action — segment-guided exploration, a growing Visual Knowledge Graph, and property-conditioned generation that keeps intent aligned across iterations.

Abstract

Designers often describe goals using ambiguous adjectives (e.g., modern); however, generative AI systems struggle to navigate this ambiguity, creating misalignment between language, intent, and outcomes. When the intended meaning remains implicit and unstructured, design generation cannot be guided precisely, resulting in semantic drift. Rather than logging past user operations, we present GenAlign, a graph-backed, human-in-the-loop design support system that externalizes cognitive visual attributes as segment–property bindings in a visual knowledge graph (VKG). GenAlign externalizes abstract attributes as segment-property bindings and maintains them as inspectable, reusable, and personalized bindings through real-time VKG updates. In a within-subject study with 24 professional designers, GenAlign transformed abstract attributes into reusable property vocabularies, supported personalized design strategies, and reduced per-step semantic drift across iterations. We discuss how VKG functions as an operational boundary object, that bridges interpretive and executable design representations rather than eliminates ambiguity.

Overview

Four key innovations drive the GenAlign framework.

From part-level segmentation to property-based generation, each stage sharpens the system’s semantic grasp of a design.

Semantic Segmentation

Multi-granularity part-level segmentation using Semantic-SAM with Swin-L Transformer, extracting precise object boundaries and hierarchical part structures.

Property Extraction

GPT-4o Vision-powered extraction of visual attributes — material, color, style, and relationships — transforming visual features into structured semantic properties.

Knowledge Graph

Structured triple extraction following VisualSem ontology with 13 standard relations plus a custom “is-style-of” relation, building a cognitive graph of design elements.

Property-based Generation

Generate and edit designs by transferring or modifying properties in the knowledge graph — enabling precise control over material, style, and form.

System architecture

From a single product image to a structured, editable graph.

The pipeline

Five stages, from pixels to knowledge graph.

Each stage refines the semantic representation — segmentation, labeling, attribute extraction, triple construction, and post-processing.

1

Segmentation

Multi-granularity part extraction using Semantic-SAM, producing hierarchical masks with object/part labels (e.g., sofa-armrest, table-leg).

2

Segment Labeling

GPT-4o Vision analyzes each segment to assign semantic labels, understanding both the part identity and its role within the overall object.

3

Visual Attribute Extraction

Extract fine-grained properties including material (wood, metal, fabric), color, texture, shape characteristics, and functional attributes.

4

Triple Extraction

Build structured knowledge graph triples following VisualSem ontology: is-a, part-of, made-of, has-property, is-style-of, and more.

5

Post-Processing

Normalize properties, inject into the cognitive graph, and save VKG logs for analysis and generation tasks.

Visual Knowledge Graph extraction process — From image to Visual Knowledge Graph.

The full extraction process — from segmented parts to a densified graph of typed relations.

HBI-based extraction and image generation — From graph to image: History-Based Injection.

Property-grounded generation and editing, driven by the accumulated knowledge graph — only the most relevant bindings are injected into the generator.

Results

Generated & edited designs across domains.

Designs produced by transferring and modifying graph properties — furniture, posters, characters, footwear, and interiors.

Generation results across multiple domains

Property-conditioned generation with a fraction of the context.

The History-Based Injection (HBI) selection feeds the generator only the designer’s most relevant bindings. It reaches comparable CLIP similarity to injecting the whole graph while using 7.2× fewer properties — an 85.9% reduction (96.7 → 13.4 properties per generation).

7.2×

fewer properties injected

85.9%

reduction in context size

0.318

CLIP similarity (vs. 0.316 full graph, n.s.)

13.4

properties per generation

User study

Twenty-four professional designers, two systems, within-subjects.

Designers (M = 5.4 yrs experience) completed furniture & product tasks defined by conflicting attribute pairs — minimal × playful, modern × sleek — comparing GenAlign against a Baseline without VKG editing, in counterbalanced order.

RQ1 · Controllability

Decomposing abstract adjectives into segment–property bindings shifted interaction from guess-and-check prompting to systematic property editing. Perceived effectiveness rose sharply (3.54 → 6.29, p<.001).

Intent matching 3.21 → 6.31 · frustration 3.38 → 1.44

RQ2 · Personalization

A user-evolving VKG captured each designer’s own meaning of an attribute. Total generations rose 2.84× while saved images dropped 3.43× — a 0.98 reuse rate shows a growing personal vocabulary, not stockpiled examples.

Generation-to-save ratio 0.25 → 0.59 (2.35×)

RQ3 · Semantic stability

Explicit bindings act as stable anchors, so intent doesn’t drift turn to turn. Per-step semantic drift fell 22× (0.67 → 0.03) and graph rewiring 11× — yet overall diversity increased.

First-to-final drift unchanged — not a narrower search, a steadier path.

Comparison of log-based and VKG-based metrics between Baseline and GenAlign — **Figure 9.** (a) Log-based metrics — prompt count, saved images, ratio, total generations · (b) VKG-based metrics — triple/property dynamics, reuse, semantic drift, and graph change rate, Baseline vs. GenAlign.

Survey results.

Post-task ratings (7-point scale; CSI on a 10-point scale) across all 24 participants. GenAlign leads on every quality item, with lower frustration.

Item	Baseline	GenAlign	p
Freely express design intentions (Q1)	3.33	6.31	<.001
System understood my intentions (Q3)	3.42	6.15	<.001
Performance (NASA-TLX, Q8)	3.54	6.00	<.001
Frustration (NASA-TLX, Q10) · lower is better	3.38	1.44	.003
Satisfied with results (Q11)	3.88	6.42	<.001
Results are diverse (Q13)	4.50	6.44	<.001
Results are creative (Q14)	3.38	6.29	<.001
Understood attributes & connections (Q18)	2.71	6.46	<.001
Enjoyment (CSI, Q23) · /10	5.56	9.39	<.001

Values are means (n = 24). Significance after Holm–Bonferroni correction; Wilcoxon signed-rank except where a paired t-test applied. The largest gap is on understanding attributes and their connections (+3.75) — the structure itself, not just the outputs.

Case P6: minimal x playful lamp, role-splitting strategy through saved segment properties — **Case (P6).** Designing a “minimal × playful” lamp, the designer split the two attributes across roles — *minimal* as geometry, *playful* as color — a strategy that surfaced from inspecting saved segment–property bindings, not from re-prompting.

Case P8: modern x sleek sofa, connecting attributes via property transfer across segments — **Case (P8).** A “modern × sleek” sofa built by *transferring* properties across segments — a “thin metal” support not found in sofa was retrieved from a saved table, then carried into the design.

What the designers said

Semi-structured interviews surfaced why the numbers moved — the relief of editing one wrong property instead of re-prompting, and seeing the system’s interpretation made visible.

“In Baseline, I kept rephrasing prompts hoping the AI would guess my intent. With GenAlign, I could just adjust the specific property that was wrong.”

P14 · on controllability

“Baseline gave me random surprises — sometimes good, often frustrating. GenAlign gave me intentional variations — I knew why each output was different.”

P9 · on controlled exploration

Designing a “minimal × playful” lamp, P6 discovered role-splitting — minimal as geometry, playful as color — not by prompting, but by inspecting saved segment properties.

P6 · on making interpretation visible

The takeaway isn’t that GenAlign generates better images — CLIP similarity is comparable to injecting the whole graph. It’s that GenAlign makes intent explicit, traceable, and stable. By externalizing abstract adjectives as inspectable segment–property bindings, it lets designers build a reusable design vocabulary and explore along dimensions they chose — turning random variation into controlled, personalized exploration.

Video

👾GenAlign