Mastering Prompts in Stable Diffusion: The Ultimate Guide

Table of Contents

1. Anatomy of a Good Prompt
2. Negative Prompt
3. Process of Building a Good Prompt
4. Prompting Techniques
5. Consistent Face
6. How Long Can a Prompt Be?
7. Checking Keywords
8. Association Effect
9. Embeddings as Keywords
10. Effect of Custom Models

---

1. Anatomy of a Good Prompt

A well-constructed prompt should be detailed, specific, and consider various aspects that influence the final image. Here are the key components, expanded with more examples:

Subject

The subject is the main focus of your image. A detailed description of the subject leads to more accurate results.

Examples:

• Basic: “A sorceress.”
• Enhanced: “A powerful, mysterious sorceress casting lightning magic, wearing a detailed leather outfit with gemstones, sitting on a rock in front of a castle.”
• Other Variations: “A knight in shining armor, riding a white horse,” “A futuristic robot with glowing blue eyes,” “A serene Buddha statue surrounded by lotus flowers.”

Medium

This refers to the style of the artwork, like illustration, oil painting, or 3D rendering.

Examples:

• “Digital art”
• “Oil painting”
• “Watercolor illustration”
• “3D rendering”
• “Charcoal sketch”

Style

Describes the artistic style, such as hyperrealistic, fantasy, or dark art.

Examples:

• “Hyperrealistic”
• “Fantasy”
• “Dark art”
• “Surrealism”
• “Abstract”
• “Minimalist”
• “Pop art”
• “Impressionist”

Art-Sharing Website

Mentioning websites like Artstation or DeviantArt can guide the style toward those found on these platforms.

Examples:

• “Artstation”
• “DeviantArt”
• “Pixiv”
• “Behance”
• “Instagram art”

Resolution

Describes the sharpness and detail of the image.

Examples:

• “Highly detailed”
• “Sharp focus”
• “Ultra HD”
• “4K resolution”
• “Fine details”
• “Crisp edges”

Additional Details

Any extra elements that can enhance the image, like sci-fi or dystopian themes.

Examples:

• “Sci-fi”
• “Dystopian”
• “Cyberpunk”
• “Steampunk”
• “Post-apocalyptic”
• “Mythological”
• “Historical”
• “Fantasy world”

Color

Specifies the dominant colors or tones.

Examples:

• “Iridescent gold”
• “Vibrant red”
• “Monochrome”
• “Pastel shades”
• “Neon colors”
• “Earth tones”
• “Sepia”
• “Cool blues”

Lighting

Describes the type of lighting, such as studio lighting or natural lighting.

Examples:

• “Studio lighting”
• “Natural lighting”
• “Dramatic lighting”
• “Soft lighting”
• “Backlighting”
• “Candlelight”
• “Moonlight”
• “Harsh shadows”
• “Golden hour”
• “Twilight glow”

---

2. Negative Prompt

Harnessing the Power of Exclusion in Stable Diffusion

Negative prompts play a pivotal role in refining and controlling text-to-image generation by specifying what you don’t want in your image. Here are more examples:

Entering Negative Prompts

Most Stable Diffusion interfaces, such as AUTOMATIC1111, provide a separate input box for negative prompts.

Use Cases for Negative Prompts

Removing Unwanted Elements

Example: To remove people from a street scene in Paris, simply add “people” to the negative prompt.

Enhancing the Exclusion: Increase the emphasis on the negative prompt (e.g., “people:1.3”) to make certain elements even less likely to appear.

Subtle Modifications

Examples:

• “Windy”
• “Underage”
• “Smiling”
• “Glasses”

Keyword Switching for Negative Prompts

Examples:

• Start with a less impactful negative prompt and switch to a more specific one at a later step.

Style Modifications

Examples:

• “Blurry” to sharpen images
• “Painting” and “cartoon” to add photorealism

Importance in Newer Models

Negative prompts are especially important in SDXL for improving aesthetics.

Crafting a Universal Negative Prompt for v2 Models

A comprehensive negative prompt for v2 models might include:

Examples:

• “Ugly”
• “Disfigured”
• “Deformed”
• “Tiling”
• “Poorly drawn hands/feet/face”
• “Extra limbs”
• “Out of frame”
• “Bad anatomy”
• “Watermark”
• “Signature”
• “Cut off”
• “Low contrast”
• “Underexposed”
• “Overexposed”
• “Bad art”
• “Beginner”
• “Amateur”
• “Distorted face”

IMPROVE FOR SDXL!!!

---

3. Process of Building a Good Prompt 🛠️

Prompt building should be iterative. Start simple and gradually add details based on the outputs.

Iterative Prompt Building

Creating an effective prompt in Stable Diffusion involves several key strategies to ensure that your AI produces exactly what you have in mind. Here’s a step-by-step guide:

1. Begin with Basic Prompts:
   • Basic: “A serene lake at sunset.”
2. Iterative Refinement:
   • Initial: “A serene lake at sunset.”
   • Refined: “A serene lake reflecting the fiery hues of sunset, bordered by shadowy hills.”
   • Further Refined: “Peaceful lake surrounded by darkened ridges reflecting the fiery color of sunset, with a secluded wooden dock.”
3. Using Negative Prompts:
   • “Setting sun over a peaceful pond – Negative Prompt: people, too cartoonish.”
4. Keyword Weighting:
   • “The peaceful (lake:1.5) at sunset, (mountains:0.5).”
5. Adjusting Strength with () and []:
   • “Sunset at a ((lake)), [mountains].”
6. Keyword Blending:
   • “Portrait of [Emma Watson: Amber Heard: 0.85], dramatic lighting.”
7. Guiding Image Composition:
   • “(Low angle view), A wizard at right location casting spell towards left under the starry sky.”
8. Art Style Experimentation:
   • “Wise wizard renaissance portrait.”
9. Creating Variations:
   • “A Wizard reading a spell book, change lighting from morning to twilight.”
10. Order and Chunking Prompt:

• Rearrange tokens to prevent color bleeding on adjacent elements.

Remember, effectively crafting prompts in Stable Diffusion is as much an art as it is a science. With increased experience, you will understand better how to create prompts that bring about desired results.

---

4. Prompting Techniques

Creating effective prompts for Stable Diffusion is an art that combines creativity with technical acumen. Here’s a practical, example-laden guide:

Keyword Weighting and Strength Adjustment 🏋️‍♂️

Synopsis:

This technique adjusts the weight on selected important elements within your picture.

Example:

If you want to draw a park in autumn but focus less on the animals:

• “park, autumn (the squirrel:0.5), (the bird:0.5).”

Strength Adjustment with () and [] Syntax

Example: To emphasize autumnal aspects:

• “(park) ((autumn)) (squirrel:0.5) (bird:0.5).”

Keyword Blending and Prompt Scheduling

Overview:

Blend two concepts or themes seamlessly over the image generation process.

Example:

For a cityscape transitioning from day to night:

• “Cityscape [daytime: night: 0.5].”

Descriptive, Short Prompts ⚡

Summary:

Very detailed yet descriptive and compact.

Example:

• Well: “A busy street with lots of people, cars, bright lights, and shops” becomes “Bustling city street at dusk, illuminated by neon signs and streetlights.”

Style Transfer

Add stylistic texture reminiscent of your favorite artists or movements.

Examples:

• “Paris street in post-impressionism Van Gogh.”
• “Landscape in the style of Claude Monet.”
• “Portrait with a touch of Renaissance.”

Super-Resolution

Enrich images with more details and clarity.

Examples:

• Refine an image using a super-resolution tool like HiResFix for clarity.
• Use upscaling options to enhance details.

Prompt Engineering 🎭

Apply quick engineering tactics to explore new artistic directions.

Examples:

• “Produce a digital painting of a futuristic cityscape that encompasses elements of both cyberpunk and art deco styles.”
• “A whimsical fairy tale scene with a surreal twist.”

Continuous Learning and Experimentation

Stay up to date on innovations and fresh AI art methods.

Resource Recommendation:

• Participate in community discussions and read academic papers regularly.
• Forums: /r/stablediffusion, /r/comfyui, etc.

Remember, mastering prompts in Stable Diffusion involves understanding language nuances, AI capabilities, and effectively communicating your creative vision.

---

5. Consistent Face

Developing guidelines for maintaining character consistency in Stable Diffusion can be verbose. Below is an improved approach:

Use Multiple Celebrity Names for Blended Faces ⭐

Blending features of different celebrities achieves a generic yet consistent face.

Prompt Example:

• “(Emma Watson:0.5), (Tara Reid:0.9), (Ana de Armas:1.2), photo of a young woman, highlighted hair, sitting outside a restaurant.”

Determining Weight of Celebrity Influence 🧐

Understand the influence each celebrity brings based on training data.

Adjusting the Weights 🏋️

Reduce or increase the weight to balance facial features.

Prompt Adjustment:

• “(Emma Watson:0.3), (Tara Reid:1.0), (Ana de Armas:1.2), young woman with a serene expression, sitting in a café.”

Precise Attribute Specification

Detail specific attributes to maintain consistency.

Examples:

• “Young woman with almond-shaped green eyes, arched brown eyebrows.”
• “Man with a square jawline, short black hair, and piercing blue eyes.”

Consistency in Emotional Expression and Pose 😃

Specify emotional expressions or poses to keep the image consistent.

Prompt Examples:

• “Serene expression, slight smile, relaxed posture.”
• “Confident stance, determined look, hands on hips.”

Reiteration of Key Terms for Pointing Out 🔁

Re-emphasize attributes to discourage non-uniform outcomes.

Prompt Examples:

• “highlight hair, highlight hair, sitting outside restaurant, wearing dress, rim lighting, studio lighting.”
• “Freckles, freckles, red hair, blue dress, soft lighting.”

Keep It Simple 🤓

Focus on the most important attributes and avoid unnecessary details.

Simplification Example:

• Instead of: “Young woman with detailed makeup, wearing a blue scarf, gold earrings, a white blouse, and a black skirt, sitting by a window.”
• Simplified: “Elegant young woman with a blue scarf and a white blouse, seated by a window.”

This guide focuses on maintaining character consistency in AI-generated images using Stable Diffusion.

---

6. How Long Can a Prompt Be?

Different versions of Stable Diffusion have varying limits on prompt length, typically around 75 tokens.

Prompt Limit in AUTOMATIC1111

The maximum length of a Stable Diffusion prompt varies around 75 to 77 tokens. In AUTOMATIC1111, if a prompt exceeds this limit, it starts a new chunk of another 75 tokens, processing each chunk independently and then concatenating the results.

Starting a New Prompt Chunk aka BREAK

The BREAK keyword in AUTOMATIC1111 helps manage long prompts by separating specific elements.

Color-Specific Objects

Example:

• “photo of a woman in white hat BREAK blue dress”

Complex Scenes with Multiple Elements

Example:

• “A bustling medieval market scene with vendors and colorful stalls BREAK In the background, a castle on a hill under a clear blue sky”

Detailed Character Descriptions

Example:

• “A warrior in detailed armor, holding a shimmering sword BREAK Standing in a mystical forest with ethereal lighting”

Using BREAK ensures that specific elements are given due attention.

---

7. Checking Keywords

To effectively test and understand the impact of specific keywords in Stable Diffusion prompts, use them as standalone prompts or within varied combinations.

Artist Names

Incorporating the names of famous artists like Pablo Picasso or Vincent van Gogh in your prompt can significantly influence the artistic style of the generated image.

Examples:

• “Portrait of a noblewoman, digital art in the style of Pablo Picasso and Vincent van Gogh.”
• “Landscape painting by Claude Monet.”
• “Abstract shapes in the style of Kandinsky.”

Website References

Including a reference to a platform like ArtStation in your prompt can guide the AI to generate images that align with the style commonly found on that platform.

Examples:

• “Futuristic cityscape, concept art resembling ArtStation styles.”
• “Character design in the style of DeviantArt.”
• “Illustration resembling Behance portfolio.”

Descriptive Adjectives

Adding descriptive adjectives and terms like ‘sharp focus’ and ‘highly detailed’ can enhance the clarity and detail of the image.

Examples:

• “Ancient castle ruins, sharp focus, highly detailed, under moonlight.”
• “Majestic mountain range, vivid colors, crystal clear.”
• “Rustic cottage, charming details, cozy atmosphere.”

Adding Extra Information

Incorporating additional descriptive words like ‘futuristic,’ ‘breathtaking,’ and ‘dystopian’ can add a specific ambiance or theme to your image.

Examples:

• “Robot in a dystopian world, with a futuristic and breathtaking design.”
• “A breathtaking view of the Milky Way over a serene beach.”
• “Dystopian cityscape with towering skyscrapers and neon lights.”

Color and Lighting

Modifying the image’s shade or brightness with keywords like ‘iridescent gold,’ ‘movie-style lighting,’ or ‘shadows’ can significantly alter the image’s tone and atmosphere.

Examples:

• “Portrait of a jazz musician, bathed in iridescent gold light, with deep movie-style shadows.”
• “Sunset over the ocean, warm hues, soft lighting.”
• “Interior of a cozy cabin, candlelit ambiance, soft shadows.”

Using the Stable Diffusion Prompt Matrix

Combine prompts to test a wide range of styles.

Example:

• “Elegant ballerina | In an abandoned building, | Art Nouveau style.”

Negative Prompts

Utilizing negative prompts by specifying what you don’t want in the image is another effective technique.

Examples:

• “Landscape painting, avoiding grainy texture and distorted perspectives.”
• “Portrait, no blurry elements, no background distractions.”
• “Sci-fi scene, without any futuristic buildings.”

Prompt Engineering

The process of building a good prompt should be iterative, where you test and refine your prompts based on the results.

Examples:

• “Serene mountain lake at sunrise, with vivid colors and a reflection.”
• “Gothic cathedral, intricate details, dark ambiance.”
• “Fantasy forest, magical creatures, glowing flora.”

These techniques help refine your ability to craft prompts that align closely with your creative vision.

---

8. Association Effect

Keywords can carry unintended associations, influencing attributes like ethnicity or style. It’s essential to be aware of these associations to avoid undesired outcomes.

Example:

Using the keyword “tribal” might unintentionally lead to specific ethnic connotations. Instead, be explicit about the desired style or elements without ambiguous terms.

Improved Example:

• Instead of: “Tribal warrior”
• Use: “Warrior with traditional patterns and accessories”

Other Examples:

• “Exotic” might imply specific ethnic features. Instead, describe the unique qualities: “Colorful, vibrant market scene.”
• “Ghetto” can have negative connotations. Instead, specify: “Urban street with lively atmosphere.”

---

9. Embeddings as Keywords

Embeddings, like textual inversions, are combinations of keywords with specific effects. They help refine and control the output by embedding complex concepts into simpler terms.

Example:

Using an embedding like “dreamy landscape” can invoke a specific style or mood consistently across different prompts.

Examples:

• “A dreamy landscape with a flowing river and distant mountains”
• “Vintage-inspired portrait with a soft focus and warm tones”
• “High-energy sports scene with dynamic motion blur”

More Examples:

• “Majestic sunset with vibrant colors and dramatic clouds”
• “Fantasy castle surrounded by lush greenery and mystical fog”
• “Steampunk airship flying over a futuristic cityscape”

---

10. Effect of Custom Models

Custom models can significantly alter the interpretation of styles and keywords. They allow for more specialized outputs tailored to specific themes or artistic directions.

Example:

Using a custom model trained on anime styles can make any prompt generate anime-like images, regardless of the original style keywords.

Examples:

• “A futuristic cityscape, highly detailed, in anime style”
• “Medieval knight, in a dynamic anime action pose”
• “Magical girl in a vibrant, colorful setting”

More Examples:

• “Cyberpunk city, with neon lights and rain, in manga style”
• “Fantasy landscape with dragons and warriors, in a comic book style”
• “Retro-futuristic robot, in a vintage sci-fi illustration style”

This comprehensive guide provides you with the tools and knowledge to master prompt crafting in Stable Diffusion. Experiment, iterate, and discover the vast potential of AI-generated art!