Stable Diffusion XL 1.0

> the progress seems to be entirely driven by the anime porn community:

Its not entirely driven by porn communities, and the porn communities driving it aren’t entirely anime porn communities (and the anime communities driving it aren’t entirely porn communities.)

But, yeah, the anime + porn/fetish art + furry + rpg art + scifi/fantasy art communities, and particularly the niches in the overlap of two or more of those are, pretty significant.

> If this works for images other than naked and cartoon women

It does, and while it may not be large proportionally compared to the anime-porn stuff, there’s a lot of publicly distributed fine tuned checkpoints, LoRas, etc., demonstrating that it does.

It absolutely works for things other than naked and cartoon women. Here are some generations of my daughter and dog (together!). I believe most of these are from a fine tuned model of them and not an extracted LoRA, though I use that sometimes too: https://imgur.com/a/naHgnel

Isn't this it? https://huggingface.co/stabilityai/stable-diffusion-xl-base-...

The release event is in like ~30 minutes on their discord, probably the announcement went out a bit early.

It does appear to be live on Clipdrop.

https://clipdrop.co/stable-diffusion

You get access to the weights instantly if you apply for them. It's basically not a hurdle.

(I've been having fun with this for a few days. https://huggingface.co/stabilityai/stable-diffusion-xl-base-... Not sure there's much of a difference with the 1.0 version.)

They have bots in their discord for generating images bases on user prompts. Those randomize some settings, compare candidate models and are used for rlhf fine-tuning and that's the main source of refining which will continue even after release.

There were, IIRC, three different post-0.9 candidate models in parallel testing to become 1.0 recently.

Diffusion is more parameter-efficient and you quickly saturate the target fidelity, especially with some refiner cascade. It's a solved problem. You do not need more than maybe 4B total. Images are far more redundant than text.

In fact, most interesting papers since Imagen show that you get more mileage out of scaling the text encoder part, which is, of course, a Transformer. This is what drives accuracy, text rendering, compositionality, parsing edge cases. In SD 1.5 the text encoder part (CLIP ViT-L/14) takes a measly 123M parameters.[1] In Imagen, it was T5-XXL with 4.6B [2]. I am interested in someone trying to use a really strong encoder baseline – maybe from a UL2-20B – to push this tactic further.

Seeing as you can throw out diffusion altogether and synthesize images with transformers [3], there is no reason to prioritize the diffusion part as such.

1. https://forums.fast.ai/t/stable-diffusion-parameter-budget-a...

2. https://arxiv.org/abs/2205.11487

3. https://arxiv.org/abs/2301.00704

They often reference this paper as the motivation for that https://arxiv.org/pdf/2203.15556.pdf I.e. training with 10x data and 10x longer can yield as good models as a gpt-3 model but with fewer weights (according to the paper) and the same principle applies in vision.

Diffusion is relatively compute intensive compared to transformers llms, and (in current implementation) doesn't quantize as well.

A 70B parameter model would be very slow and vram hungry, hence very expensive to run.

Also, image generation is more reliant on tooling surrounding the models than pure text prompting. I dont think even a 300B model would get things quite right through text prompting alone.

Do we know the amount of parameters Dall-e have these days, Firefly or Midjourney, etc?

If we are talking about Stable Diffusion, the reality is that... more parameters mean it will be hard to run locally. And let me tell you something, the community around Stable Diffusion only cares with NSFW... And want local for that...

Stable Diffusion 2 was totally boycotted by the community because they... banned NSFW from there. They had now to allow it again on SDXL.

Also, more parameters mean it will be more expensive to community finetunners to train as well.

If you want to play around with Stable Diffusion XL: https://clipdrop.co

https://playgroundai.com/create

Not affiliated in anyway and not very involved in the space. I just wanted to generate some images a few weeks ago and was looking for somewhere I could do that for free. The link above lets you do that but I suggest you look up prompts because its a lot more involved than I expected.

Midjourney right? Although, discord isn't a website I guess.

I've found https://firefly.adobe.com/ pretty good at composing images with multiple subjects. [disclaimer - I work at Adobe, but not in the Creative Cloud]

But I wouldn't say it's the "best." Just trained on images that weren't taken from unconsenting artists.

https://dreamstudio.ai/

There are toy AI things, but there is nothing quite like Stable Diffusion running on Colab. Lots of people recommended Midjourney but that is like playing with MSpaint. If you can get Stable Diffusion going with Automatic1111, its AAA tier. Especially with Control-net, and dreambooth, but that is part 2.

Google: The Last Ben Stable Diffusion Colab

for a way to not run it locally, but get all the features.

Probably Midjourney, but I like Dreamstudio better.

If you mean raster images that look like vector and contain arbitrary text and shapes, controlnets/T2I adapters do work for this. You could train your custom controlnet for this, too. (it requires understanding)

As for directly generating vector images, there's nothing yet. Your best bet is generating vector-looking raster and tracing it.

There are SD models tunes for vector like raster output. And XL has specifically focused on this use case as one of the improvements. Try SDXL 1 on Clipdrop or Dreamstudio.

A lot of people are having success by adding extra networks (lora is the most common) which are trained on the type of image you're looking for. It's still a raster image, of course, but you can produce images which look very much like rasterizations of vector images, which you can then translate back into SVGs in Inkscape or similar.

Different use case.

I can run SDXL 1.0 offline from my home. I can’t do this with Midjourney.

A closed source model that doesn’t have the limitation of running on consumer level GPUs will have certain advantages.

Midjourney is destroyed by the ecosystem around stable diffusion, especially all the features and extensions in automatic1111. It’s not even close

MJ quality is significantly worse. Everything has the Pixar look and barely follows the prompt. Its nice for a toy, but SD with Automatic1111 is miles ahead of MJ.

My go-to test is "elephant riding unicycle". Neither Midjourney nor Stable Diffusion XL is capable of doing this.

You can do this if you select the `pipe.enable_model_cpu_offload()` option. See this https://huggingface.co/stabilityai/stable-diffusion-xl-base-...

> I hope someday there’s a version of this or something comparable to it that can run on <8gb consumer hardware.

Someday is today: from the official announcement: “SDXL 1.0 should work effectively on consumer GPUs with 8GB VRAM or readily available cloud instances.” https://stability.ai/blog/stable-diffusion-sdxl-1-announceme...

Easy Diffusion (previously cmdr2 UI) can run SDXL in 768x768 in about 7 GB of VRAM. And SDXL 512x512 in about 5 GB of VRAM.

Regular SD can run in less than 2 GB of VRAM with Easy Diffusion.

1. Installation (no dependencies, python etc): https://github.com/easydiffusion/easydiffusion#installation

2. Enable beta to get access to SDXL: https://github.com/easydiffusion/easydiffusion/wiki/The-beta...

3. Use the "Low" VRAM Usage model in the Settings tab.

SDXL 0.9 runs on iPad Pro 8GiB just fine.

I feel like this is the greatest demand for LLMs at the moment too.

It's hard to believe we're only 8 months into this industry, so I imagine we'll start seeing smaller footprints soon.

Give InvokeAI a try.

https://github.com/invoke-ai/InvokeAI

Edit: Spec required from the documentation

You will need one of the following:

    An NVIDIA-based graphics card with 4 GB or more VRAM memory. 6-8 GB of VRAM is highly recommended for rendering using the Stable Diffusion XL models
    An Apple computer with an M1 chip.
    An AMD-based graphics card with 4GB or more VRAM memory (Linux only), 6-8 GB for XL rendering.

I should clarify that by <8gb I meant “less than 8gb”, which is what SD 1.5 and 2 were able to do. I’m aware that it can run on ==8gb.

There are several papers on 4/8 bit quantization, and a few implementations for Vulkan/CUDA/ROCm compilation.

TBH the UIs people run for SD 1.5 are pretty unoptimized.

SDXL is in roughly the same ballpark as MJ 5 quality-wise, but the main value is in the array of tooling immediately available for it, and the license. You can fine-tune it on your own pictures, use higher order input (not just text), and daisy-chain various non-imagegen models and algorithms (object/feature segmentation, depth detection, processing, subject control etc) to produce complex images, either procedural or one-off. It's all experimental and very improvised, but is starting to look like a very technical CGI field separate from the classic 3D CGI.

For bland stock photos and other "general-purpose" image generation, DALLE-2/Bing/Adobe etc are... the okayest. SD (with just standard model weights) is particularly weak here because of the small model size.

If you want to get arty, then state of the art for out-of-the-box typing in a prompt and clicking "generate" is probably MidJourney.

But if you're willing to spend some more time playing around with the open-source tooling, community finetunes, model augmentations (LyCORIS, etc), SD is probably going to get you the farthest.

> Also what are the most common use cases for image generation?

By sheer number of image generations? Take a guess...

SDXL 0.9 should be the state-of-the-art image generation model (in the open). It generates at 1024x1024 large resolution, with high coherency and good selection of styles out of box. It also has reasonable text-understanding comparing to other models.

That has been said, based on the configurations of these models, we are far from saturating what the best model can do. The problem is, FID is terrible metrics to evaluating these models so like LLM, we are a bit clueless about how to evaluate them now.

I don't know what the use case is for other people is, but I've been playing around with book covers. This one took about two weeks, but it was my first real try and I was still learning how. Composition is a little off. The one I'm working on now is going faster (and better).

https://imgur.com/a/CxX5eYj

I've found that I rarely get a usable image completely as-is. It might take 5 or 10 generations to find something sort of ok, and even then I end up erasing the bad parts and letting it in-paint (which again takes multiple attempts). The T-rex had like 7 legs and two jaws, but was otherwise close to what I wanted... just keep erasing extra body parts until the in-painter finally takes a hint.

I was also going to do a few book covers for some Babylon 5 books, but it does so bad on celebrity faces. Looked like Koenig's mutant love child with Ernest Borgnine. Dunno what to do about that. I keep wondering if I shouldn't spend the next 10 years putting together my own training set of fantasy and science fiction art.

Midjourney may be better for plain prompts, but Stable Diffusion is SOTA because of the tooling and finetuning surrounding it.