Meta AI announces Massive Multilingual Speech code, models for 1000+ languages

I loaded the language coverage into Datasette Lite and added some facets here:

https://lite.datasette.io/?json=https://gist.github.com/simo...

Here's how I did that: https://gist.github.com/simonw/63aa33ec827b093f9c6a2797df950...

Here are the top 20 represented language families:

    Niger-Congo 1,019
    Austronesian 609
    Sino-Tibetan 288
    Indo-European 278
    Afro-Asiatic 222
    Trans-New Guinea 219
    Otomanguean 149
    Nilo-Saharan 131
    Austro-Asiatic 100
    Dravidian 60
    Australian 51
    Creole 45
    Kra-Dai 43
    Uto-Aztecan 41
    Quechuan 36
    Language isolate 35
    Torricelli 32
    Maipurean 31
    Mayan 30
    Sepik 30

Based on the availability of STT, TTS, and translation models available to download in that github repo, a real-life babelfish is 'only' some glue code away. Wild times we're living in.

Here's a sheet I came across today...

https://www.reddit.com/r/LocalLLaMA/comments/11o6o3f/how_to_...

Granted, it's talking about quantized models, which use less memory. But you can see the 30B models taking 36 GB at 8-bit, and at least 20 GB at 4-bit.

The page even lists the recommended cards.

But as others have pointed out, you may get more bang "renting" as in purchasing cloud instances able to run these workloads. Buying a system costs about as much as buying instance time for one year. Theoretically, if you only run sporadic workloads when you're playing around it would cost less. If you're training... that's a different story.

The more VRAM the better if you'd like to run larger LLMs. Old Nvidia P40 (Pascal 24GB) cards are easily available for $200 or less and would be easy/cheap to play. Here's a recent writeup on the LLM performance you can expect for inferencing (training speeds I assume would be similar): https://www.reddit.com/r/LocalLLaMA/comments/13n8bqh/my_resu...

This repo lists very specific VRAM usage for various LLaMA models (w/ group size, and accounting for context window, which is often missing) - this are all 4-bit GPTQ quantized models: https://github.com/turboderp/exllama

Note the latest versions of llama.cpp now have decent GPU support and has both a memory tester and lets you load partial models (n-layers) into your GPU. It inferences about 2X slower than exllama from my testing on a RTX 4090, but still about 6X faster than my CPU (Ryzen 5950X).

Again this is inferencing. For training, pay attention to 4-bit bitsandbytes, coming soon: https://twitter.com/Tim_Dettmers/status/1657010039679512576

Wait why is renting a GPU in the cloud not a solution? You can even try multiple options and see which ones are capable enough for your use case.

Look into some barebones cloud GPU services, for example Lambda Labs which is significantly cheaper than AWS/GCP but offers basically nothing besides the machine with a GPU. You could even try something like Vast in which people rent out their personal GPU machines for cheap. Not something I'd use for uhhh...basically anything corporate, but for a personal project with no data security or uptime issues it would probably work great.

You can finetune whisper, stable diffusion, and LLM up to about 15B parameters with 24GB VRAM.

Which leads you to what hardware to get. Best bang for the $ right now is definitely a used 3090 at ~$700. If you want more than 24GB vram just rent the hardware as it will be cheaper.

If you're not willing to drop $700 don't buy anything just rent. I have had decent luck with vast.ai

I would recommend a 3090. It can handle everything a 4000 series can albeit slightly slower, has enough VRAM to handle most things for fun, and can be bought for around $700.

Just do it. Spend a few hours doing research and you will find out. With that said, buy as much memory as can. That makes 4090 king if you have the server that can carry it, plus the budget. For me, I settled for 3060, it's a nice compromise between cost and ram. Cheap, 12gb and 170TPW.

I think you just need to educate yourself a bit about the space. These models are very small (the large version is only 1B parameters) so should run on a 4GB gaming GPU.

TLA: “Text LLM Application.” ;)

Imagine if we used them everyday for 500 years but tune all the languages to be ever so slightly similar each day until there is one language.

Imagine if someone said this about dictionaries.

When a jet skier accidentally runs into a tree that's called a snow crash, aha

I wrote a colab to run TTS, it takes some time to install dependencies : https://colab.research.google.com/drive/11tZ1c7kQJ5uKImm9t-C...

Overall I got pretty poor results in english and french, I guess it would require some fine-tuning

I've created this environment.yml:

  name: fairseq
  channels:
    - conda-forge
    - pytorch
  dependencies:
    - python=3.9
    - Cython==0.29.21
    - librosa==0.8.0
    - matplotlib=3.3
    - numpy=1.19
    - scipy==1.5.2
    - tensorboard==2.3.0
    - pytorch=1
    - torchvision=0
    - Unidecode==1.1.1
    - pip:
        - phonemizer==2.2.1

  conda create -f environment.yml
  conda activate fairseq

  cd path/to/vits/monotonic_align
  mkdir monotonic_align
  python setup.py build_ext --inplace

  cd path/to/fairseq
  PYTHONPATH=$PYTHONPATH:path/to/vits python examples/mms/tts/infer.py --model-dir checkpoints/eng --wav outputs/eng.wav --txt "As easy as pie"

Yeah, a lot of the releases from researcg groups are woefully poorly documented.

Usually, some hero releases a friendly install system within a few days, though.

Research projects are notorious for this. The people behind them aren't very experienced in distributing software (many never heard of docker), the usual language of choice is Python which has poor packaging, native dependencies complicate things, and everything is on the bleeding edge by definition.

Usually, it needs a brave soul to reverse engineer what exact versions of dependencies are needed and make a Colab or Dockerfile that puts it all together.

This thing uses pytorch, the bane of everything that attempts to make pypi installs sane.

As a human, you need to read https://pytorch.org/get-started/locally/ and install the correct version, depending on your pytorch-version/os/packaging-system/hardware-platform combo. It's bad. It's several conflicting index-urls messing with your requirements. It also doesn't work if you pick the wrong thing.

Or we need something like a "setup.py" that is turing-complete, to dynamically pick up the correct dependency for you.

`pip install -r requirements.txt` is not enough.

They're intended for researchers/professionals not consumers, and I'm not sure how a video is going to be helpful?

And the issue with a live demo is that these are resource-intensive, they're not just webpages. It's an entire project to figure out how to host them, scale them to handle peaks, pay for them, implement rate-limiting, and so forth.

For the intended audience, download-and-run-it doesn't seem like an issue at all. I don't see how any questions are going to be answered by a video.

Here's the blog post with video of it working: https://ai.facebook.com/blog/multilingual-model-speech-recog...

I'd argue that having a "download and run" approach is so much better than videos or demos. Why do you think this is a problem?

This paragraph justifies the decision, IMHO.

> Collecting audio data for thousands of languages was our first challenge because the largest existing speech datasets cover 100 languages at most. To overcome this, we turned to religious texts, such as the Bible, that have been translated in many different languages and whose translations have been widely studied for text-based language translation research.

I think the choice was just between having any data at all, or not being able to support that language.

> I don't think the stylistic choices in the Bible are representative of how people actually speak the language, in any of the languages I can speak

The cadence and intonation sounded a little weird, but I suspect fine-tuning can improve that by a lot. I am really excited to see some low-resource language finally get some mainstream TTS support at all.

according to their blog post[1], MMS achieves ~half the error rate on words, while supporting 11x more languages. pretty impressive.

[1] https://ai.facebook.com/blog/multilingual-model-speech-recog...

Trying to attach a license to model weights seems counter-productive to me. If you argue they are copyrightable then surely they are derivative works of the training data, which is unlikely to all be public domain. Machine learning enthusiasts are better off lobbying for model weights being non-copyrightable as it doesn't have any creative input and is the result of a purely mechanical process.

The copyright on the code, on the other hand, would definitely be copyrighted and would need a clean-room implementation, as you said. The community could pool its resources and do it once, and license it under the AGPL to keep it and further improvements available to everyone.

I think this is a lot better than the other option, which would've been not releasing it at all. No company in the business of making money wants to give away their edge to would-be competitors for free.

Id imagine you could use inference from this as training for a commercial model, as that isn't currently protected under copyright.

My layman's reading is that the license does seem to allow recouping of costs, would be interested if there is a more nuanced interpretation to the contrary.

(tangentially, like another comment briefly mentioned, are models actually copyrightable? Programs are, because they are human authored creative works.)

Unless you’re in a country that doesn’t care about the IP restrictions, then you do whatever you want.

Yes, they are so far more open than open<for business>ai.

If anything, it sounds like the start of a "Meta? I don't even know her" joke.

It's not VR, it's Metaverse; VR might involve experiences that are not bullshit.