KeyArgo
/
llama-dl
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Cut anecdote; move file list to a separate gist

This commit is contained in:
Shawn Presser 2023-03-05 12:12:58 -06:00 committed by GitHub
parent 9e4af7a16a
commit d7a7c2c74c
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 2 additions and 39 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

41
README.md
View File

@ -23,18 +23,10 @@ real 98m12.980s
user 8m8.916s
sys 5m7.259s
```
This works out to an average of 40MB/s (235164838073 bytes in 5892 seconds).
This works out to 40MB/s (235164838073 bytes in 5892 seconds).
Personally, I just wanted to `curl` the weights instead of dealing with a torrent. The fact that it's several times faster was just a nice bonus.
Here's a funny screenshot from when this repo first went live. I guess I'll keep it here. The script was originally using the leaker's own private download link that Facebook emailed them, which turned out to download at up to 220MB/s:
![image](https://user-images.githubusercontent.com/59632/222940196-e763d8a0-2282-4f78-8bbe-14c559eea90f.png)
I was surprised, to say the least.
Once Facebook figured out what we were up to and shut it off, I mirrored everything to Cloudflare R2. This turned out to be fast, reliable, and almost free. (No egress fees.)
## Download
To download all model weights, `cd` into the directory you want them, then run this:
@ -58,36 +50,7 @@ Running random bash scripts generally isn't a good idea, but I'll stake my perso
## How much space do I need?
219G (235164838073 bytes) total. Here are the sizes of the individual files for reference:
```sh
./tokenizer_checklist.chk 50
./tokenizer.model 499723
./7B/checklist.chk 100
./7B/consolidated.00.pth 13476939516
./7B/params.json 101
./13B/checklist.chk 154
./13B/consolidated.00.pth 13016334699
./13B/consolidated.01.pth 13016334699
./13B/params.json 101
./30B/checklist.chk 262
./30B/consolidated.00.pth 16265763099
./30B/consolidated.01.pth 16265763099
./30B/consolidated.02.pth 16265763099
./30B/consolidated.03.pth 16265763099
./30B/params.json 101
./65B/checklist.chk 478
./65B/consolidated.00.pth 16323959449
./65B/consolidated.01.pth 16323959449
./65B/consolidated.02.pth 16323959449
./65B/consolidated.03.pth 16323959449
./65B/consolidated.04.pth 16323959449
./65B/consolidated.05.pth 16323959449
./65B/consolidated.06.pth 16323959449
./65B/consolidated.07.pth 16323959449
./65B/params.json 101
---
total 235164838073
```
219G (235164838073 bytes) total. [Here's a file list](https://gist.github.com/shawwn/bddb2f91aa45fbcdc0dd105d88816e75) with sizes for each.
## How do I know this is safe?