Add docs for formatters
This commit is contained in:
Chris Howell
parent
36f29c3058
commit
d3cd5d1254
559
README.md
559
README.md
|
|
@ -1,247 +1,312 @@
|
||||||
|
|
||||||
# YouTube Transcript/Subtitle API (including automatically generated subtitles and subtitle translations)
|
# YouTube Transcript/Subtitle API (including automatically generated subtitles and subtitle translations)
|
||||||
|
|
||||||
[](https://www.paypal.com/cgi-bin/webscr?cmd=_s-xclick&hosted_button_id=BAENLEW8VUJ6G&source=url) [](https://travis-ci.org/jdepoix/youtube-transcript-api) [](https://coveralls.io/github/jdepoix/youtube-transcript-api?branch=master) [](http://opensource.org/licenses/MIT) [](https://pypi.org/project/youtube-transcript-api/) [](https://pypi.org/project/youtube-transcript-api/)
|
[](https://www.paypal.com/cgi-bin/webscr?cmd=_s-xclick&hosted_button_id=BAENLEW8VUJ6G&source=url) [](https://travis-ci.org/jdepoix/youtube-transcript-api) [](https://coveralls.io/github/jdepoix/youtube-transcript-api?branch=master) [](http://opensource.org/licenses/MIT) [](https://pypi.org/project/youtube-transcript-api/) [](https://pypi.org/project/youtube-transcript-api/)
|
||||||
|
|
||||||
This is an python API which allows you to get the transcripts/subtitles for a given YouTube video. It also works for automatically generated subtitles, supports translating subtitles and it does not require a headless browser, like other selenium based solutions do!
|
This is an python API which allows you to get the transcripts/subtitles for a given YouTube video. It also works for automatically generated subtitles, supports translating subtitles and it does not require a headless browser, like other selenium based solutions do!
|
||||||
|
|
||||||
## Install
|
## Install
|
||||||
|
|
||||||
It is recommended to [install this module by using pip](https://pypi.org/project/youtube-transcript-api/):
|
It is recommended to [install this module by using pip](https://pypi.org/project/youtube-transcript-api/):
|
||||||
|
|
||||||
```
|
```
|
||||||
pip install youtube_transcript_api
|
pip install youtube_transcript_api
|
||||||
```
|
```
|
||||||
|
|
||||||
If you want to use it from source, you'll have to install the dependencies manually:
|
If you want to use it from source, you'll have to install the dependencies manually:
|
||||||
|
|
||||||
```
|
```
|
||||||
pip install -r requirements.txt
|
pip install -r requirements.txt
|
||||||
```
|
```
|
||||||
|
|
||||||
You can either integrate this module [into an existing application](#api), or just use it via an [CLI](#cli).
|
You can either integrate this module [into an existing application](#api), or just use it via an [CLI](#cli).
|
||||||
|
|
||||||
## API
|
## API
|
||||||
|
|
||||||
The easiest way to get a transcript for a given video is to execute:
|
The easiest way to get a transcript for a given video is to execute:
|
||||||
|
|
||||||
```python
|
```python
|
||||||
from youtube_transcript_api import YouTubeTranscriptApi
|
from youtube_transcript_api import YouTubeTranscriptApi
|
||||||
|
|
||||||
YouTubeTranscriptApi.get_transcript(video_id)
|
YouTubeTranscriptApi.get_transcript(video_id)
|
||||||
```
|
```
|
||||||
|
|
||||||
This will return a list of dictionaries looking somewhat like this:
|
This will return a list of dictionaries looking somewhat like this:
|
||||||
|
|
||||||
```python
|
```python
|
||||||
[
|
[
|
||||||
{
|
{
|
||||||
'text': 'Hey there',
|
'text': 'Hey there',
|
||||||
'start': 7.58,
|
'start': 7.58,
|
||||||
'duration': 6.13
|
'duration': 6.13
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
'text': 'how are you',
|
'text': 'how are you',
|
||||||
'start': 14.08,
|
'start': 14.08,
|
||||||
'duration': 7.58
|
'duration': 7.58
|
||||||
},
|
},
|
||||||
# ...
|
# ...
|
||||||
]
|
]
|
||||||
```
|
```
|
||||||
|
|
||||||
You can also add the `languages` param if you want to make sure the transcripts are retrieved in your desired language (it defaults to english).
|
You can also add the `languages` param if you want to make sure the transcripts are retrieved in your desired language (it defaults to english).
|
||||||
|
|
||||||
```python
|
```python
|
||||||
YouTubeTranscriptApi.get_transcripts(video_ids, languages=['de', 'en'])
|
YouTubeTranscriptApi.get_transcripts(video_ids, languages=['de', 'en'])
|
||||||
```
|
```
|
||||||
|
|
||||||
It's a list of language codes in a descending priority. In this example it will first try to fetch the german transcript (`'de'`) and then fetch the english transcript (`'en'`) if it fails to do so. If you want to find out which languages are available first, [have a look at `list_transcripts()`](#list-available-transcripts)
|
It's a list of language codes in a descending priority. In this example it will first try to fetch the german transcript (`'de'`) and then fetch the english transcript (`'en'`) if it fails to do so. If you want to find out which languages are available first, [have a look at `list_transcripts()`](#list-available-transcripts)
|
||||||
|
|
||||||
To get transcripts for a list of video ids you can call:
|
To get transcripts for a list of video ids you can call:
|
||||||
|
|
||||||
```python
|
```python
|
||||||
YouTubeTranscriptApi.get_transcripts(video_ids, languages=['de', 'en'])
|
YouTubeTranscriptApi.get_transcripts(video_ids, languages=['de', 'en'])
|
||||||
```
|
```
|
||||||
|
|
||||||
`languages` also is optional here.
|
`languages` also is optional here.
|
||||||
|
|
||||||
### List available transcripts
|
### List available transcripts
|
||||||
|
|
||||||
If you want to list all transcripts which are available for a given video you can call:
|
If you want to list all transcripts which are available for a given video you can call:
|
||||||
|
|
||||||
```python
|
```python
|
||||||
transcript_list = YouTubeTranscriptApi.list_transcripts(video_id)
|
transcript_list = YouTubeTranscriptApi.list_transcripts(video_id)
|
||||||
```
|
```
|
||||||
|
|
||||||
This will return a `TranscriptList` object which is iterable and provides methods to filter the list of transcripts for specific languages and types, like:
|
This will return a `TranscriptList` object which is iterable and provides methods to filter the list of transcripts for specific languages and types, like:
|
||||||
|
|
||||||
```python
|
```python
|
||||||
transcript = transcript_list.find_transcript(['de', 'en'])
|
transcript = transcript_list.find_transcript(['de', 'en'])
|
||||||
```
|
```
|
||||||
|
|
||||||
By default this module always picks manually created transcripts over automatically created ones, if a transcript in the requested language is available both manually created and generated. The `TranscriptList` allows you to bypass this default behaviour by searching for specific transcript types:
|
By default this module always picks manually created transcripts over automatically created ones, if a transcript in the requested language is available both manually created and generated. The `TranscriptList` allows you to bypass this default behaviour by searching for specific transcript types:
|
||||||
|
|
||||||
```python
|
```python
|
||||||
# filter for manually created transcripts
|
# filter for manually created transcripts
|
||||||
transcript = transcript_list.find_manually_created_transcript(['de', 'en'])
|
transcript = transcript_list.find_manually_created_transcript(['de', 'en'])
|
||||||
|
|
||||||
# or automatically generated ones
|
# or automatically generated ones
|
||||||
transcript = transcript_list.find_generated_transcript(['de', 'en'])
|
transcript = transcript_list.find_generated_transcript(['de', 'en'])
|
||||||
```
|
```
|
||||||
|
|
||||||
The methods `find_generated_transcript`, `find_manually_created_transcript`, `find_generated_transcript` return `Transcript` objects. They contain metadata regarding the transcript:
|
The methods `find_generated_transcript`, `find_manually_created_transcript`, `find_generated_transcript` return `Transcript` objects. They contain metadata regarding the transcript:
|
||||||
|
|
||||||
```python
|
```python
|
||||||
print(
|
print(
|
||||||
transcript.video_id,
|
transcript.video_id,
|
||||||
transcript.language,
|
transcript.language,
|
||||||
transcript.language_code,
|
transcript.language_code,
|
||||||
# whether it has been manually created or generated by YouTube
|
# whether it has been manually created or generated by YouTube
|
||||||
transcript.is_generated,
|
transcript.is_generated,
|
||||||
# whether this transcript can be translated or not
|
# whether this transcript can be translated or not
|
||||||
transcript.is_translatable,
|
transcript.is_translatable,
|
||||||
# a list of languages the transcript can be translated to
|
# a list of languages the transcript can be translated to
|
||||||
transcript.translation_languages,
|
transcript.translation_languages,
|
||||||
)
|
)
|
||||||
```
|
```
|
||||||
|
|
||||||
and provide the method, which allows you to fetch the actual transcript data:
|
and provide the method, which allows you to fetch the actual transcript data:
|
||||||
|
|
||||||
```python
|
```python
|
||||||
transcript.fetch()
|
transcript.fetch()
|
||||||
```
|
```
|
||||||
|
|
||||||
### Translate transcript
|
### Translate transcript
|
||||||
|
|
||||||
YouTube has a feature which allows you to automatically translate subtitles. This module also makes it possible to access this feature. To do so `Transcript` objects provide a `translate()` method, which returns a new translated `Transcript` object:
|
YouTube has a feature which allows you to automatically translate subtitles. This module also makes it possible to access this feature. To do so `Transcript` objects provide a `translate()` method, which returns a new translated `Transcript` object:
|
||||||
|
|
||||||
```python
|
```python
|
||||||
transcript = transcript_list.find_transcript(['en'])
|
transcript = transcript_list.find_transcript(['en'])
|
||||||
translated_transcript = transcript.translate('de')
|
translated_transcript = transcript.translate('de')
|
||||||
print(translated_transcript.fetch())
|
print(translated_transcript.fetch())
|
||||||
```
|
```
|
||||||
|
|
||||||
### By example
|
### By example
|
||||||
```python
|
```python
|
||||||
# retrieve the available transcripts
|
# retrieve the available transcripts
|
||||||
transcript_list = YouTubeTranscriptApi.list_transcripts('video_id')
|
transcript_list = YouTubeTranscriptApi.list_transcripts('video_id')
|
||||||
|
|
||||||
# iterate over all available transcripts
|
# iterate over all available transcripts
|
||||||
for transcript in transcript_list:
|
for transcript in transcript_list:
|
||||||
|
|
||||||
# the Transcript object provides metadata properties
|
# the Transcript object provides metadata properties
|
||||||
print(
|
print(
|
||||||
transcript.video_id,
|
transcript.video_id,
|
||||||
transcript.language,
|
transcript.language,
|
||||||
transcript.language_code,
|
transcript.language_code,
|
||||||
# whether it has been manually created or generated by YouTube
|
# whether it has been manually created or generated by YouTube
|
||||||
transcript.is_generated,
|
transcript.is_generated,
|
||||||
# whether this transcript can be translated or not
|
# whether this transcript can be translated or not
|
||||||
transcript.is_translatable,
|
transcript.is_translatable,
|
||||||
# a list of languages the transcript can be translated to
|
# a list of languages the transcript can be translated to
|
||||||
transcript.translation_languages,
|
transcript.translation_languages,
|
||||||
)
|
)
|
||||||
|
|
||||||
# fetch the actual transcript data
|
# fetch the actual transcript data
|
||||||
print(transcript.fetch())
|
print(transcript.fetch())
|
||||||
|
|
||||||
# translating the transcript will return another transcript object
|
# translating the transcript will return another transcript object
|
||||||
print(transcript.translate('en').fetch())
|
print(transcript.translate('en').fetch())
|
||||||
|
|
||||||
# you can also directly filter for the language you are looking for, using the transcript list
|
# you can also directly filter for the language you are looking for, using the transcript list
|
||||||
transcript = transcript_list.find_transcript(['de', 'en'])
|
transcript = transcript_list.find_transcript(['de', 'en'])
|
||||||
|
|
||||||
# or just filter for manually created transcripts
|
# or just filter for manually created transcripts
|
||||||
transcript = transcript_list.find_manually_created_transcript(['de', 'en'])
|
transcript = transcript_list.find_manually_created_transcript(['de', 'en'])
|
||||||
|
|
||||||
# or automatically generated ones
|
# or automatically generated ones
|
||||||
transcript = transcript_list.find_generated_transcript(['de', 'en'])
|
transcript = transcript_list.find_generated_transcript(['de', 'en'])
|
||||||
```
|
```
|
||||||
|
|
||||||
## CLI
|
### Using Formatters
|
||||||
|
Formatters are meant to be an additional layer of processing of the transcript you pass it. The goal is to convert the transcript from its Python data type into a consistent string of a given "format". Such as a basic text (`.txt`) or even formats that have a defined specification such as JSON (`.json`), WebVTT format (`.vtt`), Comma-separated format (`.csv`), etc...
|
||||||
Execute the CLI script using the video ids as parameters and the results will be printed out to the command line:
|
|
||||||
|
The `formatters` submodule provides a few basic formatters to wrap around you transcript data in cases where you might want to do something such as output a specific format then write that format to a file. Maybe to backup/store and run another script against at a later time.
|
||||||
```
|
|
||||||
youtube_transcript_api <first_video_id> <second_video_id> ...
|
We provided a few subclasses of formatters to use:
|
||||||
```
|
|
||||||
|
- JSONFormatter
|
||||||
The CLI also gives you the option to provide a list of preferred languages:
|
- TextFormatter
|
||||||
|
- WebVTTFormatter (a basic implementation)
|
||||||
```
|
|
||||||
youtube_transcript_api <first_video_id> <second_video_id> ... --languages de en
|
Here is how to import from the `formatters` module.
|
||||||
```
|
|
||||||
|
```python
|
||||||
You can also specify if you want to exclude automatically generated or manually created subtitles:
|
# the base class to inherit from when creating your own formatter.
|
||||||
|
from youtube_transcript_api.formatters import Formatter
|
||||||
```
|
|
||||||
youtube_transcript_api <first_video_id> <second_video_id> ... --languages de en --exclude-generated
|
# some provided subclasses, each outputs a different string format.
|
||||||
youtube_transcript_api <first_video_id> <second_video_id> ... --languages de en --exclude-manually-created
|
from youtube_transcript_api.formatters import JSONFormatter
|
||||||
```
|
from youtube_transcript_api.formatters import TextFormatter
|
||||||
|
from youtube_transcript_api.formatters import WebVTTFormatter
|
||||||
If you would prefer to write it into a file or pipe it into another application, you can also output the results as json using the following line:
|
```
|
||||||
|
|
||||||
```
|
### Provided Formatter Example
|
||||||
youtube_transcript_api <first_video_id> <second_video_id> ... --languages de en --json > transcripts.json
|
Lets say we wanted to retrieve a transcript and write that transcript as a JSON file in the same format as the API returned it as. That would look something like this:
|
||||||
```
|
|
||||||
|
```python
|
||||||
Translating transcripts using the CLI is also possible:
|
# your_custom_script.py
|
||||||
|
|
||||||
```
|
from youtube_transcript_api import YouTubeTranscriptApi
|
||||||
youtube_transcript_api <first_video_id> <second_video_id> ... --languages en --translate de
|
from youtube_transcript_api.formatters import JSONFormatter
|
||||||
```
|
|
||||||
|
# Must be a single transcript.
|
||||||
If you are not sure which languages are available for a given video you can call, to list all available transcripts:
|
transcript = YouTubeTranscriptApi.get_transcript(video_id)
|
||||||
|
|
||||||
```
|
# .format() turns the transcript into a JSON string.
|
||||||
youtube_transcript_api --list-transcripts <first_video_id>
|
json_formatted = JSONFormatter(transcript).format()
|
||||||
```
|
|
||||||
|
|
||||||
## Proxy
|
# Now we can write it out to a file.
|
||||||
|
with open('your_filename.json', 'w', encoding='utf-8') as json_file:
|
||||||
You can specify a https/http proxy, which will be used during the requests to YouTube:
|
json_file.write(json_formatted)
|
||||||
|
|
||||||
```python
|
# Now should have a new JSON file that you can easily read back into Python.
|
||||||
from youtube_transcript_api import YouTubeTranscriptApi
|
```
|
||||||
|
|
||||||
YouTubeTranscriptApi.get_transcript(video_id, proxies={"http": "http://user:pass@domain:port", "https": "https://user:pass@domain:port"})
|
**Passing extra keyword arguments**
|
||||||
```
|
|
||||||
|
Since JSONFormatter leverages `json.dumps()` you can also forward keyword arguments into `.format()` such as making your file output prettier by forwarding the `indent=2` keyword argument.
|
||||||
As the `proxies` dict is passed on to the `requests.get(...)` call, it follows the [format used by the requests library](http://docs.python-requests.org/en/master/user/advanced/#proxies).
|
|
||||||
|
```python
|
||||||
Using the CLI:
|
json_formatted = JSONFormatter(transcript).format(indent=2)
|
||||||
|
```
|
||||||
```
|
|
||||||
youtube_transcript_api <first_video_id> <second_video_id> --http-proxy http://user:pass@domain:port --https-proxy https://user:pass@domain:port
|
### Custom Formatter Example
|
||||||
```
|
You can implement your own formatter class. Just inherit from the `Formatter` base class and ensure you implement the `def format(self, **kwargs):` method which should ultimately return a string when called on your formatter instance.
|
||||||
|
|
||||||
## Cookies
|
```python
|
||||||
|
|
||||||
Some videos are age restricted, so this module won't be able to access those videos without some sort of authentication. To do this, you will need to have access to the desired video in a browser. Then, you will need to download that pages cookies into a text file. You can use the Chrome extension [cookies.txt](https://chrome.google.com/webstore/detail/cookiestxt/njabckikapfpffapmjgojcnbfjonfjfg?hl=en) or the Firefox extension [cookies.txt](https://addons.mozilla.org/en-US/firefox/addon/cookies-txt/).
|
class MyCustomFormatter(Formatter):
|
||||||
|
def format(self, **kwargs):
|
||||||
Once you have that, you can use it with the module to access age-restricted videos' captions like so.
|
# Do your custom work in here, but return a string.
|
||||||
|
return 'your processed output data as a string.'
|
||||||
```python
|
```
|
||||||
from youtube_transcript_api import YouTubeTranscriptApi
|
|
||||||
|
## CLI
|
||||||
YouTubeTranscriptApi.get_transcript(video_id, cookies='/path/to/your/cookies.txt')
|
|
||||||
|
Execute the CLI script using the video ids as parameters and the results will be printed out to the command line:
|
||||||
YouTubeTranscriptApi.get_transcripts([video_id], cookies='/path/to/your/cookies.txt')
|
|
||||||
```
|
```
|
||||||
|
youtube_transcript_api <first_video_id> <second_video_id> ...
|
||||||
Using the CLI:
|
```
|
||||||
|
|
||||||
```
|
The CLI also gives you the option to provide a list of preferred languages:
|
||||||
youtube_transcript_api <first_video_id> <second_video_id> --cookies /path/to/your/cookies.txt
|
|
||||||
```
|
```
|
||||||
|
youtube_transcript_api <first_video_id> <second_video_id> ... --languages de en
|
||||||
|
```
|
||||||
## Warning
|
|
||||||
|
You can also specify if you want to exclude automatically generated or manually created subtitles:
|
||||||
This code uses an undocumented part of the YouTube API, which is called by the YouTube web-client. So there is no guarantee that it won't stop working tomorrow, if they change how things work. I will however do my best to make things working again as soon as possible if that happens. So if it stops working, let me know!
|
|
||||||
|
```
|
||||||
## Donation
|
youtube_transcript_api <first_video_id> <second_video_id> ... --languages de en --exclude-generated
|
||||||
|
youtube_transcript_api <first_video_id> <second_video_id> ... --languages de en --exclude-manually-created
|
||||||
If this project makes you happy by reducing your development time, you can make me happy by treating me to a cup of coffee :)
|
```
|
||||||
|
|
||||||
[](https://www.paypal.com/cgi-bin/webscr?cmd=_s-xclick&hosted_button_id=BAENLEW8VUJ6G&source=url)
|
If you would prefer to write it into a file or pipe it into another application, you can also output the results as json using the following line:
|
||||||
|
|
||||||
|
```
|
||||||
|
youtube_transcript_api <first_video_id> <second_video_id> ... --languages de en --json > transcripts.json
|
||||||
|
```
|
||||||
|
|
||||||
|
Translating transcripts using the CLI is also possible:
|
||||||
|
|
||||||
|
```
|
||||||
|
youtube_transcript_api <first_video_id> <second_video_id> ... --languages en --translate de
|
||||||
|
```
|
||||||
|
|
||||||
|
If you are not sure which languages are available for a given video you can call, to list all available transcripts:
|
||||||
|
|
||||||
|
```
|
||||||
|
youtube_transcript_api --list-transcripts <first_video_id>
|
||||||
|
```
|
||||||
|
|
||||||
|
## Proxy
|
||||||
|
|
||||||
|
You can specify a https/http proxy, which will be used during the requests to YouTube:
|
||||||
|
|
||||||
|
```python
|
||||||
|
from youtube_transcript_api import YouTubeTranscriptApi
|
||||||
|
|
||||||
|
YouTubeTranscriptApi.get_transcript(video_id, proxies={"http": "http://user:pass@domain:port", "https": "https://user:pass@domain:port"})
|
||||||
|
```
|
||||||
|
|
||||||
|
As the `proxies` dict is passed on to the `requests.get(...)` call, it follows the [format used by the requests library](http://docs.python-requests.org/en/master/user/advanced/#proxies).
|
||||||
|
|
||||||
|
Using the CLI:
|
||||||
|
|
||||||
|
```
|
||||||
|
youtube_transcript_api <first_video_id> <second_video_id> --http-proxy http://user:pass@domain:port --https-proxy https://user:pass@domain:port
|
||||||
|
```
|
||||||
|
|
||||||
|
## Cookies
|
||||||
|
|
||||||
|
Some videos are age restricted, so this module won't be able to access those videos without some sort of authentication. To do this, you will need to have access to the desired video in a browser. Then, you will need to download that pages cookies into a text file. You can use the Chrome extension [cookies.txt](https://chrome.google.com/webstore/detail/cookiestxt/njabckikapfpffapmjgojcnbfjonfjfg?hl=en) or the Firefox extension [cookies.txt](https://addons.mozilla.org/en-US/firefox/addon/cookies-txt/).
|
||||||
|
|
||||||
|
Once you have that, you can use it with the module to access age-restricted videos' captions like so.
|
||||||
|
|
||||||
|
```python
|
||||||
|
from youtube_transcript_api import YouTubeTranscriptApi
|
||||||
|
|
||||||
|
YouTubeTranscriptApi.get_transcript(video_id, cookies='/path/to/your/cookies.txt')
|
||||||
|
|
||||||
|
YouTubeTranscriptApi.get_transcripts([video_id], cookies='/path/to/your/cookies.txt')
|
||||||
|
```
|
||||||
|
|
||||||
|
Using the CLI:
|
||||||
|
|
||||||
|
```
|
||||||
|
youtube_transcript_api <first_video_id> <second_video_id> --cookies /path/to/your/cookies.txt
|
||||||
|
```
|
||||||
|
|
||||||
|
|
||||||
|
## Warning
|
||||||
|
|
||||||
|
This code uses an undocumented part of the YouTube API, which is called by the YouTube web-client. So there is no guarantee that it won't stop working tomorrow, if they change how things work. I will however do my best to make things working again as soon as possible if that happens. So if it stops working, let me know!
|
||||||
|
|
||||||
|
## Donation
|
||||||
|
|
||||||
|
If this project makes you happy by reducing your development time, you can make me happy by treating me to a cup of coffee :)
|
||||||
|
|
||||||
|
[](https://www.paypal.com/cgi-bin/webscr?cmd=_s-xclick&hosted_button_id=BAENLEW8VUJ6G&source=url)
|
||||||
|
|
|
||||||
Loading…
Reference in New Issue