TVDownloader: télécharger les médias du net ! [2]

k3c · Le 22/04/2013, à 13:34

@ 11gjm

Je vais être franc, je me contente de faire un mediainfo sur les vidéos téléchargées, et je ne savais pas qu'il y avait plusieurs versions, VF, VOST...

Je vais regarder.

pacome · Le 22/04/2013, à 14:30

Bonjour!

@11gjm: Merci pour tes conseils; je pensais modifier la boucle de téléchargement du script pour ne pas avoir à le relancer à la main, mais k3c nous fournit déjà une nouvelle version parfaitement fonctionnelle
Concernant les versions VOST/VF: je préfère les VO mais les VF sont disponibles (pour l'épisode 15: rtmpe://wske.wat.tv/ondemand/mp4:vod/H264-384x288/37/29/10433729.h264) mais juste en 384x288 apparemment...

@k3c: Merci pour cette nouvelle version avec le nouveau hash swf! je reste curieux de savoir comment tu l'obtiens (la redirection avec iptables et rtmpsrv ne m'a rien donné)

À+

bibichouchou · Le 22/04/2013, à 17:53

@ k3c
à mon tour de te remercier pour la maj ! je me demandais si on ne pourrait pas intégrer le calcul des hash du lecteur flash dans le script pour éviter ces désagréments. Est-ce que tu les as calculés toi-même ou les as-tu récupérés d'un autre programme ?

@ 11gjm
les version françaises et originales sont des vidéos différentes : il ne s'agit pas d'une option comme c'est parfois le cas par exemple pour la qualités de la vidéo (qualité standard ou haute), il y a bein une adresse différente pour chacune d'elle. Exemple S02E18 VF et S02E18 VO.
Il suffit donc d'utiliser le bon lien comme argument du script. Comme signalé, la vo est en SQ et HQ tandis que la vf est uniquement en SQ.

A+

k3c · Le 22/04/2013, à 18:23

@ bibichouchou

Pluzzdl le fait, et j'y pensais

$ grep decompressSWF -A 8  /usr/share/pluzzdl/PluzzDL.py
	def decompressSWF( self, swfData ):
		"""
		Decompresse un fichier swf
		"""
		# Adapted from :
		#    Prozacgod
		#    [url]http://www.python-forum.org/pythonforum/viewtopic.php?f=2&t=14693[/url]
		if( type( swfData ) is str ):
			swfData = StringIO.StringIO( swfData )
--
		playerDataUncompress = self.decompressSWF( playerData )
		# Perform sha256 of uncompressed SWF player
		hashPlayer = hashlib.sha256( playerDataUncompress ).hexdigest()
		# Perform base64
		return base64.encodestring( hashPlayer.decode( 'hex' ) )

Dernière modification par k3c (Le 22/04/2013, à 18:24)

11gjm · Le 22/04/2013, à 20:24

Bonjour ,

Merci à vous tous , pour vos réponses .

Cordialement .

pacome · Le 23/04/2013, à 14:05

Bonjour,

Je vous propose ma petite contribution qui intègre:

une correction d'indentation qui empêchait le script de sortir à l'issue du téléchargement
une conversion ffmpeg pour corriger le conteneur de la vidéo téléchargée (qui rendait la lecture impossible sur notre téléviseur )
la suppression de quelques print

#!/usr/bin/python
#-*- coding:utf-8 -*-
# TF1 TMC NT1 HD1 V0.9.2 par k3c, bibichouchou et pacome

import subprocess, optparse, re, sys, shlex
import socket
from urllib2 import urlopen
import time, md5, random, urllib2, json
import bs4 as BeautifulSoup
import os                       # → os.remove

listeUserAgents = [ 'Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5_5; fr-fr) AppleWebKit/525.18 (KHTML, like Gecko) Version/3.1.2 Safari/525.20.1',
                                                'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/14.0.835.186 Safari/535.1',
                                                'Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US) AppleWebKit/525.13 (KHTML, like Gecko) Chrome/0.2.149.27 Safari/525.',
                                                'Mozilla/5.0 (X11; U; Linux x86_64; en-us) AppleWebKit/528.5+ (KHTML, like Gecko, Safari/528.5+) midori',
                                                'Mozilla/5.0 (Windows NT 6.0) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/13.0.782.107 Safari/535.1',
                                                'Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en-us) AppleWebKit/312.1 (KHTML, like Gecko) Safari/312',
                                                'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.12 Safari/535.11',
                                                'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.8 (KHTML, like Gecko) Chrome/17.0.940.0 Safari/535.8' ]

WEBROOTWAT          = "http://www.wat.tv"
wat_url      = "/web/"
jsonVideosInfos    = ""
ua = random.choice(listeUserAgents)

def get_soup(url, referer, ua):
    """ on récupère la soupe """
    req  = urllib2.Request(url)
    req.add_header('User-Agent', ua)
    req.add_header('Referer', referer)
    soup = urllib2.urlopen(req).read()
    return soup

def get_wat(id, HDFlag):
    """la fonction qui permet de retrouver une video sur wat"""
    def base36encode(number):
        if not isinstance(number, (int, long)):
            raise TypeError('number must be an integer')
        if number < 0:
            raise ValueError('number must be positive')
        alphabet = '0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ'
        base36 = ''
        while number:
            number, i = divmod(number, 36)
            base36 = alphabet[i] + base36
        return base36 or alphabet[0]
    if HDFlag:
        wat_url = "/webhd/"
    else:
        wat_url = "/web/"
    ts = base36encode(int(time.time())-60)
    timesec = hex(int(ts, 36))[2:]
    while(len(timesec)<8):
        timesec = "0"+timesec
    token = md5.new("9b673b13fa4682ed14c3cfa5af5310274b514c4133e9b3a81e6e3aba00912564"+wat_url+str(id)+""+timesec).hexdigest()
    id_url1 = WEBROOTWAT+"/get"+wat_url+str(id)+"?token="+token+"/"+str(timesec)+"&country=FR&getURL=1"
    return id_url1

def main():
    """ recuperation de vidéos sur TF1/TMC/NT1/HD1 (donc WAT)"""
    # timeout en secondes
    socket.setdefaulttimeout(90)
    usage   = "usage: python tmc_tf1.py     [options] <url de l'emission>"
    parser  = optparse.OptionParser( usage = usage )
    parser.add_option( "--nocolor",         action = 'store_true', default = False, help = 'desactive la couleur dans le terminal' )
    parser.add_option( "-v", "--verbose",   action = "store_true", default = False, help = 'affiche les informations de debugage' )
    ( options, args ) = parser.parse_args()
    if( len( args ) > 2 or args[ 0 ] == "" ):
        parser.print_help()
        parser.exit( 1 )
    debut_id = ''
    html = urllib2.urlopen(sys.argv[1]).read()
    nom = sys.argv[1].split('/')[-1:][0]
    no = nom.split('.')[-2:][0]
    soup = BeautifulSoup.BeautifulSoup(html)
    if 'tmc.tv' in str(soup) or 'tf1.fr' in str(soup):
        debut_id = str(soup.find('div', attrs={'class' : 'unique' }))
    if 'nt1.tv' in str(soup) or 'hd1.tv' in str(soup):
        debut_id = str(soup.find('section', attrs={'class' : 'player-unique' }))
    id = [x.strip() for x in re.findall("mediaId :([^,]*)", debut_id)][0]
    referer = [x.strip() for x in re.findall('url : "(.*?)"', debut_id)][0]
    jsonVideoInfos = get_soup(WEBROOTWAT+'/interface/contentv3/'+id, referer, ua)
    videoInfos     = json.loads(jsonVideoInfos)

    try:
        HD = videoInfos["media"]["files"][0]["hasHD"]
    except:
        HD = False

    NumberOfParts = len(videoInfos["media"]["files"])
    ListOfIds = []
    for iPart in range(NumberOfParts):
        ListOfIds.append(videoInfos["media"]["files"][iPart]["id"])

    for PartId in ListOfIds:
        id_url1 = get_wat(PartId, HD)
        req  = urllib2.Request(id_url1)
        req.add_header('User-Agent', ua)
        req.add_header('Referer', referer)
        data = urllib2.urlopen(req).read()
        # print data
        # print type(data)
        if data[0:4] == 'http':
            arguments = 'curl "%s" -C - -L -g -A "%s" -o "%s.mp4"' % (data, ua, no + "-" + str(PartId))
            print arguments
            process = subprocess.Popen(arguments, stdout=subprocess.PIPE, shell=True).communicate()[0]
        if data[0:4] == 'rtmp':
            if '.hd' in data:
                data0 = re.search('rtmpte://(.*)hd', data).group(0)
            if '.h264' in data:
                data0 = re.search('rtmpte://(.*)h264', data).group(0)
            data0 = data0.replace('rtmpte','rtmpe')
            fName=str(no) + "-" + str(PartId) # nom du fichier final sans extension
            cmds = 'rtmpdump -e -r "%s" -c 443 -m 10 -w 0818931e9bfa764b9c33e42de6d06f924ac7fc244d0d4941019b9cdfe8706705 -x 352043 -o "%s.t.mp4"' % (data0, fName)
            print cmds
            arguments = shlex.split( cmds )
            # print arguments
            cpt = 0 
            while True:
                p = subprocess.Popen( arguments,
                    stdout=subprocess.PIPE, stderr=subprocess.PIPE)
                stdout, stderr = p.communicate()
                if p.returncode != 0:
                    print "Erreur : le sous-process s\'est terminé avec (le code d\'erreur est " + str(p.returncode) + ")"
                    if cpt > 5:
                        break
                    cpt += 1
                    time.sleep(3) 
                else:
                    # conversion ffmpeg fName.t.mp4 → fName.mp4 (pour corriger le conteneur)
                    cmdFfmpeg='ffmpeg -i "%s.t.mp4" -acodec copy -vcodec copy "%s.mp4"' % (fName, fName)
                    arguments=shlex.split(cmdFfmpeg)
                    p=subprocess.Popen(arguments, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
                    stdout, stderr = p.communicate()
                    if p.returncode != 0:
                        print 'Erreur: la conversion ffmpeg s\'est terminée avec le code d\'erreur %i.\nLe fichier %s.t.mp4 est néanmois disponible' % (p.returncode, fName)
                    else:
                        # suppression du fichier temporaire
                        os.remove(fName+'.t.mp4')
                    break

if __name__ == "__main__":
    main()

J'ai utilisé os.remove pour supprimer le fichier temporaire de façon à rester indépendant de l'OS, puisque je crois que ffmpeg/avconv existe aussi pour windows et macOS.

Par contre je n'ai pas encore réussi à recalculer le hash du swf: j'ai récupéré le player à l'url http://www.wat.tv/swfnt1/264128nIc0K1110456151 et utilisé le code de PluzzDL.py, mais la chaîne obtenue ne correspond pas...
Ce n'est peut-être pas la bonne url, ou il faut lui passer des arguments?
À+

k3c · Le 23/04/2013, à 15:09

@ pacome

Ca fait plaisir de voir des personnes contribuer ! Surtout quand chaoswizard et vermouth semblent abandonner le projet.

Extrait de
http://stream-recorder.com/forum/rtmpdu … ight=flasm

SWF Verification
Download the swf player you want to use for SWFVerification, unzip it using
Code:
$ flasm -x file.swf
It will show the decompressed filesize, use it for --swfsize

Now generate the hash
Code:
$ openssl sha -sha256 -hmac "Genuine Adobe Flash Player 001" file.swf
and use the --swfhash "01234..." option to pass it. e.g.
Code:
$ ./rtmpdump --swfhash "123456..." --swfsize 987...

Et ça fait longtemps que je ne l'ai pas fait, en grand paresseux, je me contente de lancer un téléchargement par Captvty, puis je fais un

wmic process get commandline

Sinon, je vais proposer une modif mineure du script, le test pour savoir si on est sur TMC ou TF1 ... est mauvais.
/sujet à problème. Il faudrait utiliser urlparse et tester juste la bonne partie de sys.argv[1], pas soup.

k3c · Le 23/04/2013, à 17:03

Modif mineure, version 0.9.3, merci à Pacome

testé avec
http://videos.nt1.tv/fringe/saison-2-ep … 9-846.html
http://videos.tf1.fr/les-experts-manhat … 29103.html
http://videos.tmc.tv/hercule-poirot/her … 1-848.html

#!/usr/bin/python
#-*- coding:utf-8 -*-
# TF1 TMC NT1 HD1 V0.9.3 par k3c, bibichouchou et pacome

import subprocess, optparse, re, sys, shlex
import socket
from urllib2 import urlopen
import time, md5, random, urllib2, json
import bs4 as BeautifulSoup
import os                       # → os.remove
from urlparse import urlparse

listeUserAgents = [ 'Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5_5; fr-fr) AppleWebKit/525.18 (KHTML, like Gecko) Version/3.1.2 Safari/525.20.1',
                                                'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/14.0.835.186 Safari/535.1',
                                                'Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US) AppleWebKit/525.13 (KHTML, like Gecko) Chrome/0.2.149.27 Safari/525.',
                                                'Mozilla/5.0 (X11; U; Linux x86_64; en-us) AppleWebKit/528.5+ (KHTML, like Gecko, Safari/528.5+) midori',
                                                'Mozilla/5.0 (Windows NT 6.0) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/13.0.782.107 Safari/535.1',
                                                'Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en-us) AppleWebKit/312.1 (KHTML, like Gecko) Safari/312',
                                                'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.12 Safari/535.11',
                                                'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.8 (KHTML, like Gecko) Chrome/17.0.940.0 Safari/535.8' ]

WEBROOTWAT          = "http://www.wat.tv"
wat_url      = "/web/"
jsonVideosInfos    = ""
ua = random.choice(listeUserAgents)

def get_soup(url, referer, ua):
    """ on récupère la soupe """
    req  = urllib2.Request(url)
    req.add_header('User-Agent', ua)
    req.add_header('Referer', referer)
    soup = urllib2.urlopen(req).read()
    return soup

def get_wat(id, HDFlag):
    """la fonction qui permet de retrouver une video sur wat"""
    def base36encode(number):
        if not isinstance(number, (int, long)):
            raise TypeError('number must be an integer')
        if number < 0:
            raise ValueError('number must be positive')
        alphabet = '0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ'
        base36 = ''
        while number:
            number, i = divmod(number, 36)
            base36 = alphabet[i] + base36
        return base36 or alphabet[0]
    if HDFlag:
        wat_url = "/webhd/"
    else:
        wat_url = "/web/"
    ts = base36encode(int(time.time())-60)
    timesec = hex(int(ts, 36))[2:]
    while(len(timesec)<8):
        timesec = "0"+timesec
    token = md5.new("9b673b13fa4682ed14c3cfa5af5310274b514c4133e9b3a81e6e3aba00912564"+wat_url+str(id)+""+timesec).hexdigest()
    id_url1 = WEBROOTWAT+"/get"+wat_url+str(id)+"?token="+token+"/"+str(timesec)+"&country=FR&getURL=1"
    return id_url1

def main():
    """ recuperation de vidéos sur TF1/TMC/NT1/HD1 (donc WAT)"""
    # timeout en secondes
    socket.setdefaulttimeout(90)
    usage   = "usage: python tmc_tf1.py     [options] <url de l'emission>"
    parser  = optparse.OptionParser( usage = usage )
    parser.add_option( "--nocolor",         action = 'store_true', default = False, help = 'desactive la couleur dans le terminal' )
    parser.add_option( "-v", "--verbose",   action = "store_true", default = False, help = 'affiche les informations de debugage' )
    ( options, args ) = parser.parse_args()
    if( len( args ) > 2 or args[ 0 ] == "" ):
        parser.print_help()
        parser.exit( 1 )
    debut_id = ''
    html = urllib2.urlopen(sys.argv[1]).read()
    nom = sys.argv[1].split('/')[-1:][0]
    no = nom.split('.')[-2:][0]
    soup = BeautifulSoup.BeautifulSoup(html)
    site = urlparse(sys.argv[1]).netloc
    if 'tmc.tv' in site or 'tf1.fr' in site:
        debut_id = str(soup.find('div', attrs={'class' : 'unique' }))
    if 'nt1.tv' in site or 'hd1.tv' in site:
        debut_id = str(soup.find('section', attrs={'class' : 'player-unique' }))
    id = [x.strip() for x in re.findall("mediaId :([^,]*)", debut_id)][0]
    referer = [x.strip() for x in re.findall('url : "(.*?)"', debut_id)][0]
    jsonVideoInfos = get_soup(WEBROOTWAT+'/interface/contentv3/'+id, referer, ua)
    videoInfos     = json.loads(jsonVideoInfos)

    try:
        HD = videoInfos["media"]["files"][0]["hasHD"]
    except:
        HD = False

    NumberOfParts = len(videoInfos["media"]["files"])
    ListOfIds = []
    for iPart in range(NumberOfParts):
        ListOfIds.append(videoInfos["media"]["files"][iPart]["id"])

    for PartId in ListOfIds:
        id_url1 = get_wat(PartId, HD)
        req  = urllib2.Request(id_url1)
        req.add_header('User-Agent', ua)
        req.add_header('Referer', referer)
        data = urllib2.urlopen(req).read()
        # print data
        # print type(data)
        if data[0:4] == 'http':
            arguments = 'curl "%s" -C - -L -g -A "%s" -o "%s.mp4"' % (data, ua, no + "-" + str(PartId))
            print arguments
            process = subprocess.Popen(arguments, stdout=subprocess.PIPE, shell=True).communicate()[0]
        if data[0:4] == 'rtmp':
            if '.hd' in data:
                data0 = re.search('rtmpte://(.*)hd', data).group(0)
            if '.h264' in data:
                data0 = re.search('rtmpte://(.*)h264', data).group(0)
            data0 = data0.replace('rtmpte','rtmpe')
            fName=str(no) + "-" + str(PartId) # nom du fichier final sans extension
            cmds = 'rtmpdump -e -r "%s" -c 443 -m 10 -w 0818931e9bfa764b9c33e42de6d06f924ac7fc244d0d4941019b9cdfe8706705 -x 352043 -o "%s.t.mp4"' % (data0, fName)
            print cmds
            arguments = shlex.split( cmds )
            # print arguments
            cpt = 0 
            while True:
                p = subprocess.Popen( arguments,
                    stdout=subprocess.PIPE, stderr=subprocess.PIPE)
                stdout, stderr = p.communicate()
                if p.returncode != 0:
                    print "Erreur : le sous-process s\'est terminé avec (le code d\'erreur est " + str(p.returncode) + ")"
                    if cpt > 5:
                        break
                    cpt += 1
                    time.sleep(3) 
                else:
                    # conversion ffmpeg fName.t.mp4 → fName.mp4 (pour corriger le conteneur)
                    cmdFfmpeg='ffmpeg -i "%s.t.mp4" -acodec copy -vcodec copy "%s.mp4"' % (fName, fName)
                    arguments=shlex.split(cmdFfmpeg)
                    p=subprocess.Popen(arguments, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
                    stdout, stderr = p.communicate()
                    if p.returncode != 0:
                        print 'Erreur: la conversion ffmpeg s\'est terminée avec le code d\'erreur %i.\nLe fichier %s.t.mp4 est néanmois disponible' % (p.returncode, fName)
                    else:
                        # suppression du fichier temporaire
                        os.remove(fName+'.t.mp4')
                    break

if __name__ == "__main__":
    main()

11gjm · Le 23/04/2013, à 19:12

Bonjour ,

@k3c : lors de la tentative , pour essayer la v0.9.3 , je n'ai pas pu tester .

J'ai un problème avec le fichier "urllib2.py" qui date du 10/04/2012 .
Au niveau des lignes : 126,378,400,418,1177,1207.

Est-ce que quelqu'un pourrait le copier et le coller ICI , entre code et /code ? MERCI .
Je n'ai pas envie d'effectuer l'install , vu que je suis sous Windows .

A+ .

Dernière modification par 11gjm (Le 23/04/2013, à 19:13)

k3c · Le 23/04/2013, à 19:28

Enjoy (en théorie 1463 lignes, je ne suis pas sûr que tout y soit)

"""An extensible library for opening URLs using a variety of protocols

The simplest way to use this module is to call the urlopen function,
which accepts a string containing a URL or a Request object (described
below).  It opens the URL and returns the results as file-like
object; the returned object has some extra methods described below.

The OpenerDirector manages a collection of Handler objects that do
all the actual work.  Each Handler implements a particular protocol or
option.  The OpenerDirector is a composite object that invokes the
Handlers needed to open the requested URL.  For example, the
HTTPHandler performs HTTP GET and POST requests and deals with
non-error returns.  The HTTPRedirectHandler automatically deals with
HTTP 301, 302, 303 and 307 redirect errors, and the HTTPDigestAuthHandler
deals with digest authentication.

urlopen(url, data=None) -- Basic usage is the same as original
urllib.  pass the url and optionally data to post to an HTTP URL, and
get a file-like object back.  One difference is that you can also pass
a Request instance instead of URL.  Raises a URLError (subclass of
IOError); for HTTP errors, raises an HTTPError, which can also be
treated as a valid response.

build_opener -- Function that creates a new OpenerDirector instance.
Will install the default handlers.  Accepts one or more Handlers as
arguments, either instances or Handler classes that it will
instantiate.  If one of the argument is a subclass of the default
handler, the argument will be installed instead of the default.

install_opener -- Installs a new opener as the default opener.

objects of interest:

OpenerDirector -- Sets up the User Agent as the Python-urllib client and manages
the Handler classes, while dealing with requests and responses.

Request -- An object that encapsulates the state of a request.  The
state can be as simple as the URL.  It can also include extra HTTP
headers, e.g. a User-Agent.

BaseHandler --

exceptions:
URLError -- A subclass of IOError, individual protocols have their own
specific subclass.

HTTPError -- Also a valid HTTP response, so you can treat an HTTP error
as an exceptional event or valid response.

internals:
BaseHandler and parent
_call_chain conventions

Example usage:

import urllib2

# set up authentication info
authinfo = urllib2.HTTPBasicAuthHandler()
authinfo.add_password(realm='PDQ Application',
                      uri='https://mahler:8092/site-updates.py',
                      user='klem',
                      passwd='geheim$parole')

proxy_support = urllib2.ProxyHandler({"http" : "http://ahad-haam:3128"})

# build a new opener that adds authentication and caching FTP handlers
opener = urllib2.build_opener(proxy_support, authinfo, urllib2.CacheFTPHandler)

# install it
urllib2.install_opener(opener)

f = urllib2.urlopen('http://www.python.org/')


"""

# XXX issues:
# If an authentication error handler that tries to perform
# authentication for some reason but fails, how should the error be
# signalled?  The client needs to know the HTTP error code.  But if
# the handler knows that the problem was, e.g., that it didn't know
# that hash algo that requested in the challenge, it would be good to
# pass that information along to the client, too.
# ftp errors aren't handled cleanly
# check digest against correct (i.e. non-apache) implementation

# Possible extensions:
# complex proxies  XXX not sure what exactly was meant by this
# abstract factory for opener

import base64
import hashlib
import httplib
import mimetools
import os
import posixpath
import random
import re
import socket
import sys
import time
import urlparse
import bisect

try:
    from cStringIO import StringIO
except ImportError:
    from StringIO import StringIO

from urllib import (unwrap, unquote, splittype, splithost, quote,
     addinfourl, splitport, splittag,
     splitattr, ftpwrapper, splituser, splitpasswd, splitvalue)

# support for FileHandler, proxies via environment variables
from urllib import localhost, url2pathname, getproxies, proxy_bypass

# used in User-Agent header sent
__version__ = sys.version[:3]

_opener = None
def urlopen(url, data=None, timeout=socket._GLOBAL_DEFAULT_TIMEOUT):
    global _opener
    if _opener is None:
        _opener = build_opener()
    return _opener.open(url, data, timeout)

def install_opener(opener):
    global _opener
    _opener = opener

# do these error classes make sense?
# make sure all of the IOError stuff is overridden.  we just want to be
# subtypes.

class URLError(IOError):
    # URLError is a sub-type of IOError, but it doesn't share any of
    # the implementation.  need to override __init__ and __str__.
    # It sets self.args for compatibility with other EnvironmentError
    # subclasses, but args doesn't have the typical format with errno in
    # slot 0 and strerror in slot 1.  This may be better than nothing.
    def __init__(self, reason):
        self.args = reason,
        self.reason = reason

    def __str__(self):
        return '<urlopen error %s>' % self.reason

class HTTPError(URLError, addinfourl):
    """Raised when HTTP error occurs, but also acts like non-error return"""
    __super_init = addinfourl.__init__

    def __init__(self, url, code, msg, hdrs, fp):
        self.code = code
        self.msg = msg
        self.hdrs = hdrs
        self.fp = fp
        self.filename = url
        # The addinfourl classes depend on fp being a valid file
        # object.  In some cases, the HTTPError may not have a valid
        # file object.  If this happens, the simplest workaround is to
        # not initialize the base classes.
        if fp is not None:
            self.__super_init(fp, hdrs, url, code)

    def __str__(self):
        return 'HTTP Error %s: %s' % (self.code, self.msg)

    # since URLError specifies a .reason attribute, HTTPError should also
    #  provide this attribute. See issue13211 fo discussion.
    @property
    def reason(self):
        return self.msg

# copied from cookielib.py
_cut_port_re = re.compile(r":\d+$")
def request_host(request):
    """Return request-host, as defined by RFC 2965.

    Variation from RFC: returned value is lowercased, for convenient
    comparison.

    """
    url = request.get_full_url()
    host = urlparse.urlparse(url)[1]
    if host == "":
        host = request.get_header("Host", "")

    # remove port, if present
    host = _cut_port_re.sub("", host, 1)
    return host.lower()

class Request:

    def __init__(self, url, data=None, headers={},
                 origin_req_host=None, unverifiable=False):
        # unwrap('<URL:type://host/path>') --> 'type://host/path'
        self.__original = unwrap(url)
        self.__original, self.__fragment = splittag(self.__original)
        self.type = None
        # self.__r_type is what's left after doing the splittype
        self.host = None
        self.port = None
        self._tunnel_host = None
        self.data = data
        self.headers = {}
        for key, value in headers.items():
            self.add_header(key, value)
        self.unredirected_hdrs = {}
        if origin_req_host is None:
            origin_req_host = request_host(self)
        self.origin_req_host = origin_req_host
        self.unverifiable = unverifiable

    def __getattr__(self, attr):
        # XXX this is a fallback mechanism to guard against these
        # methods getting called in a non-standard order.  this may be
        # too complicated and/or unnecessary.
        # XXX should the __r_XXX attributes be public?
        if attr[:12] == '_Request__r_':
            name = attr[12:]
            if hasattr(Request, 'get_' + name):
                getattr(self, 'get_' + name)()
                return getattr(self, attr)
        raise AttributeError, attr

    def get_method(self):
        if self.has_data():
            return "POST"
        else:
            return "GET"

    # XXX these helper methods are lame

    def add_data(self, data):
        self.data = data

    def has_data(self):
        return self.data is not None

    def get_data(self):
        return self.data

    def get_full_url(self):
        if self.__fragment:
            return '%s#%s' % (self.__original, self.__fragment)
        else:
            return self.__original

    def get_type(self):
        if self.type is None:
            self.type, self.__r_type = splittype(self.__original)
            if self.type is None:
                raise ValueError, "unknown url type: %s" % self.__original
        return self.type

    def get_host(self):
        if self.host is None:
            self.host, self.__r_host = splithost(self.__r_type)
            if self.host:
                self.host = unquote(self.host)
        return self.host

    def get_selector(self):
        return self.__r_host

    def set_proxy(self, host, type):
        if self.type == 'https' and not self._tunnel_host:
            self._tunnel_host = self.host
        else:
            self.type = type
            self.__r_host = self.__original

        self.host = host

    def has_proxy(self):
        return self.__r_host == self.__original

    def get_origin_req_host(self):
        return self.origin_req_host

    def is_unverifiable(self):
        return self.unverifiable

    def add_header(self, key, val):
        # useful for something like authentication
        self.headers[key.capitalize()] = val

    def add_unredirected_header(self, key, val):
        # will not be added to a redirected request
        self.unredirected_hdrs[key.capitalize()] = val

    def has_header(self, header_name):
        return (header_name in self.headers or
                header_name in self.unredirected_hdrs)

    def get_header(self, header_name, default=None):
        return self.headers.get(
            header_name,
            self.unredirected_hdrs.get(header_name, default))

    def header_items(self):
        hdrs = self.unredirected_hdrs.copy()
        hdrs.update(self.headers)
        return hdrs.items()

class OpenerDirector:
    def __init__(self):
        client_version = "Python-urllib/%s" % __version__
        self.addheaders = [('User-agent', client_version)]
        # self.handlers is retained only for backward compatibility
        self.handlers = []
        # manage the individual handlers
        self.handle_open = {}
        self.handle_error = {}
        self.process_response = {}
        self.process_request = {}

    def add_handler(self, handler):
        if not hasattr(handler, "add_parent"):
            raise TypeError("expected BaseHandler instance, got %r" %
                            type(handler))

        added = False
        for meth in dir(handler):
            if meth in ["redirect_request", "do_open", "proxy_open"]:
                # oops, coincidental match
                continue

            i = meth.find("_")
            protocol = meth[:i]
            condition = meth[i+1:]

            if condition.startswith("error"):
                j = condition.find("_") + i + 1
                kind = meth[j+1:]
                try:
                    kind = int(kind)
                except ValueError:
                    pass
                lookup = self.handle_error.get(protocol, {})
                self.handle_error[protocol] = lookup
            elif condition == "open":
                kind = protocol
                lookup = self.handle_open
            elif condition == "response":
                kind = protocol
                lookup = self.process_response
            elif condition == "request":
                kind = protocol
                lookup = self.process_request
            else:
                continue

            handlers = lookup.setdefault(kind, [])
            if handlers:
                bisect.insort(handlers, handler)
            else:
                handlers.append(handler)
            added = True

        if added:
            bisect.insort(self.handlers, handler)
            handler.add_parent(self)

    def close(self):
        # Only exists for backwards compatibility.
        pass

    def _call_chain(self, chain, kind, meth_name, *args):
        # Handlers raise an exception if no one else should try to handle
        # the request, or return None if they can't but another handler
        # could.  Otherwise, they return the response.
        handlers = chain.get(kind, ())
        for handler in handlers:
            func = getattr(handler, meth_name)

            result = func(*args)
            if result is not None:
                return result

    def open(self, fullurl, data=None, timeout=socket._GLOBAL_DEFAULT_TIMEOUT):
        # accept a URL or a Request object
        if isinstance(fullurl, basestring):
            req = Request(fullurl, data)
        else:
            req = fullurl
            if data is not None:
                req.add_data(data)

        req.timeout = timeout
        protocol = req.get_type()

        # pre-process request
        meth_name = protocol+"_request"
        for processor in self.process_request.get(protocol, []):
            meth = getattr(processor, meth_name)
            req = meth(req)

        response = self._open(req, data)

        # post-process response
        meth_name = protocol+"_response"
        for processor in self.process_response.get(protocol, []):
            meth = getattr(processor, meth_name)
            response = meth(req, response)

        return response

    def _open(self, req, data=None):
        result = self._call_chain(self.handle_open, 'default',
                                  'default_open', req)
        if result:
            return result

        protocol = req.get_type()
        result = self._call_chain(self.handle_open, protocol, protocol +
                                  '_open', req)
        if result:
            return result

        return self._call_chain(self.handle_open, 'unknown',
                                'unknown_open', req)

    def error(self, proto, *args):
        if proto in ('http', 'https'):
            # XXX http[s] protocols are special-cased
            dict = self.handle_error['http'] # https is not different than http
            proto = args[2]  # YUCK!
            meth_name = 'http_error_%s' % proto
            http_err = 1
            orig_args = args
        else:
            dict = self.handle_error
            meth_name = proto + '_error'
            http_err = 0
        args = (dict, proto, meth_name) + args
        result = self._call_chain(*args)
        if result:
            return result

        if http_err:
            args = (dict, 'default', 'http_error_default') + orig_args
            return self._call_chain(*args)

# XXX probably also want an abstract factory that knows when it makes
# sense to skip a superclass in favor of a subclass and when it might
# make sense to include both

def build_opener(*handlers):
    """Create an opener object from a list of handlers.

    The opener will use several default handlers, including support
    for HTTP, FTP and when applicable, HTTPS.

    If any of the handlers passed as arguments are subclasses of the
    default handlers, the default handlers will not be used.
    """
    import types
    def isclass(obj):
        return isinstance(obj, (types.ClassType, type))

    opener = OpenerDirector()
    default_classes = [ProxyHandler, UnknownHandler, HTTPHandler,
                       HTTPDefaultErrorHandler, HTTPRedirectHandler,
                       FTPHandler, FileHandler, HTTPErrorProcessor]
    if hasattr(httplib, 'HTTPS'):
        default_classes.append(HTTPSHandler)
    skip = set()
    for klass in default_classes:
        for check in handlers:
            if isclass(check):
                if issubclass(check, klass):
                    skip.add(klass)
            elif isinstance(check, klass):
                skip.add(klass)
    for klass in skip:
        default_classes.remove(klass)

    for klass in default_classes:
        opener.add_handler(klass())

    for h in handlers:
        if isclass(h):
            h = h()
        opener.add_handler(h)
    return opener

class BaseHandler:
    handler_order = 500

    def add_parent(self, parent):
        self.parent = parent

    def close(self):
        # Only exists for backwards compatibility
        pass

    def __lt__(self, other):
        if not hasattr(other, "handler_order"):
            # Try to preserve the old behavior of having custom classes
            # inserted after default ones (works only for custom user
            # classes which are not aware of handler_order).
            return True
        return self.handler_order < other.handler_order


class HTTPErrorProcessor(BaseHandler):
    """Process HTTP error responses."""
    handler_order = 1000  # after all other processing

    def http_response(self, request, response):
        code, msg, hdrs = response.code, response.msg, response.info()

        # According to RFC 2616, "2xx" code indicates that the client's
        # request was successfully received, understood, and accepted.
        if not (200 <= code < 300):
            response = self.parent.error(
                'http', request, response, code, msg, hdrs)

        return response

    https_response = http_response

class HTTPDefaultErrorHandler(BaseHandler):
    def http_error_default(self, req, fp, code, msg, hdrs):
        raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)

class HTTPRedirectHandler(BaseHandler):
    # maximum number of redirections to any single URL
    # this is needed because of the state that cookies introduce
    max_repeats = 4
    # maximum total number of redirections (regardless of URL) before
    # assuming we're in a loop
    max_redirections = 10

    def redirect_request(self, req, fp, code, msg, headers, newurl):
        """Return a Request or None in response to a redirect.

        This is called by the http_error_30x methods when a
        redirection response is received.  If a redirection should
        take place, return a new Request to allow http_error_30x to
        perform the redirect.  Otherwise, raise HTTPError if no-one
        else should try to handle this url.  Return None if you can't
        but another Handler might.
        """
        m = req.get_method()
        if (code in (301, 302, 303, 307) and m in ("GET", "HEAD")
            or code in (301, 302, 303) and m == "POST"):
            # Strictly (according to RFC 2616), 301 or 302 in response
            # to a POST MUST NOT cause a redirection without confirmation
            # from the user (of urllib2, in this case).  In practice,
            # essentially all clients do redirect in this case, so we
            # do the same.
            # be conciliant with URIs containing a space
            newurl = newurl.replace(' ', '%20')
            newheaders = dict((k,v) for k,v in req.headers.items()
                              if k.lower() not in ("content-length", "content-type")
                             )
            return Request(newurl,
                           headers=newheaders,
                           origin_req_host=req.get_origin_req_host(),
                           unverifiable=True)
        else:
            raise HTTPError(req.get_full_url(), code, msg, headers, fp)

    # Implementation note: To avoid the server sending us into an
    # infinite loop, the request object needs to track what URLs we
    # have already seen.  Do this by adding a handler-specific
    # attribute to the Request object.
    def http_error_302(self, req, fp, code, msg, headers):
        # Some servers (incorrectly) return multiple Location headers
        # (so probably same goes for URI).  Use first header.
        if 'location' in headers:
            newurl = headers.getheaders('location')[0]
        elif 'uri' in headers:
            newurl = headers.getheaders('uri')[0]
        else:
            return

        # fix a possible malformed URL
        urlparts = urlparse.urlparse(newurl)
        if not urlparts.path:
            urlparts = list(urlparts)
            urlparts[2] = "/"
        newurl = urlparse.urlunparse(urlparts)

        newurl = urlparse.urljoin(req.get_full_url(), newurl)

        # For security reasons we do not allow redirects to protocols
        # other than HTTP, HTTPS or FTP.
        newurl_lower = newurl.lower()
        if not (newurl_lower.startswith('http://') or
                newurl_lower.startswith('https://') or
                newurl_lower.startswith('ftp://')):
            raise HTTPError(newurl, code,
                            msg + " - Redirection to url '%s' is not allowed" %
                            newurl,
                            headers, fp)

        # XXX Probably want to forget about the state of the current
        # request, although that might interact poorly with other
        # handlers that also use handler-specific request attributes
        new = self.redirect_request(req, fp, code, msg, headers, newurl)
        if new is None:
            return

        # loop detection
        # .redirect_dict has a key url if url was previously visited.
        if hasattr(req, 'redirect_dict'):
            visited = new.redirect_dict = req.redirect_dict
            if (visited.get(newurl, 0) >= self.max_repeats or
                len(visited) >= self.max_redirections):
                raise HTTPError(req.get_full_url(), code,
                                self.inf_msg + msg, headers, fp)
        else:
            visited = new.redirect_dict = req.redirect_dict = {}
        visited[newurl] = visited.get(newurl, 0) + 1

        # Don't close the fp until we are sure that we won't use it
        # with HTTPError.
        fp.read()
        fp.close()

        return self.parent.open(new, timeout=req.timeout)

    http_error_301 = http_error_303 = http_error_307 = http_error_302

    inf_msg = "The HTTP server returned a redirect error that would " \
              "lead to an infinite loop.\n" \
              "The last 30x error message was:\n"


def _parse_proxy(proxy):
    """Return (scheme, user, password, host/port) given a URL or an authority.

    If a URL is supplied, it must have an authority (host:port) component.
    According to RFC 3986, having an authority component means the URL must
    have two slashes after the scheme:

    >>> _parse_proxy('file:/ftp.example.com/')
    Traceback (most recent call last):
    ValueError: proxy URL with no authority: 'file:/ftp.example.com/'

    The first three items of the returned tuple may be None.

    Examples of authority parsing:

    >>> _parse_proxy('proxy.example.com')
    (None, None, None, 'proxy.example.com')
    >>> _parse_proxy('proxy.example.com:3128')
    (None, None, None, 'proxy.example.com:3128')

    The authority component may optionally include userinfo (assumed to be
    username:password):

    >>> _parse_proxy('joe:password@proxy.example.com')
    (None, 'joe', 'password', 'proxy.example.com')
    >>> _parse_proxy('joe:password@proxy.example.com:3128')
    (None, 'joe', 'password', 'proxy.example.com:3128')

    Same examples, but with URLs instead:

    >>> _parse_proxy('http://proxy.example.com/')
    ('http', None, None, 'proxy.example.com')
    >>> _parse_proxy('http://proxy.example.com:3128/')
    ('http', None, None, 'proxy.example.com:3128')
    >>> _parse_proxy('http://joe:password@proxy.example.com/')
    ('http', 'joe', 'password', 'proxy.example.com')
    >>> _parse_proxy('http://joe:password@proxy.example.com:3128')
    ('http', 'joe', 'password', 'proxy.example.com:3128')

    Everything after the authority is ignored:

    >>> _parse_proxy('ftp://joe:password@proxy.example.com/rubbish:3128')
    ('ftp', 'joe', 'password', 'proxy.example.com')

    Test for no trailing '/' case:

    >>> _parse_proxy('http://joe:password@proxy.example.com')
    ('http', 'joe', 'password', 'proxy.example.com')

    """
    scheme, r_scheme = splittype(proxy)
    if not r_scheme.startswith("/"):
        # authority
        scheme = None
        authority = proxy
    else:
        # URL
        if not r_scheme.startswith("//"):
            raise ValueError("proxy URL with no authority: %r" % proxy)
        # We have an authority, so for RFC 3986-compliant URLs (by ss 3.
        # and 3.3.), path is empty or starts with '/'
        end = r_scheme.find("/", 2)
        if end == -1:
            end = None
        authority = r_scheme[2:end]
    userinfo, hostport = splituser(authority)
    if userinfo is not None:
        user, password = splitpasswd(userinfo)
    else:
        user = password = None
    return scheme, user, password, hostport

class ProxyHandler(BaseHandler):
    # Proxies must be in front
    handler_order = 100

    def __init__(self, proxies=None):
        if proxies is None:
            proxies = getproxies()
        assert hasattr(proxies, 'has_key'), "proxies must be a mapping"
        self.proxies = proxies
        for type, url in proxies.items():
            setattr(self, '%s_open' % type,
                    lambda r, proxy=url, type=type, meth=self.proxy_open: \
                    meth(r, proxy, type))

    def proxy_open(self, req, proxy, type):
        orig_type = req.get_type()
        proxy_type, user, password, hostport = _parse_proxy(proxy)

        if proxy_type is None:
            proxy_type = orig_type

        if req.host and proxy_bypass(req.host):
            return None

        if user and password:
            user_pass = '%s:%s' % (unquote(user), unquote(password))
            creds = base64.b64encode(user_pass).strip()
            req.add_header('Proxy-authorization', 'Basic ' + creds)
        hostport = unquote(hostport)
        req.set_proxy(hostport, proxy_type)

        if orig_type == proxy_type or orig_type == 'https':
            # let other handlers take care of it
            return None
        else:
            # need to start over, because the other handlers don't
            # grok the proxy's URL type
            # e.g. if we have a constructor arg proxies like so:
            # {'http': 'ftp://proxy.example.com'}, we may end up turning
            # a request for http://acme.example.com/a into one for
            # ftp://proxy.example.com/a
            return self.parent.open(req, timeout=req.timeout)

class HTTPPasswordMgr:

    def __init__(self):
        self.passwd = {}

    def add_password(self, realm, uri, user, passwd):
        # uri could be a single URI or a sequence
        if isinstance(uri, basestring):
            uri = [uri]
        if not realm in self.passwd:
            self.passwd[realm] = {}
        for default_port in True, False:
            reduced_uri = tuple(
                [self.reduce_uri(u, default_port) for u in uri])
            self.passwd[realm][reduced_uri] = (user, passwd)

    def find_user_password(self, realm, authuri):
        domains = self.passwd.get(realm, {})
        for default_port in True, False:
            reduced_authuri = self.reduce_uri(authuri, default_port)
            for uris, authinfo in domains.iteritems():
                for uri in uris:
                    if self.is_suburi(uri, reduced_authuri):
                        return authinfo
        return None, None

    def reduce_uri(self, uri, default_port=True):
        """Accept authority or URI and extract only the authority and path."""
        # note HTTP URLs do not have a userinfo component
        parts = urlparse.urlsplit(uri)
        if parts[1]:
            # URI
            scheme = parts[0]
            authority = parts[1]
            path = parts[2] or '/'
        else:
            # host or host:port
            scheme = None
            authority = uri
            path = '/'
        host, port = splitport(authority)
        if default_port and port is None and scheme is not None:
            dport = {"http": 80,
                     "https": 443,
                     }.get(scheme)
            if dport is not None:
                authority = "%s:%d" % (host, dport)
        return authority, path

    def is_suburi(self, base, test):
        """Check if test is below base in a URI tree

        Both args must be URIs in reduced form.
        """
        if base == test:
            return True
        if base[0] != test[0]:
            return False
        common = posixpath.commonprefix((base[1], test[1]))
        if len(common) == len(base[1]):
            return True
        return False


class HTTPPasswordMgrWithDefaultRealm(HTTPPasswordMgr):

    def find_user_password(self, realm, authuri):
        user, password = HTTPPasswordMgr.find_user_password(self, realm,
                                                            authuri)
        if user is not None:
            return user, password
        return HTTPPasswordMgr.find_user_password(self, None, authuri)


class AbstractBasicAuthHandler:

    # XXX this allows for multiple auth-schemes, but will stupidly pick
    # the last one with a realm specified.

    # allow for double- and single-quoted realm values
    # (single quotes are a violation of the RFC, but appear in the wild)
    rx = re.compile('(?:.*,)*[ \t]*([^ \t]+)[ \t]+'
                    'realm=(["\'])(.*?)\\2', re.I)

    # XXX could pre-emptively send auth info already accepted (RFC 2617,
    # end of section 2, and section 1.2 immediately after "credentials"
    # production).

    def __init__(self, password_mgr=None):
        if password_mgr is None:
            password_mgr = HTTPPasswordMgr()
        self.passwd = password_mgr
        self.add_password = self.passwd.add_password
        self.retried = 0

    def reset_retry_count(self):
        self.retried = 0

    def http_error_auth_reqed(self, authreq, host, req, headers):
        # host may be an authority (without userinfo) or a URL with an
        # authority
        # XXX could be multiple headers
        authreq = headers.get(authreq, None)

        if self.retried > 5:
            # retry sending the username:password 5 times before failing.
            raise HTTPError(req.get_full_url(), 401, "basic auth failed",
                            headers, None)
        else:
            self.retried += 1

        if authreq:
            mo = AbstractBasicAuthHandler.rx.search(authreq)
            if mo:
                scheme, quote, realm = mo.groups()
                if scheme.lower() == 'basic':
                    response = self.retry_http_basic_auth(host, req, realm)
                    if response and response.code != 401:
                        self.retried = 0
                    return response

    def retry_http_basic_auth(self, host, req, realm):
        user, pw = self.passwd.find_user_password(realm, host)
        if pw is not None:
            raw = "%s:%s" % (user, pw)
            auth = 'Basic %s' % base64.b64encode(raw).strip()
            if req.headers.get(self.auth_header, None) == auth:
                return None
            req.add_unredirected_header(self.auth_header, auth)
            return self.parent.open(req, timeout=req.timeout)
        else:
            return None


class HTTPBasicAuthHandler(AbstractBasicAuthHandler, BaseHandler):

    auth_header = 'Authorization'

    def http_error_401(self, req, fp, code, msg, headers):
        url = req.get_full_url()
        response = self.http_error_auth_reqed('www-authenticate',
                                              url, req, headers)
        self.reset_retry_count()
        return response


class ProxyBasicAuthHandler(AbstractBasicAuthHandler, BaseHandler):

    auth_header = 'Proxy-authorization'

    def http_error_407(self, req, fp, code, msg, headers):
        # http_error_auth_reqed requires that there is no userinfo component in
        # authority.  Assume there isn't one, since urllib2 does not (and
        # should not, RFC 3986 s. 3.2.1) support requests for URLs containing
        # userinfo.
        authority = req.get_host()
        response = self.http_error_auth_reqed('proxy-authenticate',
                                          authority, req, headers)
        self.reset_retry_count()
        return response


def randombytes(n):
    """Return n random bytes."""
    # Use /dev/urandom if it is available.  Fall back to random module
    # if not.  It might be worthwhile to extend this function to use
    # other platform-specific mechanisms for getting random bytes.
    if os.path.exists("/dev/urandom"):
        f = open("/dev/urandom")
        s = f.read(n)
        f.close()
        return s
    else:
        L = [chr(random.randrange(0, 256)) for i in range(n)]
        return "".join(L)

class AbstractDigestAuthHandler:
    # Digest authentication is specified in RFC 2617.

    # XXX The client does not inspect the Authentication-Info header
    # in a successful response.

    # XXX It should be possible to test this implementation against
    # a mock server that just generates a static set of challenges.

    # XXX qop="auth-int" supports is shaky

    def __init__(self, passwd=None):
        if passwd is None:
            passwd = HTTPPasswordMgr()
        self.passwd = passwd
        self.add_password = self.passwd.add_password
        self.retried = 0
        self.nonce_count = 0
        self.last_nonce = None

    def reset_retry_count(self):
        self.retried = 0

    def http_error_auth_reqed(self, auth_header, host, req, headers):
        authreq = headers.get(auth_header, None)
        if self.retried > 5:
            # Don't fail endlessly - if we failed once, we'll probably
            # fail a second time. Hm. Unless the Password Manager is
            # prompting for the information. Crap. This isn't great
            # but it's better than the current 'repeat until recursion
            # depth exceeded' approach <wink>
            raise HTTPError(req.get_full_url(), 401, "digest auth failed",
                            headers, None)
        else:
            self.retried += 1
        if authreq:
            scheme = authreq.split()[0]
            if scheme.lower() == 'digest':
                return self.retry_http_digest_auth(req, authreq)

    def retry_http_digest_auth(self, req, auth):
        token, challenge = auth.split(' ', 1)
        chal = parse_keqv_list(parse_http_list(challenge))
        auth = self.get_authorization(req, chal)
        if auth:
            auth_val = 'Digest %s' % auth
            if req.headers.get(self.auth_header, None) == auth_val:
                return None
            req.add_unredirected_header(self.auth_header, auth_val)
            resp = self.parent.open(req, timeout=req.timeout)
            return resp

    def get_cnonce(self, nonce):
        # The cnonce-value is an opaque
        # quoted string value provided by the client and used by both client
        # and server to avoid chosen plaintext attacks, to provide mutual
        # authentication, and to provide some message integrity protection.
        # This isn't a fabulous effort, but it's probably Good Enough.
        dig = hashlib.sha1("%s:%s:%s:%s" % (self.nonce_count, nonce, time.ctime(),
                                            randombytes(8))).hexdigest()
        return dig[:16]

    def get_authorization(self, req, chal):
        try:
            realm = chal['realm']
            nonce = chal['nonce']
            qop = chal.get('qop')
            algorithm = chal.get('algorithm', 'MD5')
            # mod_digest doesn't send an opaque, even though it isn't
            # supposed to be optional
            opaque = chal.get('opaque', None)
        except KeyError:
            return None

        H, KD = self.get_algorithm_impls(algorithm)
        if H is None:
            return None

        user, pw = self.passwd.find_user_password(realm, req.get_full_url())
        if user is None:
            return None

        # XXX not implemented yet
        if req.has_data():
            entdig = self.get_entity_digest(req.get_data(), chal)
        else:
            entdig = None

        A1 = "%s:%s:%s" % (user, realm, pw)
        A2 = "%s:%s" % (req.get_method(),
                        # XXX selector: what about proxies and full urls
                        req.get_selector())
        if qop == 'auth':
            if nonce == self.last_nonce:
                self.nonce_count += 1
            else:
                self.nonce_count = 1
                self.last_nonce = nonce

            ncvalue = '%08x' % self.nonce_count
            cnonce = self.get_cnonce(nonce)
            noncebit = "%s:%s:%s:%s:%s" % (nonce, ncvalue, cnonce, qop, H(A2))
            respdig = KD(H(A1), noncebit)
        elif qop is None:
            respdig = KD(H(A1), "%s:%s" % (nonce, H(A2)))
        else:
            # XXX handle auth-int.
            raise URLError("qop '%s' is not supported." % qop)

        # XXX should the partial digests be encoded too?

        base = 'username="%s", realm="%s", nonce="%s", uri="%s", ' \
               'response="%s"' % (user, realm, nonce, req.get_selector(),
                                  respdig)
        if opaque:
            base += ', opaque="%s"' % opaque
        if entdig:
            base += ', digest="%s"' % entdig
        base += ', algorithm="%s"' % algorithm
        if qop:
            base += ', qop=auth, nc=%s, cnonce="%s"' % (ncvalue, cnonce)
        return base

    def get_algorithm_impls(self, algorithm):
        # algorithm should be case-insensitive according to RFC2617
        algorithm = algorithm.upper()
        # lambdas assume digest modules are imported at the top level
        if algorithm == 'MD5':
            H = lambda x: hashlib.md5(x).hexdigest()
        elif algorithm == 'SHA':
            H = lambda x: hashlib.sha1(x).hexdigest()
        # XXX MD5-sess
        KD = lambda s, d: H("%s:%s" % (s, d))
        return H, KD

    def get_entity_digest(self, data, chal):
        # XXX not implemented yet
        return None


class HTTPDigestAuthHandler(BaseHandler, AbstractDigestAuthHandler):
    """An authentication protocol defined by RFC 2069

    Digest authentication improves on basic authentication because it
    does not transmit passwords in the clear.
    """

    auth_header = 'Authorization'
    handler_order = 490  # before Basic auth

    def http_error_401(self, req, fp, code, msg, headers):
        host = urlparse.urlparse(req.get_full_url())[1]
        retry = self.http_error_auth_reqed('www-authenticate',
                                           host, req, headers)
        self.reset_retry_count()
        return retry


class ProxyDigestAuthHandler(BaseHandler, AbstractDigestAuthHandler):

    auth_header = 'Proxy-Authorization'
    handler_order = 490  # before Basic auth

    def http_error_407(self, req, fp, code, msg, headers):
        host = req.get_host()
        retry = self.http_error_auth_reqed('proxy-authenticate',
                                           host, req, headers)
        self.reset_retry_count()
        return retry

class AbstractHTTPHandler(BaseHandler):

    def __init__(self, debuglevel=0):
        self._debuglevel = debuglevel

    def set_http_debuglevel(self, level):
        self._debuglevel = level

    def do_request_(self, request):
        host = request.get_host()
        if not host:
            raise URLError('no host given')

        if request.has_data():  # POST
            data = request.get_data()
            if not request.has_header('Content-type'):
                request.add_unredirected_header(
                    'Content-type',
                    'application/x-www-form-urlencoded')
            if not request.has_header('Content-length'):
                request.add_unredirected_header(
                    'Content-length', '%d' % len(data))

        sel_host = host
        if request.has_proxy():
            scheme, sel = splittype(request.get_selector())
            sel_host, sel_path = splithost(sel)

        if not request.has_header('Host'):
            request.add_unredirected_header('Host', sel_host)
        for name, value in self.parent.addheaders:
            name = name.capitalize()
            if not request.has_header(name):
                request.add_unredirected_header(name, value)

        return request

    def do_open(self, http_class, req):
        """Return an addinfourl object for the request, using http_class.

        http_class must implement the HTTPConnection API from httplib.
        The addinfourl return value is a file-like object.  It also
        has methods and attributes including:
            - info(): return a mimetools.Message object for the headers
            - geturl(): return the original request URL
            - code: HTTP status code
        """
        host = req.get_host()
        if not host:
            raise URLError('no host given')

        h = http_class(host, timeout=req.timeout) # will parse host:port
        h.set_debuglevel(self._debuglevel)

        headers = dict(req.unredirected_hdrs)
        headers.update(dict((k, v) for k, v in req.headers.items()
                            if k not in headers))

        # We want to make an HTTP/1.1 request, but the addinfourl
        # class isn't prepared to deal with a persistent connection.
        # It will try to read all remaining data from the socket,
        # which will block while the server waits for the next request.
        # So make sure the connection gets closed after the (only)
        # request.
        headers["Connection"] = "close"
        headers = dict(
            (name.title(), val) for name, val in headers.items())

        if req._tunnel_host:
            tunnel_headers = {}
            proxy_auth_hdr = "Proxy-Authorization"
            if proxy_auth_hdr in headers:
                tunnel_headers[proxy_auth_hdr] = headers[proxy_auth_hdr]
                # Proxy-Authorization should not be sent to origin
                # server.
                del headers[proxy_auth_hdr]
            h.set_tunnel(req._tunnel_host, headers=tunnel_headers)

        try:
            h.request(req.get_method(), req.get_selector(), req.data, headers)
        except socket.error, err: # XXX what error?
            h.close()
            raise URLError(err)
        else:
            try:
                r = h.getresponse(buffering=True)
            except TypeError: # buffering kw not supported
                r = h.getresponse()

        # Pick apart the HTTPResponse object to get the addinfourl
        # object initialized properly.

        # Wrap the HTTPResponse object in socket's file object adapter
        # for Windows.  That adapter calls recv(), so delegate recv()
        # to read().  This weird wrapping allows the returned object to
        # have readline() and readlines() methods.

        # XXX It might be better to extract the read buffering code
        # out of socket._fileobject() and into a base class.

        r.recv = r.read
        fp = socket._fileobject(r, close=True)

        resp = addinfourl(fp, r.msg, req.get_full_url())
        resp.code = r.status
        resp.msg = r.reason
        return resp


class HTTPHandler(AbstractHTTPHandler):

    def http_open(self, req):
        return self.do_open(httplib.HTTPConnection, req)

    http_request = AbstractHTTPHandler.do_request_

if hasattr(httplib, 'HTTPS'):
    class HTTPSHandler(AbstractHTTPHandler):

        def https_open(self, req):
            return self.do_open(httplib.HTTPSConnection, req)

        https_request = AbstractHTTPHandler.do_request_

class HTTPCookieProcessor(BaseHandler):
    def __init__(self, cookiejar=None):
        import cookielib
        if cookiejar is None:
            cookiejar = cookielib.CookieJar()
        self.cookiejar = cookiejar

    def http_request(self, request):
        self.cookiejar.add_cookie_header(request)
        return request

    def http_response(self, request, response):
        self.cookiejar.extract_cookies(response, request)
        return response

    https_request = http_request
    https_response = http_response

class UnknownHandler(BaseHandler):
    def unknown_open(self, req):
        type = req.get_type()
        raise URLError('unknown url type: %s' % type)

def parse_keqv_list(l):
    """Parse list of key=value strings where keys are not duplicated."""
    parsed = {}
    for elt in l:
        k, v = elt.split('=', 1)
        if v[0] == '"' and v[-1] == '"':
            v = v[1:-1]
        parsed[k] = v
    return parsed

def parse_http_list(s):
    """Parse lists as described by RFC 2068 Section 2.

    In particular, parse comma-separated lists where the elements of
    the list may include quoted-strings.  A quoted-string could
    contain a comma.  A non-quoted string could have quotes in the
    middle.  Neither commas nor quotes count if they are escaped.
    Only double-quotes count, not single-quotes.
    """
    res = []
    part = ''

    escape = quote = False
    for cur in s:
        if escape:
            part += cur
            escape = False
            continue
        if quote:
            if cur == '\\':
                escape = True
                continue
            elif cur == '"':
                quote = False
            part += cur
            continue

        if cur == ',':
            res.append(part)
            part = ''
            continue

        if cur == '"':
            quote = True

        part += cur

    # append last part
    if part:
        res.append(part)

    return [part.strip() for part in res]

def _safe_gethostbyname(host):
    try:
        return socket.gethostbyname(host)
    except socket.gaierror:
        return None

class FileHandler(BaseHandler):
    # Use local file or FTP depending on form of URL
    def file_open(self, req):
        url = req.get_selector()
        if url[:2] == '//' and url[2:3] != '/' and (req.host and
                req.host != 'localhost'):
            req.type = 'ftp'
            return self.parent.open(req)
        else:
            return self.open_local_file(req)

    # names for the localhost
    names = None
    def get_names(self):
        if FileHandler.names is None:
            try:
                FileHandler.names = tuple(
                    socket.gethostbyname_ex('localhost')[2] +
                    socket.gethostbyname_ex(socket.gethostname())[2])
            except socket.gaierror:
                FileHandler.names = (socket.gethostbyname('localhost'),)
        return FileHandler.names

    # not entirely sure what the rules are here
    def open_local_file(self, req):
        import email.utils
        import mimetypes
        host = req.get_host()
        filename = req.get_selector()
        localfile = url2pathname(filename)
        try:
            stats = os.stat(localfile)
            size = stats.st_size
            modified = email.utils.formatdate(stats.st_mtime, usegmt=True)
            mtype = mimetypes.guess_type(filename)[0]
            headers = mimetools.Message(StringIO(
                'Content-type: %s\nContent-length: %d\nLast-modified: %s\n' %
                (mtype or 'text/plain', size, modified)))
            if host:
                host, port = splitport(host)
            if not host or \
                (not port and _safe_gethostbyname(host) in self.get_names()):
                if host:
                    origurl = 'file://' + host + filename
                else:
                    origurl = 'file://' + filename
                return addinfourl(open(localfile, 'rb'), headers, origurl)
        except OSError, msg:
            # urllib2 users shouldn't expect OSErrors coming from urlopen()
            raise URLError(msg)
        raise URLError('file not on local host')

class FTPHandler(BaseHandler):
    def ftp_open(self, req):
        import ftplib
        import mimetypes
        host = req.get_host()
        if not host:
            raise URLError('ftp error: no host given')
        host, port = splitport(host)
        if port is None:
            port = ftplib.FTP_PORT
        else:
            port = int(port)

        # username/password handling
        user, host = splituser(host)
        if user:
            user, passwd = splitpasswd(user)
        else:
            passwd = None
        host = unquote(host)
        user = user or ''
        passwd = passwd or ''

        try:
            host = socket.gethostbyname(host)
        except socket.error, msg:
            raise URLError(msg)
        path, attrs = splitattr(req.get_selector())
        dirs = path.split('/')
        dirs = map(unquote, dirs)
        dirs, file = dirs[:-1], dirs[-1]
        if dirs and not dirs[0]:
            dirs = dirs[1:]
        try:
            fw = self.connect_ftp(user, passwd, host, port, dirs, req.timeout)
            type = file and 'I' or 'D'
            for attr in attrs:
                attr, value = splitvalue(attr)
                if attr.lower() == 'type' and \
                   value in ('a', 'A', 'i', 'I', 'd', 'D'):
                    type = value.upper()
            fp, retrlen = fw.retrfile(file, type)
            headers = ""
            mtype = mimetypes.guess_type(req.get_full_url())[0]
            if mtype:
                headers += "Content-type: %s\n" % mtype
            if retrlen is not None and retrlen >= 0:
                headers += "Content-length: %d\n" % retrlen
            sf = StringIO(headers)
            headers = mimetools.Message(sf)
            return addinfourl(fp, headers, req.get_full_url())
        except ftplib.all_errors, msg:
            raise URLError, ('ftp error: %s' % msg), sys.exc_info()[2]

    def connect_ftp(self, user, passwd, host, port, dirs, timeout):
        fw = ftpwrapper(user, passwd, host, port, dirs, timeout,
                        persistent=False)
##        fw.ftp.set_debuglevel(1)
        return fw

class CacheFTPHandler(FTPHandler):
    # XXX would be nice to have pluggable cache strategies
    # XXX this stuff is definitely not thread safe
    def __init__(self):
        self.cache = {}
        self.timeout = {}
        self.soonest = 0
        self.delay = 60
        self.max_conns = 16

    def setTimeout(self, t):
        self.delay = t

    def setMaxConns(self, m):
        self.max_conns = m

    def connect_ftp(self, user, passwd, host, port, dirs, timeout):
        key = user, host, port, '/'.join(dirs), timeout
        if key in self.cache:
            self.timeout[key] = time.time() + self.delay
        else:
            self.cache[key] = ftpwrapper(user, passwd, host, port, dirs, timeout)
            self.timeout[key] = time.time() + self.delay
        self.check_cache()
        return self.cache[key]

    def check_cache(self):
        # first check for old ones
        t = time.time()
        if self.soonest <= t:
            for k, v in self.timeout.items():
                if v < t:
                    self.cache[k].close()
                    del self.cache[k]
                    del self.timeout[k]
        self.soonest = min(self.timeout.values())

        # then check the size
        if len(self.cache) == self.max_conns:
            for k, v in self.timeout.items():
                if v == self.soonest:
                    del self.cache[k]
                    del self.timeout[k]
                    break
            self.soonest = min(self.timeout.values())

    def clear_cache(self):
        for conn in self.cache.values():
            conn.close()
        self.cache.clear()
        self.timeout.clear()

11gjm · Le 23/04/2013, à 20:52

Re-... ,

@k3c : Merci .

Je viens de télécharger "partiellement" la vidéo de NT1 . Ça fonctionne .
Je testerai les 2 autres .

Cordialement .

11gjm · Le 23/04/2013, à 22:01

Re-... ,

Résultats des téléchargements : obtenu 3 vidéos fonctionnelles (en version française) .
Ci-dessous anomalies indiquées à la fin de chaque process .

===
saison-2-episode-18-le-passage-7927329-846-10434745.t.mp4
---
TF1_093.py <= mon fichier

=> ligne 135
p=subprocess.Popen(arguments, stdout=subprocess.PIPE, stderr=subprocess.PIPE)

=> ligne 145
main()
---
subprocess.py

=> 679
errread, errwrite)

=> 876
startupinfo)

WindowsError: [Error 2] le fichier spécifié est introuvable
===
===
episode-13-saison-02-le-dernier-metro-7929103-10456585.t.mp4

Plusieurs erreurs .
===
===
hercule-poirot-mine-perdue-7918591-848-10397917.t.mp4

Mêmes erreurs que pour la 1ère vidéo .
===

Cordialement .

thom83 · Le 23/04/2013, à 22:44

Pour le fichier
saison-2-episode-18-le-passage-7927329-846-10434745.t.mp4
La commandertmpdump -e -r ...etc
est tout de suite suivie de

Erreur : le sous-process s'est terminé avec (le code d'erreur est 1)

qui se répète à l'envie.
Pour le fichier de tf1, j'ai la même erreur, que j'ai stoppée après la deuxième répétition du message et là j'obtiens une info supplémentaire :

^CTraceback (most recent call last):
  File "Bureau/fichiers_TVDownloader/tf1-tmc-v093.py", line 145, in <module>
    main()
  File "Bureau/fichiers_TVDownloader/tf1-tmc-v093.py", line 130, in main
    time.sleep(3)

Pour hercule Poirot, même comportement que pour tf1

Dernière modification par thom83 (Le 23/04/2013, à 22:55)

pacome · Le 23/04/2013, à 23:16

@k3c: La disponibilité de chacun est variable avec le temps et d'autres facteurs (famille, santé, boulot, projets, etc.), mais tant mieux si je peux contribuer à améliorer ces outils qui me rendent bien service

Ceci dit...

@11gjm: je crains que ce soient mes modifications, et plus précisément l'appel à ffmpeg, qui provoquent les erreurs que tu rencontres.
Je n'ai plus de windows depuis bien longtemps, je ne pourrais donc pas t'aider beaucoup de ce coté-là, mais as-tu installé ffmpeg (windows build) sur ton système? si oui il faudra modifier l'appel pour lui donner le chemin complet ou la casse (FFmpeg?); v0.9.3 ligne 133:

                    # conversion ffmpeg fName.t.mp4 → fName.mp4 (pour corriger le conteneur)
                    cmdFfmpeg='ffmpeg -i "%s.t.mp4" -acodec copy -vcodec copy "%s.mp4"' % (fName, fName)

Comme ffmpeg est utilisé dans PluzzDL.py je ne me suis pas posé la question, mais il faudra que je rajoute un test pour vérifier qu'il est là et un petit message explicite si ce n'est pas le cas...

J'ai bien l'intention de creuser cette histoire de hash swf aussi (merci pour le lien k3c) puisque je n'ai pas de windows !

@thom83: Je viens de re-tester et je n'ai aucune erreur; essaye de copier-coller le script à nouveau pour être sur.

À+

JUL1EN094 · Le 24/04/2013, à 02:33

Bonjour à tous,

juste au cas où, c'est pas ca le player de wat : http://www.wat.tv/images/v40/PlayerWat.swf ?

k3c · Le 24/04/2013, à 07:47

JUL1EN094 a écrit :

Bonjour à tous,
juste au cas où, c'est pas ca le player de wat : http://www.wat.tv/images/v40/PlayerWat.swf ?

Merci

Si on le met dans
www.showmycode.com

ça permet de voir ce qu'il fabrique

ynad · Le 24/04/2013, à 11:38

Bonjour,

@ thom83

j'ai les mêmes erreurs que toi et je crois que cela vient de la version de rtmpdump
j'ai fait un essai avec l'utilisation de rtmpdump de windows et en modifiant la ligne 116 cmds = avec wine rtmptdump.exe dans"tf1.0.9.3" ça marche ...

Dernière modification par ynad (Le 24/04/2013, à 11:44)

11gjm · Le 24/04/2013, à 13:21

Bonjour ,

Nouvel essai après modifications :

_ encodé le fichier "TF1_TMC_NT1_HD1_v0.9.3.py" en utf-8
_ ajouté ffmpeg
_ ajouté rtmpdump (v2.5)

Résultat pour "fringe/saison-2-episode-18-le-passage-7927329-846.html" :
Aucune anomalie constatée .

Merci , à tous .

Cordialement .

thom83 · Le 24/04/2013, à 14:42

j'ai les mêmes erreurs que toi et je crois que cela vient de la version de rtmpdump
Avec Ubuntu 10.04 en version amd64, j'ai rtmpdump 2.4~20110711.gitc28f1bab-1 et n'en ai pas trouvé de plus récent.
J'ai refait l'essai avec le dernièr script qui porte le numéro 0.9.3 qui lance la commande

rtmpdump -e -r "rtmpe://wske.wat.tv/ondemand/mp4:vod/HD-1280x720/65/85/10456585.hd" -c 443 -m 10 -w 0818931e9bfa764b9c33e42de6d06f924ac7fc244d0d4941019b9cdfe8706705 -x 352043 -o "episode-13-saison-02-le-dernier-metro-7929103-10456585.t.mp4"

Comme indiqué précédemment, cela se termine par

Erreur : le sous-process s'est terminé avec (le code d'erreur est 1)
^CTraceback (most recent call last):
  File "fichiers_TVDownloader/tf1-tmc-v093.py", line 145, in <module>
    main()
  File "fichiers_TVDownloader/tf1-tmc-v093.py", line 130, in main
    time.sleep(3) 
KeyboardInterrupt

Par curiosité, j'ai utilisé, pour la même émission, le fichier touvé au # de ce fil. Il me donne la commande suivante

rtmpdump -e -r "rtmpe://wske.wat.tv/ondemand/mp4:vod/HD-1280x720/65/85/10456585.hd" -c 443 -m 10 -w b23434cbed89c9eaf520373c4c6f26e1f7326896dee4b1719e8d9acda0c19e99 -x 343427 -o "episode-13-saison-02-le-dernier-metro-7929103-10456585.mp4"

Quand je la lance, cela se termine par

Connecting ...
WARNING: HandShake: Type mismatch: client sent 6, server answered 9
WARNING: HandShake: Server not genuine Adobe!
ERROR: RTMP_Connect1, handshake failed.

Il est à noter que ce qui suit le -w n'est pas identique dans les deux commandes...

bibichouchou · Le 24/04/2013, à 15:06

@thom83
je pense que tu fais référence à la version que j'ai postée, donc au premier message de ce fil.
le script devrait être encore fonctionnel à condition de changer les hashs du lecteur flash donc les chaînes qui se trouvent après les options -w et -x de la commande rtmpdump. regarde dans la nouvelle version du programme et édite la version précédente en conséquence...
je n'ai pas testé la nouvelle version du programme mais d'après ce que j'ai lu, c'est un peu de cosmétique et rien de fondamental, non ?

Dernière modification par bibichouchou (Le 24/04/2013, à 15:07)

11gjm · Le 24/04/2013, à 15:39

Bonjour ,

@thom83 :

Fichier .py utilisé , voir #133
(http://forum.ubuntu-fr.org/viewtopic.ph … #p13300011)

Je viens de tester la commande ( sous windows avec RTMPDUMP V2.5 ) :

rtmpdump -e -r "rtmpe://wske.wat.tv/ondemand/mp4:vod/HD-1280x720/65/85/10456585.hd" -c 443 -m 10 -w 0818931e9bfa764b9c33e42de6d06f924ac7fc244d0d4941019b9cdfe8706705  -x 352043 -o "episode-13-saison-02-le-dernier-metro-7929103-10456585.mp4"
...

client send 6 , server answered 10

1) aucun problème
2) pense à mettre les 3 points , afin de facilter la copie du lien .

A+ .

thom83 · Le 24/04/2013, à 16:11

@ bibichouchou et ynad
Je faisais référence au script bibichouchou effectivement
Je viens de passer sous Ubuntu 12.04 avec aussi rtmpdump 2.4~20110711.gitc28f1bab-1.
Le dernier script de k3c charge l'émission de tf1 sans se plaindre. Le script du 20 mars n'est plus adapté en effet.

pacome · Le 25/04/2013, à 14:04

Bonjour,

J'ai ré-essayé d'utiliser les fonctions de calculs du hash swf de PluzzDL.py dans le tmc_tf1.py, mais il doit y avoir un truc qui m'échappe parce que je ne récupère pas le hash qui fonctionne... et puis à force de l'avoir devant les yeux régulièrement j'ai fini par voir cette option de rtmpdump: --swfVfy|-W url URL to player swf file, compute hash/size automatically

Je vous propose donc une nouvelle version qui intègre:

l'utilisation de l'option -W <URL Player swf> (j'ai trouvé sur www.wat.tv un autre player que celui proposé par JUL1EN094, mais je n'ai pas automatisé cette recherche)
la vérification de la présence des programmes externes requis (il faudra que je revois la vérification des versions)
et du coup l'utilisation optionnelle de ffmpeg (si il n'est pas installé c'est pas grave )
la déclaration de variables concernant ces points en début de code (c'est plus facile à modifier si nécessaire)

#!/usr/bin/python
#-*- coding:utf-8 -*-
# TF1 TMC NT1 HD1 V0.9.4 par k3c, bibichouchou et pacome

import subprocess, optparse, re, sys, shlex
import socket
from urllib2 import urlopen
import time, md5, random, urllib2, json
import bs4 as BeautifulSoup
import os                       # → os.remove
from urlparse import urlparse

# swf players
swfWatPlayer="www.wat.tv/images/v40/PlayerWat.swf" # fourni par JUL1EN094
swfWatPlayerLite="www.wat.tv/images/v70/PlayerLite.swf" # trouvé sur www.wat.tv
# programmes externes utilisés
ffmpegEx='ffmpeg'
rtmpdumpEx='rtmpdump'

listeUserAgents = [ 'Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5_5; fr-fr) AppleWebKit/525.18 (KHTML, like Gecko) Version/3.1.2 Safari/525.20.1',
                    'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/14.0.835.186 Safari/535.1',
                    'Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US) AppleWebKit/525.13 (KHTML, like Gecko) Chrome/0.2.149.27 Safari/525.',
                    'Mozilla/5.0 (X11; U; Linux x86_64; en-us) AppleWebKit/528.5+ (KHTML, like Gecko, Safari/528.5+) midori',
                    'Mozilla/5.0 (Windows NT 6.0) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/13.0.782.107 Safari/535.1',
                    'Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en-us) AppleWebKit/312.1 (KHTML, like Gecko) Safari/312',
                    'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.12 Safari/535.11',
                    'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.8 (KHTML, like Gecko) Chrome/17.0.940.0 Safari/535.8' ]

WEBROOTWAT="http://www.wat.tv"
wat_url="/web/"
jsonVideosInfos=""
ua=random.choice(listeUserAgents)

def checkExternalProgram(prog, optArg='', expectedValue=''):
    """ Permet de vérifier la présence des programmes externes requis """
    args=shlex.split('%s %s' % (prog, optArg))
    try:
        process=subprocess.Popen(args,stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
        stdout, stderr = process.communicate()
        if expectedValue == '':
            return True
        else:
            if expectedValue in stdout: # à améliorer pour versions > ...
                return True
            else:
                return False
    except OSError:
        print 'Le programme %s n\'est pas présent sur votre système' % (prog)
        return False

def get_soup(url, referer, ua):
    """ on récupère la soupe """
    req  = urllib2.Request(url)
    req.add_header('User-Agent', ua)
    req.add_header('Referer', referer)
    soup = urllib2.urlopen(req).read()
    return soup

def get_wat(id, HDFlag):
    """la fonction qui permet de retrouver une video sur wat"""
    def base36encode(number):
        if not isinstance(number, (int, long)):
            raise TypeError('number must be an integer')
        if number < 0:
            raise ValueError('number must be positive')
        alphabet = '0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ'
        base36 = ''
        while number:
            number, i = divmod(number, 36)
            base36 = alphabet[i] + base36
        return base36 or alphabet[0]
    if HDFlag:
        wat_url = "/webhd/"
    else:
        wat_url = "/web/"
    ts = base36encode(int(time.time())-60)
    timesec = hex(int(ts, 36))[2:]
    while(len(timesec)<8):
        timesec = "0"+timesec
    token = md5.new("9b673b13fa4682ed14c3cfa5af5310274b514c4133e9b3a81e6e3aba00912564"+wat_url+str(id)+""+timesec).hexdigest()
    id_url1 = WEBROOTWAT+"/get"+wat_url+str(id)+"?token="+token+"/"+str(timesec)+"&country=FR&getURL=1"
    return id_url1

def main():
    """ recuperation de vidéos sur TF1/TMC/NT1/HD1 (donc WAT)"""
    # vérification de la présence de rtmpdump v2.4 ou v2.5
    if not checkExternalProgram(rtmpdumpEx, '-help', 'v2.4'):
        if not checkExternalProgram(rtmpdumpEx, '-help', 'v2.5'): # pas top...
            print 'Ce script requiert %s v2.4 ou v2.5' % (rtmpdumpEx)
            return
    # timeout en secondes
    socket.setdefaulttimeout(90)
    usage   = "usage: python tmc_tf1.py     [options] <url de l'emission>"
    parser  = optparse.OptionParser( usage = usage )
    parser.add_option( "--nocolor",         action = 'store_true', default = False, help = 'desactive la couleur dans le terminal' )
    parser.add_option( "-v", "--verbose",   action = "store_true", default = False, help = 'affiche les informations de debugage' )
    ( options, args ) = parser.parse_args()
    if( len( args ) > 2 or args[ 0 ] == "" ):
        parser.print_help()
        parser.exit( 1 )
    debut_id = ''
    html = urllib2.urlopen(sys.argv[1]).read()
    nom = sys.argv[1].split('/')[-1:][0]
    no = nom.split('.')[-2:][0]
    soup = BeautifulSoup.BeautifulSoup(html)
    site = urlparse(sys.argv[1]).netloc
    if 'tmc.tv' in site or 'tf1.fr' in site:
        debut_id = str(soup.find('div', attrs={'class' : 'unique' }))
    if 'nt1.tv' in site or 'hd1.tv' in site:
        debut_id = str(soup.find('section', attrs={'class' : 'player-unique' }))
    id = [x.strip() for x in re.findall("mediaId :([^,]*)", debut_id)][0]
    referer = [x.strip() for x in re.findall('url : "(.*?)"', debut_id)][0]
    jsonVideoInfos = get_soup(WEBROOTWAT+'/interface/contentv3/'+id, referer, ua)
    videoInfos     = json.loads(jsonVideoInfos)

    try:
        HD = videoInfos["media"]["files"][0]["hasHD"]
    except:
        HD = False

    NumberOfParts = len(videoInfos["media"]["files"])
    ListOfIds = []
    for iPart in range(NumberOfParts):
        ListOfIds.append(videoInfos["media"]["files"][iPart]["id"])

    for PartId in ListOfIds:
        id_url1 = get_wat(PartId, HD)
        req  = urllib2.Request(id_url1)
        req.add_header('User-Agent', ua)
        req.add_header('Referer', referer)
        data = urllib2.urlopen(req).read()
        # print data
        # print type(data)
        if data[0:4] == 'http':
            arguments = 'curl "%s" -C - -L -g -A "%s" -o "%s.mp4"' % (data, ua, no + "-" + str(PartId))
            print arguments
            process = subprocess.Popen(arguments, stdout=subprocess.PIPE, shell=True).communicate()[0]
        if data[0:4] == 'rtmp':
            if '.hd' in data:
                data0 = re.search('rtmpte://(.*)hd', data).group(0)
            if '.h264' in data:
                data0 = re.search('rtmpte://(.*)h264', data).group(0)
            data0 = data0.replace('rtmpte','rtmpe')
            fName=str(no) + "-" + str(PartId) # nom du fichier final sans extension
            fileName=fName+".mp4"             # nom du fichier final avec extension
            # ffmpeg est-il disponible?
            if not checkExternalProgram(ffmpegEx):
                print "L'installation de ffmpeg sur votre système permettrait de corriger automatiquement le conteneur de la vidéo (flash→mp4)."
                tmpFileName=fileName
                ffmpegAvailable=False
            else:
                tmpFileName=fName+".tmp.mp4"
                ffmpegAvailable=True
            rtmpCmd = '%s -e -r "%s" -c 443 -m 10 -W %s -o "%s"' % (rtmpdumpEx, data0, swfWatPlayerLite, tmpFileName)
            print rtmpCmd
            arguments = shlex.split( rtmpCmd )
            # print arguments
            cpt = 0 
            while True:
                p = subprocess.Popen(arguments, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
                stdout, stderr = p.communicate()
                if p.returncode != 0:
                    print "Erreur : le sous-process s\'est terminé avec (le code d\'erreur est " + str(p.returncode) + ")"
                    if cpt > 5:
                        break
                    cpt += 1
                    time.sleep(3) 
                else:
                    if ffmpegAvailable:
                        # conversion ffmpeg tmpFileName → fileName (pour corriger le conteneur)
                        ffmpegCmd='%s -i "%s" -acodec copy -vcodec copy "%s"' % (ffmpegEx, tmpFileName, fileName)
                        arguments=shlex.split(ffmpegCmd)
                        p=subprocess.Popen(arguments, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
                        stdout, stderr = p.communicate()
                        if p.returncode != 0:
                            print 'Erreur: la conversion ffmpeg s\'est terminée avec le code d\'erreur %i.\nLe fichier %s est néanmois disponible' % (p.returncode, tmpFileName)
                        else:
                            # suppression du fichier temporaire
                            os.remove(tmpFileName)
                    break

if __name__ == "__main__":
    main()

Je me rend compte à l'instant que je ne me suis absolument pas intéressé au téléchargement par curl ... ça sera pour une prochaine version.

À+

11gjm · Le 25/04/2013, à 14:11

Bonjour ,

@pacome : Merci pour ton travail .

J'avoue être un peu dépassé , par tes recherches .
Mes essais avec la V0.9.3 sont concluants .

Existe-t-il d'autres points qui nous empêchent d'obtenir les vidéos ?

Cordialement .

k3c · Le 25/04/2013, à 16:02

@ pacome

Ca a l'air une très bonne idée de passer le player en ligne de commande !

par contre je ne réussis pas à charger le dernier Experts.

gg@gg-SATELLITE-L755:~$ python pac.py http://videos.tf1.fr/les-experts-las-vegas/episode-7-saison-12-le-cerveau-de-la-bande-7929865.html
rtmpdump -e -r "rtmpe://wske.wat.tv/ondemand/mp4:vod/HD-1280x720/09/73/10460973.hd" -c 443 -m 10 -W www.wat.tv/images/v70/PlayerLite.swf -o "episode-7-saison-12-le-cerveau-de-la-bande-7929865-10460973.tmp.mp4"
Erreur : le sous-process s'est terminé avec (le code d'erreur est 2)
Erreur : le sous-process s'est terminé avec (le code d'erreur est 2)
Erreur : le sous-process s'est terminé avec (le code d'erreur est 2)
Erreur : le sous-process s'est terminé avec (le code d'erreur est 2)
Erreur : le sous-process s'est terminé avec (le code d'erreur est 2)
Erreur : le sous-process s'est terminé avec (le code d'erreur est 2)
Erreur : le sous-process s'est terminé avec (le code d'erreur est 2)
gg@gg-SATELLITE-L755:~$

Ubuntu-fr

Navigation

Liens de recherche

Annonce

#126 Le 22/04/2013, à 13:34

Re : TVDownloader: télécharger les médias du net ! [2]

#127 Le 22/04/2013, à 14:30

Re : TVDownloader: télécharger les médias du net ! [2]

#128 Le 22/04/2013, à 17:53

Re : TVDownloader: télécharger les médias du net ! [2]

#129 Le 22/04/2013, à 18:23

Re : TVDownloader: télécharger les médias du net ! [2]

#130 Le 22/04/2013, à 20:24

Re : TVDownloader: télécharger les médias du net ! [2]

#131 Le 23/04/2013, à 14:05

Re : TVDownloader: télécharger les médias du net ! [2]

#132 Le 23/04/2013, à 15:09

Re : TVDownloader: télécharger les médias du net ! [2]

#133 Le 23/04/2013, à 17:03

Re : TVDownloader: télécharger les médias du net ! [2]

#134 Le 23/04/2013, à 19:12

Re : TVDownloader: télécharger les médias du net ! [2]

#135 Le 23/04/2013, à 19:28

Re : TVDownloader: télécharger les médias du net ! [2]

#136 Le 23/04/2013, à 20:52

Re : TVDownloader: télécharger les médias du net ! [2]

#137 Le 23/04/2013, à 22:01

Re : TVDownloader: télécharger les médias du net ! [2]

#138 Le 23/04/2013, à 22:44

Re : TVDownloader: télécharger les médias du net ! [2]

#139 Le 23/04/2013, à 23:16

Re : TVDownloader: télécharger les médias du net ! [2]

#140 Le 24/04/2013, à 02:33

Re : TVDownloader: télécharger les médias du net ! [2]

#141 Le 24/04/2013, à 07:47

Re : TVDownloader: télécharger les médias du net ! [2]

#142 Le 24/04/2013, à 11:38

Re : TVDownloader: télécharger les médias du net ! [2]

#143 Le 24/04/2013, à 13:21

Re : TVDownloader: télécharger les médias du net ! [2]

#144 Le 24/04/2013, à 14:42

Re : TVDownloader: télécharger les médias du net ! [2]

#145 Le 24/04/2013, à 15:06

Re : TVDownloader: télécharger les médias du net ! [2]

#146 Le 24/04/2013, à 15:39

Re : TVDownloader: télécharger les médias du net ! [2]

#147 Le 24/04/2013, à 16:11

Re : TVDownloader: télécharger les médias du net ! [2]

#148 Le 25/04/2013, à 14:04

Re : TVDownloader: télécharger les médias du net ! [2]

#149 Le 25/04/2013, à 14:11

Re : TVDownloader: télécharger les médias du net ! [2]

#150 Le 25/04/2013, à 16:02

Re : TVDownloader: télécharger les médias du net ! [2]

Pied de page des forums