관리-도구
편집 파일: _inputstream.cpython-36.pyc
3 �Rea~ � @ s� d dl mZmZmZ d dlmZ d dlmZmZ d dl Z d dl Z d dl m Z m Z d dlmZ ddlmZmZmZmZ dd lmZ dd lmZ ed d � eD ��Zed d � eD ��Zedd � eD ��Zeeddg�B Z dZ ej �r e dE dkr�e j d�dks�t �e j!e ddF� e"d� d �Z#n e j!e �Z#ddddddddd d d d d d!d"d#d$d%d&d'd(d)d*d+d,d-d.d/d0d1d2d3h Z$e j!d4�Z%i Z&G d5d6� d6e'�Z(d7d8� Z)G d9d:� d:e'�Z*G d;d<� d<e*�Z+G d=d>� d>e,�Z-G d?d@� d@e'�Z.G dAdB� dBe'�Z/dCdD� Z0dS )G� )�absolute_import�division�unicode_literals)� text_type)� http_client�urllibN)�BytesIO�StringIO)� webencodings� )�EOF�spaceCharacters� asciiLetters�asciiUppercase)�_ReparseException)�_utilsc C s g | ]}|j d ��qS )�ascii)�encode)�.0�item� r ��/builddir/build/BUILDROOT/alt-python36-pip-20.2.4-5.el8.x86_64/opt/alt/python36/lib/python3.6/site-packages/pip/_vendor/html5lib/_inputstream.py� <listcomp> s r c C s g | ]}|j d ��qS )r )r )r r r r r r s c C s g | ]}|j d ��qS )r )r )r r r r r r s � >� <u� [- - --]�]z"\uD800-\uDFFF"i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� z[ - -/:-@\[-`{-~]c @ sH e Zd ZdZdd� Zdd� Zdd� Zdd � Zd d � Zd d � Z dd� Z dS )�BufferedStreamz�Buffering for streams that do not have buffering of their own The buffer is implemented as a list of chunks on the assumption that joining many strings will be slow since it is O(n**2) c C s || _ g | _ddg| _d S )Nr r ���)�stream�buffer�position)�selfr r r r �__init__: s zBufferedStream.__init__c C s@ d}x(| j d | jd � D ]}|t|�7 }qW || jd 7 }|S )Nr r )r r �len)r! �pos�chunkr r r �tell? s zBufferedStream.tellc C sX || j � kst�|}d}x0t| j| �|k rH|t| j| �8 }|d7 }qW ||g| _d S )Nr r )�_bufferedBytes�AssertionErrorr# r r )r! r$ �offset�ir r r �seekF s zBufferedStream.seekc C sT | j s| j|�S | jd t| j �krF| jd t| j d �krF| j|�S | j|�S d S )Nr r r )r � _readStreamr r# �_readFromBuffer)r! �bytesr r r �readO s zBufferedStream.readc C s t dd� | jD ��S )Nc S s g | ] }t |��qS r )r# )r r r r r r Y s z1BufferedStream._bufferedBytes.<locals>.<listcomp>)�sumr )r! r r r r' X s z BufferedStream._bufferedBytesc C s< | j j|�}| jj|� | jd d7 < t|�| jd< |S )Nr r )r r/ r �appendr r# )r! r. �datar r r r, [ s zBufferedStream._readStreamc C s� |}g }| j d }| j d }x�|t| j�k r�|dkr�|dks@t�| j| }|t|�| krn|}||| g| _ n"t|�| }|t|�g| _ |d7 }|j|||| � � ||8 }d}q W |r�|j| j|�� dj|�S )Nr r � )r r# r r( r1 r, �join)r! r. �remainingBytes�rv� bufferIndex� bufferOffset� bufferedData� bytesToReadr r r r- b s& z BufferedStream._readFromBufferN) �__name__� __module__� __qualname__�__doc__r" r& r+ r/ r' r, r- r r r r r 3 s r c K s� t | tj�s(t | tjj�r.t | jtj�r.d}n&t| d�rJt | jd�t �}n t | t �}|r�dd� |D �}|rvt d| ��t | f|�S t | f|�S d S )NFr/ r c S s g | ]}|j d �r|�qS )� _encoding)�endswith)r �xr r r r � s z#HTMLInputStream.<locals>.<listcomp>z3Cannot set an encoding with a unicode input, set %r) � isinstancer � HTTPResponser �response�addbase�fp�hasattrr/ r � TypeError�HTMLUnicodeInputStream�HTMLBinaryInputStream)�source�kwargs� isUnicode� encodingsr r r �HTMLInputStream} s rO c @ sp e Zd ZdZdZdd� Zdd� Zdd� Zd d � Zd d � Z d d� Z ddd�Z dd� Z dd� Z d dd�Zdd� ZdS ) rI z�Provides a unicode stream of characters to the HTMLTokenizer. This class takes care of character encoding and removing or replacing incorrect byte-sequences and also provides column and line tracking. i ( c C sZ t jsd| _n td�dkr$| j| _n| j| _dg| _td�df| _| j |�| _ | j � dS )a� Initialises the HTMLInputStream. HTMLInputStream(source, [encoding]) -> Normalized stream from source for use by html5lib. source can be either a file-object, local filename or a string. The optional encoding parameter must be a string that indicates the encoding. If specified, that encoding will be used, regardless of any BOM or later declaration (such as in a meta element) Nu r r zutf-8�certain) r �supports_lone_surrogates�reportCharacterErrorsr# �characterErrorsUCS4�characterErrorsUCS2�newLines�lookupEncoding� charEncoding� openStream� dataStream�reset)r! rK r r r r" � s z HTMLUnicodeInputStream.__init__c C s. d| _ d| _d| _g | _d| _d| _d | _d S )N� r )r% � chunkSize� chunkOffset�errors� prevNumLines� prevNumCols�_bufferedCharacter)r! r r r rZ � s z HTMLUnicodeInputStream.resetc C s t |d�r|}nt|�}|S )zvProduces a file object from source. source can be either a file object, local filename or a string. r/ )rG r )r! rK r r r r rX � s z!HTMLUnicodeInputStream.openStreamc C sT | j }|jdd|�}| j| }|jdd|�}|dkr@| j| }n ||d }||fS )N� r r r )r% �countr_ �rfindr` )r! r) r% �nLines� positionLine� lastLinePos�positionColumnr r r � _position� s z HTMLUnicodeInputStream._positionc C s | j | j�\}}|d |fS )z:Returns (line, col) of the current position in the stream.r )ri r] )r! �line�colr r r r � s z HTMLUnicodeInputStream.positionc C s6 | j | jkr| j� stS | j }| j| }|d | _ |S )zo Read one character from the stream or queue if available. Return EOF when EOF is reached. r )r] r\ � readChunkr r% )r! r] �charr r r rm � s zHTMLUnicodeInputStream.charNc C s� |d kr| j }| j| j�\| _| _d| _d| _d| _| jj|�}| j rX| j | }d | _ n|s`dS t |�dkr�t |d �}|dks�d| ko�dkn r�|d | _ |d d� }| j r�| j |� |j dd �}|j d d �}|| _t |�| _d S )Nr[ r Fr � i � i�� z rb � Tr r r )�_defaultChunkSizeri r\ r_ r` r% r] rY r/ ra r# �ordrR �replace)r! r\ r2 �lastvr r r rl � s0 z HTMLUnicodeInputStream.readChunkc C s, x&t ttj|���D ]}| jjd� qW d S )Nzinvalid-codepoint)�ranger# �invalid_unicode_re�findallr^ r1 )r! r2 �_r r r rS s z*HTMLUnicodeInputStream.characterErrorsUCS4c C s� d}x�t j|�D ]�}|rqt|j� �}|j� }tj|||d � �rttj|||d � �}|tkrn| j j d� d}q|dkr�|dkr�|t |�d kr�| j j d� qd}| j j d� qW d S )NF� zinvalid-codepointTi � i�� r ) ru �finditerrq �group�startr �isSurrogatePair�surrogatePairToCodepoint�non_bmp_invalid_codepointsr^ r1 r# )r! r2 �skip�match� codepointr$ �char_valr r r rT # s z*HTMLUnicodeInputStream.characterErrorsUCS2Fc C s yt ||f }W nl tk r| x |D ]}t|�dk s&t�q&W djdd� |D ��}|s^d| }tjd| � }t ||f< Y nX g }x||j| j| j �}|dkr�| j | j kr�P n0|j � }|| j kr�|j | j| j |� � || _ P |j | j| j d� � | j � s�P q�W dj|�} | S )z� Returns a string of characters from the stream up to but not including any character in 'characters' or EOF. 'characters' must be a container that supports the 'in' method and iteration over its characters. � r[ c S s g | ]}d t |� �qS )z\x%02x)rq )r �cr r r r H s z5HTMLUnicodeInputStream.charsUntil.<locals>.<listcomp>z^%sz[%s]+N)�charsUntilRegEx�KeyErrorrq r( r4 �re�compiler� r% r] r\ �endr1 rl ) r! � characters�opposite�charsr� �regexr6 �mr� �rr r r � charsUntil: s2 z!HTMLUnicodeInputStream.charsUntilc C sT |t k rP| jdkr.|| j | _| jd7 _n"| jd8 _| j| j |ksPt�d S )Nr r )r r] r% r\ r( )r! rm r r r �ungeti s z HTMLUnicodeInputStream.unget)N)F)r; r< r= r> rp r" rZ rX ri r rm rl rS rT r� r� r r r r rI � s & /rI c @ sL e Zd ZdZddd�Zdd� Zd d � Zdd d �Zd d� Zdd� Z dd� Z dS )rJ z�Provides a unicode stream of characters to the HTMLTokenizer. This class takes care of character encoding and removing or replacing incorrect byte-sequences and also provides column and line tracking. N� windows-1252Tc C sn | j |�| _tj| | j� d| _d| _|| _|| _|| _|| _ || _ | j |�| _ | j d dk sbt �| j� dS )a� Initialises the HTMLInputStream. HTMLInputStream(source, [encoding]) -> Normalized stream from source for use by html5lib. source can be either a file-object, local filename or a string. The optional encoding parameter must be a string that indicates the encoding. If specified, that encoding will be used, regardless of any BOM or later declaration (such as in a meta element) i �d r N)rX � rawStreamrI r" � numBytesMeta�numBytesChardet�override_encoding�transport_encoding�same_origin_parent_encoding�likely_encoding�default_encoding�determineEncodingrW r( rZ )r! rK r� r� r� r� r� � useChardetr r r r" � s z HTMLBinaryInputStream.__init__c C s& | j d jj| jd�| _tj| � d S )Nr rr )rW � codec_info� streamreaderr� rY rI rZ )r! r r r rZ � s zHTMLBinaryInputStream.resetc C sL t |d�r|}nt|�}y|j|j� � W n tk rF t|�}Y nX |S )zvProduces a file object from source. source can be either a file object, local filename or a string. r/ )rG r r+ r& � Exceptionr )r! rK r r r r rX � s z HTMLBinaryInputStream.openStreamc C s� | j � df}|d d k r |S t| j�df}|d d k r:|S t| j�df}|d d k rX|S | j� df}|d d k rt|S t| j�df}|d d k r�|d jjd� r�|S t| j�df}|d d k r�|S |�rtyddl m } W n t k r� Y n�X g }|� }xF|j �s>| j j| j�}t|t��s t�|�s(P |j|� |j|� q�W |j� t|jd �}| j jd� |d k �rt|dfS t| j�df}|d d k �r�|S td�dfS )NrP r � tentativezutf-16)�UniversalDetector�encodingz windows-1252)� detectBOMrV r� r� �detectEncodingMetar� �name� startswithr� �%pip._vendor.chardet.universaldetectorr� � ImportError�doner� r/ r� rB r. r( r1 �feed�close�resultr+ r� )r! �chardetrW r� �buffers�detectorr r� r r r r� � sR z'HTMLBinaryInputStream.determineEncodingc C s� | j d dkst�t|�}|d kr&d S |jdkrFtd�}|d k s�t�nT|| j d krf| j d df| _ n4| jjd� |df| _ | j� td| j d |f ��d S ) Nr rP �utf-16be�utf-16lezutf-8r z Encoding changed from %s to %s)r� r� )rW r( rV r� r� r+ rZ r )r! � newEncodingr r r �changeEncoding s z$HTMLBinaryInputStream.changeEncodingc C s� t jdt jdt jdt jdt jdi}| jjd�}t|t �s<t �|j |dd� �}d}|s~|j |�}d}|s~|j |dd � �}d }|r�| jj |� t |�S | jj d � dS dS ) z�Attempts to detect at BOM at the start of the stream. If an encoding can be determined from the BOM return the name of the encoding otherwise return Nonezutf-8zutf-16lezutf-16bezutf-32lezutf-32be� N� rx r )�codecs�BOM_UTF8� BOM_UTF16_LE� BOM_UTF16_BE� BOM_UTF32_LE� BOM_UTF32_BEr� r/ rB r. r( �getr+ rV )r! �bomDict�stringr� r+ r r r r� s$ z HTMLBinaryInputStream.detectBOMc C sV | j j| j�}t|t�s t�t|�}| j jd� |j� }|dk rR|j dkrRt d�}|S )z9Report the encoding declared by the meta element r N�utf-16be�utf-16lezutf-8)r� r� ) r� r/ r� rB r. r( �EncodingParserr+ � getEncodingr� rV )r! r �parserr� r r r r� 3 s z(HTMLBinaryInputStream.detectEncodingMeta)NNNNr� T)T) r; r< r= r> r" rZ rX r� r� r� r� r r r r rJ z s ( >"rJ c @ s� e Zd ZdZdd� Zdd� Zdd� Zdd � Zd d � Zd d � Z dd� Z dd� Z e e e �Z dd� Ze e�Zefdd�Zdd� Zdd� Zdd� Zd S ) � EncodingBytesz�String-like object with an associated position and various extra methods If the position is ever greater than the string length then an exception is raisedc C s t |t�st�tj| |j� �S )N)rB r. r( �__new__�lower)r! �valuer r r r� F s zEncodingBytes.__new__c C s d| _ d S )Nr r )ri )r! r� r r r r" J s zEncodingBytes.__init__c C s | S )Nr )r! r r r �__iter__N s zEncodingBytes.__iter__c C s> | j d }| _ |t| �kr"t�n |dk r.t�| ||d � S )Nr r )ri r# � StopIterationrH )r! �pr r r �__next__Q s zEncodingBytes.__next__c C s | j � S )N)r� )r! r r r �nextY s zEncodingBytes.nextc C sB | j }|t| �krt�n |dk r$t�|d | _ }| ||d � S )Nr r )ri r# r� rH )r! r� r r r �previous] s zEncodingBytes.previousc C s | j t| �krt�|| _ d S )N)ri r# r� )r! r r r r � setPositionf s zEncodingBytes.setPositionc C s* | j t| �krt�| j dkr"| j S d S d S )Nr )ri r# r� )r! r r r � getPositionk s zEncodingBytes.getPositionc C s | | j | j d � S )Nr )r )r! r r r �getCurrentByteu s z EncodingBytes.getCurrentBytec C sL | j }x:|t| �k r@| ||d � }||kr6|| _|S |d7 }qW || _dS )z Skip past a list of charactersr N)r r# ri )r! r� r� r� r r r r z s zEncodingBytes.skipc C sL | j }x:|t| �k r@| ||d � }||kr6|| _|S |d7 }qW || _d S )Nr )r r# ri )r! r� r� r� r r r � skipUntil� s zEncodingBytes.skipUntilc C s( | j || j�}|r$| jt|�7 _|S )z�Look for a sequence of bytes at the start of a string. If the bytes are found return True and advance the position to the byte after the match. Otherwise return False and leave the position alone)r� r r# )r! r. r6 r r r � matchBytes� s zEncodingBytes.matchBytesc C s> y | j || j�t|� d | _W n tk r8 t�Y nX dS )z�Look for the next sequence of bytes matching a given sequence. If a match is found advance the position to the last byte of the matchr T)�indexr r# ri � ValueErrorr� )r! r. r r r �jumpTo� s zEncodingBytes.jumpToN)r; r< r= r> r� r" r� r� r� r� r� r� �propertyr r� � currentByte�spaceCharactersBytesr r� r� r� r r r r r� B s r� c @ sX e Zd ZdZdd� Zdd� Zdd� Zdd � Zd d � Zd d � Z dd� Z dd� Z dd� Z dS )r� z?Mini parser for detecting character encoding from meta elementsc C s t |�| _d| _dS )z3string - the data to work on for encoding detectionN)r� r2 r� )r! r2 r r r r" � s zEncodingParser.__init__c C s� d| j krd S d| jfd| jfd| jfd| jfd| jfd| jff}x�| j D ]|}d}y| j jd� W n tk rx P Y nX xD|D ]<\}}| j j|�r�y |� }P W q� tk r� d}P Y q�X q�W |sJP qJW | j S ) Ns <metas <!--s </s <!s <?r TF) r2 � handleComment� handleMeta�handlePossibleEndTag� handleOther�handlePossibleStartTagr� r� r� r� )r! �methodDispatchrw � keepParsing�key�methodr r r r� � s2 zEncodingParser.getEncodingc C s | j jd�S )zSkip over commentss -->)r2 r� )r! r r r r� � s z EncodingParser.handleCommentc C s� | j jtkrdS d}d }x�| j� }|d kr.dS |d dkr^|d dk}|r�|d k r�|| _dS q|d dkr�|d }t|�}|d k r�|| _dS q|d dkrtt|d ��}|j� }|d k rt|�}|d k r|r�|| _dS |}qW d S ) NTFr s http-equivr s content-types charsets content) r2 r� r� � getAttributer� rV �ContentAttrParserr� �parse)r! � hasPragma�pendingEncoding�attr�tentativeEncoding�codec� contentParserr r r r� � s: zEncodingParser.handleMetac C s | j d�S )NF)�handlePossibleTag)r! r r r r� � s z%EncodingParser.handlePossibleStartTagc C s t | j� | jd�S )NT)r� r2 r� )r! r r r r� � s z#EncodingParser.handlePossibleEndTagc C sf | j }|jtkr(|r$|j� | j� dS |jt�}|dkrD|j� n | j� }x|d k r`| j� }qNW dS )NTr )r2 r� �asciiLettersBytesr� r� r� �spacesAngleBracketsr� )r! �endTagr2 r� r� r r r r� � s z EncodingParser.handlePossibleTagc C s | j jd�S )Nr )r2 r� )r! r r r r� s zEncodingParser.handleOtherc C s | j }|jttdg�B �}|dks2t|�dks2t�|d kr>dS g }g }xt|dkrX|rXP nX|tkrl|j� }P nD|d kr�dj|�dfS |tkr�|j|j � � n|dkr�dS |j|� t |�}qHW |dkr�|j � dj|�dfS t |� |j� }|d k�rT|}x�t |�}||k�r(t |� dj|�dj|�fS |tk�rB|j|j � � n |j|� � q�W nJ|dk�rldj|�dfS |tk�r�|j|j � � n|dk�r�dS |j|� x^t |�}|t k�r�dj|�dj|�fS |tk�r�|j|j � � n|dk�r�dS |j|� �q�W dS ) z_Return a name,value pair for the next attribute in the stream, if one is found, or None� /Nr r � =r3 � '� ")r N)r� r )r� r� ) r2 r r� � frozensetr# r( r4 �asciiUppercaseBytesr1 r� r� r� r� )r! r2 r� �attrName� attrValue� quoteCharr r r r� sh zEncodingParser.getAttributeN) r; r< r= r> r" r� r� r� r� r� r� r� r� r r r r r� � s $r� c @ s e Zd Zdd� Zdd� ZdS )r� c C s t |t�st�|| _d S )N)rB r. r( r2 )r! r2 r r r r" a s zContentAttrParser.__init__c C s y�| j jd� | j jd7 _| j j� | j jdks8d S | j jd7 _| j j� | j jdkr�| j j}| j jd7 _| j j}| j j|�r�| j || j j� S d S nF| j j}y | j jt� | j || j j� S tk r� | j |d � S X W n tk �r d S X d S )Ns charsetr r� r� r� )r� r� )r2 r� r r r� r� r� r� )r! � quoteMark� oldPositionr r r r� e s. zContentAttrParser.parseN)r; r< r= r" r� r r r r r� ` s r� c C s` t | t�r.y| jd�} W n tk r, dS X | dk rXy tj| �S tk rT dS X ndS dS )z{Return the python codec name corresponding to an encoding or None if the string doesn't correspond to a valid encoding.r N)rB r. �decode�UnicodeDecodeErrorr �lookup�AttributeError)r� r r r rV � s rV r r )1� __future__r r r Zpip._vendor.sixr �pip._vendor.six.movesr r r� r� �ior r � pip._vendorr � constantsr r r r r r[ r r� r� r� r� r� � invalid_unicode_no_surrogaterQ rc r( r� �evalru r~ �ascii_punctuation_rer� �objectr rO rI rJ r. r� r� r� rV r r r r �<module> sP J g Ib ='