o …ðÔi+ã@sÐddlZddlmZmZmZddlZddlmmZ ddlmZddl m Z ddl m Z eeeeeffZzddlmZWn eyJdZYnwGdd „d ejƒZGd d „d ejƒZGd d „d ejƒZdS)éN)ÚUnionÚTupleÚOptional)Únn)Ú rearrangeé)ÚClsToken)Úindirect_grid_samplecs–eZdZ           d9dededed ed ed ed eed edededeedeedef‡fdd„ Zde j de j fdd„Z e dd„ƒZ e dd„ƒZe dd„ƒZe dd „ƒZe d!d"„ƒZd#d$„Zd%e j d&ejfd'd(„Zd)e j d*e j fd+d,„Zde j de j fd-d.„Z  d:d/e j d0ee j d1eeeefde j fd2d3„Z  d:d4ed0ee j d1eeeefde j fd5d6„Zd4edeeeffd7d8„Z‡ZS);ÚViTPatchGeneratorTFNçrÚ patch_sizeÚ embed_dimÚ input_dimsÚabs_posÚnormalize_patchesÚ cls_tokenÚmax_input_dimsÚ pos_dropoutÚreturn_pos_encÚnum_cls_tokensÚregister_multipleÚ num_registersÚ patch_biascsTtƒ ¡t|tƒr||f}|dur|}t|tƒr||f}t‡fdd„|Dƒƒ}||k|_||_| |_t||d�}ˆ|_ ||_ ||_ |dˆ|_ |dˆ|_ t‡fdd„|Dƒƒ|_|j |j |_||_tˆƒ|_tˆ|fd| i|¤Ž|_|r�|d}t tjd|j|fi|¤Ž|¡|_t|| || | d �|_|r£t |¡|_dSt ¡|_dS) Nc3s&�|]}tt |ˆ¡ˆƒVqdS©N)ÚintÚmathÚceil©Ú.0Úd©r ©ú@/home/zhangtao/hf_model/vectorllm_hf_0407/vit_patch_generator.pyÚ 6s €ÿ ÿz-ViTPatchGenerator.__init__..)ÚdeviceÚdtyperrc3s�|]}|ˆVqdSrr!rr r!r"r#Gs€Úbiasgà¿)Ú num_tokensÚenabledrr)ÚsuperÚ__init__Ú isinstancerÚtupleÚcpe_moderrÚdictr rr Únum_rowsÚnum_colsrÚ num_patchesrÚ Im2PatchesÚ im_to_patchesÚViTPatchLinearÚembedderrÚ ParameterÚtorchÚrandnÚ pos_embedrrÚ LayerNormÚIdentityÚpatch_normalizer)Úselfr r rrrrrrrrrrrr$r%ÚfactoryÚscale©Ú __class__r r"r*sF    þ   $û"zViTPatchGenerator.__init__ÚxÚreturncCsL| |¡}|j||jdd…d�\}}| |¡}| |¡}|jr$||fS|S)Né)Ú input_size)Ú embed_patchesÚ apply_pos_encÚshaperr<r)r=rBÚpatchesÚpos_encr!r!r"Úforward\s   zViTPatchGenerator.forwardcCó|jjSr)rr(©r=r!r!r"Úapply_cls_tokeneóz!ViTPatchGenerator.apply_cls_tokencCrLr)rr'rMr!r!r"rirOz ViTPatchGenerator.num_cls_tokenscCrLr)rr1rMr!r!r"Únum_cls_patchesmrOz!ViTPatchGenerator.num_cls_patchescCrLr)rrrMr!r!r"rqrOzViTPatchGenerator.num_registerscCs |j|jSr)rrrMr!r!r"Únum_skipus zViTPatchGenerator.num_skipcCsdgS)Nr9r!rMr!r!r"Úno_weight_decayysÿz!ViTPatchGenerator.no_weight_decayÚ src_embedÚ targ_embedcCs€|j|jkr8tt |jd¡ƒ}|d|jdksJdƒ‚t|d||d�}tj||j|jfdddd �}t|d ƒ}|j   |¡dS) NrrDz*Unable to interpolate non-square embeddingzb (h w) c -> b c h w)ÚhÚwÚbicubicTF©ÚsizeÚmodeÚ align_cornersÚ antialiaszb c h w -> b (h w) c) rHrrÚsqrtrÚFÚ interpolater/r0ÚdataÚcopy_)r=rSrTÚsrc_sizer!r!r"Ú _load_embed~s  zViTPatchGenerator._load_embedÚsrc_proj_weightÚtarg_proj_weightcCsŠ|j|jkr=tt |jdd¡ƒ}|dd|jdks!Jdƒ‚t|dd||d�}tj||j|jfddd d �}t|d ƒ}|j  |¡dS) NrérDz+Unable to interpolate non-square patch sizezb (c h w) -> b c h w)ÚcrUrVrWTFrXzb c h w -> b (c h w)) rHrrr]rr^r_r r`ra)r=rdreZsrc_patch_sizer!r!r"Ú_load_projection‰s  z"ViTPatchGenerator._load_projectioncCs0| |¡}|j tj¡|_| | tj¡¡}|Sr)r3r5Útor7Úbfloat16)r=rBrIr!r!r"rF”s zViTPatchGenerator.embed_patchesrIÚ patch_idxsrEcCsr|js|S| |jd||¡}|jr1|jdkr1tj|jddd|j|jd�|jk}t  ||d¡}n|}|||fS)Nrr)r%r$) rÚ get_pos_encrHÚtrainingrr7Úrandr%r$Úwhere)r=rIrkrErJZkeepsZ pos_enc_dropr!r!r"rGšs$ zViTPatchGenerator.apply_pos_encÚ batch_sizecs||durˆj}n t‡fdd„|Dƒƒ}ˆ ||¡}|dur|S| d¡ dd|jd¡}tj| |jddd¡d|d�}|S)Nc3s�|]}|ˆjVqdSrr rrMr!r"r#´s€z0ViTPatchGenerator.get_pos_enc..éÿÿÿÿrr)ÚdimÚindex)rr,Ú_get_pos_embeddingsÚ unsqueezeÚexpandrHr7Úgather)r=rprkrErr9Zexp_patch_idxsr!rMr"rl¬s  zViTPatchGenerator.get_pos_enccs<|j|jfˆkr |jS|j d|j|jd¡ dddd¡}‡fdd„}|jrõ|jrÛt d¡}t j |dd|j d �d||}t  d ¡}| }t   t j |dd|j d �|||¡} || } |d| } t j| | gdd � dd¡} t j |ddd|j d �d| } t jddˆd|j d �d  |ˆdd¡}t jddˆd|j d �ddd…df |dˆd¡}t j||gdd �}|| | }| d¡ d¡tj| ¡ |ddd¡|dddd� |j¡}ntˆƒ}tj| ¡||fddd� |j¡}||ƒ}n||ƒ}|jdd…ˆk�rtj| ¡ˆddd� |j¡}| d¡ ddd¡}|S)NrrqrrfrDcs\ˆd|jdkr|ddˆd…dd…f}ˆd|jdkr,|ddd…dˆd…f}|S)Nréþÿÿÿ.rrq)rH)r9©rr!r"Ú window_selectÇs z.window_selectgš™™™™™¹?)r$gè?)rr)Ústepsr$©NNÚbilinearÚzerosT)ÚgridrZÚ padding_moder[)rYr[rZrx)r/r0r9ÚreshapeÚpermuter-rmrr]r7rnr$ÚlogÚexpÚstackÚclamp_ÚlinspacervÚmul_Úsub_r^Ú grid_sampleÚfloatrir%Úmaxr_rHÚflatten)r=rprr9rzZ min_scaler?Z aspect_minZ aspect_maxÚaspectÚscale_xÚscale_yZscale_xyZpos_xyZlin_xZlin_yZlin_xyZgrid_xyÚmax_dimr!ryr"rtÁsJ"    & *4 ûú" z%ViTPatchGenerator._get_pos_embeddings) TFFNr FrNNFNNr|)Ú__name__Ú __module__Ú __qualname__rÚ input_dim_tÚboolrr‹r*r7ÚTensorrKÚpropertyrNrrPrrQrRrr6rcrhrFrrGrlrtÚ __classcell__r!r!r@r"r sšòÿþýüûúùø ÷ ö õ ô ó@       ýÿþý üýÿþý ü"r cs8eZdZdef‡fdd„ Zdejdejfdd„Z‡ZS)r2r cstƒ ¡||_dSr©r)r*r )r=r r@r!r"r*s  zIm2Patches.__init__rBrCcCsb|jdkr| d¡}| ddd¡}|S|jd|j}|jd|j}t|d||j||jd�}|S)NrrDrrxrqz*b c (py yy) (px xx) -> b (py px) (c yy xx))ÚpyÚyyÚpxÚxx)r r�r‚rHr)r=rBrIr›r�r!r!r"rK s  þzIm2Patches.forward) r’r“r”rr*r7r—rKr™r!r!r@r"r2sr2cs,eZdZddededef‡fdd„ Z‡ZS)r4Fr r r&c s,tƒjd|d|fd|i|¤Ž||_dS)NrfrDr&rš)r=r r r&r>r@r!r"r*s þýü zViTPatchLinear.__init__)F)r’r“r”rr–r*r™r!r!r@r"r4s$r4)rÚtypingrrrr7Ztorch.nn.functionalrÚ functionalr^Úeinopsrrrrr•r Ú ImportErrorÚModuler r2ÚLinearr4r!r!r!r"Ús"    ÿj