fig1 illustrates an image - guided surgery system with which the present invention may be employed . the system includes a surgical or medical instrument 11 , such as an endoscope , having an elongate axis 12 and tip 13 , which is used to probe an internal target site 14 of a patient 15 . a plurality of fiducials or markers 16 are placed on the patient 15 near the target site . the fiducials are used to register corresponding points on the preoperative or intraoperative 2 - d image scans of patient target site 14 . instrument 11 has a plurality of tracking elements 17 on its shaft 12 which emit signals to sensors 33 positioned in view of the instrument . both the instrument and the sensors are in communication with a tracking controller which is in communication with a computer system that processes the signals received by sensors 33 to track the movement of instrument 11 during surgery . the tracking controller may be a separate element or it may be physically integrated with the computer system and may even be embodied in an option card which is inserted into an available card slot in the computer . the computer system is also used to render and display the 2 - d preoperative or intraoperative images and render 3 - d surface or volumetric images , either of which may be perspective or orthographic , of target site 14 on a display device . various aspects of the image - guided surgery procedure , such as registration , tracking , image generation and navigation , may be implemented by a program of instructions ( e . g ., software ) in response to user input supplied by various input devices such as a keyboard , mouse , track ball or joystick . the software is executed by a processor , such as a central processing unit ( cpu ) which may be in the form of a microprocessor . other processors may also be used in conjunction with the cpu such as a graphics chip . as shown in fig1 a dynamic frame - of - reference ( dfr ) 21 is rigidly attached to a support 22 that is securely attached to the portion of patient 15 ( e . g ., head 23 ) where the surgery is to be performed . thus , when the patient &# 39 ; s head moves , support 22 and dfr 21 also move . securely attached to the dfr is a rigid elongated element 24 extending outwardly from a main surface of dfr 21 at a known angle and terminating in a tip 25 at its distal end . dfr 21 includes a plurality of sensors 35 in communication with tracking controller which is also in communication with the computer system , instrument 11 and sensors 33 to determine the location and orientation of the dfr including element 24 and its tip 25 . in accordance with one variation of the preferred embodiment of the invention , a pointer 26 has its tip 27 attached to one end of an elastic , telescopic or otherwise length - adjustable potentiometer 28 or equivalent device , the other end of which is attached to tip 25 . if pointer 26 is not attached to dfr 21 , then the pointer will include tracking elements / position sensors 26 a that are in communication with sensors 33 and the computer system to track the movement and determine the position and orientation of the pointer . the remaining figures . taken in conjunction with fig1 describe the various embodiments of the invention . fig2 illustrates a first mode of operation in accordance with embodiments of the invention . first , the user selects a target point 29 in patient target site 14 and establishes the orientation of the instrument view axis ( e . g ., the endoscope view axis ) with respect to target point 29 . next , selected preoperative or intraoperative scan data representing internal scans of the patient target site are used to construct an image of the patient target site , with respect to the orientation of the instrument , such as viewed along the view axis of the instrument . that is , the constructed image is orthogonal to the view axis of the instrument but is viewed along the axis of the instrument . the image is then displayed in various 2 - d orientations and in 3 - d on the display device for viewing by the user . fig3 illustrates a second mode of operation , in accordance with embodiments of the invention , where a view orientation with respect to a virtual target point 31 , external to the patient , is transformed to a view orientation with respect to a selected target point 29 in the patient . the virtual target point may be established at any external location that can be determined relative to the tracking system , e . g ., relative to dfr 21 . for example , virtual target point 31 can be at the tip of elongated support 24 as shown in fig1 or it can be located in free space and initially defined by the tip of pointer 26 . as illustrated in fig3 the user selects an orientation with which to view the virtual target point and the corresponding image of patient target site 14 . the view orientation of the virtual target point is defined by an imaginary line extending between a known point on the pointer , e . g ., its tip 27 and virtual target point 31 . thus , selecting the view orientation also involves selecting the location of the virtual target point , if it is not at a fixed location , e . g ., at the tip of the elongated support , as shown in fig1 . the virtual target point can be defined in free space by moving the tip of the pointer to the desired point in space and clicking a selector button on pointer 26 to establish virtual target 31 point at the tip of the pointer . the virtual target point will remain at that location even after the pointer is moved , until a new virtual target point is established by the user . the selected view orientation and virtual target point location is then input into the computer where this information is processed and stored . in particular , the computer establishes a correspondence between selected target point 29 in patient target site 14 and virtual target point 31 in “ virtual ” space , which may be accomplished , for example , using point - to - point mapping . point - to - point mapping essentially involves determining a transformation matrix that maps the coordinates of point 29 to another set of coordinates representing point 31 . a transformation is then made between the view orientation with respect to virtual target point 31 and the view orientation with respect to selected point 29 . scan data is used to construct an image of the patient target site , as viewed along the view orientation with respect to the virtual target point . the image is then displayed on the display device . fig4 and 5 illustrate a preferred embodiment of the second mode of operation , where the virtual target point is located a specified distance from a known point on the pointer 26 which is used to view the virtual target point and the corresponding image of the patient target site 14 along the axis of the pointer 26 . it should be noted that in this embodiment as well as in the other disclosed embodiments pointer 26 may be the medical instrument used at the patient target site 14 or it may be a separate instrument . the user selects a target point b in the patient target site 14 that lies on the axis of the instrument 11 at a selected distance d from a known point a on the instrument 11 and inputs this information into the computer . the coordinates x 1 , y 1 , z 1 of selected point b in the patient target site are determined with respect to known point a . next , the user specifies an external virtual target point b ′ with respect to the axis of pointer 26 . this may be accomplished , for example , by clicking a selector button on pointer 26 to establish b ′ at the tip of the pointer relative to its axis . the coordinates x 1 , y 1 , z 1 of b are transformed to a second set of coordinates x 1 ′, y 1 ′, z 1 ′ representing b ′ to establish correspondence between points b and b ′ in their respective spaces . a transformation is then made between the view orientation along the axis of pointer 26 with respect to point b ′ at distance d ′ and the view orientation along the axis of instrument 11 with respect to point b at distance d , where d and d ′ may or may not be equal . these transformations establish correspondence between the two view orientations , so that the view that the user sees along the axis of pointer 26 at distance d ′ is the same view as the user sees ( or would see ) along the axis of instrument 11 at distance d in the corresponding view orientation . now , as pointer 26 is moved , say , counterclockwise through an angle θ to the position shown in fig5 the user now sees patient target site 14 through point b ′ as if instrument 11 was moved that same angular distance to the position shown in a dashed line in fig5 . also , if the user were to move pointer 26 along its axis , say away from b ′ thereby increasing d ′, the user would view target site 14 as if instrument 11 was also moved along its axis away from b to increase d . the computer may maintain a constant ratio between d and d ′, or alternatively may employ other linear or nonlinear mathematical relationships between d and d ′ in correlating the view distances . scan data is used to construct images of the patient target site 14 , as viewed along the axis of the pointer 26 , with respect to b ′ , and these images are displayed on the display device . a variation on the above - described embodiment is to use a “ universal swivel ” device connecting the tip of elongated element 24 ( which in this case represents virtual target point b ′) to the tip of the pointer 26 ( point a ′ in fig5 ). the universal swivel device may be potentiometer 28 or equivalent device . here , point b ′ is at a known location and d ′ may easily be determined using the potentiometer 28 . preferably , the potentiometer is adjustable along its axis to allow the pointer to be moved in and out but keeps the axis of the pointer aligned with the axis of the potentiometer . thus , this arrangement enables pointer 26 to rotate about virtual point b ′ while maintaining co - linearity between any two points on the pointer 26 and b ′. the “ universal swivel ” may also be implemented using a joystick , or similar device . here the ball of the joystick represents virtual target point b ′ and the joystick itself represents pointer 26 . view distance d ′ is fixed and corresponds to some view distance d in the patient target site . the joystick need not be tracked using sensors but would interface with the computer as an input device so that its position can be determined . the user inputs information to the computer to establish distance d ′, a reference view orientation with respect to point b ′, and a corresponding reference joystick position . based on that information , the computer then employs an algorithm to correlate different positions of the joystick with different view orientations with respect to point b ′, which , in turn , is correlated with various view orientations of instrument 11 with respect to target point b . as the joystick is moved , the view orientation that the user sees on the display device changes . the “ universal swivel ” may also be implemented using a track ball as a computer input device . once calibrated , manipulating the track ball can be used to change the view orientation vector a ′ b ′ which is correlated to various view orientations of instrument 11 represented by vector ab . here there is no pointer 26 , and the view orientation with respect to b ′ is relative to the position of the track ball . as with the joystick , view distance d ′ is fixed and corresponds to a view distance d in the patient target site . the user inputs information to the computer to establish d ′ which is correlated with d . in another embodiment of the second mode of operation , which is more general than the first embodiment , a transformation is made between a view orientation with respect to a selected virtual target point and a view orientation with respect a selected patient target site point , where the view orientations may or may not be along the axis of the instrument / pointer . an example of this embodiment is illustrated in fig6 and 7 , where the view orientation with respect to the target point is along the axis of the instrument but the view orientation with respect to the virtual target point is not along the axis of the pointer . this is merely one example of this embodiment ; other variations are possible . referring now specifically to fig6 and 7 , the user supplies input specifying a target point b in patient target site 14 , where b is defined relative to the instrument , e . g ., b lies along the axis of the instrument 11 . the coordinates x 1 , y 1 , z 1 of selected point b in the patient target site are determined with respect to the position of the instrument . the coordinates x 1 , y 1 , z 1 and a distance d may be determined relative to a known point , say a , on the tip of instrument 11 . the imaginary line between b and a defines the view orientation with respect to the selected patient target site point . next , the user specifies an external virtual target point b ′ with respect to the position of pointer 26 . this may be accomplished , for example , by using a position selector on the pointer as previously described . the coordinates x 1 , y 1 , z 1 are transformed to a second set of coordinates x 1 ′, y 1 ′, z 1 ′ representing b ′, and the view orientation in the “ virtual ” space is determined by the line between a ′ and b ′. a transformation is then made between the view orientation defined by vector a ′ b ′ and the view orientation defined by vector ab . thus , as the tip of pointer 26 is moved , say , counterclockwise through an angle θ , as shown in fig7 the user now sees patient target site 14 as if instrument 11 were moved that same angular distance to the position shown in dashed lines in fig7 . that is , view orientation a ′ 2 b ′ in the “ virtual ” space corresponds to view orientation a 2 b in the “ real ” space . scan data is used to construct images of the patient target site 14 , as viewed along the transformed view orientation , and these images are displayed on the display device . in another embodiment of the second mode of operation , a transformation is made between a view orientation along the axis of pointer 26 and a view orientation along the axis of instrument 11 . here , the view orientations are maintained along the axis of the instrument and the pointer respectively but not necessarily with respect to points b and b ′ when the pointer 26 is moved . an example of this embodiment is illustrated in fig8 and 9 . this is merely one example of this embodiment ; other variations are possible . referring now specifically to fig8 and 9 , the user supplies input specifying a patient target site point b along the axis of instrument 11 a specified distance d from a known point , say a , on the instrument . the coordinates x 1 , y 1 , z 1 of the selected point b are determined with respect to a known point , say a , on the instrument . next , the user specifies an external virtual target point b ′ with respect to the axis of pointer 26 , as previously described with regard to the first embodiment . the coordinates x 1 , y 1 , z 1 are transformed to a second set of coordinates x 1 , y 1 , z 1 representing b ′. a transformation is then made between the view orientation along the axis of pointer 26 and the view orientation along the axis of instrument 11 . in this embodiment , the view orientations are maintained along the axis of the respective viewing device . thus , if pointer 26 is moved laterally to the right , as shown in fig9 the view orientation remains along the axis of the pointer , and corresponds to the view orientation along the axis of instrument 11 , as if it had been moved laterally to the right the same distance , as shown in dashed line in fig9 . similarly , if pointer 26 is rotated , with or without accompanying lateral motion , the view orientation remains along the axis of the pointer , and corresponds to the view orientation along the axis of instrument 11 , as if it had been moved in the same manner . scan data is used to construct images of the patient target site 14 , as viewed along the transformed view orientation , and these images are displayed on the display device . as previously noted , various aspects of the image - guided surgery procedure , such as registration , tracking , image generation and navigation , may be implemented by a program of instructions ( i . e ., software ). various aspects of the present invention including transforming a target point in a patient target site to an external virtual target point and transforming the corresponding view orientations , which are part of the navigation step , may likewise be implemented by software . software implementing one or more of the various aspects of the present invention may be written to run with existing software used for image - guided surgery . the software for any or all of these tasks of the present invention may be fetched by the cpu from random - access memory ( ram ) for execution . the software may be stored in read - only memory ( rom ) on the computer system and transferred to ram when in use . alternatively , the software may be transferred to ram or directly to the cpu for execution from rom , or through a storage medium such as a disk drive or through an i / o device such as a modem . more broadly , the software may be conveyed by any medium that is readable by the cpu . such media may include , for example , various magnetic media such as disks or tapes , various optical media such as compact disks , as well as various communication paths throughout the electromagnetic spectrum including infrared signals , signals transmitted through a network or the internet , and carrier waves encoded to transmit the software . as an alternative to software implementation , the above - described aspects of the invention may be implemented with functionally equivalent hardware using discrete components , application specific integrated circuits ( asics ), digital signal processing circuits , or the like . such hardware may be physically integrated with the computer processor ( s ) or may be a separate device which may be embodied on a computer card that can be inserted into an available card slot in the computer . thus , the above - described aspects of the invention can be implemented using software , hardware , or combination thereof . with that in mind , it is to be understood that the flow diagrams used to illustrate the various aspects of the present invention show the performance of certain specified functions and relationships therebetween . the boundaries of these functional blocks have been defined herein for convenience of description . alternate boundaries may be defined so long as the appropriate functions are performed and the appropriate relationships therebetween are maintained . the diagrams and accompanying description provide the functional information one skilled in the art would require to implement a system to perform the functions required . each of the functions depicted in the flow diagrams may be implemented , for example , by software , functionally equivalent hardware , or a combination thereof . while embodiments of the invention have been described , it will be apparent to those skilled in the art in light of the foregoing description that many further alternatives , modifications and variations are possible . the invention described herein is intended to embrace all such alternatives , modifications and variations as may fall within the spirit and scope of the appended claims .
systems and methods for performing transformations between a view orientation in a virtual space and a view orientation with respect to a target site in a patient &# 39 ; s body where surgery is being performed . the view in virtual space enables the surgeon to see the internal structures in the target site in a variety of ways and orientations including from behind the site .